Three-dimensional mandible image restoration method based on comparative learning prior and mask autoregression

Abstract

The invention relates to a three-dimensional lower jawbone image restoration method based on comparative learning prior and mask autoregression. The method comprises the following steps: acquiring a maxillofacial region three-dimensional CT image of a patient; for the image of the mandible of the patient, respectively obtaining coarse repair results on a coronal plane, a sagittal plane and a cross section through a two-dimensional repair model; the two-dimensional repair model and the mask autoregression strategy are adopted, comparison learning loss is added, global sample feature information is obtained by using a pre-trained ViT model, and the lambda model is guided to complete intra-layer repair of global information and local fine-grained information; and fusing the coarse repair results of different layers obtained by the two-dimensional model by adopting a fine repair module based on residual fusion to obtain a three-dimensional repair result. Compared with the prior art, the method has the advantages that the deep learning technology is utilized, a brand new medical image restoration algorithm is constructed, the full-automatic restoration result of the maxillofacial region three-dimensional CT is achieved, and effective assistance is provided for preoperative accurate planning.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image reconstruction and restoration, and in particular to a three-dimensional mandibular image restoration method based on contrastive learning prior and mask autoregression. Background Art

[0002] Mandibular reconstruction is widely used to treat mandibular tumors and restore the patient's oral function and facial appearance. This surgery restores the structure and function of the mandible by transplanting healthy bone tissue or artificial implants. The position of the mandible needs to be accurately measured and planned before surgery to ensure the best results after surgery. However, there are some difficulties in surgical planning for mandibular tumors, one of which is determining the ideal mandibular position.

[0003] Traditional measurement methods are based on anatomical landmarks, which is time-consuming and labor-intensive, and it is difficult to intuitively determine the ideal normal shape of the mandible. With the development of generative artificial intelligence, image restoration technology based on artificial intelligence helps solve this difficulty. By analyzing and processing the patient's maxillofacial CT and other imaging data, artificial intelligence can generate a three-dimensional reconstruction model to simulate the effects of different mandibular positions and help doctors determine the most appropriate surgical plan.

[0004] In the process of medical image restoration, image restoration technology can complete the patient's diseased area and predict its corresponding normal morphology for surgical planning. In particular, in this scenario, image restoration technology can predict the normal morphology of the diseased area based on the non-diseased area, that is, the normal morphology corresponding to the mandible, as the main reference for surgical planning. In particular, in recent years, image restoration technology based on deep learning has made rapid progress, and deep neural networks have demonstrated powerful capabilities in the field of image generation, which can produce fast and high-quality restoration results. Among them, there are methods based on partial convolution, which can gradually complete the restoration of the image; methods based on gated convolution, which can use flexible mask step-by-step update strategies to obtain better restoration results; methods based on edge information such as edge-connect; and methods based on Fourier convolution such as lama that can produce high-quality and high-resolution results; these methods can ensure the edge rationality and content rationality of the restoration results.

[0005] However, this restoration method has the following limitations: (1) The convolutional neural network is used as the model architecture. The direct training process is often limited by the receptive field of the convolutional neural network. Therefore, there is a lack of understanding of global features, and the restoration results have problems such as content consistency; (2) The model is easy to learn a template, and then simply apply the template to obtain the same restoration results when repairing different patients, without considering the individual characteristics of the patient's own anatomical structure; (3) When repairing a three-dimensional image, it is often split into two dimensions and repaired layer by layer, so there is a serious problem of inter-layer discontinuity. For example, when it is split into several two-dimensional cross-sectional layers for repair and stacked into three-dimensional volume data, when viewed from the coronal and sagittal planes, it will be found that the restoration image quality is seriously insufficient, which is often manifested as burrs and artifacts, interfering with the doctor's actual observation and judgment process. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a three-dimensional mandibular image restoration method based on contrastive learning prior and mask autoregression.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A three-dimensional mandibular image restoration method based on contrastive learning prior and mask autoregression, the method steps comprising:

[0009] Obtain three-dimensional CT images of the patient's maxillofacial region;

[0010] For the patient's mandibular image, a two-dimensional rough restoration result is obtained in the coronal plane, sagittal plane, and transverse plane respectively through a two-dimensional restoration model; the two-dimensional restoration model is based on a mask autoregressive strategy and adopts contrastive learning loss, uses a pre-trained ViT model to obtain global sample feature information, and guides the lama model to complete the intra-layer restoration of global information and local fine-grained information;

[0011] A fine restoration module based on residual fusion is used to fuse the two-dimensional rough restoration results of different layers of the two-dimensional model to obtain a three-dimensional restoration result.

[0012] As a preferred technical solution, the ViT model has global attention, uses global different element tokens to interact with each other, and learns the relationship between elements; a masked autoregression task is used to guide the ViT model to obtain global understanding information, and serves as a priori for the lama model.

[0013] As an optimal technical solution, the ViT model includes: a multi-head attention layer, which enables different element tokens to interact and calculate similarity; an MLP layer, which transforms features; a residual connection is used between the multi-head attention layer and the MLP layer; and a normalization layer is respectively provided before the multi-head attention layer and the MLP layer.

[0014] As a preferred technical solution, the ViT model adopts contrastive learning loss based on infoNCE, where a patient covers different areas as a pair of positive samples, and different patients cover the same area as a pair of negative samples. Each optimization sample is a small batch of patients and repair areas, as follows:

[0015] Calculate the feature representation z for each repair region j for each patient i ij ;

[0016] Select the feature representation z of different repair regions j′ of the same patient i ij′ As a positive sample pair;

[0017] Select the feature representation z of the same repair area j′ of different patients i′ i′j′ As a negative sample pair;

[0018] Calculate the infoNCE loss L for each sample pair infoNCE,(i,j) ;

[0019] The infoNCE losses of all patients and repaired regions are summed and averaged to obtain the total contrastive learning loss L total , which is used to guide network learning and mine the patient's features in the non-repaired area for subsequent detail restoration.

[0020] As a preferred technical solution, the infoNCE loss is calculated as follows:

[0021]

[0022] Where: z ij is the feature representation of the jth repair region of patient i; z ij′ For z ij Feature representation of positive sample pairs, i.e. different regions of the same patient; z i′j′ For z ij The feature representation of negative sample pairs, i.e., the restoration results of the same area in different patients; τ is a hyperparameter;

[0023] The infoNCE loss is calculated for all N patients and the M repaired regions of each patient and averaged to obtain the total contrastive learning loss L total .

[0024] As a preferred technical solution, the lama model uses the global feature prior obtained by the ViT model to complete further repair through a convolutional network and a Fourier convolution module; the attention feature map output by the Vit model decoder is used as an additional input before the Fourier convolution network to provide global personal features.

[0025] As a preferred technical solution, the Fourier convolution module performs the following calculations on a feature map with a dimension of H*W*C:

[0026] Using fast Fourier transform, we get a complex tensor with the dimension of H*W / 2*C;

[0027] Concatenate the real and imaginary parts of the complex tensor to obtain a real tensor with a dimension of H*W / 2*2C;

[0028] In the frequency domain, convolution is used for further processing, and the final result dimension remains unchanged through batch normalization layer and ReLU activation function layer;

[0029] The inverse Fourier transform is used to obtain the original spatial data, which is first divided into the real part and the imaginary part, and then the inverse Fourier transform is performed to obtain the final real tensor with the dimension of H*W*C.

[0030] As a preferred technical solution, the refinement module is specifically a multi-level cascaded residual fusion module, which stacks multiple residual fusion modules, takes the average frame of K frames of the previous level sequence each time, stacks them to form a higher level sequence, and returns to the previous level after residual fusion; multi-level fusion of maxillofacial three-dimensional CT sequences to obtain consistent results between layers.

[0031] As a preferred technical solution, in each of the residual fusion modules:

[0032] For each group of layers, the corresponding rough restoration result is obtained based on the previous two-dimensional restoration model;

[0033] Based on the rough restoration result, the average frame of this group of layers is extracted, and the residual of each frame compared with the average frame is extracted through the convolution module. The residuals of different frames are fused to obtain continuous residual results;

[0034] After adding the residual and average frame results pixel by pixel, the corresponding fusion result is obtained, which is then further refined and improved through a convolution module to obtain a refined restoration result.

[0035] As a preferred technical solution, during the training phase, the model of the restoration method inputs an image of a normal person, randomly masks a part of the area, and based on the relevant information of the non-masked area, allows the model to predict the normal shape of the masked area; supervised by using the actual image of the corresponding area of ​​a normal person as the gold standard, the model learns the basic shape of a normal person;

[0036] During inference, the model infers on the lesion data, masks the lesion area, and estimates the normal morphology within the lesion area based on information from other non-lesion areas.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1) The present invention constructs a two-stage image restoration neural network, using a global attention mechanism to first restore the basic content and then add high-frequency details, completing the restoration task of mandibular CT images in steps. The present invention realizes the restoration of mandibular CT 3D images for the first time, which can provide important guidance to doctors, including the results of bone defect-related areas.

[0039] 2) The present invention uses a transformer model with global attention to provide a powerful global feature prior for the convolutional neural network, expands the receptive field of the corresponding model, and can use global relevant information and fuse long-range information to guide the repair of specific locations, thereby greatly improving the content consistency of the original convolutional network.

[0040] 3) The present invention proposes a loss based on contrastive learning to mine the unique characteristics of an individual and avoid the problem of the network falling into shortcuts due to large uncertainty, thereby ensuring personalized restoration results.

[0041] 4) The present invention proposes a fine restoration module based on residual fusion to comprehensively utilize the restoration results of different layers to achieve consistency. This greatly alleviates the limitations of using single-layer information and the uncertainty of estimation, as well as the problem of inter-layer discontinuity, and ensures the details of image restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of a three-dimensional mandibular image restoration method based on contrastive learning prior and mask autoregression of the present invention;

[0043] Figure 2 This is a schematic diagram of the overall architecture of Transformer Block;

[0044] Figure 3 Schematic diagram of the loss architecture for contrastive learning;

[0045] Figure 4 Schematic diagram of the overall architecture of the repair network based on global individual feature prior guidance;

[0046] Figure 5 Schematic diagram of the refined restoration architecture based on the residual fusion module;

[0047] Figure 6Schematic diagram of global consistent restoration based on cascaded residual fusion module. DETAILED DESCRIPTION

[0048] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0049] Example 1

[0050] Before implementing 3D reconstruction, doctors usually analyze the patient's mandibular condition through maxillofacial 3D CT images. Based on this, the corresponding surgical plan is completed. The present invention constructs a priori-guided image restoration and network model based on contrastive learning priors and implicit functions, and for the first time realizes a robust restoration algorithm for 3D image restoration of the mandible and for surgical planning based on skull CT.

[0051] In terms of network design, the present invention uses a mixed architecture of convolution and self-attention layers, and is divided into two stages: in the first stage, it is a two-dimensional repair model, based on a masked autoregressive strategy, and adding contrastive learning loss, using the global sample feature information obtained by the pre-trained ViT (Vision Transformers) model to guide the lama model to complete the intra-layer repair of global information and local fine-grained information. The two-dimensional model obtains separate reconstruction results in the coronal plane, sagittal plane, and cross-section. Then, in the second stage, a fine repair module with residual fusion is used to fuse the results of different layers to avoid inter-layer continuity problems. Based on this strategy, we obtain a complete three-dimensional repair result with intra-layer continuity and reasonable and consistent content. Therefore, the present invention selects a three-dimensional CT image of the mandible that can be provided as input, completes image repair on this basis, inputs the patient's image, and completes the generation of surgical reference targets for the mandibular part to determine the condition of the mandibular defect.

[0052] The model development process of the image restoration technology used in the present invention is divided into two stages: the training stage, in which a normal person's image is input and a part of the area is randomly covered. Based on the relevant information of the non-covered area, the model can predict the normal shape of the covered area, and supervised by the actual normal person's corresponding area image as the gold standard, so that the model can learn the basic shape of normal people. During reasoning, the model infers on the lesion data, covers the lesion area, and estimates the normal shape in the lesion area based on the information of other non-lesion areas.

[0053] Two sub-models are used in the first stage, one is the ViT model for extracting global personal feature priors, and the other is the Lama model for repairing local information.

[0054] In image restoration, the size of the model's receptive field is related to the global prior. The ViT model uses a masked autoregressive strategy for training. The ViT model has global attention and proposes to use different global elements (tokens) for pairwise interaction, which can better learn the relationship between elements and make full use of the different features of the entire three-dimensional image. Therefore, we use the mainstream masked autoregressive task to guide the ViT (Vision Transformers) model to obtain global understanding information as a priori. During the training process, 75% of the image blocks are randomly masked, and the model is restored based on the remaining 25% of the image blocks. The loss is calculated between the restored results and the original image blocks, which prompts the training network to learn the relationship between different global image blocks and understand the global scene, which is used to guide another sub-model lama.

[0055] like Figure 2 As shown in the figure, in order to complete the global feature interaction, multi-head attention is first used to allow different tokens to interact and calculate the similarity; then MLP is used to transform the features; before the two-step operation, layernorm technology is used for normalization, and residual connection is used to avoid gradient explosion in deep networks.

[0056] The multi-head attention mechanism is used for global feature interaction. The basic approach is as follows: for a sequence X of image block features (patch token), calculate q, k, v:

[0057] Q=XW q , k = XW k , V = XW v

[0058] Calculate the attention score:

[0059] Attention Score=QK^T

[0060] Attention Weights=softmax(Attention Scores / sqrt(d_k))

[0061] Weighted Value=Attention Weights*V

[0062] In this way, the corresponding multi-head attention layer output can be obtained, the global token interaction can be completed, and finally used for further processing in the subsequent network.

[0063] Based on this, the network can understand the global scene through global attention and mask self-encoding strategies, and can complete basic repair tasks under a larger masking rate.

[0064] However, only using global priors often makes it easy for the network to learn a basic mandibular template. When the repair area is large, such as when most of the patient's chin area needs to be repaired, the problem is difficult and has large uncertainty, and a shortcut problem will be encountered. Specifically, the network easily learns a basic human skeleton template, and directly fills in the memorized template for larger repair areas. However, this seemingly reasonable result does not take into account the different conditions of different patients, such as the relationship between the skull and other anatomical regions and the mandible.

[0065] In order to enhance the network’s understanding of the anatomical structures of different patients, we additionally introduced a contrastive learning loss to allow the network to compare the differences between different patients, thereby completing personalized repairs based on the unique features extracted from the patient’s non-repaired areas.

[0066] Among them, for the definition of positive and negative sample pairs, we believe that when a patient covers different areas, it belongs to a pair of positive samples; when different patients cover the same area, they belong to a pair of negative samples.

[0067] Therefore, the present invention adopts a contrastive learning loss based on information noise contrast estimation (infoNCE), which can network mine the characteristics of patients in non-repaired areas for subsequent further detail restoration.

[0068] The loss function of infoNCE is defined as follows:

[0069]

[0070] Among them, z i and z j is the feature representation of the positive sample pair; z k is the feature representation of negative samples; τ is a hyperparameter; K is the number of all samples including positive samples.

[0071] like Figure 3 As shown, the present invention samples a small batch of patients and repair areas for each optimization: for N patients, each patient has M repair areas, and the repair result of each area corresponds to a feature representation; a feature representation z is generated for each patient N and repair area j ij ,We hope to mine the features of non-repaired areas through contrastive learning loss function.

[0072] In this case, the loss function is defined as follows:

[0073]

[0074] Where: z ijand z ij′ is the feature representation of the positive sample pair (different regions of the same patient); other i′, j combinations are the feature representation of the negative sample pair (the restoration results of the same or different restoration regions of different patients)

[0075] The infoNCE loss needs to be calculated for all N patients and the M repaired regions of each patient and averaged:

[0076]

[0077] The specific steps are:

[0078] 1.1 Calculate the feature representation z of each repair region j for each patient i ij .

[0079] 1.2. Selecting the feature representation z of different repaired regions j′ of the same patient i ij′ as a positive sample pair.

[0080] 1.3 Selecting the feature representation z of the same repair region j′ for different patients i′ i′j′ as negative sample pairs.

[0081] 1.4 Use infoNCE formula to calculate the loss L for each sample pair infoNCE,(i,j) .

[0082] 1.5 Sum and average the losses of all patients and repaired regions to obtain the total contrastive learning loss L total .

[0083] Finally, we get the total loss function L total It can be used to guide the network to better learn and mine the patient's features in the non-repaired area for subsequent detail restoration.

[0084] The second sub-model uses the lama model. This model is designed to further restore image details, especially the lost high-frequency features, and use the global feature priors of the first stage to complete further repair and restore more image details through the convolutional network and the additional Fourier convolution module. Among them, the attention feature map output by the decoder of the first stage is used as an additional input before the Fourier convolution network to provide global personal features, further improve the network's receptive field, and ensure content consistency. Among them, the specific method of using global personal feature priors is as follows Figure 4 Hint:

[0085] Among them, the Fourier convolution used further increases the receptive field of the downstream repair network. The specific steps are as follows:

[0086] For a feature map with dimensions H*W*C:

[0087] 2.1 Use fast Fourier transform (FFT2d) to obtain a complex tensor with dimension H*W / 2*C;

[0088] 2.2 Concatenate the real and imaginary parts of the complex tensor to obtain a real tensor with a dimension of H*W / 2*2C;

[0089] 2.3 In the frequency domain, convolution is used for further processing, and the final result dimension remains unchanged through batch normalization layer and ReLU activation function layer;

[0090] 2.4 Use inverse Fourier transform to obtain the original spatial data, first divide it into real part and imaginary part, and then perform inverse Fourier transform to get the final real tensor with dimension H*W*C.

[0091] Through such processing, we analyze and process the frequency space of the feature map, which can capture more high-frequency details and expand the receptive field of the convolutional network, further improving the restoration effect.

[0092] In the second stage, the information fusion of different layers based on the residual fusion module is used for refined restoration to alleviate the inter-layer discontinuity problem existing in the two-dimensional restoration model in the first stage. The refinement module adopts the residual learning and post-fusion method, which is specifically implemented as a multi-level cascaded residual fusion module.

[0093] The design in each residual fusion module is as follows Figure 5 As shown. First, for each three-dimensional data, it can be regarded as a stack of several layers of coronal / sagittal / cross-sectional two-dimensional images (hereinafter referred to as a two-dimensional sequence, referring to the three-dimensional result of stacking two-dimensional data). Specifically, when the length, width and height of the three-dimensional data itself is H*W*D, it can be split into H coronal two-dimensional data of size W*D, or W sagittal data of size H*D, or D cross-sectional data of size H*W. By splitting it into several two-dimensional data, performing two-dimensional repair, and then stacking them up, we can obtain a preliminary three-dimensional estimate result. Then, for a group of layers that should have been continuous, the corresponding rough repair results can be obtained based on the first-stage model in the front. However, the rough repair results are not continuous, so we first extract the average frame of this group of layers as the basic content of this group of two-dimensional images. Then, the residual of each frame compared to the average frame is extracted through the convolution module, and the residuals of different frames are fused to obtain a continuous residual result. After adding the residual and average frame results pixel by pixel, the corresponding fusion result can be obtained, and then further refined and improved through a convolution module to obtain a refined restoration result.

[0094] However, a simple residual fusion module is not enough to fuse images with strong variability in the depth dimension, especially the more complex maxillofacial 3D CT anatomical structure. In particular, since the number of layers of general maxillofacial 3D CT images is large, the overhead of directly fusing all layers is too high. Therefore, by stacking multiple residual fusion modules, we can fuse longer maxillofacial 3D CT sequences at multiple levels and obtain consistent results between layers while ensuring that the amount of computation is acceptable.

[0095] like Figure 6 As shown in the figure, each time the K frames of the previous level sequence are averaged, they are stacked to form a higher level sequence, and then the residuals are fused and returned to the previous level. This design can expand the global receptive field represented by the average frame and achieve a more consistent effect.

[0096] Based on the refined restoration module designed by cascade residual learning, we finally obtained consistent 3D restoration results between layers. The specific results of the fusion of the three-level results are as follows: For each 3D image, we can obtain a 2D sequence of coronal, sagittal, and transverse restorations, which are spliced ​​together in the channel dimension to become H*W*D*3 data. The residual fusion module is passed once in each of the three directions to achieve continuity in the three dimensions of length, width, and height. Finally, the convolution transform becomes an image of H*W*D*1, which is the final fusion result.

[0097] The present invention aims to combine artificial intelligence technologies such as deep learning to construct a new mandibular medical image restoration technology based on the idea of ​​masked autoencoders, using artificial intelligence to learn the ability of normal people, restore the patient's image, and ensure the continuity of the three-dimensional structure. Using deep learning technology, a new medical image restoration algorithm is constructed, and the fully automatic restoration results of mandibular CBCT films are achieved for the first time. The research results of the present invention will provide effective assistance for precise preoperative planning, assist in determining the bone reconstruction goals of mandibular repair surgery, and improve the clinical efficacy of mandibular reconstruction surgery.

[0098] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A three-dimensional mandibular image restoration method based on contrastive learning prior and mask autoregression, characterized in that: The method steps include: Obtain three-dimensional CT images of the patient's maxillofacial region; For the patient's mandibular image, a two-dimensional rough restoration result is obtained in the coronal plane, sagittal plane, and transverse plane respectively through a two-dimensional restoration model; the two-dimensional restoration model is based on a mask autoregressive strategy and adopts contrastive learning loss, uses a pre-trained ViT model to obtain global sample feature information, and guides the lama model to complete the intra-layer restoration of global information and local fine-grained information; A fine restoration module based on residual fusion is used to fuse the two-dimensional rough restoration results of different layers of the two-dimensional model to obtain a three-dimensional restoration result.

2. The three-dimensional mandibular image restoration method based on contrastive learning prior and mask autoregression according to claim 1 is characterized in that: The ViT model has global attention, uses global different element tokens to interact with each other and learns the relationship between elements; a masked autoregression task is used to guide the ViT model to obtain global understanding information and serve as a priori for the lama model.

3. The three-dimensional mandibular image restoration method based on contrastive learning prior and mask autoregression according to claim 2 is characterized in that: The ViT model includes: a multi-head attention layer, which enables different element tokens to interact and calculate similarity; an MLP layer, which transforms features; a residual connection is used between the multi-head attention layer and the MLP layer; and a normalization layer is respectively set before the multi-head attention layer and the MLP layer.

4. The three-dimensional mandibular image restoration method based on contrastive learning prior and mask autoregression according to claim 1, characterized in that: The ViT model uses a contrastive learning loss based on infoNCE, where a patient covering different areas is a pair of positive samples, and different patients covering the same area are a pair of negative samples. Each optimization samples a small batch of patients and repair areas, as follows: Calculate the feature representation z for each repair region j for each patient i ij ; Select the feature representation z of different repair regions j′ of the same patient i ij′ As a positive sample pair; Select the feature representation z of the same repair area j′ of different patients i′ i′j′ As a negative sample pair; Calculate the infoNCE loss L for each sample pair infoNCE,(i,j) ; The infoNCE losses of all patients and repaired regions are summed and averaged to obtain the total contrastive learning loss L total , which is used to guide network learning and mine the patient's features in the non-repaired area for subsequent detail restoration.

5. The three-dimensional mandibular image restoration method based on contrastive learning prior and mask autoregression according to claim 4, characterized in that: The infoNCE loss is calculated as follows: Where: z ij is the feature representation of the jth repair region of patient i; z ij′ For z ij Feature representation of positive sample pairs, i.e. different regions of the same patient; z i′j′ For z ij The feature representation of negative sample pairs, i.e., the restoration results of the same area in different patients; τ is a hyperparameter; The infoNCE loss is calculated for all N patients and the M repaired regions of each patient and averaged to obtain the total contrastive learning loss L total .

6. The three-dimensional mandibular image restoration method based on contrastive learning prior and mask autoregression according to claim 1, characterized in that: The lama model uses the global feature prior obtained by the ViT model to complete further repair through the convolutional network and Fourier convolution module; the attention feature map output by the Vit model decoder is used as an additional input before the Fourier convolution network to provide global personal features.

7. The method for three-dimensional mandibular image restoration based on contrastive learning prior and mask autoregression according to claim 6, characterized in that: The Fourier convolution module operates on a feature map with a dimension of H*W*C as follows: Using fast Fourier transform, we get a complex tensor with the dimension of H*W / 2*C; Concatenate the real and imaginary parts of the complex tensor to obtain a real tensor with a dimension of H*W / 2*2C; In the frequency domain, convolution is used for further processing, and the final result dimension remains unchanged through batch normalization layer and ReLU activation function layer; The inverse Fourier transform is used to obtain the original spatial data, which is first divided into the real part and the imaginary part, and then the inverse Fourier transform is performed to obtain the final real tensor with the dimension of H*W*C.

8. The three-dimensional mandibular image restoration method based on contrastive learning prior and mask autoregression according to claim 1, characterized in that: The refinement module is specifically a multi-level cascaded residual fusion module. By stacking multiple residual fusion modules, after taking the average frame of K frames of the previous level sequence each time, a higher level sequence is stacked, and after residual fusion, it returns to the previous level; the maxillofacial three-dimensional CT sequence is fused at multiple levels to obtain consistent results between layers.

9. The three-dimensional mandibular image restoration method based on contrastive learning prior and mask autoregression according to claim 1, characterized in that: In each of the residual fusion modules: For each group of layers, the corresponding rough restoration result is obtained based on the previous two-dimensional restoration model; Based on the rough restoration result, the average frame of this group of layers is extracted, and the residual of each frame compared with the average frame is extracted through the convolution module. The residuals of different frames are fused to obtain continuous residual results; After adding the residual and average frame results pixel by pixel, the corresponding fusion result is obtained, which is then further refined and improved through a convolution module to obtain a refined restoration result.

10. The three-dimensional mandibular image restoration method based on contrastive learning prior and mask autoregression according to claim 1, characterized in that: In the training phase, the model of the restoration method inputs an image of a normal person, randomly masks a part of the area, and allows the model to predict the normal shape of the masked area based on the relevant information of the non-masked area; supervised by the actual image of the corresponding area of ​​the normal person as the gold standard, the model learns the basic shape of the normal person; During inference, the model infers on the lesion data, masks the lesion area, and estimates the normal morphology within the lesion area based on information from other non-lesion areas.

Images