Oral and maxillofacial region CT metal artifact elimination method

Abstract

The technical scheme of the invention discloses an oral and maxillofacial region CT metal artifact elimination method. The invention firstly provides a synthesis method of metal artifact data, and constructs a paired synthesis artifact-free and artifact-free oral maxillofacial region CT image data set and a non-paired real artifact-free and artifact-free oral maxillofacial region CT image data set in combination with clinical data collection. The invention further provides a metal artifact elimination model training framework based on semi-supervised domain adaptation, and a metal artifact elimination model capable of effectively removing metal artifacts in a real oral maxillofacial region clinical CT image can be trained by utilizing the constructed simulation data set and the real data set. The method can effectively eliminate the metal artifacts in the oral maxillofacial region CT image, improves the image quality, and assists in clinical diagnosis and treatment.

Description

Technical Field

[0001] The invention relates to a method for eliminating metal artifacts in oral and maxillofacial CT images, and belongs to the technical field of computer medical imaging. Background Art

[0002] Computed tomography (CT) systems can detect the internal structure of the human body non-destructively, and have become an important means of medical diagnosis, evaluation, and formulation of treatment plans and guidance. For oral and maxillofacial CT images, when the patient has metal implants, such as metal dentures, implants, orthodontic brackets, titanium plates and titanium nails, etc., it will cause inconsistent ray projection information. Inconsistent projection information will cause strong emission-like or stripe-like light and dark metal artifacts to appear in the reconstructed CT image. These artifacts interfere with the interpretation of image information, greatly reduce the image quality and seriously hinder medical clinical diagnosis and treatment.

[0003] In the past few decades, relevant researchers have made a lot of efforts to eliminate CT metal artifacts. The first type is the projection interpolation correction algorithm, the basic principle of which is to reconstruct the corrected projection data using the filtered back projection reconstruction algorithm. Due to the presence of metal implants, the projection data undergoes nonlinear changes, and the filtered back projection (FBP) is particularly sensitive to the incompleteness of the original projection. Therefore, direct reconstruction will lead to severe metal artifacts in the reconstructed image. In order to reduce the interference of metal artifacts, relevant researchers start from the projection and complete and correct the incomplete projection information. Another type of method for correcting metal artifacts starts from the reconstructed image, aiming to directly estimate and remove metal artifacts in the original contaminated image through image processing technology.

[0004] In the past decade, deep learning has made great progress and has achieved good results in various tasks. Deep learning extracts various statistical information of features based on a large amount of training data, and processes different tasks based on the obtained large amount of feature information. Existing supervised metal artifact removal methods based on deep learning require training with artifact-affected data and artifact-free label pairs, one containing metal implants and the other without metal implants. Paired images cannot be obtained under actual conditions, and most supervised methods can only be trained with simulated data. However, due to the rich variety of metal materials in the oral and maxillofacial region, the metal artifacts of different materials have different shapes, and it is difficult for synthetic artifacts to accurately reproduce real scenes. Therefore, the existing metal artifact removal methods cannot effectively remove metal artifacts in the oral and maxillofacial region in practical applications. Summary of the invention

[0005] The technical problem to be solved by the present invention is that the existing metal artifact removal method cannot effectively remove the metal artifacts in the oral and maxillofacial region in practical applications.

[0006] In order to solve the above technical problems, the technical solution of the present invention is to provide a method for removing metal artifacts in oral and maxillofacial CT, which is characterized by comprising the following steps:

[0007] Step 1, generating an oral and maxillofacial CT image dataset for model training, wherein the oral and maxillofacial CT image dataset includes a simulated dataset and a real dataset, wherein the simulated dataset includes paired simulated oral and maxillofacial CT images with metal artifacts and simulated oral and maxillofacial CT images without metal artifacts and corresponding metal masks, and the real dataset includes unpaired real oral and maxillofacial CT images with metal artifacts and real oral and maxillofacial CT images without metal artifacts;

[0008] Step 2: Construct a metal artifact elimination model, which includes a three-stage encoder and a two-stage decoder, where:

[0009] Characteristics of the first stage encoder output Figure 1 On the one hand, it is used as the input of the second-level encoder, and on the other hand, it is fed into the corresponding first-level decoder through a jump connection; the first-level encoder includes a tile embedding layer and several spatial frequency domain Transformer blocks. After the input image is processed by the tile embedding layer, it is processed by several spatial frequency domain Transformer blocks. The tile embedding layer does not implement the downsampling process, and all spatial frequency domain Transformer blocks only contain spatial frequency domain convolutional layers;

[0010] Characteristics of the second-stage encoder output Figure 1 On the one hand, it is used as the input of the third-level encoder, and on the other hand, it is fed into the corresponding second-level decoder through a jump connection; the second-level encoder includes a tile embedding layer and several spatial frequency domain Transformer blocks. After the tile embedding layer downsamples the input feature map, it is processed by several spatial frequency domain Transformer blocks. The first N spatial frequency domain Transformer blocks in the several spatial frequency domain Transformer blocks only contain spatial frequency domain convolutional layers, and the last M spatial frequency domain Transformer blocks contain both window self-attention layers and spatial frequency domain convolutional layers, N≥1, M≥1;

[0011] The feature map output by the third-level encoder is directly used as the input of the second-level decoder; the third-level encoder includes a tile embedding layer implemented by convolution and several spatial-frequency domain Transformer blocks, all of which contain a windowed self-attention layer and a spatial-frequency domain convolution layer;

[0012] The second-stage decoder is located after the third-stage encoder. The second-stage decoder first upsamples and transforms the dimension of the feature map output by the third-stage encoder through the tile de-embedding layer, and then concatenates it with the feature map output by the second-stage encoder according to the channel dimension. After a 1×1 convolution dimensionality reduction, the obtained feature map is input into several subsequent spatial frequency domain Transformer blocks, which only contain spatial frequency domain convolution layers.

[0013] The first-level decoder first upsamples and transforms the feature map output by the second-level decoder through the tile de-embedding layer, and then concatenates it with the feature map output by the first-level encoder according to the channel dimension, and undergoes a 1×1 convolution to reduce the dimension. The resulting feature map is input into several subsequent spatial-frequency domain Transformer blocks, which only contain spatial-frequency domain convolution layers.

[0014] The feature map output by the first-level decoder is transformed by the tile de-embedding layer to obtain the prediction residual map After global residual connection, the final output of the metal artifact removal model is obtained Where X represents the input image of the metal artifact removal model;

[0015] Step 3: Train the metal artifact removal model:

[0016] The training framework of the metal artifact removal model includes an initial warm-up phase and a semi-supervised domain adaptation phase, where:

[0017] In the initial warm-up stage, the metal artifact removal model is supervisedly trained using paired simulated data in the simulated data set, so that the metal artifact removal model has preliminary knowledge about the removal of metal artifacts in the oral and maxillofacial region. The network is trained by minimizing the L1 loss between the input image with simulated metal artifacts in the simulated data and the corresponding artifact-free label map. The loss function is as follows:

[0018]

[0019] Among them, D sim represents a simulated data set, X sim is a simulated oral and maxillofacial CT image with metal artifacts, Y sim is the corresponding simulated oral and maxillofacial CT image without metal artifacts, M sim is the corresponding metal mask;

[0020] In the semi-supervised domain adaptation stage, a domain discriminator is introduced, and a mixture of simulated data and real data is used to perform adversarial training on the metal artifact removal model at the feature level. During the adversarial training process, the parameters of the metal artifact removal model and the domain discriminator are updated alternately until convergence. The domain discriminator is optimized by minimizing the following loss function:

[0021]

[0022] Among them, P is the label of the discrimination result. If X comes from the real data set D real , then P is an all-one matrix, and the output probability of the shape and domain discriminator Figure 1 If X comes from the simulated data set D sim , then P is an all-zero matrix;

[0023] The loss function of the metal artifact removal model training is the reconstruction loss in the first stage. Based on this, an additional domain invariance loss is added It is necessary not only to remove the artifacts in the image, but also to make it difficult for the domain discriminator to distinguish whether the features extracted by the encoder are from simulated metal artifact images or real metal artifact images. The overall loss function is specifically defined as follows:

[0024]

[0025] Among them, λ is a hyperparameter that balances the weights of the two losses, domain invariance loss The definitions are as follows.

[0026]

[0027] That is, the encoder f of the metal artifact removal model is trained using simulated or real data θ (·), adjust the parameter θ so that the domain discriminator g φ (·) Misidentifying simulated data features as real or simulated data features.

[0028] Preferably, in step 1, the following method is used to generate a paired simulated oral and maxillofacial CT image with metal artifacts and a simulated oral and maxillofacial CT image without metal artifacts and a corresponding metal mask:

[0029] Step 101: Given an oral and maxillofacial CT image without metal artifacts, roughly segment the teeth in the image using a threshold segmentation algorithm to obtain a coarse segmentation mask; after filling the holes in the coarse segmentation mask and eliminating unnecessary connections between teeth using morphological operations, extract each connected component in the coarse segmentation mask, perform a final screening according to the area standard of normal teeth, and finally obtain a fine segmentation mask for each tooth;

[0030] Step 102, setting different scanning and simulation parameters, including at least energy level;

[0031] Step 103: given an artifact-free oral and maxillofacial CT image and the fine segmentation mask of each tooth obtained in step 101, a plurality of corresponding simulated images affected by metal artifacts are generated, wherein the types and positions of metals simulated in these simulated images are different, and the steps include:

[0032] According to the fine segmentation masks of teeth corresponding to the oral and maxillofacial CT images, a certain number of teeth are randomly selected and fused to obtain multiple metal masks, which contain independent fine segmentation masks of one or more teeth in different positions and are used in the metal artifact synthesis process of the oral and maxillofacial CT images;

[0033] For the multiple energy levels set in step 102, the projection data of the tissue part, the bone part and the metal part in the oral and maxillofacial CT image are calculated according to the metal mask, the projections of the parts at different energy levels are summed to form a multi-energy synthetic projection, random Poisson noise is added, and a polynomial water correction model is applied to correct the errors in the imaging process;

[0034] The corrected multi-energy synthetic projections were back-projected and reconstructed to obtain simulated oral and maxillofacial CT images with metal artifacts.

[0035] Preferably, each of the spatial frequency domain Transformer blocks comprises layer normalization, a spatial frequency primitive mixer, a multi-layer perceptron and corresponding residual connections, wherein the spatial frequency primitive mixer comprises a window self-attention layer and / or a spatial frequency domain convolution layer;

[0036] The input feature map of the spatial frequency domain Transformer block is represented as F in , then the output feature map F of the current spatial frequency domain Transformer block out The shape and F in Keeping the same, the processing of the spatial frequency domain Transformer block is expressed as:

[0037] F out =MLP(F mix )+F mix

[0038] F mix =MixerJLN(F in )K+F in

[0039] Among them, Mixer(·) is a spatial frequency primitive mixer, LN(·) is a layer normalization operation, and MLP(·) is a multi-layer perceptron.

[0040] Preferably, in the window self-attention layer:

[0041] After the input feature map F is linearly transformed to obtain the query matrix Q, key matrix K and value matrix V, it is further split into and

[0042] Q i =XW i Q

[0043]

[0044] Among them, W @ E , and is the linear transformation matrix for the i-th head, i=1,…,N H , N H is the number of attention heads, d = C / N H is the characteristic dimension of each head;

[0045] The matrix Q of each head @ , K @ 、V @ It is divided into multiple M×M windows, and the window size M is set to 8. The query, key, and value matrices of each window are and The query, key, and value matrices for each head under each window are calculated as follows:

[0046]

[0047] in, is the output result of the i-th head;

[0048] Connect the output results of all heads and pass them through a linear transformation matrix Restore to the original channel dimension and get the multi-head attention result A under each window w :

[0049]

[0050] Combine the multi-head attention results of all windows to get the final multi-head attention result A.

[0051] Preferably, a sliding window mechanism is introduced in the window self-attention layer. Then, for multiple consecutive window self-attention layers, the window position of the odd-numbered layers remains unchanged, while the window position of the even-numbered layers slides M / 2 units to the lower right, where M is the window size.

[0052] Preferably, in the spatial frequency domain convolution layer:

[0053] The input feature map V is subjected to frequency convolution as shown in the following formula:

[0054]

[0055] Among them, conv 0×0 represents a 1×1 convolution, c(·) represents a composite transformation of a 1×1 convolution and ReLU activation, represents the fast Fourier transform, represents the inverse fast Fourier transform, V freq is the frequency domain feature obtained through residual connection;

[0056] At the same time, the input feature map V is spatially convolved to obtain the spatial domain feature V spat ;

[0057] The frequency domain feature V freq Finally, with the spatial domain feature V spat Sum and obtain the output of the spatial frequency domain convolution layer.

[0058] Preferably, in the spatial frequency primitive mixer, the output of the spatial frequency domain convolution layer is added to the multi-head attention result A, and fused through a linear layer to obtain the output of the spatial frequency primitive mixer.

[0059] Preferably, in step 3, the domain discriminator is composed of 4 “convolution-instance normalization-LeakyReLU” composite convolution layers, a tile embedding layer and a final layer of sigmoid function;

[0060] The domain discriminator takes the feature F1 output by the first-level encoder of the metal artifact removal model as input. The feature F1 is first downsampled to feature F through four layers of composite convolutional layers. disc , feature F disc After the tile embedding layer and sigmoid function, it is converted into a probability map Probability plot A pixel in contains the information of a 70×70 pixel block in feature F1. The pixel value range is [0,1], indicating whether the information of each patch comes from the real clinical image.

[0061] Preferably, in step 3, a hidden layer feature domain discriminator is introduced into the training framework, and the hidden layer feature domain discriminator includes several block embedding layers and a Leaky ReLU nonlinear activation function. The hidden layer feature domain discriminator takes the hidden layer features of the metal artifact removal model as input, downsamples it and determines whether the information of each block in the input feature comes from a real clinical image, and finally returns the discrimination result in the form of a probability map.

[0062] The present invention first proposes a method for synthesizing metal artifact data, and combines clinical data collection to construct paired synthetic oral and maxillofacial CT image datasets with and without artifacts and unpaired real oral and maxillofacial CT image datasets with and without artifacts. Then, the present invention proposes a metal artifact removal model training framework based on semi-supervised domain adaptation, which includes a metal artifact removal model and a latent feature domain discriminator. The framework can use the simulated dataset and real dataset constructed above to train a metal artifact removal model that can effectively remove metal artifacts in real oral and maxillofacial clinical CT images. The present invention effectively eliminates metal artifacts caused by metal dentures, implants, orthodontic brackets and archwires of different materials, titanium plates and titanium nails in oral and maxillofacial CT images through an improved deep learning model, improves image quality, and assists clinical diagnosis and treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is an overview of the training framework of the metal artifact removal model based on semi-supervised domain adaptation, where “FDMAR” is the metal artifact removal model and “Domain Disc.” is the domain discriminator introduced in the training process. DETAILED DESCRIPTION

[0064] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0065] The embodiment of the present invention discloses a method for removing metal artifacts in oral and maxillofacial CT, which specifically comprises the following steps:

[0066] The first step is to generate an oral and maxillofacial CT image dataset for model training, which includes a simulated dataset and a real dataset.

[0067] 1) Simulate metal artifact data synthesis to construct a simulated dataset

[0068] In order to effectively train the metal artifact removal model (abbreviated as "FDMAR") proposed in the present invention, a large number of CT images affected by artifacts and corresponding artifact-free images are required. In the absence of available paired real data, the present invention proposes an artifact simulation technology that synthesizes artifact data by understanding the optical properties of metal materials and the physical mechanism of the CT imaging process, determining the reasonable position of metal in the oral and maxillofacial region. This technology can simulate the situation of embedding different types of metal in multiple positions in an artifact-free oral and maxillofacial CT image, thereby obtaining multiple pairs of paired artifact-affected CT images and artifact-free CT images.

[0069] The simulation process of constructing multiple pairs of simulated oral and maxillofacial CT images with metal artifacts and simulated oral and maxillofacial CT images without metal artifacts from each oral and maxillofacial CT image without metal artifacts includes three stages: tooth segmentation and post-processing, simulation parameter setting, and artifact simulation:

[0070] The first stage: Given a CT image of the oral and maxillofacial region without metal artifacts, the teeth in the image are roughly segmented by the threshold segmentation algorithm to obtain a rough segmentation mask. After filling the holes in the rough segmentation mask and eliminating unnecessary connections between teeth through morphological operations, each connected component in the rough segmentation mask is extracted and finally screened according to the area standard of normal teeth to obtain a fine segmentation mask for each tooth.

[0071] The second stage: different scanning and simulation parameters need to be set, including energy level, noise scale, actual pixel size, scanning angle, metal type, etc., and then the attenuation coefficients, noise scale, energy composition and correction coefficients of water, bone and metal at different energies are set according to the table. These parameters will directly affect the quality and characteristics of the synthesized artifact image, and need to be set according to the actual application scenario, so that the distribution of synthesized metal artifacts is close to the distribution of metal artifacts in the image scanned by the actual CT machine.

[0072] The third stage: given an artifact-free oral and maxillofacial CT image (also called "given CT image") and the fine segmentation mask of each tooth obtained in the first stage, multiple corresponding images affected by metal artifacts are generated. The types and locations of metals simulated in these images are different to increase the diversity of the data. Specifically:

[0073] Firstly, according to the fine segmentation mask of the teeth corresponding to the given CT image, a certain number of teeth are randomly selected to fuse to obtain multiple metal masks, which contain independent fine segmentation masks of one or more teeth in different positions and will be used in the metal artifact synthesis process of the given CT image.

[0074] Then, for multiple energy levels set in the second stage, the projection data of the tissue part, bone part and metal part in the given CT image are calculated according to the metal mask. The projections of each part at different energy levels are summed to form a multi-energy synthetic projection. After adding random Poisson noise, a polynomial water correction model is applied to correct the errors in the imaging process.

[0075] Finally, the corrected multi-energy synthetic projections were back-projected and reconstructed to obtain simulated oral and maxillofacial CT images with metal artifacts.

[0076] Through the above simulation process, multiple pairs of simulated oral and maxillofacial CT images with metal artifacts and simulated oral and maxillofacial CT images without metal artifacts, as well as corresponding metal masks can be constructed from each oral and maxillofacial CT image without metal artifacts, which can be used for the subsequent training process of the metal artifact elimination model.

[0077] 2) Collection of real metal artifact data to build a real dataset

[0078] In addition to generating simulated paired metal artifact data in the first part mentioned above, the present invention also needs to collect a batch of real, unpaired CT images with metal artifacts and CT images without metal artifacts from the oral and maxillofacial CT scanning equipment used clinically as training data.

[0079] In this way, limited real data and a large amount of simulated data can be fully utilized to improve the generalization ability of the model on real data. To ensure the diversity of data, CT images should be collected from different patients, and the metals in them should include different materials (such as metal dentures, implants, titanium plates, titanium nails, etc.), sizes and numbers as much as possible.

[0080] The second step is to build and train the metal artifact removal model.

[0081] The present invention adopts a novel hourglass network as a metal artifact elimination model, called FDMAR, such as Figure 1 As shown in the figure, each encoder and decoder layer of this architecture contains several spatial-frequency transformer blocks, which mine the metal artifact features and anatomical structure features of the input CT image in both the spatial domain and the frequency domain, and efficiently fuse these features in multiple subspaces and multiple scales. After training, it can effectively remove metal artifacts and retain the details of the image content.

[0082] The architecture of FDMAR is as follows Figure 1As shown, the whole includes three levels of encoders and two levels of decoders. Each level of encoder includes a patch embedding layer and several spatial-frequency domain Transformer blocks. Each level of decoder consists of a spatial-frequency domain Transformer block and a corresponding patch unembedding layer. The features of the first and second level encoders are re-integrated into the corresponding decoder through skip connections. In the present invention, in order to learn to remove metal artifacts in the input CT image while retaining other image content, FDMAR is also configured with a global residual connection, whose input will be directly added to the output element by element.

[0083] 1) Encoder

[0084] The tile embedding layer of the first-level encoder is implemented with a 3×3 convolution, with a stride of 1, padding of 1, and an output channel of 24. This tile embedding layer does not implement the downsampling process, so the resolution of the feature map output by the first-level encoder is consistent with the input image. After that, the first-level encoder is configured with 8 spatial-frequency domain Transformer blocks. To avoid the additional computational cost introduced by high resolution, these spatial-frequency domain Transformer blocks do not contain windowed self-attention layers, but only spatial-frequency domain convolutional layers. Given an input image The feature map output by the first-level encoder On the one hand, it will serve as the input of the second-level encoder, and on the other hand, it will be fed into the corresponding first-level decoder through a skip connection.

[0085] The tile embedding layer of the second-level encoder uses a 2×2 convolution implementation with a stride of 2, padding of 0, and an output channel number of 48, which downsamples the feature map. After that, 8 spatial-frequency domain Transformer blocks are configured, but the first 4 spatial-frequency domain Transformer blocks only contain spatial-frequency domain convolutional layers, and the last 4 spatial-frequency domain Transformer blocks contain both windowed self-attention layers and spatial-frequency domain convolutional layers. The feature map output by the second-level encoder On the one hand, it will serve as the input of the third-level encoder, and on the other hand, it will be fed into the corresponding second-level decoder through a skip connection.

[0086] The tile embedding layer of the third-level encoder is implemented using a 2×2 convolution with a stride of 2, padding of 0, and 96 output channels. After that, 8 spatial-frequency domain Transformer blocks are configured, all of which contain a windowed self-attention layer and a spatial-frequency domain convolution layer. The feature map output by the third-level encoder is It will be directly used as the input of the second stage decoder.

[0087] 1) Spatial-frequency domain Transformer block

[0088] Each spatial-frequency domain Transformer block contains the following components: layer normalization, spatial-frequency token mixer, multi-layer perceptron (MLP), and some corresponding residual connections. Unless otherwise specified, the spatial-frequency token mixer contains windowed multi-head self-attention (W-MHSA) layers and spatial-frequency domain convolution layers. Windowed multi-head self-attention is an effective tool for capturing local features and long-range dependencies from different representation subspaces, and thus constitutes the backbone of the spatial-frequency token mixer. In addition, FDMAR also extracts spatial and frequency features in different representation subspaces by performing convolution operations in the spatial domain and frequency domain respectively, thereby enhancing the expressiveness of features.

[0089] like Figure 1 As shown, the input feature map of a spatial frequency domain Transformer block is represented as Among them, H′ and W′ are the height and width of the feature map, respectively, and C is the number of channels. Then the output feature map of the current spatial frequency domain Transformer block is The shape and F in Keeping it consistent, the processing of the spatial frequency domain Transformer block can be written as:

[0090] F out =MLP(F mix )+F mix

[0091] F mix =MixerJLN(F in )K+F in

[0092] Among them, Mixer(·) is the spatial frequency primitive mixer, LN(·) is the layer normalization operation, and MLP(·) is the multi-layer perceptron.

[0093] Inside the spatial frequency primitive mixer that includes a windowed multi-head self-attention layer and a spatial frequency domain convolution layer, the layer-normalized feature map The query matrix will first be generated through three different linear transformations Key Matrix Sum Matrix Q, K and V will be subsequently calculated in the window multi-head attention layer, and V will also be sent to the spatial frequency domain convolution layer in parallel to extract features.

[0094] a) Windowed multi-head attention layer

[0095] The windowed multi-head attention layer is an effective tool for capturing local features and long-range dependencies from different representation subspaces. The calculation process is described in detail below.

[0096] Using the previous symbolic definition, assume that the input feature map size is For each head of the multi-head attention, the query matrix Q, key matrix K and value matrix V obtained by linear transformation of the input features will be further split into and

[0097] Q i =XW i Q

[0098]

[0099] in, and is the linear transformation matrix for the i-th head, i=1,…,N h , N h is the number of attention heads, d = C / N h is the feature dimension of each head. The above three matrices Q for each head i , K i 、V i will be divided into multiple M×M windows, and the window size M is set to 8. The query, key, and value matrices of each window are and The query, key, and value matrices for each head under each window are calculated as follows:

[0100]

[0101] in, is the output result of the i-th head.

[0102] Then, the output results of all heads are connected and transformed through a linear transformation matrix Restore to the original channel dimension and get the multi-head attention result under each window The process is as follows:

[0103]

[0104] Finally, the multi-head attention results of all windows are combined to obtain the final multi-head attention result.

[0105] In order to increase the interaction between windows and capture a wider range of information, it is usually necessary to introduce a sliding window mechanism. In the sliding window mechanism, the window position will shift between different layers, usually by half the window size M. Specifically, for multiple consecutive window multi-head self-attention layers, the window position of the odd-numbered layers remains unchanged, while the window position of the even-numbered layers slides to the lower right by M / 2 units.

[0106] b) Spatial frequency domain convolution layer

[0107] The spatial frequency domain convolution can extract the spatial and frequency features in different representation subspaces and enhance the expressiveness of the features. Assume that the input feature map is Spatial convolution obtains spatial domain features through 3×3 convolution The frequency convolution process first performs a composite transformation of 1×1 convolution and ReLU activation on the input feature map, which is denoted as c(·), and then passes through the fast Fourier transform Mapped to the frequency domain, a composite transformation c(·) is performed in the frequency domain, and then the inverse fast Fourier transform is performed. Map back to the spatial domain, and finally obtain the frequency domain features through residual connection It is expressed as:

[0108]

[0109] Among them, conv 1×1 Represents a 1×1 convolution.

[0110] Frequency domain characteristics V freq Finally, with the spatial domain feature V spat Sum and add the features output by the window multi-head attention layer The sum is added and fused through a linear layer to obtain the output of the spatial-frequency domain Transformer block.

[0111] 2) Decoder

[0112] The second-level decoder is located after the third-level encoder, and its tile de-embedding layer is implemented using 2×2 transposed convolution, with a step size of 2, padding of 0, and an output channel number of 48. It first uses the tile de-embedding layer to transform the feature map output by the third-level encoder into Perform upsampling and dimension transformation to obtain Then, the feature map output by the second-level encoder The feature map is concatenated according to the channel dimension and reduced by a 1×1 convolution. As the input of the subsequent 4 spatial-frequency domain Transformer blocks, the output feature map The spatial-frequency domain Transformer block of the decoder contains only spatial-frequency domain convolutional layers.

[0113] The tile de-embedding layer of the first-level decoder is implemented using a 2×2 transposed convolution with a stride of 2, padding of 0, and an output channel number of 24. It first uses the tile de-embedding layer to transform the feature map output by the second-level decoder Perform upsampling and dimension transformation to obtain Then, the feature map output by the first-level encoder The feature map is concatenated according to the channel dimension and reduced by a 1×1 convolution. As the input of the subsequent 4 spatial-frequency domain Transformer blocks, the output feature map

[0114] The last layer of the decoder is the tile de-embedding layer, which is implemented using 3×3 transposed convolution with a step size of 1, a padding of 1, and an output channel number of 24. The feature map F5 is transformed by the tile de-embedding layer to obtain the predicted residual map After global residual connection, the final output of FDMAR is obtained Here X represents the input image of FDMAR.

[0115] The FDMAR training framework proposed in the present invention includes two stages: an initial warm-up stage and a semi-supervised domain adaptation stage.

[0116] In the initial warm-up stage, the FDMAR is supervisedly trained using the paired simulated data in the simulated data set, so that FDMAR has preliminary knowledge about the removal of metal artifacts in the oral and maxillofacial region. The FDMAR network is denoted as f θ (·), simulated data set D sim Each sample in contains a CT image X with simulated metal artifacts. sim , the corresponding artifact-free label image Y sim And the corresponding metal mask Msim , the network is trained by minimizing the L1 loss between the input images with simulated metal artifacts and the corresponding artifact-free label maps in the simulated data. The loss function is as follows:

[0117]

[0118] In the semi-supervised domain adaptation stage, a domain discriminator is introduced to perform adversarial training on FDMAR at the feature level using a mixture of simulated data and real data. During the adversarial training process, the parameters of FDMAR and the domain discriminator are updated alternately until convergence.

[0119] 1) Domain Discriminator Architecture

[0120] The domain discriminator consists of 4 composite convolutional layers of "convolution-instance normalization-LeakyReLU", a tile embedding layer, and a final sigmoid function. The convolution kernel size of each of the 4 composite convolutional layers is 4×4. The stride of the first three convolutional layers is 2 and the padding is 1. The stride and padding of the fourth convolutional layer are both 1. The number of output channels of each convolution layer is 32, 64, 128, and 256 respectively. The final tile embedding layer is implemented as a 4×4 convolutional layer with a stride and padding of 1.

[0121] The domain discriminator uses the features output by the first-level encoder of FDMAR is the input. F1 is first downsampled through 4 layers of composite convolutional layers to Finally, F disc After the tile embedding layer and sigmoid function, it is converted into a probability map A pixel in contains the information of a block of 70×70 pixels in F1. The pixel value range is [0,1], indicating whether the information of each patch comes from the real clinical image.

[0122] The training process of the domain discriminator is as follows: The domain discriminator is denoted as g φ (·), given a CT image X with artifacts, the image may come from a simulated dataset D sim , which may also come from a real clinical dataset D cli It first passes through the FDMAR encoder Get feature map Each pixel on the feature map F contains the information of the corresponding block in X. This feature will be passed through g φ (·) Get a probability map As a discriminant result, each pixel value on the probability map ranges from [0, 1], indicating the probability that the corresponding patch in X comes from the real clinical image. The domain discriminator is optimized by minimizing the following loss function:

[0123]

[0124] Among them, P is the label of the discrimination result. If X comes from the real data set D real , then P is an all-one matrix, and the output probability of the shape and domain discriminator Figure 1 If X comes from the simulated data set D sim , then P is an all-zero matrix.

[0125] The loss function of FDMAR training is the reconstruction loss in the first stage. Based on this, an additional domain invariance loss is added It is necessary not only to remove the artifacts in the image, but also to make it difficult for the domain discriminator to distinguish whether the features extracted by the encoder are from simulated metal artifact images or real metal artifact images. The overall loss function is specifically defined as follows:

[0126]

[0127] Among them, λ is a hyperparameter that balances the weights of the two losses, domain invariance loss The definitions are as follows.

[0128]

[0129] That is, use simulated (real) data to train f θ (·), adjust the parameter θ so that g φ (·) Mistakenly identifies simulated data features as real (simulated) data features. Reconstruction loss function Used to ensure the detail quality of the generated image, domain invariant loss function The domain invariance of the hidden features encoded by FDMAR is improved. Finally, FDMAR can learn to effectively remove metal artifacts and retain the details of image content through supervised training, while encoding domain-invariant features, thereby ensuring the generalization of the model on oral and maxillofacial CT images contaminated by real metal artifacts. The training process uses data augmentation such as random rotation and flipping to avoid overfitting.

[0130] The training framework proposed in the present invention also needs to introduce a hidden layer feature domain discriminator, which forces the metal artifact removal model to learn domain-invariant features during the training process through adversarial training. That is, features that can be used to eliminate simulated metal artifacts and can also be generalized to real metal artifact removal tasks.

[0131] like Figure 1 As shown in Figure 1, the hidden feature domain discriminator contains several patch embedding layers and Leaky ReLU nonlinear activation functions. The hidden feature domain discriminator takes the hidden features of FDMAR as input, downsamples them, and determines whether the information of each patch in the input features comes from the real clinical image, and finally returns the discrimination result in the form of a probability map.

Claims

1. A method for removing metal artifacts in oral and maxillofacial CT, characterized in that: The following steps are involved: Step 1, generating an oral and maxillofacial CT image dataset for model training, wherein the oral and maxillofacial CT image dataset includes a simulated dataset and a real dataset, wherein the simulated dataset includes paired simulated oral and maxillofacial CT images with metal artifacts and simulated oral and maxillofacial CT images without metal artifacts and corresponding metal masks, and the real dataset includes unpaired real oral and maxillofacial CT images with metal artifacts and real oral and maxillofacial CT images without metal artifacts; Step 2: Construct a metal artifact elimination model, which includes a three-stage encoder and a two-stage decoder, where: The feature map output by the first-level encoder is used as the input of the second-level encoder on the one hand, and is also fed into the corresponding first-level decoder through skip connections on the other hand; the first-level encoder includes a tile embedding layer and several spatial frequency domain Transformer blocks. After the input image is processed by the tile embedding layer, it is processed by several spatial frequency domain Transformer blocks. The tile embedding layer does not implement the downsampling process, and all spatial frequency domain Transformer blocks only contain spatial frequency domain convolutional layers; The feature map output by the second-level encoder is used as the input of the third-level encoder on the one hand, and is also fed into the corresponding second-level decoder through a jump connection on the other hand; the second-level encoder includes a tile embedding layer and several spatial frequency domain Transformer blocks. The tile embedding layer downsamples the input feature map and then processes it by several spatial frequency domain Transformer blocks. The first N spatial frequency domain Transformer blocks in the several spatial frequency domain Transformer blocks only contain spatial frequency domain convolutional layers, and the last M spatial frequency domain Transformer blocks contain both window self-attention layers and spatial frequency domain convolutional layers, N≥1, M≥1; The feature map output by the third-level encoder is directly used as the input of the second-level decoder; the third-level encoder includes a tile embedding layer implemented by convolution and several spatial-frequency domain Transformer blocks, all of which contain a windowed self-attention layer and a spatial-frequency domain convolution layer; The second-stage decoder is located after the third-stage encoder. The second-stage decoder first upsamples and transforms the dimension of the feature map output by the third-stage encoder through the tile de-embedding layer, and then concatenates it with the feature map output by the second-stage encoder according to the channel dimension. After a 1×1 convolution dimensionality reduction, the obtained feature map is input into several subsequent spatial frequency domain Transformer blocks, which only contain spatial frequency domain convolution layers. The first-level decoder first upsamples and transforms the feature map output by the second-level decoder through the tile de-embedding layer, and then concatenates it with the feature map output by the first-level encoder according to the channel dimension, and undergoes a 1×1 convolution to reduce the dimension. The resulting feature map is input into several subsequent spatial-frequency domain Transformer blocks, which only contain spatial-frequency domain convolution layers. The feature map output by the first-level decoder is transformed by the tile de-embedding layer to obtain the prediction residual map After global residual connection, the final output of the metal artifact removal model is obtained Where X represents the input image of the metal artifact removal model; Step 3: Train the metal artifact removal model: The training framework of the metal artifact removal model includes an initial warm-up phase and a semi-supervised domain adaptation phase, where: In the initial warm-up stage, the metal artifact removal model is supervisedly trained using paired simulated data in the simulated data set, so that the metal artifact removal model has preliminary knowledge about the removal of metal artifacts in the oral and maxillofacial region. The network is trained by minimizing the L1 loss between the input image with simulated metal artifacts in the simulated data and the corresponding artifact-free label map. The loss function is as follows: Among them, D sim represents a simulated data set, X sim is a simulated oral and maxillofacial CT image with metal artifacts, Y sim is the corresponding simulated oral and maxillofacial CT image without metal artifacts, M sim is the corresponding metal mask; In the semi-supervised domain adaptation stage, a domain discriminator is introduced, and a mixture of simulated data and real data is used to perform adversarial training on the metal artifact removal model at the feature level. During the adversarial training process, the parameters of the metal artifact removal model and the domain discriminator are updated alternately until convergence. The domain discriminator is optimized by minimizing the following loss function: Among them, P is the label of the discrimination result. If X comes from the real data set D real , then P is an all-one matrix, and its shape is consistent with the output probability map of the domain discriminator; if X comes from the simulated data set D sim , then P is an all-zero matrix; The loss function of the metal artifact removal model training is the reconstruction loss in the first stage. Based on this, an additional domain invariance loss is added It is necessary not only to remove the artifacts in the image, but also to make it difficult for the domain discriminator to distinguish whether the features extracted by the encoder are from simulated metal artifact images or real metal artifact images. The overall loss function is specifically defined as follows: Among them, λ is a hyperparameter that balances the weights of the two losses, domain invariance loss The definitions are as follows. That is, the encoder f of the metal artifact removal model is trained using simulated or real data , (·), the parameter θ is adjusted to make the domain discriminator f6(·) mistakenly discriminate the simulated data features as real or simulated data features.

2. The method for removing metal artifacts in oral and maxillofacial CT according to claim 1, characterized in that: In step 1, the following method is used to generate a paired simulated oral and maxillofacial CT image with metal artifacts and a simulated oral and maxillofacial CT image without metal artifacts and a corresponding metal mask: Step 101: Given an oral and maxillofacial CT image without metal artifacts, roughly segment the teeth in the image using a threshold segmentation algorithm to obtain a coarse segmentation mask; after filling the holes in the coarse segmentation mask and eliminating unnecessary connections between teeth using morphological operations, extract each connected component in the coarse segmentation mask, perform a final screening according to the area standard of normal teeth, and finally obtain a fine segmentation mask for each tooth; Step 102, setting different scanning and simulation parameters, including at least energy level; Step 103: given an artifact-free oral and maxillofacial CT image and the fine segmentation mask of each tooth obtained in step 101, a plurality of corresponding simulated images affected by metal artifacts are generated, wherein the types and positions of metals simulated in these simulated images are different, and the steps include: According to the fine segmentation masks of teeth corresponding to the oral and maxillofacial CT images, a certain number of teeth are randomly selected and fused to obtain multiple metal masks, which contain independent fine segmentation masks of one or more teeth in different positions and are used in the metal artifact synthesis process of the oral and maxillofacial CT images; For the multiple energy levels set in step 102, the projection data of the tissue part, the bone part and the metal part in the oral and maxillofacial CT image are calculated according to the metal mask, the projections of the parts at different energy levels are summed to form a multi-energy synthetic projection, random Poisson noise is added, and a polynomial water correction model is applied to correct the errors in the imaging process; The corrected multi-energy synthetic projections were back-projected and reconstructed to obtain simulated oral and maxillofacial CT images with metal artifacts.

3. The method for removing metal artifacts in oral and maxillofacial CT according to claim 1, characterized in that: Each of the spatial frequency domain Transformer blocks includes layer normalization, a spatial frequency primitive mixer, a multi-layer perceptron, and corresponding residual connections, wherein the spatial frequency primitive mixer includes a window self-attention layer and / or a spatial frequency domain convolution layer; The input feature map of the spatial frequency domain Transformer block is represented as F .7 , then the output feature map F of the current spatial frequency domain Transformer block out The shape and F in Keeping the same, the processing of the spatial frequency domain Transformer block is expressed as: F out =MLP(F mix )+F mix F mix =Mixer(LN(F in ))+F in Among them, Mixer(·) is a spatial frequency primitive mixer, LN(·) is a layer normalization operation, and MLP(·) is a multi-layer perceptron.

4. The method for removing metal artifacts in oral and maxillofacial CT according to claim 3, characterized in that: In the windowed self-attention layer: After the input feature map F is linearly transformed to obtain the query matrix Q, key matrix K and value matrix V, it is further split into and Q i =XW i Q K i =XW i K V i =XW i V Among them, W i Q , W i K and W i V is the linear transformation matrix for the i-th head, i=1,...,N h , N h is the number of attention heads, d = C / N h is the characteristic dimension of each head; The matrix Q of each head i , K i 、V i It is divided into multiple M×M windows, and the window size M is set to 8. The query, key, and value matrices of each window are and The query, key, and value matrices for each head under each window are calculated as follows: in, is the output result of the i-th head; Connect the output results of all heads and pass them through a linear transformation matrix Restore to the original channel dimension and get the multi-head attention result A under each window w : Combine the multi-head attention results of all windows to get the final multi-head attention result A.

5. The method for removing metal artifacts in oral and maxillofacial CT according to claim 4, characterized in that: A sliding window mechanism is introduced in the window self-attention layer. Then, for multiple consecutive window self-attention layers, the window position of the odd-numbered layers remains unchanged, while the window position of the even-numbered layers slides M / 2 units to the lower right, where M is the window size.

6. The method for removing metal artifacts in oral and maxillofacial CT according to claim 4, characterized in that: In the spatial frequency domain convolution layer: The input feature map V is subjected to frequency convolution as shown in the following formula: Among them, conv 1×1 represents a 1×1 convolution, c(·) represents a composite transformation of a 1×1 convolution and ReLU activation, represents the fast Fourier transform, represents the inverse fast Fourier transform, V freq is the frequency domain feature obtained through residual connection; At the same time, the input feature map V is spatially convolved to obtain the spatial domain feature V spat ; The frequency domain feature V freq Finally, with the spatial domain feature V spat Sum and obtain the output of the spatial frequency domain convolution layer.

7. The method for removing metal artifacts in oral and maxillofacial CT according to claim 6, characterized in that: In the spatial frequency primitive mixer, the output of the spatial frequency domain convolution layer is added to the multi-head attention result A, and fused through a linear layer to obtain the output of the spatial frequency primitive mixer.

8. The method for removing metal artifacts in oral and maxillofacial CT according to claim 1, characterized in that: In step 3, the domain discriminator consists of four "convolution-instance normalization-LeakyReLU" composite convolutional layers, a tile embedding layer, and the last layer of sigmoid function; The domain discriminator takes the feature F1 output by the first-level encoder of the metal artifact removal model as input. The feature F1 is first downsampled to feature F through four layers of composite convolutional layers. disc , feature F disc After the tile embedding layer and sigmoid function, it is converted into a probability map Probability plot A pixel in contains the information of a 70×70 pixel block in feature F1. The pixel value range is [0, 1], indicating whether the information of each block comes from the real clinical image.

9. The method for removing metal artifacts in oral and maxillofacial CT according to claim 1, characterized in that: In step 3, a hidden feature domain discriminator is introduced into the training framework. The hidden feature domain discriminator includes several block embedding layers and a Leaky ReLU nonlinear activation function. The hidden feature domain discriminator takes the hidden features of the metal artifact removal model as input, downsamples it and determines whether the information of each block in the input feature comes from a real clinical image, and finally returns the discrimination result in the form of a probability map.

Images