A 2D-3D registration method integrating dual attention mechanism and residual structure

Abstract

The present invention discloses a 2D-3D registration method integrating a dual attention mechanism and a residual structure, and the steps are as follows: 1) image preprocessing: preprocessing the 2D image to facilitate step 2) DA-ResNet regression prediction; 2) DA-ResNet regression prediction: establishing a DA-ResNet deep learning network, sending the preprocessed image into the DA-ResNet deep learning network to regress and predict the PCA coefficients used to characterize the 3D image; 3) PCA restoration of the 3D image: multiplying the PCA coefficients obtained in step 2) with their eigenvectors and summing them to obtain a 3D image array, thereby restoring the 2D image to a 3D image. The method is faster and more accurate in registration, has a wide range of adaptability, and good versatility.

Description

Technical Field

[0001] The present invention relates to multimodal registration technology, and specifically uses a deep learning method to train an algorithm model. After the training is completed, the model can be used to realize automatic registration from 2D images to 3D images, which belongs to the field of artificial intelligence computer vision. Background Art

[0002] Image registration is an important part of image processing and a necessary prerequisite for image fusion, analysis and target recognition. It is mainly a process of converting two or more images into a target image through certain geometric transformation methods. Multimodal registration is mainly a registration process for different types of images obtained by different imaging devices.

[0003] The registration of 2D images to 3D images is a branch of multimodal registration and is the premise of 3D reconstruction. It is widely used in practical scenarios such as medical image processing, unmanned driving, and augmented reality. For example, in medical image processing, 2D-3D registration uses X-ray images to restore the actual location of the tumor and guide the radiotherapy process; in unmanned driving, 2D-3D registration uses the captured 2D images to reconstruct the 3D scene and systematically judge the current road conditions; in augmented reality, 2D-3D registration uses historical images to create virtual reality scenes and realize human-computer interaction.

[0004] 2D-3D image registration can be divided into traditional processing methods and deep learning methods.

[0005] The traditional processing method is an iterative process. By continuously deforming the initialized 3D image, the 2D projection of the deformed 3D image has the highest similarity with the source 2D image. The final deformed 3D image is the target image. The entire deformation process is 2D-3D registration. Common algorithms include the motion model (MM) method and the motion model-free (MM-FD) method. Since iteration is a long process, this method has great limitations for some low-latency scenarios.

[0006] With the development of deep learning, deep neural networks have been widely used in various fields due to their automatic feature extraction and powerful nonlinear mapping capabilities, and have become an important means of image processing. In 2019, Wei et al. successfully applied deep neural networks to 2D-3D image registration using convolutional neural networks. However, convolution is a local area operation in deep neural networks, which captures long-distance dependencies by deepening the network depth. As the network depth increases, gradient diffusion occurs in back propagation, resulting in a decrease in the network's learning ability. In addition, the noise information extracted by the convolutional neural network will disrupt the established mapping relationship, resulting in poor results in 2D-3D image registration.

[0007] From the above analysis, it can be seen that the prior art has the following deficiencies:

[0008] 1. The existing traditional method is an iterative process that takes a long time to achieve high-precision alignment, and is difficult to apply to scenarios such as unmanned driving and medical radiotherapy that require low latency.

[0009] 2. Existing deep learning methods have the characteristics of low registration accuracy and poor generalization ability. In the registration process of poor source image quality, it is difficult to avoid the problem of image noise disturbing the registration effect; in the registration process of video sequences, it is difficult to avoid the problem of inaccurate registration details caused by similar scenes.

[0010] 3. Most existing registration algorithms are designed for specific scenarios and are not very versatile. There are large differences in images in different application scenarios. For example, medical imaging modalities include X-rays, computed tomography (CT), ultrasound (US), magnetic resonance imaging (MRI), etc. There is currently a lack of registration algorithms that can achieve good results for both medical images and natural images. Summary of the invention

[0011] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a 2D-3D registration method integrating a dual attention mechanism and a residual structure. The method has faster and more accurate registration, wide adaptability and good versatility.

[0012] The technical solution of the present invention is achieved in this way:

[0013] A 2D-3D registration method that integrates dual attention mechanism and residual structure, the steps are as follows:

[0014] 1) Image preprocessing: Preprocess the 2D image to facilitate step 2) DA-ResNet regression prediction;

[0015] 2) DA-ResNet regression prediction: Establish a DA-ResNet deep learning network, send the pre-processed image into the DA-ResNet deep learning network to regress and predict the PCA coefficients used to characterize the 3D image;

[0016] 3) PCA restores the 3D image: The PCA coefficients obtained in step 2) are multiplied by their eigenvectors and summed to obtain a 3D image array, thereby restoring the 2D image to a 3D image.

[0017] The image preprocessing described in step 1) includes, in sequence, denoising the 2D image to remove some useless information, normalizing the image to adjust the intensity of the image, and scaling the image to a fixed size to facilitate the DA-ResNet regression prediction in step 2).

[0018] The PCA restoration of the 3D image in step 3) is specifically as follows: first, a large number of 3D images are subjected to PCA dimensionality reduction for training a PCA eigenvector matrix, and then the eigenvector matrix is ​​multiplied by the PCA coefficients obtained by DA-ResNet regression prediction to obtain a 3D image array, thereby restoring the real 3D image; the function is expressed as:

[0019]

[0020] Among them, Img 3D represents a 3D image, p and q represent the PCA eigenvector and its corresponding PCA coefficient, respectively, and the subscript i is the index of the eigenvector.

[0021] Among them, the DA-ResNet deep learning network described in step 2) includes a ResNet-18 structure, a residual structure, a CBAM attention module and a SPA attention module; the ResNet-18 structure is composed of a convolution block, a pooling layer, five layer layers and two fully connected layers, wherein the residual structure exists in the layer layer; the CBAM attention module is located between the convolution block and the pooling layer, and is used to extract important channel and spatial information in the feature map in the low-level network structure; the SPA attention module is introduced at the connection structure in each layer layer, and is used to extract important correlation information in the feature map in the high-level network structure.

[0022] Furthermore, the residual structure is mainly composed of a skip-layer branch and a residual branch, wherein the residual branch is composed of three Conv-BN-ReLU blocks, and the skip-layer branch and the residual branch are merged between the last BN and ReLU.

[0023] The CBAM attention module is a module that combines channel and space attention mechanisms. The entire CBAM attention module is divided into two cascaded operation processes. The first operation is the channel attention module, which is used to measure the weight of each channel. The second operation is the spatial attention module, which is used to perceive the proportion of information in each region.

[0024] In the channel attention module, the input feature map is subjected to MaxPool and AvgPool operations in the layer dimension to obtain a 1×1×C vector representing global information, and a multi-layer perceptron MLP is used to learn deeper information. Finally, the outputs of the two are added and sent to the sigmoid function to obtain the attention weight; the calculation formula is as follows:

[0025] A C (F)=Sigmoid(MLP(AvgPool(F))+MLP(MaxPool(F)))

[0026] Among them, F and Ac (F) represents the input and output of the channel attention module, AvgPool() and MaxPool() represent average pooling and maximum pooling, MLP() represents multi-layer perceptron learning, and Sigmoid() is the sigmoid function;

[0027] In the spatial attention module, the feature map after the channel attention module is subjected to MaxPool and AvgPool operations in the channel dimension to obtain a vector of H×W×1. These two vectors are aggregated into a vector of H×W×2 through the Concat operation, and then the attention weight of the spatial attention module is obtained through a two-dimensional convolution operation with kernel_size of 7 and padding of 3 and a sigmoid function. The calculation formula is as follows:

[0028] A s (F c )=sigmoid(Conv([AvgPool(F c );MaxPool(F c )]))

[0029] where F c and A S (F c ) represent the input and output of the spatial attention module respectively, and Conv represents the convolution operation with a convolution kernel of 7 and a padding of 3.

[0030] The SPA attention module is a channel attention mechanism module that uses a pyramid structure to achieve structural regularization while exploring structural information; the SPA attention module consists of three submodules: a channel matcher, a spatial pyramid structure, and a channel communication component;

[0031] The channel matcher consists of a Conv-BN-ReLU block, which mainly realizes channel number matching and channel information integration; the spatial pyramid structure includes three different scales of adaptive average pooling of 4×4, 2×2 and 1×1, which integrates structural regularization and structural information into a feature map; the channel communication component passes the output of the spatial pyramid structure through FC-BN-ReLU and FC-BN-Sigmoid to finally obtain the attention weight map; the spatial pyramid structure is expressed as:

[0032] S(F)=[R(ρ global (F)),R(ρ coarse (F)),R(ρ fine (F))]

[0033] Where S represents the spatial pyramid function, S(F) represents the output of the spatial pyramid structure, ρ() represents the adaptive average pooling layer, global represents global pooling, the output result is 1×1, coarse represents coarse pooling, the output result is 4×4, fine represents fine pooling, the output result is 2×2, R() represents the Resize function, and finally the three outputs are concat-operated into a feature map; the output of the SPA attention module is expressed as:

[0034] A p (F) = Sigmoid(W fc (W fc (S(F))))

[0035] Among them A p (F) represents the output of the SPA attention module, W fc () represents the fully connected operation, and Sigmoid() represents the sigmoid function.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. The present invention uses the principal component analysis (PCA) method to characterize the key features of 3D images, greatly reducing the amount of data, thereby greatly reducing the amount of calculation in the 2D image to 3D image registration mapping process, making the registration faster and making real-time registration possible, which can better meet the low-latency requirements of unmanned driving and medical radiotherapy.

[0038] 2. The present invention proposes a DA-ResNet deep learning network model, which integrates a residual structure and a dual attention mechanism. The residual structure establishes a long connection by skipping layers, which alleviates the problems of gradient diffusion and network degradation, and can then build a deeper network to achieve more accurate regression mapping. The attention mechanism assigns more weights to more important feature areas or feature channels, so that the network pays attention to more effective features, ignores the noise part in the features, and improves the generalization ability of the algorithm. The dual attention mechanism is to use two different attention mechanisms in combination, so that different attention mechanisms perform their duties and pay attention to different feature forms respectively. The combined use method can better extract important information in the features, making the alignment results more accurate.

[0039] 3. This method has wide adaptability and good versatility, and can be widely used in multimodal image registration in fields such as unmanned driving, augmented reality and medical image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 -2D-3D real-time registration flow chart provided by the present invention.

[0041] Figure 2-Schematic diagram of the DA-ResNet network model provided by the present invention.

[0042] Figure 3 - Schematic diagram of the residual structure provided by the present invention.

[0043] Figure 4 - Schematic diagram of the CBAM module provided by the present invention.

[0044] Figure 5 - Schematic diagram of the SPA module provided by the present invention.

[0045] Figure 6 -Comparison chart of reconstruction effects using different registration algorithms in the present invention. DETAILED DESCRIPTION

[0046] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] The entire real-time 2D-3D image registration process of the present invention is as follows Figure 1 As shown in the figure, it is mainly divided into three stages: image preprocessing, DA-ResNet regression prediction and PCA restoration. For the convenience of explanation, the following uses medical 2D-3D image registration to illustrate the whole process. The input 2D X-ray image undergoes a series of image preprocessing operations such as denoising, normalization and scale transformation, and is sent to the DA-ResNet network for regression prediction of the PCA coefficients used to characterize the 3D-CT image. Finally, the PCA technology is combined to restore the real 3D-CT target image.

[0048] 1. Image preprocessing: In order to ensure stable performance of the entire real-time registration process, the input image needs to be properly adjusted. The input 2D image is denoised to remove some useless information, and the image intensity is adjusted by normalization. Finally, the image is scaled to a fixed size to facilitate the subsequent DA-ResNet regression prediction.

[0049] 2. DA-ResNet deep learning network: DA-ResNet deep learning network is mainly used for regression prediction. The input 2D image is predicted by the deep learning network to obtain the PCA coefficients used to represent the 3D image. The DA-ResNet network structure is as follows: Figure 2As shown, it is mainly improved on ResNet-18. The basic ResNet-18 structure is composed of 1 convolution block, 1 pooling layer, 5 layer layers and 2 fully connected layers, in which the residual structure exists in the layer layer. The present invention introduces a dual attention mechanism into the ResNet-18 structure, introduces a CBAM attention module between the convolution block and the pooling layer, and is used to extract important channel and spatial information in the feature map in the low-level network structure; introduces an SPA attention module at the connection structure in each layer layer, and is used to extract important correlation information in the feature map in the high-level network structure.

[0050] 1. Residual structure: The characteristic of the residual structure is to bypass some layers through a skip layer structure and merge its input and output. Figure 3 As shown in the figure, it mainly consists of a skip branch and a residual branch, where the residual branch consists of three Conv-BN-ReLU blocks, and the skip branch and the residual branch are merged between the last BN and ReLU. The residual structure allows the input signal to reach the end of the residual branch directly without passing through the residual branch, effectively preventing the gradient disappearance and network degradation problems in deeper neural networks.

[0051] 2. CBAM attention module: CBAM is a channel and space attention mechanism module, its structure is as follows Figure 4 The entire CBAM is divided into two cascaded operation processes. The first operation is the channel attention module, which is used to measure the weight of each channel; the second operation is the spatial attention module, which is used to perceive the proportion of information in each region.

[0052] In the channel attention module, the input feature map is subjected to MaxPool and AvgPool operations in the layer dimension to obtain a 1×1×C vector representing global information, and a multi-layer perceptron (MLP) is used to learn deeper information. Finally, the outputs of the two are added and sent to the sigmoid function to obtain the attention weight. The calculation process is summarized as follows:

[0053] A C (F)=Sigmoid(MLP(AvgPool(F))+MLP(MaxPool(F)))

[0054] Among them, F and A c (F) represents the input and output of the channel attention module respectively, AvgPool() and MaxPool() represent average pooling and maximum pooling, MLP() represents multi-layer perceptron learning, and Sigmoid() is the sigmoid function.

[0055] In the spatial attention module, the feature map after the channel attention module is subjected to MaxPool and AvgPool operations in the channel dimension to obtain a vector of H×W×1. These two vectors are aggregated into a vector of H×W×2 through the Concat operation, and then the attention weight of the spatial attention module is obtained through a two-dimensional convolution operation with kernel_size of 7 and padding of 3 and a sigmoid function. The calculation process is summarized as follows:

[0056] A s (F c )=sigmoid(Conv([AvgPool(F c );MaxPool(F c )]))

[0057] where F c and A S (F c ) represent the input and output of the spatial attention module respectively, and Conv represents the convolution operation with a convolution kernel of 7 and a padding of 3.

[0058] 3. SPA attention module: SPA is a channel attention mechanism module that uses a pyramid structure to achieve structural regularization while exploring structural information. The SPA attention module consists of three submodules: channel matcher, spatial pyramid structure, and channel communication component. Figure 5 .

[0059] The channel matcher consists of a Conv-BN-ReLU block, which mainly realizes channel number matching and channel information integration. The spatial pyramid structure includes three different scales of adaptive average pooling, 4×4, 2×2 and 1×1, which integrates structural regularization and structural information into a feature map. The channel communication component passes the output of the spatial pyramid structure through FC-BN-ReLU and FC-BN-Sigmoid to finally obtain the attention weight map. The spatial pyramid component can be expressed as:

[0060] S(F)=[R(ρ global (F)),R(ρ coarse (F)),R(ρ fine (F))]

[0061] Among them, S represents the spatial pyramid function, S(F) represents the output of the spatial pyramid structure, ρ() represents the adaptive average pooling layer, and global represents global pooling, the output result is 1×1, coarse represents coarse pooling, the output result is 4×4, fine represents fine pooling, the output result is 2×2, and R() represents the Resize function. Finally, the three outputs are concat-operated into a feature map. The output of the SPA attention module is expressed as:

[0062] A p (F) = Sigmoid(W fc (W fc (S(F))))

[0063] Among them A p (F) represents the output of the SPA attention module, W fc () represents the fully connected operation, Sigmoid() represents the sigmoid function

[0064] 3. PCA restoration of 3D images: PCA restoration can be described as a process of multiplying a series of PCA coefficients with their eigenvectors and summing them to represent a high-dimensional array. In actual use, PCA dimensionality reduction is first performed on a large number of 3D images to train a PCA eigenvector matrix, and then the eigenvector matrix is ​​multiplied with the PCA coefficients predicted by DA-ResNet to obtain a 3D image array, thereby restoring the real 3D image. The entire PCA technology restoration process can be expressed as:

[0065]

[0066] Among them, Img 3D represents a 3D image, p and q represent the PCA eigenvector and its corresponding PCA coefficient, respectively, and the subscript i is the index of the eigenvector.

[0067] This paper takes the 2D-3D registration of medical lung images as an example. The 2D source image is an X-ray image and the 3D target image is a CT image. In the experiment, 9000 X-ray images were used to train the model and 1000 images were used for testing. 2), Mean Absolute Difference (MAD), Normalized Cross Correlation (NCC) and Structural Similarity Index (SSIM) to evaluate algorithm performance. The present invention uses the DA-ResNet algorithm to compare with multiple algorithms such as CNN, Unet and ResNet, where the algorithm comparison results are shown in Table 1. In addition, the embodiment of the present invention performs 3D image reconstruction on the algorithm results, and the reconstruction effect is shown in Table 1. Figure 6 As shown in the figure (the solid line box in the figure represents the detailed comparison of the reconstruction effect of each algorithm, and the dotted line box represents the detailed comparison of the reconstruction effect of each algorithm relative to the ground truth), and the indicators are used to quantify it. The quantification results are shown in Table 2. The experimental results show that the DA-ResNet algorithm has the best registration performance in this embodiment.

[0068] Table 1

[0069]

[0070] Table 2

[0071]

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the applicant has described the present invention in detail with reference to the preferred embodiments, ordinary technicians in the field should understand that modifications or equivalent substitutions of the technical solution of the present invention without departing from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.

Claims

1. A 2D-3D registration method integrating dual attention mechanism and residual structure, characterized by: Here are the steps: 1) Image preprocessing: Preprocess the 2D image to facilitate step 2) DA-ResNet regression prediction; 2) DA-ResNet regression prediction: Establish a DA-ResNet deep learning network, send the pre-processed image into the DA-ResNet deep learning network to regress and predict the PCA coefficients used to characterize the 3D image; 3) PCA restores the 3D image: The PCA coefficients obtained in step 2) are multiplied by their eigenvectors and summed to obtain a 3D image array, thereby restoring the 2D image to a 3D image.

2. According to claim 1, a 2D-3D registration method integrating dual attention mechanism and residual structure is characterized by: The image preprocessing described in step 1) includes, in sequence, denoising the 2D image to remove some useless information, normalizing the image to adjust the intensity of the image, and scaling the image to a fixed size to facilitate DA-ResNet regression prediction in step 2).

3. According to claim 1, a 2D-3D registration method integrating dual attention mechanism and residual structure is characterized by: The PCA restoration of the 3D image in step 3) is specifically as follows: first, a large number of 3D images are subjected to PCA dimensionality reduction to train a PCA eigenvector matrix, and then the eigenvector matrix is ​​multiplied by the PCA coefficients obtained by DA-ResNet regression prediction to obtain a 3D image array, thereby restoring the real 3D image; the function is expressed as: Among them, Img 3D represents a 3D image, p and q represent the PCA eigenvector and its corresponding PCA coefficient, respectively, and the subscript i is the index of the eigenvector.

4. According to claim 1, a 2D-3D registration method integrating dual attention mechanism and residual structure is characterized by: The DA-ResNet deep learning network described in step 2) includes a ResNet-18 structure, a residual structure, a CBAM attention module and a SPA attention module; the ResNet-18 structure is composed of a convolution block, a pooling layer, five layer layers and two fully connected layers, wherein the residual structure exists in the layer layer; the CBAM attention module is located between the convolution block and the pooling layer, and is used to extract important channel and spatial information in the feature map in the low-level network structure; the SPA attention module is introduced at the connection structure within each layer layer, and is used to extract important correlation information in the feature map in the high-level network structure.

5. According to claim 4, a 2D-3D registration method integrating dual attention mechanism and residual structure is characterized in that: The residual structure mainly consists of a skip-layer branch and a residual branch, wherein the residual branch consists of three Conv-BN-ReLU blocks, and the skip-layer branch and the residual branch are merged between the last BN and ReLU.

6. The 2D-3D registration method integrating dual attention mechanism and residual structure according to claim 4, characterized in that: The CBAM attention module is a module that combines channel and space attention mechanisms. The entire CBAM attention module is divided into two cascaded operation processes. The first operation is the channel attention module, which is used to measure the weight of each channel. The second operation is the spatial attention module, which is used to perceive the proportion of information in each region. In the channel attention module, the input feature map is subjected to MaxPool and AvgPool operations in the layer dimension to obtain a 1×1×C vector representing global information, and a multi-layer perceptron MLP is used to learn deeper information. Finally, the outputs of the two are added and sent to the sigmoid function to obtain the attention weight; the calculation formula is as follows: A C (F)=Sigmoid(MLP(AvgPool(F))+MLP(MaxPool(F))) Among them, F and A c (F) represents the input and output of the channel attention module, AvgPool() and MaxPool() represent average pooling and maximum pooling, MLP() represents multi-layer perceptron learning, and Sigmoid() is the sigmoid function; In the spatial attention module, the feature map after the channel attention module is subjected to MaxPool and AvgPool operations in the channel dimension to obtain a vector of H×W×1. These two vectors are aggregated into a vector of H×W×2 through the Concat operation, and then the attention weight of the spatial attention module is obtained through a two-dimensional convolution operation with kernel_size of 7 and padding of 3 and a sigmoid function. The calculation formula is as follows: A s (F c )=sigmoid(Conv([AvgPool(F c );MaxPool(F c )])) where F c and A S (F c ) represent the input and output of the spatial attention module respectively, and Conv represents the convolution operation with a convolution kernel of 7 and a padding of 3.

7. The 2D-3D registration method integrating dual attention mechanism and residual structure according to claim 4, characterized in that: The SPA attention module is a channel attention mechanism module that uses a pyramid structure to achieve structural regularization while exploring structural information; the SPA attention module consists of three submodules: a channel matcher, a spatial pyramid structure, and a channel communication component; The channel matcher consists of a Conv-BN-ReLU block, which mainly realizes channel number matching and channel information integration; the spatial pyramid structure includes adaptive average pooling of three different scales: 4×4, 2×2 and 1×1, which integrates structural regularization and structural information into a feature map; the channel communication component passes the output of the spatial pyramid structure through FC-BN-ReLU and FC-BN-Sigmoid to finally obtain the attention weight map; The spatial pyramid structure is expressed as: S(F)=[R(ρ global (F)),R(ρ coarse (F)),R(ρ fine (F))] Where S represents the spatial pyramid function, S(F) represents the output of the spatial pyramid structure, ρ() represents the adaptive average pooling layer, global represents global pooling, the output result is 1×1, coarse represents coarse pooling, the output result is 4×4, fine represents fine pooling, the output result is 2×2, R() represents the Resize function, and finally the three outputs are concat-operated into a feature map; the output of the SPA attention module is expressed as: A p (F)=Sigmoid(W fc (W fc (S(F)))) Among them A p (F) represents the output of the SPA attention module, W fc () represents the fully connected operation, and Sigmoid() represents the sigmoid function.

Images