Multimodal image registration method based on predictive correction and convergent attention transformer

Abstract

The present invention discloses a multimodal image registration method based on prediction correction and convergent attention transformer. The method first uses a convolutional neural network to extract features from the target image and the source image, then performs self-attention feature modification on the target image / source image feature map based on the self-attention mechanism, and further predicts the residual registration parameters based on the cross-attention mechanism; the above process is repeated a set number of times, and the residual registration parameters are accumulated to obtain the final registration parameters, so that the source image can be deformed. The method of the present invention introduces an iterative reasoning framework based on model prediction to guide deformation correction, which improves the robustness of the model and enhances the accuracy of registration. Experimental results and analysis prove the effectiveness and superiority of the multimodal image registration method based on prediction correction and convergent attention transformer proposed by the present invention in solving the problem of multimodal image registration.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a multimodal image registration method based on prediction correction and convergent attention transformer. Background Art

[0002] Multimodal image registration involves spatially aligning and matching images from different sensors or modalities to facilitate cross-modal data analysis and integration. Images from different modalities may have different feature representations and information content, so registering them can establish correspondence between them, enabling consistency analysis and cross-validation of cross-modal data. However, multimodal image registration presents significant challenges due to factors such as modality differences, geometric deformation, noise, and artifacts.

[0003] Traditional image registration methods are mainly based on feature point construction and feature descriptor matching algorithms. Feature point construction detects feature points (such as corners and edges) in the image and then searches for matching feature point pairs in the two images. Common feature point construction algorithms include SIFT (Scale Invariant Feature Transform) and SURF (Speeded Up Robust Features). Feature descriptor matching algorithms use the local area around the feature point to describe its features and then perform matching by comparing descriptors. Common feature descriptor matching methods include RANSAC and MAGSAC.

[0004] The above-mentioned traditional image registration methods still play an important role in many applications. They perform well in image registration tasks with small deformations and modal differences. However, for image registration problems with large-scale deformations, nonlinear transformations and modal differences, the performance of traditional methods may be limited. In recent years, with the development of deep learning, neural network-based methods have made significant progress in multimodal image registration tasks. End-to-end deep learning models can directly learn to predict the deformation between two images, improving the accuracy and robustness of registration. Related deep learning methods include DHN (Deep image homography estimation) and MHN (Deep homography estimation for dynamic scenes). The above deep learning methods have improved the performance of multimodal image registration problems with large deformations, but still cannot achieve high-precision cross-modal image registration. Summary of the Invention

[0005] In response to the above-mentioned deficiencies in the prior art, the present invention provides a multimodal image registration method based on predictive correction and convergent attention transformer, which can be used to align multimodal images with high precision and achieve advanced matching performance.

[0006] The present invention is achieved by adopting the following technical solutions:

[0007] A multimodal image registration method based on predictive correction and convergent attention transformer is proposed. Given a multimodal image pair to be registered, the multimodal image pair includes a target image and a source image. The target image is used as a reference and the source image is spatially deformed using the predictive registration parameters. The method includes the following steps:

[0008] 1) First, convolutional neural networks are used to extract features from the target image and source image in two ways to obtain target image / source image feature maps. Then, self-attention features are modified on the target image / source image feature maps based on the self-attention mechanism. Then, mutual attention modification is performed on the target image feature map and the source image feature map after self-attention feature modification based on the cross-attention mechanism. Finally, the registration parameter residual is predicted based on the cross-attention mechanism.

[0009] 2) Using the registration parameter residuals to update the current predicted registration parameters, and then using the current predicted registration parameters to perform deformation correction on the source image; replacing the previous source image with the deformation corrected source image, and continuing with step 1) to output the next registration parameter residuals;

[0010] 3) Execute step 2) until the preset number of iterations has passed and output the final predicted registration parameters;

[0011] 4) The final output predicted registration parameters are used to perform deformation correction on the source image, thereby achieving multimodal image registration.

[0012] In the above technical solution, further, in the step 1), a convolutional neural network is used to extract the features of the target image and the source image. The specific method is: first, a convolutional layer with a kernel size of 7×7 is used to perform preliminary processing on the target image / source image; then, multiple 3×3 convolutional layers are used to process the image, wherein the 3×3 convolutional layer includes a downsampling convolutional layer and a non-downsampling convolutional layer, and the downsampling convolutional layer downsamples the features in a ratio of 2×2 in the spatial dimension; finally, a 1×1 convolutional layer is used to perform dimension projection on the feature map to obtain the target image / source image feature map.

[0013] Furthermore, in step 1), the target image / source image feature map is modified with self-attention features based on the self-attention mechanism, and then the target image feature map and the source image feature map after self-attention feature modification are modified with mutual attention based on the cross-attention mechanism. Finally, the registration parameter residual is predicted based on the cross-attention mechanism. This is specifically implemented based on the converged attention transformer structure, and the specific steps are as follows:

[0014] S1: The target image feature map or source image feature map extracted by the convolutional neural network is divided into two paths, and the self-attention mechanism is used to calculate the correlation between the extracted features at different coordinate positions, thereby capturing the dependency relationship between the spatial positions of the image and allowing information interaction between different positions to achieve self-attention feature modification; the calculation of the self-attention mechanism is shown in formula (1):

[0015]

[0016] Where x is the coordinate index of the target image / source image feature map; Q, K, and V are the query matrix, key matrix, and value matrix, respectively, which are obtained by processing the target image / source image feature map using a fully connected layer; Represents the area around the coordinate position x, that is, the scope of the self-attention mechanism; if If the whole feature map is set, it is a global self-attention mechanism, otherwise it is a non-global or local self-attention mechanism with a certain radius. During the iteration process, the scope of the self-attention mechanism decreases as the number of iterations increases.

[0017] After the self-attention mechanism calculation is completed, the attention features obtained after the self-attention mechanism calculation are processed using the layer normalization (LayerNorm) method, and added to the target image / source image feature map obtained by feature extraction to complete the feature modification of the self-attention;

[0018] S2: The cross-attention mechanism is used to calculate the correlation between the target image feature map and the source image feature map after self-attention feature modification at different coordinate positions to capture the correlation between the spatial positions of the two. The calculation of the cross-attention mechanism is shown in formula (2):

[0019]

[0020] Wherein, the subscripts i and j indicate that the query matrix Q comes from one of the feature maps, while the key matrix K and value matrix V come from the other feature map; Represents the area around x where the cross attention mechanism is located, i.e., the scope of the cross attention mechanism. If the whole feature map is used, it is a global cross-attention mechanism, otherwise it is a non-global or local cross-attention mechanism with a certain radius. During the iteration process, the scope of the cross-attention mechanism decreases as the number of iterations increases.

[0021] After completing the cross-attention mechanism calculation, the attention features obtained after the cross-attention mechanism calculation are processed using layer normalization, and added to the target image / source image feature map modified by the self-attention feature to complete the feature modification of the mutual attention; then the fully connected layer is used to process the features modified by the mutual attention, and the features processed by the fully connected layer are added to the features modified by the mutual attention to obtain the final self-attention and cross-attention modified target image / source image feature map;

[0022] S3: Prediction of registration parameter residuals; For the self-attention and cross-attention modified target image / source image feature maps, use formula (2) to perform cross-attention calculation, directly calculate the mutual correlation between the two, and input the calculated cross-attention into the fully connected layer to output the registration parameter residuals.

[0023] Furthermore, in step 2), the current predicted registration parameters are updated using the registration parameter residuals, and then the source image is deformed using the current predicted registration parameters; specifically:

[0024] First, the matrix multiplication of the last iteration prediction registration parameter and the current iteration prediction registration parameter residual is performed to obtain the current prediction registration parameter, as shown in formula (3).

[0025] D n =D n-1 ΔD n ,#(3)

[0026] Among them, D n-1 Denotes the last iteration prediction registration parameter, ΔD n Represents the residual error of the current iteration prediction registration parameter, D n Current predicted registration parameters, n represents the number of iterations, D 0 Indicates the identity transformation, that is, the registration parameters predicted in the previous iteration are unchanged when n=1 for the first time;

[0027] Then, the source image is deformed and corrected using the current predicted registration parameters, as calculated in formula (4):

[0028]

[0029] Among them, I2 represents the source image, π represents the coordinate mapping, Represents the source image after Dn deformation correction

[0030] The present invention also provides a registration model for executing the multimodal image registration method based on prediction correction and converged attention transformer, characterized in that the training process of the registration model includes the following steps:

[0031] 1) Construct a multimodal image registration dataset; use one modality as the target image and the other modality as the source image, and have the true values ​​of the registration parameters of both for supervision;

[0032] 2) Using convolutional neural networks to extract features from the target image and source image in two ways to obtain target image / source image feature maps: the input image is first preliminarily processed by a convolutional layer with a kernel size of 7×7, and then processed by multiple 3×3 convolutional layers. The 3×3 convolutional layers are divided into downsampling convolutional layers and non-downsampling convolutional layers. The downsampling convolutional layers downsample the features by a ratio of 2×2 in the spatial dimension as needed; finally, a 1×1 convolutional layer is used to project the feature map to obtain the target image / source image feature map; the converged attention transformer structure is used to perform feature modification on the target image / source image feature map based on self-attention and cross-attention, and finally the registration parameter residual is predicted based on the cross-attention mechanism;

[0033] 3) Using the registration parameter residual to update the current predicted registration parameter to obtain the current predicted registration parameter; performing deformation correction on the source image based on the current predicted registration parameter; replacing the previous source image with the deformation corrected source image, and continuing with step 2) to output the next registration parameter residual;

[0034] 4) Repeat step 3) until the number of iterations n reaches the preset number of iterations, obtain the final predicted registration parameters, and use the final output predicted registration parameters to correct the deformation of the source image; for two-modal images, if the number of iterations n = 1, then process them separately in two ways; if the number of iterations n > 1, then only process the source image; for the predicted registration parameters D in each iteration n , the true value of the registration parameter D gt and the predicted registration parameter D n The L1 distance between them is used for supervision, and the sum of all the registration parameters output during the n iterations is used as the loss function, as shown in formula (5):

[0035]

[0036] Where N represents the total number of iterations, and n ranges from 1 to N.

[0037] The beneficial effects of the present invention are:

[0038] To address the problem of inconsistent feature content in multimodal images, the present invention uses a converged attention transformer to perform feature modification, improving the expressive power of features and thus improving feature consistency. Unlike ordinary transformer usage methods, this method improves the ordinary attention mechanism to converged attention, significantly improving the accuracy of multimodal image registration while reducing the amount of computation. Convolutional neural networks generally do not have the identity for deformations such as rotation, scale, shearing, aspect ratio scaling, and perspective, and will extract inconsistent features for the content of the matching position. Model prediction is used to guide the source image to perform deformation correction, and then extract features, thereby significantly improving the accuracy of multimodal image registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flow chart for the implementation of the present invention;

[0040] Figure 2 The feature extraction method implemented in the present invention;

[0041] Figure 3 The convergent attention transformer implemented by the present invention;

[0042] Figure 4 The attention-gathering method of the attention-gathering transformer implemented by the present invention;

[0043] Figure 5 The model prediction guides the deformation correction operation implemented by the present invention;

[0044] Figure 6 This is the registration result implemented by the present invention. DETAILED DESCRIPTION

[0045] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0046] like Figure 1 As shown in the figure, the multimodal image registration method based on model prediction-guided deformation correction and a converged attention transformer structure described in the present invention takes two multimodal images as input, with one modality as the target image and the other as the source image. Using the target image as a reference, the source image is deformed and corrected to the target image through a registration process. The specific steps include:

[0047] (1) Using a convolutional neural network, feature extraction is performed on the two modal images in two ways to obtain the target image feature map and the source image feature map. Preferably, the parameters of the two convolutional neural networks are not shared. Using a converged attention transformer structure, feature modification based on self-attention and cross-attention mechanisms is performed in two ways, and finally the registration parameter residual is predicted based on the cross-attention mechanism. Preferably, the learnable parameters in the two-way converged self-attention and cross-attention mechanisms are independent of each other;

[0048] (2) Use the registration parameter residual to update the current predicted registration parameter, and then use the current predicted registration parameter to perform deformation correction on the source image; replace the previous source image with the deformation corrected source image, continue to perform step 1), and output the next registration parameter residual.

[0049] (3) Execute step 2) until the preset number of iterations N, and then stop and output the final predicted registration parameters. Since the target image path is extracted when n = 1, and the source image is continuously deformed during the iteration process. If the number of iterations n > 1, only the source image path is processed.

[0050] (4) The final output predicted registration parameters are used to perform deformation correction on the source image, thereby achieving multimodal image registration.

[0051] Figure 2 The feature extraction method implemented by the present invention is demonstrated. It uses two convolutional layers with a kernel size of 7×7 to perform preliminary processing on the target image / source image, and then uses a 3×3 convolutional layer to implement 2×2 downsampling of the features in the spatial dimension. Then, a 3×3 convolutional layer is used to convolve the downsampled features, and a 1×1 convolutional layer is used to perform dimension projection on the feature map, and a feature map of scale 2 (larger spatial resolution) is output; next, a 3×3 convolutional layer is continued to be used to implement 2×2 downsampling of the features in the spatial dimension, and then a 3×3 convolutional layer is used to convolve the downsampled features, and a 1×1 convolutional layer is used to perform dimension projection on the feature map, and a feature map of scale 1 (smaller spatial resolution) is output. Preferably, this method directly uses a downsampling convolutional layer with a stride of 2 to reduce the amount of inference calculation, which is marked as a bidirectional arrow, and the number of convolution kernel channels is marked in the figure.

[0052] Figure 3The present invention shows a converged attention transformer structure implemented by the present invention, which consists of operations such as self-attention, normalization, cross-attention, and a fully connected layer. The converged attention transformer structure first implements feature modification of the target image feature map and the source image feature map based on self-attention and feature modification based on cross-attention between the two in two ways. The two types of attention implementation methods are shown in formulas (1) and (2). The converged attention transformer structure finally uses cross-attention in conjunction with the fully connected layer to predict the alignment parameters. Based on the obtained alignment parameter residual ΔD, the alignment parameters are updated using formula (3). Preferably, only one self-attention layer and one cross-attention layer are used in the converged attention transformer structure. Since it is used iteratively and to improve computational efficiency, the attention layer only uses one attention head.

[0053] Figure 4 This paper demonstrates the convergent attention method of the convergent attention transformer architecture implemented in the present invention. As the network iterates and infers, the attention scope of the convergent attention transformer shifts from global to non-global, and then from non-global to local. Preferably, the present invention uses an attention scope of [global, 9, 7, 7, 5, 3, ...].

[0054] Figure 5 This paper demonstrates the model prediction-guided deformation correction operation implemented in the present invention, which requires the use of an iterative network inference process. After passing through the feature extractor, the target and source images are fed into a converged attention transformer, which predicts the residual registration parameters for the current iteration. Based on these predicted residuals, the predicted registration parameters are updated, guiding the source image to undergo deformation correction. The updated source image features are then fed into the corresponding feature extractor, which extracts them and feeds them into the converged attention transformer for the next round of residual registration parameter prediction.

[0055] Figure 6 The registration effects of the present invention and other existing methods are demonstrated, and the error is described by ACE (average corner error). It can be seen that the method of the present invention has higher cross-modal image registration accuracy.

[0056] The above is only a specific implementation method of the present invention, which cannot be used to limit the scope of the present invention. Equal changes made by ordinary technicians in this technical field based on this creation, as well as changes well known to technicians in this field, should still fall within the scope of the present invention.

Claims

1. A multimodal image registration method based on predictive correction and convergent attention transformer, characterized by: Given a multimodal image pair to be registered, the multimodal image pair includes a target image and a source image; using the target image as a reference, the source image is deformed and corrected using the predicted registration parameters; specifically, the following steps are included: 1) First, convolutional neural networks are used to extract features from the target image and source image in two ways to obtain target image / source image feature maps. Then, self-attention features are modified on the target image / source image feature maps based on the self-attention mechanism. Then, mutual attention modification is performed on the target image feature map and the source image feature map after self-attention features modification based on the cross-attention mechanism. Finally, the registration parameter residual is predicted based on the cross-attention mechanism. 2) Use the registration parameter residual to update the current predicted registration parameter, and then use the current predicted registration parameter to correct the source image; replace the previous source image with the corrected source image, and continue with step 1) to output the next registration parameter residual; 3) Execute step 2) until the preset number of iterations has passed and output the final predicted registration parameters; 4) Use the final output predicted registration parameters to perform deformation correction on the source image, thereby achieving multimodal image registration; In step 1), the target image / source image feature map is modified with self-attention features based on the self-attention mechanism, and then the target image feature map and the source image feature map after self-attention feature modification are modified with mutual attention based on the cross-attention mechanism. Finally, the registration parameter residual is predicted based on the cross-attention mechanism. This is specifically implemented based on the converged attention transformer structure. The specific steps are as follows: S1: Use the self-attention mechanism in two ways to calculate the correlation between the extracted features at different coordinate positions to achieve self-attention feature modification; the calculation of the self-attention mechanism is shown in formula (1): ， Where x is the coordinate index of the target image / source image feature map; They are the query matrix, key matrix, and value matrix, which are obtained by processing the target image / source image feature map using a fully connected layer; Indicates coordinate position The surrounding area is the range of the self-attention mechanism; if If the whole feature map is set, it is a global self-attention mechanism, otherwise it is a non-global or local self-attention mechanism with a certain radius. During the iteration process, the scope of the self-attention mechanism decreases as the number of iterations increases. After the self-attention mechanism calculation is completed, the attention features obtained after the self-attention mechanism calculation are processed using layer normalization and added to the target image / source image feature map obtained by feature extraction to complete the feature modification of the self-attention; S2: The cross-attention mechanism is used to calculate the correlation between the target image feature map and the source image feature map after self-attention feature modification at different coordinate positions. The calculation of the cross-attention mechanism is shown in formula (2): ， Among them, the subscript and Represents the query matrix from one of the feature maps, and the key matrix Sum Matrix From another feature map; Indicates that the cross attention mechanism is The surrounding area is the scope of the cross attention mechanism; if If the whole feature map is used, it is a global cross-attention mechanism, otherwise it is a non-global or local cross-attention mechanism with a certain radius. During the iteration process, the scope of the cross-attention mechanism decreases as the number of iterations increases. After completing the cross-attention mechanism calculation, the attention features obtained after the cross-attention mechanism calculation are processed using layer normalization, and added to the target image / source image feature map modified by the self-attention feature to complete the feature modification of the mutual attention; then the fully connected layer is used to process the features modified by the mutual attention, and the features processed by the fully connected layer are added to the features modified by the mutual attention to obtain the final self-attention and cross-attention modified target image / source image feature map; S3: Registration parameter residual prediction; for the self-attention and cross-attention modified target image / source image feature map, use formula (2) to calculate the cross-attention, input the calculated cross-attention into the fully connected layer, and output the registration parameter residual.

2. The multimodal image registration method based on predictive correction and convergent attention transformer according to claim 1, characterized in that In step 1), a convolutional neural network is used to extract the features of the target image and the source image. The specific method is as follows: first, a kernel size of The convolution layer performs preliminary processing on the target image / source image; then continues to use multiple Convolutional layers process images, where The convolution layer includes the downsampling convolution layer and the non-downsampling convolution layer. The downsampling convolution layer performs feature extraction on the spatial dimension. Finally, use The convolution layer performs dimension projection on the feature map to obtain the target image / source image feature map.

3. The multimodal image registration method based on predictive correction and convergent attention transformer according to claim 1, characterized in that In step 2), the current predicted registration parameters are updated using the registration parameter residuals, and then the source image is deformed using the current predicted registration parameters; specifically: First, the matrix multiplication of the last iteration prediction registration parameter and the current iteration prediction registration parameter residual is performed to obtain the current prediction registration parameter, as shown in formula (3). ， in, represents the last iteration prediction registration parameters, Represents the residual error of the current iteration prediction registration parameter, Current predicted registration parameters, represents the number of iterations, Represents the identity transformation, that is, in the first When , the registration parameters predicted in the previous iteration are unchanged; Then, the source image is deformed and corrected using the current predicted registration parameters, as calculated in formula (4): ， in, represents the source image, represents the coordinate mapping, Indicates passing The source image after distortion correction.

Images