Multi-modal image registration method based on modal alignment distillation

Abstract

The invention relates to a multi-modal image registration method based on modal alignment distillation, and belongs to the technical field of image processing. The method comprises the steps of adopting a distillation learning guide type training process, performing guide type supervision on a multi-modal registration student model by using intermediate features of a same-modal registration teacher model with excellent performance and an output deformation field, migrating modals of the intermediate features of the student model, and effectively reducing modal differences, a high-quality deformation field close to teacher model estimation is output, and the multi-modal registration performance is improved; a deformable mask training strategy is adopted to deal with the problem that a registration result is inconsistent with a label image boundary caused by a black edge effect of spatial transformation in a registration process (namely, a black edge of the registration result and an original boundary of the label image), so that the negative influence on registration is effectively reduced, and the registration performance of the model is improved. According to the invention, a multi-modal image registration task can be realized, registration challenges caused by modal differences are effectively solved, and the effect of multi-modal image space alignment is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and relates to a multimodal image registration method based on modality alignment distillation. Background Art

[0002] Multimodal image registration is a key technology that aims to accurately align images collected by different sensors, such as infrared, visible light, and synthetic aperture radar (SAR) images, to achieve accurate pixel-level matching. This technology has a wide range of applications, including but not limited to medical diagnosis, remote sensing monitoring, and security monitoring systems. In the medical field, multimodal image registration can help doctors diagnose diseases more accurately; in the remote sensing field, it can improve the recognition accuracy of ground objects; in monitoring systems, it can enhance the tracking ability of targets. With the advancement of multimodal technology, it has become an inevitable trend to use these complementary information for more comprehensive analysis. However, if unregistered multimodal images are used for analysis, significant errors may be introduced, such as artifacts in the image fusion process or inconsistent target recognition in target detection. Therefore, multimodal image registration is not only a prerequisite for the application of multimodal technology, but also a key step to ensure the accuracy of analysis results. Solving the problem of multimodal image registration has become an important challenge in current research and application.

[0003] In the existing field of image registration, methods are mainly divided into three categories: traditional feature-based methods, traditional region-based methods, and deep learning-based methods. Feature-based methods, such as scale-invariant feature transform (SIFT), speeded up robust features (SURF), and oriented rapid rotation (ORB), etc., identify and match key feature points in the image and use spatial transformation models to achieve image alignment. This method performs well when processing single-modality images, but when facing multi-modal image registration, due to the significant differences between different modalities, it often leads to failure in matching key feature points, which in turn affects the accuracy of image registration, such as Figure 1As shown in Figure 1. In contrast, region-based methods focus on a larger range of images rather than a single feature point. This type of method measures the similarity between different image regions by designing indicators, and performs spatial transformation based on this to achieve registration. However, due to the nonlinear intensity differences between multimodal images, region-based methods often find it difficult to accurately identify truly similar regions when evaluating regional similarity, which limits their effectiveness in multimodal image registration. In recent years, deep learning-based methods have made some progress in the field of image registration, but they also face challenges brought by modality differences. Although deep learning models can learn complex feature representations, when processing multimodal images, due to the significant differences between modalities, the registration performance and generalization ability of the models are often unsatisfactory. Since images of different modalities have significant differences in vision and information content - for example, infrared images mainly reflect the thermal radiation characteristics of objects, while visible light images focus more on texture details and color information - these differences make multimodal image registration more challenging than single-modality image registration. Therefore, reducing modality differences is currently a major challenge in multimodal image registration.

[0004] At present, some image registration methods focus on how to reduce modal differences. For example, the style transfer method based on adversarial learning converts the visible light image into a pseudo infrared image, and then the pseudo infrared image is used as the reference image, and the deformed infrared image is output as the deformed image to the model to predict the registration flow field. However, this method is extremely dependent on the performance of the style transfer module. The converted image often has large noise, and the converted thermal radiation information is uncertain. If the thermal radiation information of some scenes is incorrect, it may even affect the performance of subsequent registration. Some methods also improve the registration performance by combining multiple tasks to find a mutually reinforcing strategy. For example, multimodal image registration is jointly trained with multimodal image fusion tasks, and the generated fused image can reduce the modal difference to a certain extent, but this method is limited to multimodal images with good fusion performance, and is not suitable for all multimodal image registrations, and the registration performance still needs to be improved. Some studies have also focused on how to extract homomodal information from multimodal images and then use the homomodal information to predict flow fields. However, multimodal image information is often unbalanced, and the forced extraction of homomodal information often lacks some important information. The final predicted flow field often cannot fully align all the information of the multimodal image.

[0005] Therefore, there is an urgent need for a multimodal image registration method that can simultaneously solve the problems of poor matching feature recognition ability, difficulty in multimodal model optimization, and poor registration performance caused by large modality differences in multimodal image registration tasks. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a multimodal image registration method based on modal alignment distillation to solve the problems of poor matching feature recognition ability, difficult multimodal model optimization and poor registration performance caused by excessive modal differences in multimodal image registration tasks.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A multimodal image registration method based on modality alignment distillation adopts a basic distillation learning guided training process, uses a homomodal registration teacher model with excellent performance, and guides the supervision of the multimodal registration student model from intermediate features to output deformation fields, so that the intermediate features of the student model can be modally transferred to effectively reduce the modal difference, and the output is a high-quality deformation flow field close to that produced by the teacher model, thereby improving the registration performance; a deformable mask training strategy is adopted to limit the unequal edge information of the unregistered image (i.e., the black edge without information in the output result and the boundary containing information in the label image) and the black edge effect of spatial transformation in the registration process, thereby effectively reducing its negative impact on the registration and improving the model registration performance.

[0009] The method specifically comprises the following steps:

[0010] S1: Data preprocessing: Obtain the original multimodal image, generate a random deformation flow field to act on the original multimodal image, so that the original image produces a random deformation (including rigid and non-rigid deformation), and establish an unregistered multimodal dataset, including a training set and a test set, in which the original image is used as a label for self-supervised learning;

[0011] S2: Build the registration model, including the encoder and the transformer-based U-shaped registration network;

[0012] S3: Training model registration model: Use deformable mask training strategy and distillation learning technology to train teacher model and student model to obtain the same modality teacher model and multimodal student model based on distillation learning;

[0013] S4: Input the unregistered multimodal test set established in step S1 into the trained model registration model and output the registered image.

[0014] Further, in step S2, the encoder is a classic U-shaped network with lateral connections, including four downsampling and four upsampling operations, with excellent feature mapping capabilities, and uses shared parameters to extract features from the deformed image and the reference image respectively, which is an intermediate feature of 16 channels. The reference image of the student model is output by the encoder through the intermediate features, and the modality is transferred under distillation learning, thereby reducing the modality difference;

[0015] The transformer-based U-shaped registration network includes a downsampling encoding part composed of a feature enhancement module and a multi-scale attention mechanism module, an upsampling decoding module composed of a disabled connection module, and a lateral connection part composed of a transformer; the feature enhancement module is mainly composed of diffuse convolutions with different diffusion rates, combined with a multi-scale attention mechanism, which can better extract features from different receptive fields, extract deformation information of different sizes, and effectively improve the non-rigid registration performance; the transformer can better obtain global information and effectively improve the rigid registration performance; the upsampling decoding module synthesizes the previously extracted information and predicts the final output registration flow field.

[0016] Further, in step S3, the deformable mask training strategy is specifically: setting a single-channel all-1 mask matrix, which is successively subjected to multiple effects of the data preprocessing distortion flow field and the model output registration flow field, and is filled with 0 during the registration resampling process to generate a black edge (0 boundary) equivalent to the registered image; during the parameter optimization process, the mask is respectively multiplied by the Hadamard matrix with the registered image and the label image, and the mask black edge will limit the redundant information of the black edge of the registered image and the corresponding area of ​​the label image, so that the model only focuses on the registration result of the central deformation area, and ignores the influence of zero filling (i.e., black edge) of the output image boundary on model training and performance, effectively promoting the performance optimization of the registration task; this strategy is applied to both the teacher model and the student model training process to reduce the unequal edge information of the unregistered image (i.e., the black edge without information in the output result and the boundary containing information in the label image) and the negative impact of the black edge generated by the spatial transformation resampling on the registration performance during the registration process.

[0017] Further, in step S3, the deformable mask training strategy specifically comprises the following steps:

[0018] S21: Set a corresponding single-channel all-1 mask for each modality image to be registered in the training set. The generated random deformation flow field acts on the image and the mask at the same time. The 0-filled areas of the two are consistent, that is, the black edges are the same. The calculation process is as follows:

[0019]

[0020]

[0021] Among them, I mov is the deformed image, φ r is the deformed flow field randomly generated in data preprocessing, M is the original single-channel all-1 mask, It is the result of the deformation image being affected by the deformation flow field, M W is the result of M being deformed by the same flow field, that is, M W and have the same 0 boundaries;

[0022] S22: The model outputs the registration flow field to register the deformed image, generating a secondary black edge. At the same time, the mask is affected by the inverse transformation of the deformed flow field, and the secondary black edge of the mask is consistent with the secondary black edge of the registration image. The calculation process is as follows:

[0023]

[0024]

[0025] Among them, φ reg is the model-predicted registration flow field, is the registered image, Yes r The flow field after the reverse operation, M′ is M W by The mask after the action is ideally the same as The 0 boundary of is the same;

[0026] S23: The mask M′ is multiplied by the Hadamard matrix with the original image and the registered image respectively to limit the additional information of the original image compared to the 0 boundary of the registered image. Then the loss of the image is calculated to make the network pay more attention to the central area and solve the parameter optimization disorder problem caused by the unequal edge information.

[0027] Furthermore, in step S3, the distillation learning uses a homomodal teacher model that has been trained and has excellent registration performance to perform multi-dimensional guided supervision on the multimodal registration student model, including feature mapping supervision of the homomodal features output by the encoder to the multimodal features (such as the transfer of visible light features to infrared features), thereby reducing modal differences and improving the student model's recognition ability for multimodal matching features, as well as supervision of the predicted flow field output by the transformer-based U-shaped registration network (the deformed images have the same deformation and the true value of the registration flow field is the same); in this process, the registration performance of the multimodal student model continues to approach the registration performance of the homomodal teacher model, thereby completing the multimodal image registration task based on distillation learning.

[0028] Further, in step S3, the training of the teacher model is specifically as follows: select a unimodal image in a multimodal data set for training (such as an infrared visible light data set, select the infrared modality as the teacher model training object); generate a random deformation flow field to act on the unimodal original image, and use the image as a deformed image after spatial deformation. The undeformed original image is input into the model as a reference image, and parameter optimization is performed under the set loss function to complete the training and obtain a homomodal teacher model.

[0029] Furthermore, in step S3, the loss functions used for teacher model training include L1 loss and SSIM loss, which optimize the registration performance of the model from the perspectives of pixel level and structure level respectively. The loss function is calculated as follows:

[0030]

[0031] L tea =L pix +L sim (7)

[0032] Among them, L tea represents the teacher model training loss function, L pix represents the same-modal pixel loss, L sim represents the same-modal structure loss, h represents the image length, w represents the image width, I w represents the deformed image after deformation, I gt represents the original image, M′ represents the mask matrix obtained by the deformable mask training strategy, represents the model prediction registration flow field, represents the registration resampling operation, ⊙ represents the Hadamard matrix multiplication operation, and ||·1 represents the L1 loss.

[0033] Further, in step S3, the training of the student model is specifically as follows: the parameters of the teacher model remain fixed; a random deformation flow field is generated and simultaneously acts on the modal image selected by the teacher model and the corresponding modal image in the student model to generate unregistered images with the same deformation as the deformed images of the two models, the teacher model uses the original image of the same modality as the reference image, and the student model uses another modality (such as the visible light modality in the infrared-visible light dataset) as the reference image; during the training process, the encoder outputs intermediate features of the teacher model with fixed parameters and the output registration flow field of the transformer-based U-shaped registration network as the target, and the intermediate features and output registration flow field corresponding to the student model are calculated for loss to achieve multimodal image registration distillation learning.

[0034] Furthermore, in step S3, the loss function used for student model training includes, in addition to L1 loss and SSIM loss, distillation-level supervision of the teacher model on the student model. The loss function is calculated as follows:

[0035]

[0036] L stu =L' pix +L' sim +L distillation (11)

[0037] Among them, L' pix represents the multimodal pixel loss, L'sim represents the multimodal structure loss, L distillation represents the loss function corresponding to distillation learning, h represents the image length, w represents the image width, M′ represents the mask matrix obtained by the deformable mask training strategy, and I w represents the deformed image after deformation, I gt represents the original image, E s is the student model encoder, E t is the teacher model encoder, I md is a multimodal image, is the predicted registration flow field of the deformable image with the multimodal image as the target, represents the model prediction registration flow field; L distillation is the loss function corresponding to distillation learning, which guides the optimization of the registration performance of the student model. Part I || E s (I md )-E t (I gt )||1 Promote the multimodal reference image features of the student model to continuously approach the unimodal reference image features of the teacher model, greatly reducing the modality difference and improving the registration performance; Part II The predicted registration flow field of the student model is promoted to be close to the teacher model with excellent performance, and the deformation of the displacement image of the student model and the teacher model during the distillation process is the same, so, and Ideally, they are the same, which is why it is important to distill the output registered flow field.

[0038] The beneficial effects of the present invention are:

[0039] (1) The method of the present invention can realize the registration of multimodal images (such as infrared and visible light images), and guide the multimodal registration student model with the same-modal registration teacher model, thereby overcoming the difficulties of registration and low registration accuracy caused by excessive modality differences in multimodal image registration. In addition, a deformable mask training strategy is proposed to reduce the negative impact of the unequal edge information of multimodal images and the black edge effect in the registration resampling process, thereby making further breakthroughs on the previous achievements.

[0040] (2) The present invention can realize multimodal image registration tasks (such as remote sensing multimodal image registration, etc.), effectively solve the registration challenges brought about by modal differences, and achieve the effect of multimodal image spatial alignment.

[0041] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0043] Figure 1 This is a visual comparison chart;

[0044] Figure 2 It is the overall network framework of the present invention;

[0045] Figure 3 Learning training process for multimodal distillation;

[0046] Figure 4 Training strategies for deformable masks;

[0047] Figure 5 This is a network architecture diagram. DETAILED DESCRIPTION

[0048] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0049] See also Figure 1 to Figure 5 The present invention provides a multimodal image registration method based on distillation learning modality alignment, which can realize the multimodal image registration task. The overall method block diagram is as follows Figure 2As shown. Before distillation learning, the teacher model training must be completed first: select the modality to be registered in the multimodal image (generally RGB is selected as the reference image, and the common deformed image modalities to be registered are: infrared, near-infrared, SAR, etc.); during the training process, the randomly generated deformation flow field acts on the modality image to be registered to generate a deformed image after deformation, and the original image to be registered before deformation is used as the reference image, which is input into the encoder with shared parameters to extract features from the deformed image and the reference image at the same time, and then the extracted features of the deformed image and the reference image are stacked on the channel and input into the transformer-based U-shaped registration network. After multi-scale feature extraction, horizontal connection transformer action and feature decoding, the registration flow field is predicted; the predicted registration flow field acts on the deformed image, and the registration resampling operation is performed to obtain the registered image; the registered image and the reference image calculate the loss, and the registration parameters are optimized, including L1 loss and SSIM loss, which optimize the registration performance of the model from the pixel level and structure level respectively. The loss function is calculated as follows:

[0050]

[0051] L tea =L pix +L sim

[0052] Based on the unimodal teacher model, superior registration performance can be achieved. The trained unimodal teacher model with fixed parameters is used to guide the multimodal student model to achieve registration optimization from multiple dimensions, completing the registration distillation learning from unimodal to multimodal. The specific distillation process is as follows: Figure 3 As shown in the figure, the parameters of the teacher model remain fixed; the randomly generated flow field acts on the same modality image of the teacher model and the learning model at the same time, the deformed images of the teacher model and the student model are deformed in the same way, and the corresponding required registration flow field is also the same; the teacher model uses the original same image as the reference image, and the student model uses the multimodal image as the reference image; during the training process, the teacher model outputs features from the encoder to the transformer-based U-shaped network output registration flow field to perform multi-dimensional guided supervision on the student model, and the student model loss is calculated as follows:

[0053]

[0054] L stu =L' pix +L' sim +L distillation

[0055] Among them, L distillationIt is the loss function corresponding to distillation learning, which guides the optimization of the registration performance of the student model. The first part promotes the multimodal reference image features of the student model to continuously converge with the unimodal reference image features of the teacher model, greatly reducing the modal difference and improving the registration performance; the second part promotes the predicted registration flow field of the student model to be close to the teacher model with excellent performance (the two have the same deformation and the corresponding registration real flow field is the same).

[0056] Aiming at the negative impact of the unequal edge information of unregistered images and the black edges generated by spatial transformation during the registration process on the registration performance, a deformable mask training strategy is proposed. Figure 4 As shown in the figure, it is used in the training process of the teacher model and the learning model. The specific steps are as follows:

[0057] S1: Set a corresponding single-channel all-1 mask for each modality image to be registered in the training set. The generated random deformation flow field acts on the image and the mask at the same time. The 0-filled areas of the two are consistent, that is, the black edges are the same. The calculation process is as follows:

[0058]

[0059] M W =R(M,φ r )

[0060] Among them, I mov is the deformed image, φ r is the deformed flow field randomly generated in data preprocessing, M is the original single-channel all-1 mask, It is the result of the deformation image being affected by the deformation flow field, M W It is the result of M being deformed by the same flow field, that is: W and have the same 0 boundaries;

[0061] S2: The model outputs the registration flow field to register the deformed image, generating a secondary black edge. At the same time, the mask is affected by the inverse transformation of the deformed flow field, and the secondary black edge of the mask is consistent with the secondary black edge of the registration image. The calculation process is as follows:

[0062]

[0063] Among them, φ reg is the model-predicted registration flow field, is the registered image, Yes r The flow field after the reverse operation, M′ is M W by The mask after the action is ideally the same as The 0 boundary of is the same;

[0064] S3: The mask M′ is multiplied by the Hadamard matrix with the original image and the registered image respectively to limit the additional information of the original image compared to the 0 boundary of the registered image. Then the loss of the image is calculated to make the network pay more attention to the central area and solve the parameter optimization disorder problem caused by the unequal edge information.

[0065] Among them, the deformed flow field is not the real value of the registration flow field after inverse transformation, but because the mask values ​​are 1 and 0, they are less affected by the resampling bilinear interpolation. Therefore, after the inverse transformed deformed flow field resamples the mask, the black edge of the mask can be equivalent to the real black edge.

[0066] In order to more efficiently and accurately predict the registration flow field, a new encoder is designed - a transformer-based U-shaped registration model. The specific structure is shown as follows Figure 5 As shown in the figure, the entire registration model is divided into two parts: the encoder and the transformer-based U-shaped registration network:

[0067] (1) The encoder is a classic U-shaped network with horizontal connections, including four downsampling and four upsampling operations, and has excellent feature mapping capabilities. The encoder with shared parameters extracts features from the deformed image and the reference image respectively, and outputs 16-channel intermediate features. The reference image of the student model is transferred to the model under distillation learning through the intermediate features output by the encoder, as shown in formula (10), L distillation The first part is the intermediate feature supervision of the teacher model on the student model, which guides the multimodal feature mapping and thus reduces the modal difference.

[0068] (2) The transformer-based U-shaped registration network consists of a downsampling encoding part consisting of a feature enhancement module, a multi-scale attention mechanism module, an upsampling decoding module consisting of a residual connection module, and a lateral connection part consisting of a transformer. The feature enhancement module is composed of a diffusion convolution with different diffusion rates, an instance normalization layer, and a LeakyRelu activation layer, and adopts a residual connection structure, such as Figure 5 As shown in the figure, it is conducive to extracting multi-scale information without increasing the number of parameters. Combined with the multi-scale attention mechanism, it can better extract deformation information of different sizes from different receptive fields and effectively improve the non-rigid registration performance. The horizontal connection transformer can better obtain global information and effectively improve the rigid registration performance. The upsampling part is a classic residual connection structure, which synthesizes the information extracted in the previous text and predicts the final output registration flow field with the same resolution as the input.

[0069] Specific implementation details: The entire network architecture and training process are implemented on pytorch. The experimental running environment is as follows: Intel Core i9-10900X 3.70GHz CPU and NVIDIA RTX 3090GPU. Adam optimizer is used for parameter optimization. For the infrared-visible light dataset, both the teacher model and the student model are trained for 300 rounds; for the SAR-visible light dataset, both the teacher model and the student model are trained for 800 rounds. The learning rate is set to 0.0001, and it decreases by 50% every 200 rounds; batchsize is set to 32.

[0070] Comparative experiment:

[0071] The method proposed in the present invention is compared with multiple registration methods, including SIFT (traditional), ORB (traditional), Superfusion (deep learning), and CrossModalFlow (deep learning) on ​​multiple multimodal datasets, including ordinary scene infrared-visible light MSRS dataset, remote sensing scene infrared-visible light dataset, remote sensing scene near-infrared-visible light dataset, and remote sensing scene SAR-visible light dataset, to verify the effectiveness of our proposed method on multimodal datasets.

[0072] Evaluation indicators: mean square error (MSE), structural similarity (SSIM), correlation coefficient (CC). MSE is used to evaluate the degree of alignment between the registered image and the original image at the pixel level, while SSIM is used to evaluate the degree of alignment in terms of image structure, and CC is used to measure the degree of correlation between the registered image and the original image. The lower the MSE, the higher the SSIM and CC, which means the better the image registration effect. Therefore, the registration performance of our method is evaluated from multiple angles.

[0073] Experimental results: As shown in Tables 1 to 4, the method proposed in the present invention has achieved the best registration performance in all index evaluations on the four multimodal datasets, especially on the three infrared datasets, which are significantly ahead of other methods and have excellent registration effects; on the SAR dataset, although the test index is lower, it is also ahead of other methods. SAR modality image registration is still an extremely difficult task at present.

[0074] Table 1 Experimental results of the algorithm on the common scene infrared dataset

[0075]

[0076] Table 2 Experimental results of the algorithm on the remote sensing scene infrared dataset

[0077]

[0078]

[0079] Table 3 Experimental results of the algorithm on the near-infrared dataset of remote sensing scenes

[0080]

[0081] Table 4 Experimental results of the algorithm on the remote sensing scene SAR dataset

[0082]

[0083] In summary, the multimodal image registration method based on modality alignment distillation proposed in the present invention is advanced to a certain extent, and can effectively solve the problems of poor matching feature recognition ability, difficult multimodal model optimization, and poor registration performance caused by excessive modality differences in multimodal image registration tasks, and has excellent multimodal image registration performance.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A multimodal image registration method based on modality alignment distillation, characterized in that: The method specifically comprises the following steps: S1: Data preprocessing: obtain the original pre-registered multimodal image, generate a random deformation flow field to act on the original multimodal image, so that the original image produces a random deformation, and establish an unregistered multimodal dataset, including a training set and a test set, in which the original image is used as a label for self-supervised learning; S2: Build the registration model, including the encoder and the transformer-based U-shaped registration network; S3: Training multimodal registration model: Based on the deformable mask training strategy and modality distillation learning technology, the teacher model and student model are trained to obtain a homomodal teacher model and a multimodal student model based on modality alignment distillation; S4: Input the unregistered multimodal test set established in step S1 into the trained model registration model and output the registered image.

2. The multimodal image registration method based on modality alignment distillation according to claim 1, characterized in that: In step S2, the encoder is a U-shaped network with lateral connections, including four downsampling and four upsampling operations, and uses shared parameters to extract features from the deformed image and the reference image respectively, which is an intermediate feature of 16 channels. The reference image of the student model is output by the encoder through the intermediate features, and the modality is transferred under distillation learning, thereby reducing the modality difference; The transformer-based U-shaped registration network includes a downsampling encoding part composed of a feature enhancement module and a multi-scale attention mechanism module, an upsampling decoding module composed of a disabled connection module, and a lateral connection part composed of a transformer; the feature enhancement module is composed of diffuse convolutions with different diffusion rates, and combined with a multi-scale attention mechanism, it can extract features from different receptive fields and extract deformation information of different sizes, effectively improving the non-rigid registration performance; the transformer can obtain global information and effectively improve the rigid registration performance; the upsampling decoding module synthesizes the previously extracted information and predicts the final output registration flow field.

3. The multimodal image registration method based on modality alignment distillation according to claim 1, characterized in that: In step S3, the deformable mask training strategy is specifically: setting a single-channel all-1 mask matrix, which is subjected to multiple effects of the data preprocessing distortion flow field and the model output registration flow field, and is filled with 0 during the registration resampling process to generate a black edge equivalent to the registered image; During the parameter optimization process, the mask is multiplied with the registration image and the label image by the Hadamard matrix. The black edges of the mask will limit the redundant information of the black edges of the registration image and the corresponding areas of the label image, so that the model only focuses on the registration results of the central deformation area, and ignores the impact of zero padding on the output image boundary on model training and performance, which effectively promotes the performance optimization of the registration task; this strategy is also applied to the teacher model and student model training processes to reduce the negative impact of the asymmetric edge information of the unregistered image and the black edges generated by the spatial transformation resampling during the registration process on the registration performance.

4. The multimodal image registration method based on modality alignment distillation according to claim 3, characterized in that: In step S3, the deformable mask training strategy includes the following specific steps: S21: Set a corresponding single-channel all-1 mask for each modality image to be registered in the training set. The generated random deformation flow field acts on the image and the mask at the same time. The 0-filled areas of the two are consistent, that is, the black edges are the same. The calculation process is as follows: M W =R(M,φ r ) (2) Among them, I mov is the deformed image, φ r is the deformed flow field randomly generated in data preprocessing, M is the original single-channel all-1 mask, It is the result of the deformation image being affected by the deformation flow field, M W is the result of M being deformed by the same flow field, that is, M W and have the same 0 boundaries; S22: The model outputs the registration flow field to register the deformed image, generating a secondary black edge. At the same time, the mask is affected by the inverse transformation of the deformed flow field, and the secondary black edge of the mask is consistent with the secondary black edge of the registration image. The calculation process is as follows: Among them, φ reg is the model-predicted registration flow field, is the registered image, Yes r The flow field after the reverse operation, M′ is M W by The mask after the action is ideally the same as The 0 boundary of is the same; S23: The mask M′ is multiplied by the Hadamard matrix with the original image and the registered image respectively to limit the additional information of the original image compared to the 0 boundary of the registered image. Then the loss of the image is calculated to make the network pay more attention to the central area and solve the parameter optimization disorder problem caused by the unequal edge information.

5. The multimodal image registration method based on modality alignment distillation according to claim 1, characterized in that: In step S3, the distillation learning is to use a homomodal teacher model that has been trained and has excellent registration performance to perform multi-dimensional guided supervision on the multimodal registration student model, including feature mapping supervision of the homomodal features output by the encoder to the multimodal features, thereby reducing the modal difference and improving the student model's recognition ability of multimodal matching features, as well as supervision of the prediction flow field output by the transformer-based U-shaped registration network; In this process, the registration performance of the multimodal student model continues to approach the registration performance of the same-modal teacher model, thus completing the multimodal image registration task based on distillation learning.

6. The multimodal image registration method based on modality alignment distillation according to claim 1, characterized in that: In step S3, the training of the teacher model specifically includes: selecting a single-modality image in the multi-modality data set for training; A random deformation flow field is generated to act on the unimodal original image. The image is used as the deformed image after spatial deformation, and the undeformed original image is input into the model as the reference image. The parameters are optimized under the set loss function to complete the training and obtain the same-modal teacher model.

7. The multimodal image registration method based on modality alignment distillation according to claim 6, characterized in that: In step S3, the loss functions used for teacher model training include L1 loss and SSIM loss, which optimize the registration performance of the model from the perspectives of pixel level and structure level respectively. The loss function is calculated as follows: L tea =L pix +L sim (7) Among them, L tea represents the teacher model training loss function, L pix represents the same-modal pixel loss, L sim represents the same-modal structure loss, h represents the image length, w represents the image width, I w represents the deformed image after deformation, I gt represents the original image, M′ represents the mask matrix obtained by the deformable mask training strategy, represents the model prediction registration flow field, represents the registration resampling operation, ⊙ represents the Hadamard matrix multiplication operation, and ||·1 represents the L1 loss.

8. The multimodal image registration method based on modality alignment distillation according to claim 1, characterized in that: In step S3, the training of the student model is specifically as follows: the parameters of the teacher model remain fixed; a random deformation flow field is generated and simultaneously acts on the modal image selected by the teacher model and the corresponding modal image in the student model to generate unregistered images with the same deformation as the deformed images of the two models, the teacher model uses the original image of the same modality as the reference image, and the student model uses the other modality as the reference image; during the training process, the encoder of the teacher model with fixed parameters outputs intermediate features and the registration flow field output by the transformer-based U-shaped registration network as the target, and the intermediate features and output registration flow field corresponding to the student model are calculated for loss to achieve multimodal image registration distillation learning.

9. The multimodal image registration method based on modality alignment distillation according to claim 8, characterized in that: In step S3, the loss function used for student model training includes not only L1 loss and SSIM loss, but also distillation-level supervision of the teacher model on the student model. The loss function is calculated as follows: L stu =The pix +The sim +L distillati where (11) Among them, L' pix represents the multimodal pixel loss, L' sim represents the multimodal structure loss, L distillation represents the loss function corresponding to distillation learning, h represents the image length, w represents the image width, I w represents the deformed image after deformation, I gt represents the original image, M′ represents the mask matrix obtained by the deformable mask training strategy, and E s is the student model encoder, E t is the teacher model encoder, I md is a multimodal image, is the predicted registration flow field of the deformable image with the multimodal image as the target, represents the model prediction registration flow field; L distillation is the loss function corresponding to distillation learning, which guides the optimization of the registration performance of the student model. Part I || E s (I md )-E t (I gt )1 Promote the multimodal reference image features of the student model to continuously approach the unimodal reference image features of the teacher model, greatly reducing the modality difference and improving the registration performance; Part II The predicted registration flow field of the student model is promoted to be close to the teacher model with excellent performance, and the deformation of the displacement image of the student model and the teacher model during the distillation process is the same, so, and Ideally they are the same.

Images