A multi-modal image registration method based on modal alignment distillation

Abstract

The present application relates to a kind of multi-modal image registration methods based on modal alignment distillation, belong to image processing technical field.The method includes: using a kind of distillation learning guided training procedure, using the intermediate feature of excellent performance of same modal registration teacher model and output deformation field to guide type supervision to multi-modal registration student model, migrate the modal of student model intermediate feature and effectively reduce modal difference, output close to the high-quality deformation field estimated by teacher model, improve multi-modal registration performance;Adopt a deformable mask training strategy, deal with the problem that the registration result is inconsistent with the boundary of label image (i.e. the black edge of registration result and the original boundary of label image) caused by black edge effect of spatial transformation in registration process, effectively reduce its negative influence on registration, improve the registration performance of model.The present application can realize multi-modal image registration task, effectively solve the registration challenge caused by modal difference, achieve the effect of multi-modal image spatial alignment.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and relates to a multi-modal image registration method based on modal alignment distillation. BACKGROUND

[0002] Multi-modal image registration is a key technology that aims to accurately align images captured by different sensors, such as infrared, visible light, synthetic aperture radar (SAR) images, etc., and achieve pixel-level accurate matching. This technology has a wide range of applications, including but not limited to medical diagnosis, remote sensing monitoring, and security monitoring systems. In the medical field, multi-modal image registration can help doctors more accurately diagnose diseases; in the field of remote sensing, it can improve the recognition accuracy of ground objects; in monitoring systems, it can enhance the tracking ability of targets. With the advancement of multi-modal technology, it has become an inevitable trend to use these complementary information for more comprehensive analysis. However, if unregistered multi-modal images are used for analysis, significant errors may be introduced, such as artifacts in image fusion or inconsistent target recognition in target detection. Therefore, multi-modal image registration is not only a prerequisite for the application of multi-modal technology, but also a key step to ensure the accuracy of analysis results. Solving the problem of multi-modal image registration has become an important challenge in current research and application.

[0003] In the existing field of image registration, methods are mainly divided into three categories: traditional feature-based methods, traditional region-based methods, and deep learning-based methods. Feature-based methods, such as scale-invariant feature transform (SIFT), speeded up robust features (SURF), oriented FAST and rotated BRIEF (ORB), etc., identify and match key feature points in images, and use spatial transformation models to achieve image alignment. This method performs well in processing single-modal images, but when faced with multi-modal image registration, due to significant differences between different modalities, key feature point matching often fails, which in turn affects the accuracy of image registration, such as Figure 1In contrast, region-based methods focus on larger ranges of images rather than single feature points. Such methods measure the similarity between different image regions by designing indicators and perform spatial transformation to achieve registration accordingly. However, due to the nonlinear intensity differences between multi-modal images, region-based methods often struggle to accurately identify truly similar regions when evaluating regional similarity, which limits their effectiveness in multi-modal image registration. In recent years, deep learning-based methods have made some progress in image registration, but they also face challenges due to modal differences. Although deep learning models can learn complex feature representations, their registration performance and generalization ability often fall short when dealing with multi-modal images due to significant differences between modalities. Since images of different modalities have significant differences in visual and information content - for example, infrared images mainly reflect the thermal radiation characteristics of objects, while visible light images focus more on texture details and color information - these differences make multi-modal image registration more challenging than single-modal image registration. Therefore, reducing modal differences is a major problem in current multi-modal image registration.

[0004] Currently, some image registration methods focus on how to reduce modal differences. For example, style transfer methods based on adversarial learning convert visible light images into pseudo-infrared images, which are then used as reference images to deform infrared images as deformation images to output to the model to predict the registration flow field. However, this method is highly dependent on the performance of the style transfer module, and the converted images often have significant noise, and the thermal radiation information obtained by conversion is uncertain. If the thermal radiation information of some scenes is incorrect, it may even affect the performance of subsequent registration. Some methods use multiple tasks simultaneously to find a mutually reinforcing strategy to improve registration performance. For example, multi-modal image registration and multi-modal image fusion tasks are trained jointly, and the generated fusion images can reduce modal differences to some extent, but this approach is limited to multi-modal images with good fusion performance and is not suitable for all multi-modal image registration, and the registration performance still needs to be improved. Some research focuses on how to extract same-modal information from multi-modal images and then use same-modal information to predict flow fields, but multi-modal image information is often unbalanced, and forced extraction of same-modal information often misses important information, and the final predicted flow field often cannot fully register all information of multi-modal images.

[0005] Therefore, there is an urgent need for a multi-modal image registration method that can simultaneously solve the problems of poor matching feature recognition ability, difficult multi-modal model optimization, and poor registration performance caused by excessive modal differences in multi-modal image registration tasks. SUMMARY

[0006] Therefore, the present application aims to provide a multi-modal image registration method based on modal alignment distillation, which solves the problems of poor matching feature recognition ability, difficult multi-modal model optimization, and poor registration performance caused by excessive modal difference in the multi-modal image registration task.

[0007] To achieve the above-mentioned purpose, the present application provides the following technical solutions.

[0008] The multi-modal image registration method based on modal alignment distillation adopts a basic distillation learning guided training process, uses a high-performance same-modal registration teacher model to guide and supervise the multi-modal registration student model from the intermediate feature to the output deformation field, effectively reduces the modal difference through modal migration of the intermediate feature of the student model, and outputs a high-quality deformation flow field close to that generated by the teacher model, thereby improving the registration performance; a deformable mask training strategy is adopted to limit the problem of non-equivalence of edge information of unregistered images (i.e., the black edge without information in the output result and the boundary containing information in the label image) and the black edge effect of spatial transformation in the registration process, effectively reducing the negative impact on registration and improving the registration performance of the model.

[0009] The method specifically comprises the following steps:

[0010] S1: data preprocessing: obtaining original multi-modal images, generating a random deformation flow field to act on the original multi-modal images, causing the original images to produce random deformation (including rigid and non-rigid deformation), and establishing an unregistered multi-modal dataset, including a training set and a test set, wherein the original images are used as labels for self-supervised learning;

[0011] S2: constructing a registration model, including an encoder and a transformer-based U-shaped registration network;

[0012] S3: training the registration model: training the teacher model and the student model by using the deformable mask training strategy and the distillation learning technology to obtain a same-modal teacher model and a multi-modal student model based on distillation learning;

[0013] S4: inputting the unregistered multi-modal test set established in step S1 into the trained model registration model to output a registered image.

[0014] Further, in step S2, the encoder is a classic U-shaped network with horizontal connection, which contains four downsampling and four upsampling operations and has excellent feature mapping capability, and is used for feature extraction of the deformation image and the reference image respectively, and the intermediate feature is 16 channels; the intermediate feature of the reference image of the student model output by the encoder is subjected to modal transfer under distillation learning, thereby reducing the modal difference.

[0015] The transformer-based U-shaped registration network comprises a down-sampling encoding part composed of a feature enhancement module and a multi-scale attention mechanism module, an up-sampling decoding module composed of a residual connection module, and a transverse connection part composed of a transformer; the feature enhancement module is mainly composed of diffusion convolution with different diffusion rates, combined with a multi-scale attention mechanism, which can better extract features from different receptive field ranges and extract deformation information of different sizes, effectively improving the non-rigid registration performance; the transformer can better obtain global information, effectively improving the rigid registration performance; the up-sampling decoding module synthesizes the extracted information in the foregoing, and predicts the final output registration flow field.

[0016] Further, in step S3, the deformable mask training strategy is specifically: a single-channel all-1 mask matrix is set, and is subjected to the action of a data preprocessing distortion flow field and a model output registration flow field in sequence for multiple times, and is filled with 0 in the registration resampling process to generate an equivalent black border (0 border) of the registration image; in the parameter optimization process, the mask is subjected to Hadamard matrix multiplication with the registration image and the label image, respectively, the mask black border limits the redundant information of the registration image black border and the corresponding area of the label image, so that the model only focuses on the registration result of the central deformation area and ignores the influence of the output image boundary zero padding (i.e., the black border) on the model training and performance, effectively promoting the performance optimization of the registration task; the strategy is simultaneously applied to the teacher model and the student model training process, reduces the negative influence of the non-equivalent edge information of the unregistered image (i.e., the black border of the output result without information and the boundary containing information in the label image) and the black border generated by the spatial transformation resampling in the registration process on the registration performance.

[0017] Further, in step S3, the deformable mask training strategy comprises the following specific steps:

[0018] S21: a corresponding single-channel all-1 mask is set for each registration modality image in the training set, a random deformation flow field is simultaneously applied to the image and the mask, the 0 padding areas of the two are consistent, i.e., the black borders are the same, and the calculation process is as follows:

[0019]

[0020]

[0021] wherein, I mov is a deformation image, φ r is a random deformation flow field in the data preprocessing, M is an original single-channel all-1 mask, is a result of the deformation image after being subjected to the action of the deformation flow field, M W is a result of M after being subjected to the action of the same flow field, i.e., M W and M have the same 0 boundary;

[0022] S22: The model outputs a registration flow field to register the deformed image, generating a secondary black border, and at the same time, the mask is affected by the inverse transformation of the deformed flow field, and the secondary black border of the mask is consistent with the secondary black border of the registered image. The calculation process is as follows:

[0023]

[0024]

[0025] wherein φ reg is a model predicted registration flow field, is a registered image, is a flow field after inverse operation of φ r , and M' is M W affected by the mask, which is ideally the same as the 0 boundary of ;

[0026] S23: The mask M' respectively performs Hadamard matrix multiplication on the original image and the registered image, limits the additional information of the 0 boundary of the original image compared with the registered image, and then calculates the loss of the image, so that the network pays more attention to the central region, and solves the parameter optimization disorder problem caused by the non-equivalence of edge information.

[0027] Further, in step S3, the distillation learning is multi-dimensional guiding supervision of a multi-modal registration student model by a same-modal teacher model which has completed training and has excellent registration performance, including feature mapping type supervision of same-modal features output by an encoder to multi-modal features (such as transfer of visible light features to infrared features), so as to reduce modal differences, improve the recognition ability of the student model to multi-modal matching features, and supervision of a predicted flow field output by a transformer-based U-shaped registration network (the deformed images are deformed in the same way, and the true values of the registration flow field are the same); in this process, the registration performance of the multi-modal student model continuously approaches the registration performance of the same-modal teacher model, thereby completing the multi-modal image registration task based on distillation learning.

[0028] Further, in step S3, the training of the teacher model is as follows: single-modal images in a multi-modal data set are selected for training (such as an infrared-visible light data set, and infrared modal is selected as a training object of the teacher model); a random deformation flow field is generated to act on the single-modal original image, and the spatial deformation is used as a deformed image. The original image without deformation is used as a reference image input into the model, and parameter optimization is performed under the setting of a loss function, thereby completing the training and obtaining a same-modal teacher model.

[0029] Further, in step S3, the loss function used for teacher model training includes L1 loss and SSIM loss, which optimizes the registration performance of the model from two angles of pixel level and structure level respectively, and the loss function is calculated as follows:

[0030]

[0031] L tea =L pix +L sim (7)

[0032] Wherein, L tea represents the teacher model training loss function, L pix represents the same modality pixel loss, L sim represents the same modality structure loss, h represents the length of the image, w represents the width of the image, I w represents the deformed image, I gt represents the original image, M' represents the mask matrix obtained by the deformable mask training strategy, represents the model predicted registration flow field, represents the registration resampling operation, represents the Hadamard matrix multiplication operation, and ||·1 represents the L1 loss.

[0033] Further, in step S3, the training of the student model is as follows: the teacher model parameters remain fixed; a random deformation flow field is generated and simultaneously acts on the teacher model selected modality image and the corresponding modality image in the student model, producing the same deformation of the unregistered image as the deformed image of the two models, the teacher model takes the same modality original image as the reference image, and the student model takes another modality (such as the visible light modality in the infrared-visible light dataset) as the reference image; during the training process, the teacher model encoder outputs the intermediate features and the transformer-based U-shaped registration network outputs the registration flow field as the target, and the student model performs loss calculation on the corresponding intermediate features and the output registration flow field, realizing multi-modal image registration distillation learning.

[0034] Further, in step S3, the loss function used for student model training includes L1 loss and SSIM loss, and also includes distillation level supervision of the teacher model on the student model, and the loss function is calculated as follows:

[0035]

[0036] L stu =L' pix +L' sim +L distillation (11)

[0037] Wherein, L' pix represents the multi-modal pixel loss, L'sim denotes a multi-modal structure loss, L distillation denotes a loss function corresponding to distillation learning, h denotes image length, w denotes image width, M' denotes a mask matrix obtained by a deformable mask training strategy, I w denotes a deformed deformation image, I gt denotes an original image, E s is a student model encoder, E t is a teacher model encoder, I md is a multi-modal image, is a predicted registration flow field of the deformation image with the multi-modal image as a target, denotes a model predicted registration flow field; L distillation is a loss function corresponding to distillation learning, guiding optimization of registration performance of the student model, a first part ||E s (I md )-E t (I gt )||1 facilitates continuous feature convergence of multi-modal reference image features of the student model to single-modal reference image features of the teacher model, greatly reduces modal difference, and improves registration performance; a second part facilitates the predicted registration flow field of the student model to be close to the teacher model with excellent performance, and the deformation of the displacement image of the student model and the teacher model is the same during the distillation process, so, and are the same in an ideal case, which is an important reason for distilling the output registration flow field.

[0038] The present application has the following advantages:

[0039] (1) The method of the present application can realize registration of multi-modal images (such as infrared and visible light images), and a same-modal registration teacher model guides a multi-modal registration student model, which overcomes registration difficulties and low registration accuracy caused by excessive modal difference in multi-modal image registration, and a deformable mask training strategy is proposed for unequal edge information of multi-modal images and black edge effect in the registration resampling process, which reduces the negative impact of the model registration performance, and further breakthroughs are made on the basis of previous achievements.

[0040] (2) The present application can realize a multi-modal image registration task (such as remote sensing multi-modal image registration), effectively solve registration challenges caused by modal difference, and achieve the effect of spatial alignment of multi-modal images.

[0041] Additional advantages, objects, and features of the application will be apparent to those skilled in the art upon examination of the following specification. It is intended to BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to make the purposes, technical solutions and advantages of the present application clearer, the preferred embodiments of the present application will be described in detail below with reference to the drawings, in which:

[0043] Figure 1 For visual comparison chart;

[0044] Figure 2 For the overall network framework of the present application;

[0045] Figure 3 For multi-modal distillation learning training process;

[0046] Figure 4 For deformable mask training strategy;

[0047] Figure 5 For network architecture diagram. DETAILED DESCRIPTION

[0048] The embodiments of the present application will be described in detail below with reference to specific examples. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in the specification. The present application can also be implemented or applied in different specific embodiments, and the details in the specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0049] Please refer to Figures 1-5 The present application provides a multi-modal image registration method based on distillation learning modal alignment, which can realize the registration task of multi-modal images, and the overall method block diagram is as follows Figure 2shown. Before distillation learning, the teacher model training needs to be completed first: select the modality to be registered in the multi-modal image (generally select RGB as the reference image, and the common to-be-registered deformation image modalities are: infrared, near-infrared, SAR, etc.); in the training process, the randomly generated deformation flow field acts on the to-be-registered modality image to generate a deformed deformation image, and the original to-be-registered image before deformation is taken as the reference image, which is input into the encoder with shared parameters to extract features of the deformation image and the reference image, then the extracted features of the deformation image and the reference image are stacked in the channel, and then input into the U-shaped registration network based on the transformer, after multi-scale feature extraction, transverse connection transformer action and feature decoding, the registration flow field is predicted; the predicted registration flow field acts on the deformation image to perform registration resampling operation to obtain a registration image; the registration image and the reference image are used to calculate the loss, and the registration parameter is optimized, including L1 loss and SSIM loss, which optimize the registration performance of the model from two angles of pixel level and structure level respectively, and the loss function is calculated as follows:

[0050]

[0051] L tea pix sim

[0052] Based on the single-modal teacher model, superior registration performance can be achieved, and the single-modal teacher model with trained and fixed parameters is used to guide the multi-modal student model from multiple dimensions to achieve registration optimization, and the registration distillation learning from single-modal to multi-modal is completed. The specific distillation process is shown in Figure 3 , the teacher model parameters remain fixed; the randomly generated flow field acts on the same modality images of the teacher model and the learning model at the same time, the deformation images of the teacher model and the student model are deformed the same, and the corresponding required registration flow field is also the same; while the teacher model takes the original same image as the reference image, the student model takes the multi-modal image as the reference image; in the training process, the teacher model from the encoder output feature to the transformer-based U-shaped network output registration flow field supervises the student model in multiple dimensions, and the student model loss is calculated as follows:

[0053]

[0054] L stu pix sim distillation

[0055] Wherein, L distillation ​​​​​is the loss function corresponding to distillation learning, guiding the optimization of the registration performance of the student model, the first part promotes the student model's multi-modal reference image features to constantly converge to the teacher model's single-modal reference image features, greatly reducing the modal difference and improving the registration performance; the second part promotes the student model's predicted registration flow field to be close to the teacher model with excellent performance (both have the same deformation, and the corresponding registration real flow field is the same).

[0056] To solve the problem of unequal edge information of unregistered images and the negative impact of black edges generated by spatial transformation in the registration process on registration performance, a deformable mask training strategy is proposed, as shown in Figure 4 The specific steps are as follows:

[0057] S1: Set a corresponding single-channel all-1 mask for each unregistered modal image in the training set, and generate a random deformation flow field that acts on the image and the mask at the same time. The 0 padding areas of the two are consistent, that is, the black edges are the same, and the calculation process is as follows:

[0058]

[0059] M W =R(M,φ r )

[0060] Where I mov is the deformed image, φ r is a random deformation flow field generated in data preprocessing, M is the original single-channel all-1 mask, is the result of the deformed image after being acted on by the deformation flow field, M W is the result of M after being deformed by the same flow field, that is, M W and have the same 0 boundary;

[0061] S2: The model outputs the registration flow field to register the deformed image, generating a secondary black edge. At the same time, the mask is acted on after being inversely transformed by the deformation flow field. The secondary black edge of the mask is consistent with the secondary black edge of the registered image, and the calculation process is as follows:

[0062]

[0063] Where φ reg is the model-predicted registration flow field, is the registered image, is the flow field after inverse operation of φ r , M' is the mask after M W is acted on by , and in an ideal case, it has the same 0 boundary as ;

[0064] S3: The mask M' respectively performs Hadamard matrix multiplication with the original image and the registered image, limits the additional information of the original image compared to the 0 boundary of the registered image, and then calculates the loss of the image, so that the network pays more attention to the central area, and solves the parameter optimization disorder problem caused by the unequal edge information.

[0065] Wherein, the inverse transformation of the deformation flow field is not the true value of the registration flow field, but due to the mask value being 1 and 0, it is less affected by the resampling bilinear interpolation, and thus the mask black border after the resampling of the inverse transformed deformation flow field can be equivalent to the true black border.

[0066] In order to more efficiently and accurately predict the registration flow field, a new type of encoder, a transformer-based U-shaped registration model, is designed, and the specific structure is shown in Figure 5 The entire registration model is divided into an encoder and a transformer-based U-shaped registration network:

[0067] (1) The encoder is a classic U-shaped network with horizontal connection, which contains four downsampling operations and four upsampling operations, has excellent feature mapping ability, and the encoder with shared parameters extracts features from the deformation image and the reference image respectively, and outputs 16-channel intermediate features. The reference image of the student model is transferred to the modal under the distillation learning of the intermediate feature output by the encoder, as shown in formula (10), L distillation The first part is the intermediate feature supervision of the teacher model to the student model, which guides the multi-modal feature mapping, thereby reducing the modal difference.

[0068] (2) The transformer-based U-shaped registration network is composed of a feature enhancement module, a multi-scale attention mechanism module, a downsampling encoding part, a residual connection module, an upsampling decoding module and a transformer. The feature enhancement module is composed of diffusion convolution with different diffusion rates, Instance normalization layer and LeakyRelu activation layer, and adopts a residual connection structure, as shown in Figure 5 It is beneficial to extract multi-scale information without increasing the number of parameters, and combined with the multi-scale attention mechanism, it can better extract deformation information of different sizes from different receptive fields, effectively improve the non-rigid registration performance; the transformer of the horizontal connection can better obtain global information, effectively improve the rigid registration performance; the upsampling part is a classic residual connection structure, which integrates the information extracted in the foregoing to predict the final output registration flow field with the same resolution as the input.

[0069] Specific implementation details: The entire network architecture and training process are implemented on pytorch, and the experimental running environment is as follows: Intel Core i9-10900X 3.70GHz CPU and Nvidia RTX 3090 GPU. The parameter optimization uses the Adam optimizer, and the teacher model and the student model are trained for 300 rounds for the infrared-visible light dataset; the teacher model and the student model are trained for 800 rounds for the SAR-visible light dataset. The learning rate is set to 0.0001, and is reduced by 50% every 200 rounds; the batchsize is set to 32.

[0070] Comparative experiment:

[0071] The method proposed in the application is compared with a plurality of registration methods, including: SIFT (traditional), ORB (traditional), Superfusion (deep learning), CrossModalFlow (deep learning), on a plurality of multi-modal data sets, including: ordinary scene infrared-visible light MSRS data set, remote sensing scene infrared-visible light data set, remote sensing scene near-infrared-visible light data set, remote sensing scene SAR-visible light data set, to verify the effectiveness of the method proposed in the application on the multi-modal data set.

[0072] Evaluation index: mean square error (MSE), structural similarity (SSIM), correlation coefficient (CC), MSE is used to evaluate the registration degree of the registered image and the original image at the pixel level, while SSIM is used to evaluate the registration degree of the image structure, and CC is used to measure the correlation degree between the registered image and the original image. The lower the MSE, the higher the SSIM and CC, indicating that the image registration effect is better, so that the registration performance of the method is evaluated from multiple angles.

[0073] Experimental results: as shown in Tables 1-4, the method proposed in the application has achieved the best registration performance on all index evaluations on the four multi-modal data sets, especially on the three infrared data sets, which is significantly ahead of other methods and has superior registration effect; on the SAR data set, although the test index is low, it is still ahead of other methods, and SAR modality image registration is still a very difficult task at present.

[0074] Table 1 Experimental results of the algorithm on the ordinary scene infrared data set

[0075]

[0076] Table 2 Experimental results of the algorithm on the remote sensing scene infrared data set

[0077]

[0078]

[0079] Table 3 Experimental results of the algorithm on the near-infrared dataset of the remote sensing scene

[0080]

[0081] Table 4 Experimental results of the algorithm on the SAR dataset of the remote sensing scene

[0082]

[0083] In summary, the multi-modal image registration method based on the modal alignment distillation provided by the application has certain advancement, and can effectively solve the problems of poor matching feature recognition ability, difficult multi-modal model optimization, and poor registration performance caused by too large modal difference in the multi-modal image registration task, and has excellent multi-modal image registration performance.

[0084] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the application and are not limiting. Although the application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the application can be modified or replaced equivalently without departing from the purpose and scope of the technical solutions, and they should be covered in the scope of the claims of the application.

Claims

1. A multimodal image registration method based on modality alignment distillation, characterized in that: The method specifically comprises the following steps: S1: Data preprocessing: Obtain the original pre-registered multimodal images, generate a random deformation flow field to act on the original multimodal images, causing the original images to produce random deformations, and establish an unregistered multimodal dataset, including a training set and a test set, in which the original images are used as labels for self-supervised learning; S2: Build the registration model, including the encoder and the transformer-based U-shaped registration network; The encoder is a U-shaped network with horizontal connections, including four downsampling and four upsampling operations. It uses shared parameters to extract features from the deformed image and the reference image respectively, which are 16-channel intermediate features. The reference image of the student model is output by the encoder through the intermediate features, and modality transfer is performed under distillation learning to reduce modality differences. The transformer-based U-shaped registration network includes a downsampling encoding part consisting of a feature enhancement module and a multi-scale attention mechanism module, an upsampling decoding module consisting of a residual connection module, and a lateral connection part consisting of a transformer; the feature enhancement module adopts parallel diffusion convolution with different diffusion rates, an instance normalization layer, and a leaky Relu activation layer, and adopts a residual connection structure. In combination with the multi-scale attention mechanism, it can extract features from different receptive fields and extract deformation information of different sizes, effectively improving the non-rigid registration performance; the transformer can obtain global information and effectively improve the rigid registration performance; the upsampling decoding module synthesizes the previously extracted information and predicts the final output registration flow field; S3: Training the multimodal registration model: Based on the deformable mask training strategy and modality distillation learning technology, the teacher model and student model are trained to obtain a homomodal teacher model and a multimodal student model based on modality alignment distillation; The specific steps of the deformable mask training strategy are: S31: Set a corresponding single-channel all-1 mask for each modality image to be registered in the training set. The generated random deformation flow field acts on both the image and the mask at the same time. The zero-filled areas of the two are consistent, that is, the black edges are the same. The calculation process is as follows: M W =R(M,φ r ) (2) Among them, I mov is the deformed image, φ r is the deformed flow field randomly generated in data preprocessing, M is the original single-channel all-1 mask, It is the result of the deformation image being affected by the deformation flow field, M W It is the result of M being deformed by the same flow field, that is, M W and have the same 0 boundaries; S32: The model outputs the registration flow field to register the deformed image, generating a secondary black edge. At the same time, the mask is affected by the inverse transformation of the deformed flow field. The secondary black edge of the mask is consistent with the secondary black edge of the registration image. The calculation process is as follows: Among them, φ reg is the model-predicted registration flow field, is the registered image, is φ r The flow field after the reverse operation, M′ is M W by The mask after the action is ideally the same as The 0 boundary of is the same; S33: Mask M′ is multiplied by the Hadamard matrix with the original image and the registered image respectively to limit the extra information of the original image compared to the zero boundary of the registered image. Then the loss calculation is performed on the image to make the network pay more attention to the central area and solve the parameter optimization disorder caused by the asymmetric edge information. The loss function used for student model training includes not only L1 loss and SSIM loss, but also distillation-level supervision of the teacher model on the student model. The loss function is calculated as follows: L stu =The pix +The sim +L distillati where (8) Among them, L' pix represents the multimodal pixel loss, L' sim represents the multimodal structure loss, L distillation represents the loss function corresponding to distillation learning, h represents the image length, w represents the image width, I w Represents the deformed image after deformation, I gt represents the original image, M′ represents the mask matrix obtained by the deformable mask training strategy, and E s is the student model encoder, E t is the teacher model encoder, I md is a multimodal image, It is the predicted registration flow field of the deformable image with the multimodal image as the target, represents the model prediction and registration flow field; L distillation Is the loss function corresponding to distillation learning, guiding the optimization of the registration performance of the student model. Part I || E s (I md )-E t (I gt )||1 Promote the multimodal reference image features of the student model to continuously move closer to the unimodal reference image features of the teacher model, greatly reducing the modality difference and improving the registration performance; Part II The predicted registration flow field of the student model is promoted to be close to the teacher model with excellent performance, and the deformation of the displacement image of the student model and the teacher model during the distillation process is the same, so, and Ideally, they are the same; S4: Input the unregistered multimodal test set established in step S1 into the trained model registration model and output the registered image.

2. The multimodal image registration method based on modality alignment distillation according to claim 1, characterized in that: In step S3, the deformable mask training strategy specifically includes: setting a single-channel all-one mask matrix, subjecting it to the data preprocessing distortion flow field and the model output registration flow field multiple times, and filling it with 0 during the registration resampling process to produce a black edge equivalent to the registered image; During the parameter optimization process, the mask is multiplied by the Hadamard matrix with the registered image and the label image respectively. The black edges of the mask will limit the redundant information of the black edges of the registered image and the corresponding areas of the label image, so that the model only focuses on the registration results of the central deformation area, and ignores the impact of zero padding on the output image boundary on model training and performance, effectively promoting the performance optimization of the registration task; this strategy is applied to the teacher model and student model training processes at the same time, reducing the negative impact of the edge information asymmetry of the unregistered image and the black edges generated by the spatial transformation resampling during the registration process on the registration performance.

3. The multimodal image registration method based on modality alignment distillation according to claim 1, characterized in that: In step S3, the distillation learning uses a trained homomodal teacher model with excellent registration performance to perform multi-dimensional guided supervision on the multimodal registration student model, including feature mapping supervision of the homomodal features output by the encoder to the multimodal features, thereby reducing modal differences and improving the student model's ability to recognize multimodal matching features, as well as supervision of the predicted flow field output by the transformer-based U-shaped registration network; In this process, the registration performance of the multimodal student model continues to approach the registration performance of the same-modal teacher model, thus completing the multimodal image registration task based on distillation learning.

4. The multimodal image registration method based on modality alignment distillation according to claim 1, characterized in that: In step S3, the training of the teacher model is specifically as follows: selecting a single modality image in the multimodal dataset for training; A random deformation flow field is generated and applied to the unimodal original image. The spatially deformed image is used as the deformed image, and the undeformed original image is input into the model as the reference image. Parameters are optimized under the set loss function to complete the training and obtain the homomodal teacher model.

5. The multimodal image registration method based on modality alignment distillation according to claim 4, characterized in that: In step S3, the loss functions used for teacher model training include L1 loss and SSIM loss, which optimize the registration performance of the model from the pixel level and structure level perspectives respectively. The loss function is calculated as follows: L tea =L pix +L sim (11) Among them, L tea Represents the teacher model training loss function, L pix represents the same-modal pixel loss, L sim represents the same modal structure loss, h represents the image length, w represents the image width, I w Represents the deformed image after deformation, I gt represents the original image, M′ represents the mask matrix obtained by the deformable mask training strategy, represents the model prediction registration flow field, represents the registration resampling operation, ⊙ represents the Hadamard matrix multiplication operation, and ||·||1 represents the L1 loss.

6. The multimodal image registration method based on modality alignment distillation according to claim 1, characterized in that: In step S3, the training of the student model is specifically as follows: the parameters of the teacher model remain fixed; a random deformation flow field is generated and simultaneously acts on the modal image selected by the teacher model and the corresponding modal image in the student model to produce unregistered images with the same deformation as the deformed images of the two models. The teacher model uses the original image of the same modality as the reference image, and the student model uses the other modality as the reference image; during the training process, the encoder of the teacher model with fixed parameters outputs intermediate features and the transformer-based U-shaped registration network outputs the registration flow field as the target, and the intermediate features and output registration flow field corresponding to the student model are calculated for loss to realize multimodal image registration distillation learning.

Images