A multimodal remote sensing image registration method and storage medium based on mixed deformation

By combining rigid and non-rigid registration models, using feature extractors and feature enhancement modules with mask learning, integrating transformation parameters, the problem of insufficient single deformation processing capability in the existing technology is solved, and the complex deformation scenarios are realized multimodal image registration improves robustness and applicability.

CN118071804BActive Publication Date: 2024-11-26ANHUI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410194034.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-02-21
Publication Date
2024-11-26
Estimated Expiration
2044-02-21

AI Technical Summary

Technical Problem

The existing multi-stage remote sensing image registration method mainly focuses on processing single deformation, and it is difficult to effectively deal with complex multiple deformations and large position differences, which limits its applicability and robustness in practical applications.

Method used

Using a multimodal remote sensing image registration method based on hybrid deformation, combining rigid and non-rigid registration models, a bidirectional registration strategy is designed through a feature extractor and feature enhancement module with mask learning, integrating transformation parameters and calculating errors. , update model parameters to improve registration accuracy and applicability.

Benefits of technology

It realizes multimodal image registration in complex deformation scenarios, enhances the robustness and accuracy of the model, can process images from conventional viewing angles and remote sensing angles, and is suitable for large-scale and small-scale deformations, improving registration Performance and reliability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118071804B_ABST
    Figure CN118071804B_ABST
Patent Text Reader

Abstract

A multimodal remote sensing image registration method and storage medium based on mixed deformation, belonging to the technical field of computer vision, solves the problem of multimodal image registration in complex deformation scenes, obtains remote sensing image pairs of different modes to be registered, uses a feature extractor of mask learning to obtain image features, sends the image features to a rigid registration model, obtains transformation parameters and a rough registration result, processes the rough registration result and a target image again through a feature extractor, sends the image features to a non-rigid registration model to obtain transformation parameters, processes the original image to be registered to obtain a registration result, calculates a transformation parameter error and a reprojection error, updates model parameters according to the error, uses a joint registration network to process the image to obtain a registration result, and calculates the corner error loss of the image; the present invention combines rigid registration and non-rigid registration methods to register remote sensing images, and adopts a bidirectional registration strategy to increase the robustness and accuracy of the model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision and relates to a multimodal remote sensing image registration method based on mixed deformation and a storage medium. Background Art

[0002] In recent years, with the rapid development of sensor technology and imaging equipment, image acquisition methods have shown a trend of richness and diversity. This diversity not only shows an explosive growth in the amount of data, but also covers multiple aspects such as multi-dimensionality, multi-scale and high resolution of data. In the context of such massive data, multimodal images have become an important data basis for human beings to explore the world and promote the development of social intelligence, providing researchers with rich subject content. The application of multimodal images has widely covered many fields, including but not limited to remote sensing ground object monitoring, medical auxiliary diagnosis and computer vision analysis. In these fields, multimodal images are not only used for basic image tasks such as recognition, detection, segmentation, tracking and matching, but also gradually penetrate into more complex application scenarios. For example, in remote sensing data analysis, multi-source heterogeneous image information provides comprehensive and multi-level support for earth observation and resource monitoring; in the medical field, the combination of multimodal images enables auxiliary diagnosis systems to analyze patients' conditions more comprehensively and accurately. However, as mentioned above, massive multimodal image data also brings a series of challenges. The influence of time sequence, perspective, illumination, noise, spectral characteristics and other factors often lead to inconsistent intensity differences and visual characteristics between images, making the effective use of data more complicated. Therefore, how to eliminate these differences and mine potential feature association information has become an important issue to be solved in current multimodal image research. Solving this problem will further promote the development of automatic image processing and intelligent interpretation, and enhance the comprehensive level of human cognition of the world.

[0003] In the field of remote sensing, there are many types of multimodal images. According to the source of electromagnetic wave signals, they can be divided into active and passive imaging technologies. Common active remote sensing images include lidar, synthetic aperture radar, etc. Passive remote sensing images can be divided into visible light, infrared, ultraviolet, etc. according to the wavelength of electromagnetic waves. Image data of the same scene from different sources usually differ in resolution, spectrum, landscape features, local texture structure and noise distribution due to differences in imaging time, viewing angle, illumination and imaging mechanism. It is these complex differences that make the comprehensive utilization of multimodal image information quite difficult. In practical applications, how to effectively integrate these heterogeneous data and give full play to the advantages of various images has become an important issue that needs to be solved in the field of remote sensing.

[0004] Various types of multimodal images are widely used in the fields of image change detection, stitching, fusion, super-resolution, etc. Regardless of the type of multimodal image, in order to make full use of the complementary, synergistic and redundant information between images, the first preprocessing to be completed is registration. The most fundamental reason why image registration is so important is that multimodal images of the same target or scene often have geometric deviations. Only when an accurate point-to-point mapping of objects with the same name is established between different modalities can further in-depth processing and analysis be effective. By aligning these multimodal images together, information fusion can be achieved, etc., providing a more comprehensive observation of the surface, thereby supporting many remote sensing applications such as geological exploration, resource management, agricultural and forestry monitoring, and urban planning.

[0005] Registration methods can be divided into rigid registration and non-rigid registration according to the deformation type. Rigid registration is a commonly used image registration method that aims to map the corresponding feature points or regions of different images to a common coordinate system through rigid transformations such as translation, rotation, and scaling to achieve spatial alignment of images. Non-rigid registration is an important type of image registration, which refers to the image registration process that allows more complex local deformations or bends between images and is not limited to rigid transformations.

[0006] In recent years, deep learning methods have gradually been integrated into the field of remote sensing images, such as image fusion, change detection, segmentation, etc. The advantage of deep learning methods in image registration tasks is that they can learn richer image features and have stronger representation and generalization capabilities. Deep learning-based methods can directly learn feature representations and transformation parameters from the original input image data. This end-to-end learning process can overcome changes and interference in the image and achieve more accurate and efficient image registration. In the paper (L. Li, L. Han, M. Ding, Z. Liu and H. Cao, "Remote Sensing Image Registration Based on Deep Learning Regression Model," in IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1-5, 2022, Art no. 8002905, doi:10.1109 / LGRS.2020.3032439.), a single-stage two-stream DenseNet model was constructed to learn features in images of different modalities. Finally, the angular displacement was regressed using the fully connected layer, and the corresponding rigid transformation parameters were derived from the displacement to achieve registration. Compared with some traditional methods, this method has significantly improved the registration accuracy. However, this method lacks interaction between the two different modal images. To address this problem, the literature (L. Li, L. Han, M. Ding and H. Cao, "Multimodal Image Fusion Framework for End-to-End Remote Sensing ImageRegistration," in IEEE Transactions on Geoscience and Remote Sensing, vol.61, pp. 1-14, 2023, Art no. 5607214, doi: 10.1109 / TGRS.2023.3247642.) proposed a multimodal image fusion network with self-attention to merge the feature representations of different modal images. The fused information is then used to regress the displacement parameters of the specified points to obtain the transformation parameters between the images. The structure of this method has great advantages over other model structures (VGG-16, ResNet-50, SENet).

[0007] However, deep learning models usually have a high degree of freedom in learning feature representations of images, which makes the learned transformation parameters may not strictly follow the geometric constraints of rigid transformations. Although deep learning models such as CNN can learn good feature representations on large-scale datasets, the learned feature representations do not always accurately capture the local and global characteristics of rigid transformations. In the literature (Y. Ye, T. Tang, B. Zhu, C. Yang, B. Li and S. Hao,"A Multiscale Framework With Unsupervised Learning for Remote Sensing Image Registration," in IEEE Transactions on Geoscience and Remote Sensing(TGRS),vol. 60, pp. 1-15, 2022, Art no. 5622215, doi: 10.1109 / TGRS.2022.3167644.), a multi-scale framework MU-Net is proposed, which stacks multiple deep neural network (DNN) models at multiple scales to generate a coarse-to-fine registration pipeline. This iterative idea can well solve the problem of low precision of single-stage networks, and experimental results also prove the effectiveness of MU-Net on multiple datasets. Existing CNN-based rigid registration methods focus on local misalignment or global orientation and position of the input to predict the rigid transformation matrix. These methods are sensitive to spatial initialization and show limited generalizability beyond the training dataset. A fast and robust coarse-to-fine algorithm (C2FViT) is proposed in the literature (TCW Mok and ACSChung, "Affine Medical Image Registration with Coarse-to-Fine VisionTransformer," 2022 IEEE / CVF Conference on Computer Vision and PatternRecognition (CVPR), New Orleans, LA, USA, 2022, pp. 20803-20812, doi:10.1109 / CVPR52688.2022.02017.), which naturally exploits the global connectivity and locality of the convolutional vision transformer and the multi-resolution strategy to learn global rigid registration. C2FViT outperforms existing CNN-based rigid registration methods in terms of registration accuracy, robustness, and generalizability.

[0008] Rigid registration methods are only applicable to linear transformation differences. For complex nonlinear transformations, deformable image registration methods are required. This type of method predicts a displacement for each pixel in the image. The paper (Sokooti, ​​Hessamet al. "3D Convolutional Neural Networks Image Registration Based on Efficient Supervised Learning from Artificial Deformations." ArXiv abs / 1908.10235) proposes a supervised deformable image registration network structure RegNet, which uses CNN to directly predict the displacement vector field (DVF) from a pair of input images, and uses a large number of artificially synthesized smooth DVFs to train the network. However, directly estimating DVF cannot guarantee that the transformation is smooth and reversible. In order to better ensure the differential homeomorphism of the transformation, the literature (Sokooti, ​​H., de Vos, B., Berendsen, F., Lelieveldt, BPF, Išgum, I., Staring, M. “Nonrigid Image Registration Using Multi-scale 3D ConvolutionalNeural Networks.” in Medical Image Computing and Computer AssistedIntervention(MICCAI), 2017, vol 10433. Springer, Cham. https: / / doi.org / 10.1007 / 978-3-319-66182-7_27) proposed the SVF-Net network, which uses a fully convolutional network similar to U-Net to estimate the stationary velocity field SVF of a pair of images. The mean square error between the real SVF parameters and the predicted SVF parameters is also used as the loss function to train the network.

[0009] In order to more accurately predict DVF with large displacement, the literature (Rohé, MM., Datar, M., Heimann,T., Sermesant, M., Pennec, X. “SVF-Net: Learning Deformable Image Registration Using Shape Matching.” in Medical Image Computing and ComputerAssisted Intervention (MICCAI), 2017, vol 10433.) proposed a multi-stage supervised deformable image registration method based on the RegNet model. The network consists of RegNet4, RegNet2 and RegNet1 networks. The three RegNet networks are divided into three stages to perform image registration step by step and generate the final DVF. Also using a multi-stage network structure, the paper (Y. Xu, J. Li, C. Du and H. Chen, "NBR-Net: A NonrigidBidirectional Registration Network for Multitemporal Remote Sensing Images," in IEEE Transactions on Geoscience and Remote Sensing(TGRS), vol. 60, pp. 1-15, 2022, Art no. 5620715, doi: 10.1109 / TGRS.2022.3162094.) proposed a bidirectional registration network. First, VGG-16 is used to extract features from the two images, which are then sent to the optical flow estimation decoder to predict the corresponding deformation field. After multiple stages, the final registered image is obtained, and then the registered image and the reference image distorted by the true value are sent to the model again to obtain the original image to be registered. This bidirectional structure is used to enhance the reversibility and geometric consistency of the registration. Similarly, the paper (Zhang, Jun. "Inverse-Consistent Deep Networks for Unsupervised Deformable Image Registration." ArXiv abs / 1809.03443) also proposed an inverse consistency network ICNet to encourage a pair of images to deform symmetrically with each other, thereby ensuring the inverse consistency of the transformation between a pair of images. At the same time, in order to better avoid the folding of the deformation field, an anti-folding constraint is added to the original smoothness constraint.

[0010] When processing remote sensing image pairs, existing single-stage registration methods often have difficulty achieving high-precision registration when faced with situations with large modality differences. In particular, when there are significant positional differences between image pairs, single-stage methods often show unsatisfactory results. To overcome these challenges, multi-stage registration strategies have emerged. By introducing multi-stage processing, the impact of modality differences and positional differences on registration accuracy can be effectively mitigated, thereby further improving registration performance and providing a more reliable foundation for subsequent tasks.

[0011] However, current multi-stage methods mainly focus on dealing with a single deformation, which limits their applicability in practical applications. In actual remote sensing image registration scenarios, image pairs may involve multiple deformations, and existing methods are limited in their performance for complex deformation situations. In order to more comprehensively address the challenges of multimodality and large position differences, it is necessary to further explore and develop more comprehensive and multi-faceted multi-stage registration strategies to improve the robustness and applicability of registration. This will help promote research in the field of remote sensing image registration and provide a more reliable data foundation for a wider range of remote sensing applications. Summary of the invention

[0012] The technical solution of the present invention is used to solve the problem of multimodal image registration in complex deformation scenes.

[0013] The present invention solves the above technical problems through the following technical solutions:

[0014] A multimodal remote sensing image registration method based on mixed deformation, comprising: a training process and a testing process;

[0015] The method of the training process is as follows:

[0016] 1) Obtain remote sensing image pairs of different modalities to be registered;

[0017] 2) Obtain image features using a feature extractor with mask learning;

[0018] 3) Send the image features into the rigid registration model to obtain the transformation parameters and rough registration results;

[0019] 4) The rough registration result and the target image are processed again by the feature extractor;

[0020] 5) Send the image features into the non-rigid registration model to obtain the transformation parameters;

[0021] 6) Integrate the two types of transformation parameters and process the original image to be registered to obtain the registration result;

[0022] 7) Calculate the transformation parameter error and reprojection error, and update the model parameters based on the error;

[0023] The method of the testing process is: taking the visible light image and the infrared image used for testing as input, using the joint registration network to process the image to obtain the registration result, and after the registration, calculating the corner error loss of the image.

[0024] Furthermore, the method for obtaining remote sensing image pairs of different modalities to be registered is as follows: aligning the visible light images and infrared images The infrared image in the center is randomly distorted to obtain the image to be registered. The distorted infrared image, i.e. the image to be registered, is recorded as .

[0025] Furthermore, the method for obtaining image features using a feature extractor with mask learning is as follows: using a feature extractor with mask learning to perform image registration and visible light images The processing process is as follows: the image to be registered and visible light images The learned mask is multiplied by the image to be registered and the visible light image to obtain the final image feature, which is recorded as and .

[0026] Furthermore, the method of sending the image features into the rigid registration model to obtain the transformation parameters and the rough registration results is as follows:

[0027] The extracted image features are fed into the rigid registration network to directly predict the 6 transformation parameters , and use the parameters to treat the registered image Perform coarse registration and obtain coarse registration results , which is expressed as follows:

[0028]

[0029]

[0030] Where Rmodel represents the rigid registration module and warp represents the warping operation.

[0031] Furthermore, the method of processing the rough registration result and the target image again through the feature extractor is as follows: and visible light images Then it is sent to the feature extractor again to obtain the corresponding image features, which are recorded as: , .

[0032] Furthermore, the method of sending the image features into the non-rigid registration model to obtain the transformation parameters is as follows: the extracted image features are sent together into the non-rigid registration network to directly predict the transformation parameters , which is expressed as follows:

[0033]

[0034] Where NRmodel represents the rigid registration module.

[0035] Furthermore, the method of integrating the two types of transformation parameters and processing the original image to be registered to obtain the registration result is as follows:

[0036] The rigid transformation parameters Convert to non-rigid transformation parameters The expression form of Perform a warping operation to obtain the final registration result , expressed as follows:

[0037]

[0038] in, Indicates that the transformation parameters Convert to transformation parameters form of expression.

[0039] Furthermore, it is characterized in that the formula for calculating the transformation parameter error and the reprojection error is:

[0040] Rigid transformation parameter error ;

[0041] Non-rigid transformation parameter error ;

[0042] Reprojection Error ;

[0043] in, represents the rigid transformation parameters predicted by the model, represents the non-rigid transformation parameters predicted by the model, and represents the truth value transformation parameter, Represents the pixel value of the corresponding coordinate in the image.

[0044] Furthermore, the formula for calculating the corner error loss of the image is as follows:

[0045]

[0046] Where n represents the number of image pairs, is the jth displacement parameter of the ith test sample, is the true value of the corresponding component.

[0047] A storage medium stores a computer program, which executes the steps of the above-mentioned multi-modal remote sensing image registration method based on mixed deformation when executed by a processor.

[0048] The advantages of the present invention are:

[0049] The present invention combines rigid registration and non-rigid registration methods to register remote sensing images, and adopts a two-way registration strategy to increase the robustness and accuracy of the model. In addition, the scope of application of the model is expanded, and it can not only process images under conventional perspectives but also images under remote sensing perspectives, and whether the deformation between the image pairs is large-scale or small-scale, it is within the processing capability of the model. In the specific module, a feature enhancement module and a residual attention module are designed to further improve the performance of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of a multimodal remote sensing image registration method based on mixed deformation according to an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of a network model structure of a multimodal remote sensing image registration method based on mixed deformation according to an embodiment of the present invention;

[0052] Figure 3 It is a structural schematic diagram of a feature enhancement module in a rigid registration model of a multimodal remote sensing image registration method based on mixed deformation according to an embodiment of the present invention;

[0053] Figure 4 Schematic diagram of the structure of an attention module in a non-rigid registration model of a multimodal remote sensing image registration method based on mixed deformation according to an embodiment of the present invention;

[0054] Figure 5 Ablation experiment results on feature enhancement (FE);

[0055] Figure 6 Ablation experiment results on attention (ATT). DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments:

[0058] Embodiment 1

[0059] like Figure 1 and Figure 2 As shown, the multimodal remote sensing image registration method based on hybrid deformation in an embodiment of the present invention is used for cross-modal remote sensing image registration of visible light and infrared light. Visible light images have rich visual information such as texture and shape. Considering that infrared images are usually used as a supplement to visible light image information, the method in an embodiment of the present invention takes the visible light image as the target image and the infrared image as the moving image, and aligns the infrared image with the visible light image; specifically, it includes a training process and a testing process.

[0060] 1. Training process:

[0061] 1) Obtain remote sensing image pairs of different modalities to be registered

[0062] Align the visible light images and infrared images The infrared image in the center is randomly distorted to obtain the image to be registered, and the distorted infrared image (image to be registered) is recorded as .

[0063] 2) Obtain image features using a feature extractor with mask learning

[0064] In step 2, the feature extractor with mask learning is used to treat the registration image and visible light images The processing process is as follows: the image to be registered and visible light images The learned mask is multiplied by the image to be registered and the visible light image to obtain the final image feature, which is recorded as and Among them, the role of mask learning of the feature extractor is to make the extracted features more helpful for the registration task features.

[0065] 3) Send the image features into the rigid registration model to obtain the transformation parameters and rough registration results

[0066] The extracted image features are fed into the rigid registration network to directly predict the 6 transformation parameters , and use the parameters to treat the registered image Perform coarse registration and obtain coarse registration results , which is expressed as follows:

[0067]

[0068]

[0069] Where Rmodel represents the rigid registration module and warp represents the warping operation.

[0070] The extracted visible light image features and infrared image features It is sent as input to the parameter regression module; this module uses ResNet as the backbone to obtain the final transformation parameters.

[0071]

[0072] 4) The rough registration result and the target image are processed again by the feature extractor

[0073] The obtained rough registration result and visible light images Then it is sent to the feature extractor again to obtain the corresponding image features, which are recorded as: , .

[0074] 5) Send image features into the non-rigid registration model to obtain transformation parameters

[0075] The extracted image features are fed into the non-rigid registration network to directly predict the transformation parameters , which is expressed as follows:

[0076]

[0077] Where NRmodel represents the rigid registration module.

[0078] Step 4) is the same as step 5), except that the parameter regression network is changed and the backbone of the non-rigid registration module is UNet, that is:

[0079]

[0080] 6) Integrate the two types of transformation parameters and process the original image to be registered to obtain the registration result;

[0081] The rigid transformation parameters Convert to non-rigid transformation parameters The expression form of Perform a warping operation to obtain the final registration result , expressed as follows:

[0082]

[0083] in, Indicates that the transformation parameters Convert to transformation parameters form of expression.

[0084] 7) Calculate the transformation parameter error and reprojection error, and update the model parameters based on the error

[0085] The parameter error and reprojection error formula are:

[0086] Rigid transformation parameter error

[0087] Non-rigid transformation parameter error

[0088] Reprojection Error

[0089] in, represents the rigid transformation parameters predicted by the model, represents the non-rigid transformation parameters predicted by the model, and represents the truth value transformation parameter, Represents the pixel value of the corresponding coordinate in the image.

[0090] 2. Testing process:

[0091] The model parameters in the testing process should be the parameters that have been trained in the training process. The model parameters remain unchanged during the training process. The multimodal image to be registered is taken as input, and the model gives the registered multimodal image as a result.

[0092] Specifically, the visible light image and infrared image used for testing are used as input, and the image is processed using the joint registration network to obtain the registration result. After registration, the corner error loss of the image is calculated as follows:

[0093]

[0094] Where n represents the number of image pairs, is the jth displacement parameter of the ith test sample, is the true value of the corresponding component.

[0095] The present invention first analyzes two types of registration methods. For rigid registration methods, the advantage is that the global geometric structure of the image can be retained during the transformation process, including transformations such as translation, rotation and scaling. This is very advantageous for processing large-scale transformations because the overall shape and direction of the image are maintained. At the same time, there are also shortcomings: the goal of rigid registration is overall transformation, so local details and minor changes in the image may be ignored. In remote sensing images, some important objects or features may exist in small-scale areas of the image, and rigid registration may not be able to capture these subtle changes, resulting in inaccurate registration. For non-rigid registration, the advantage is that it can better capture subtle changes between images, which means that non-rigid registration can align images more accurately in local areas, thereby improving the overall registration accuracy. However, it may perform poorly when processing large-scale deformations. When there are large overall displacements, scaling or distortions between images, non-rigid methods are difficult to capture these global transformations, resulting in increased registration errors. In addition, in the field of remote sensing, although most existing methods use multi-stage or multi-scale strategies to improve registration accuracy, only one deformation method is involved in their models.

[0096] This paper integrates multiple types of deformation into one framework for the first time and proposes a registration method for multimodal remote sensing images based on mixed deformation. Compared with the existing multi-stage methods involving only a single type of deformation, this paper is more widely applicable and has better effects. Specifically in the rigid registration stage, a feature enhancement module is designed, such as Figure 3 As shown in Figure 1, by connecting dilated convolution operations with different dilation rates in series and parallel, the receptive field of the feature points is expanded without reducing the resolution of the feature map. The number of convolution blocks and the choice of the dilation rate can be customized based on the dimension of the input feature map. For example, consecutive dilated convolution layers with dilation rates of 1, 2, and 4 will produce receptive fields of size 3, 7, and 15, respectively. For a 16×16 feature map, using dilated convolutions with these dilation rates ensures that the points in the final layer contain contextual cues that span the 15×15 region of the initial feature map.

[0097] In fact, this design allows the network to absorb most of the initial input information by gradually expanding. Specifically in the non-rigid registration stage, an attention module is designed, such as Figure 4 As shown in Figure 1, in this module, we design a residual dilated convolution block (RDC) to enhance the correlation between similar features. This is achieved by weighting and aggregating multi-scale representations originating from different spatial locations. In the end, this method integrates the two types of parameters and only warps the image once, reducing the impact of the interpolation operation on image quality and registration accuracy. Figure 5 and Figure 6 The effectiveness of the two modules can be seen.

[0098] For multimodal remote sensing images, previous methods only involve a single deformation category. The single-stage rigid registration has low accuracy, and the single-stage non-rigid registration cannot effectively handle large-scale position differences. Even if both categories derive multi-stage strategies, the improvement effect of a single deformation type is limited and there are few application scenarios. This invention proposes for the first time a multi-stage registration strategy that combines rigid deformation and non-rigid deformation. It has a wider range of application scenarios. It can process multimodal images from different perspectives (conventional perspective, remote sensing perspective), and can also process images with different scale differences (large scale, small scale).

[0099] Embodiment 2

[0100] A storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a multi-modal remote sensing image registration method based on mixed deformation in embodiment 1 are executed.

[0101] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal remote sensing image registration method based on mixed deformation, characterized in that: include: Training process and testing process; The method of the training process is as follows: 1) Obtain remote sensing image pairs of different modalities to be registered; 2) Obtain image features using a feature extractor with mask learning; 3) Send the image features into the rigid registration model to obtain the transformation parameters and rough registration results; 4) The rough registration result and the target image are processed again by the feature extractor; 5) Send the image features into the non-rigid registration model to obtain the transformation parameters; 6) Integrate the two types of transformation parameters and process the original image to be registered to obtain the registration result. The specific method is as follows: The rigid transformation parameters Convert to non-rigid transformation parameters The expression form of Perform a warping operation to obtain the final registration result , expressed as follows: in, Indicates that the transformation parameters Convert to transformation parameters The form of expression; 7) Calculate the transformation parameter error and the reprojection error, and update the model parameters according to the error. The formula for calculating the transformation parameter error and the reprojection error is: Rigid transformation parameter error ; Non-rigid transformation parameter error ; Reprojection Error ; in, represents the rigid transformation parameters predicted by the model, represents the non-rigid transformation parameters predicted by the model, and represents the truth value transformation parameter, Represents the pixel value of the corresponding coordinate in the image; The method of the testing process is: taking the visible light image and the infrared image used for testing as input, using the joint registration network to process the image to obtain the registration result, and after the registration, calculating the corner error loss of the image.

2. The multimodal remote sensing image registration method based on mixed deformation according to claim 1, characterized in that: The method for obtaining remote sensing image pairs of different modalities to be registered is as follows: aligning the visible light images and infrared images The infrared image in the center is randomly distorted to obtain the image to be registered. The distorted infrared image, i.e. the image to be registered, is recorded as .

3. The multimodal remote sensing image registration method based on mixed deformation according to claim 1, characterized in that: The method for obtaining image features by using a feature extractor with mask learning is as follows: using a feature extractor with mask learning to treat the registration image and visible light images The processing process is as follows: the image to be registered and visible light images The learned mask is multiplied by the image to be registered and the visible light image to obtain the final image feature, which is recorded as and .

4. The multimodal remote sensing image registration method based on mixed deformation according to claim 3 is characterized in that: The method of sending the image features into the rigid registration model to obtain the transformation parameters and the rough registration results is as follows: The extracted image features are fed into the rigid registration network to directly predict the 6 transformation parameters , and use the parameters to treat the registered image Perform coarse registration and obtain coarse registration results , which is expressed as follows: Where Rmodel represents the rigid registration module and warp represents the warping operation.

5. The multimodal remote sensing image registration method based on mixed deformation according to claim 1, characterized in that: The method of processing the rough registration result and the target image again through the feature extractor is as follows: and visible light images Then it is sent to the feature extractor again to obtain the corresponding image features, which are recorded as: , .

6. The multimodal remote sensing image registration method based on mixed deformation according to claim 1, characterized in that: The method of sending the image features into the non-rigid registration model to obtain the transformation parameters is as follows: the extracted image features are sent together into the non-rigid registration network to directly predict the transformation parameters , which is expressed as follows: Where NRmodel represents the rigid registration module.

7. The multimodal remote sensing image registration method based on mixed deformation according to claim 1, characterized in that: The formula for calculating the corner error loss of the image is as follows: Where n represents the number of image pairs, is the jth displacement parameter of the ith test sample, is the true value of the corresponding component.

8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multimodal remote sensing image registration method based on mixed deformation according to any one of claims 1 to 7 are executed.