Image registration method and system based on joint guidance of common features and imaging parameters

Abstract

The application discloses a kind of based on common feature and imaging parameter joint guiding image registration method and system, the method of the present application includes extracting deep feature map to target modality and reference modality, after projection to unified structure representation space, using bidirectional interactive attention mechanism carries out geometric alignment to obtain coarse registration transformation matrix, the coarse registration image of target modality and the input image of reference modality are respectively using depth estimation network to predict dense depth map, dense depth map is used to construct three-dimensional point cloud graph using imaging parameter, and fine registration transformation matrix is calculated to the three-dimensional point cloud graph of target modality and reference modality;Coarse registration transformation matrix and fine registration transformation matrix are fused to obtain the final registration transformation matrix, and the input image of target modality is used to generate the final coarse registration image of target modality using the final registration transformation matrix.The present application aims to solve the problem that the existing multi-modal image registration method is not good in complex scene registration effect.

Description

Technical Field

[0001] This invention relates to image registration technology in the field of image processing, specifically to an image registration method and system based on common features and imaging parameters jointly guided by these features. Background Technology

[0002] Multimodal image registration, a key technology for fusing images acquired from different sensors, has been widely applied in fields such as UAV navigation, target recognition, change detection, and image fusion. With the rapid development of imaging technology, multiple modalities of images, including optical, SAR, and infrared images, are playing an increasingly important role in applications such as military reconnaissance, disaster monitoring, and smart cities. However, due to differences in imaging mechanisms, different modal images exhibit significant differences in radiometric characteristics, geometric distortion, and texture representation, making it difficult for traditional image registration methods to achieve high-precision cross-modal registration. Furthermore, the complex and variable imaging conditions of UAVs and satellite platforms, along with factors such as changing viewpoints, scale differences, and occlusion, further increase the difficulty of multimodal image registration. Therefore, how to effectively overcome modal differences and achieve accurate registration of multi-source images has become a research hotspot in the field of multimodal image processing.

[0003] Existing multimodal image registration methods can be broadly categorized into feature-based methods, region-based methods, and deep learning-based methods. Feature-based methods perform registration by extracting and matching key points in the image, but the invariance of feature descriptors is difficult to guarantee in cross-modal scenarios. Region-based methods perform registration by calculating similarity metrics between image regions, but they are sensitive to radiometric differences and have poor robustness under multimodal conditions. In recent years, deep learning-based methods have received widespread attention, and researchers have proposed various cross-modal feature learning networks. However, existing methods still suffer from insufficient registration accuracy and poor generalization ability when dealing with large viewpoint changes and complex geometric distortions. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide an image registration method and system based on common features and imaging parameters, which addresses the above-mentioned problems in the prior art. This invention aims to solve the problem that the existing multimodal image registration methods have poor registration results in complex scenes.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] An image registration method based on common features and imaging parameters jointly guided includes the following steps: S1, extracting depth feature maps from the input images of the target modality and the reference modality using feature extractors, projecting the obtained depth feature maps onto a unified structural representation space, establishing cross-modal feature associations using a bidirectional interactive attention mechanism, and performing geometric alignment on the target modality image to obtain a coarse registration transformation matrix, and generating a coarse registration image of the target modality from the input image of the target modality using the coarse registration transformation matrix; S2, predicting dense depth maps from the coarse registration image of the target modality and the input image of the reference modality using a depth estimation network, constructing a 3D point cloud map from the dense depth map using imaging parameters, and calculating a fine registration transformation matrix for the 3D point cloud map of the target modality and the reference modality; S3, fusing the coarse registration transformation matrix and the fine registration transformation matrix to obtain the final registration transformation matrix, and generating the final registration image of the target modality from the input image of the target modality using the final registration transformation matrix.

[0007] Optionally, step S1 includes:

[0008] S1.1, extract features from the input images of the target mode and the reference mode using feature extractors with independent parameters, and obtain depth feature maps of the target mode and the reference mode respectively;

[0009] S1.2, the depth feature maps of the target mode and the reference mode are projected onto a unified structural representation space using their respective learnable projection matrices;

[0010] S1.3, the projected target mode and reference mode depth feature maps are enhanced across modes using a bidirectional interactive attention mechanism to obtain the enhanced depth feature maps;

[0011] S1.4, the cosine similarity of each pixel in the enhanced depth feature map of the target mode and the reference mode is calculated by the feature matcher, so as to obtain the coarse registration similarity matrix composed of the cosine similarity of the pixels;

[0012] S1.5, Based on the coarse registration similarity matrix, the transformation matrix estimation algorithm is used to estimate the rotation matrix of the coarse registration. Translation vector And construct the coarse registration transformation matrix:

[0013] ;

[0014] in, This is the coarse registration transformation matrix;

[0015] S1.6, using the coarse registration transformation matrix The target modality image is converted into a coarsely registered target modality image.

[0016] Optionally, step S1.3, which involves using a bidirectional interactive attention mechanism to enhance the depth feature maps of the projected target modality and the reference modality across modalities, includes: using the projected depth feature map of the target modality as a query and the projected depth feature map of the reference modality as a key-value pair to obtain the enhanced depth feature map of the target modality through the bidirectional interactive attention mechanism; and using the projected depth feature map of the reference modality as a query and the projected depth feature map of the target modality as a key-value pair to obtain the enhanced depth feature map of the reference modality through the bidirectional interactive attention mechanism.

[0017] Optionally, the function expression for constructing a 3D point cloud map from the dense depth map using imaging parameters in step S2 is:

[0018] ;

[0019] in, This is a 3D point cloud map. For dense depth maps, For the camera intrinsic parameter matrix, For pixels, For pixels The coordinates.

[0020] Optionally, before calculating the fine registration transformation matrix for the 3D point cloud maps of the target modality and the reference modality in step S2, the method further includes performing structured feature enhancement on the 3D point cloud maps of the target modality and the reference modality using a Transformer-based multi-scale visual structure encoding network according to the following formula:

[0021] ;

[0022] in, This is a 3D point cloud image with enhanced structured features. This is a 3D point cloud map. For the learned geometric modulation weight matrix, Pixels extracted by a Transformer-based multi-scale visual structure encoding network The surrounding geometric context information, including local curvature and surface normals.

[0023] Optionally, in step S2, when calculating the fine registration transformation matrix for the 3D point cloud maps of the target mode and the reference mode, the least-squares optimization problem of constructing the optimal rigid body transformation between the 3D point cloud maps of the aligned target mode and the reference mode is included:

[0024] ;

[0025] in, Let be the rotation matrix of the rigid body transformation. Let be the translation vector of the rigid body transformation. For the k-th 3D point in the 3D point cloud map after target modal structured feature enhancement, For the k-th 3D point in the 3D point cloud map after modal structure enhancement, the rotation matrix of the rigid body transformation is obtained by fitting a least-squares optimization problem to each 3D point in the 3D point cloud maps of the target mode and the reference mode. Translation vector And construct the fine registration transformation matrix:

[0026] ;

[0027] in, This is the fine registration transformation matrix.

[0028] Optionally, the function expression for fusing the coarse registration transformation matrix and the fine registration transformation matrix in step S3 to obtain the final registration transformation matrix is:

[0029] ;

[0030] in, This is the final registration transformation matrix. This is an element-wise multiplication operation. This is the coarse registration transformation matrix. For fine registration transformation matrix, Let be the spatial adaptive weight matrix, where any nth element in the spatial adaptive weight matrix... OK Column elements And its calculation function expression is:

[0031] ;

[0032] ;

[0033] in, The alignment residual of the k-th 3D point in the 3D point cloud image after modal structured feature enhancement is used as a reference. To find the maximum alignment residual among all points in the 3D point cloud image after modal structured feature enhancement, For the k-th 3D point in the 3D point cloud map after target modal structured feature enhancement, Let be the rotation matrix of the rigid body transformation. For the k-th 3D point in the 3D point cloud map enhanced with reference modal structured features, Let be the translation vector for the rigid body transformation.

[0034] The present invention also provides an image registration system based on common features and imaging parameters jointly guided, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the image registration method based on common features and imaging parameters jointly guided.

[0035] The present invention also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the image registration method based on common features and imaging parameters jointly guided by a processor.

[0036] The present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the image registration method based on common features and imaging parameters jointly guided by a processor.

[0037] Compared with existing technologies, the present invention mainly achieves the following beneficial effects: 1. The present invention involves projecting the obtained depth feature map onto a unified structural representation space, establishing cross-modal feature associations using a bidirectional interactive attention mechanism, and performing geometric alignment on the target modal image to obtain a coarse registration transformation matrix. Through the bidirectional interactive attention mechanism, common feature representations of cross-modal images can be effectively extracted, and the two-dimensional registration problem is upgraded to a three-dimensional structural alignment problem through a depth estimation network. 2. The present invention, through a carefully designed two-stage architecture of "coarse registration - fine registration," achieves an effective combination of common feature extraction and imaging parameter estimation, significantly improving registration accuracy and robustness. Simultaneously, by combining the coarse and fine registration transformation matrices to obtain the final registration transformation matrix, the model can maintain high-precision registration results under complex conditions such as cross-spectral interference, viewpoint changes, and illumination differences. This solves the problem of poor registration performance of existing multimodal image registration methods in complex scenes, achieving high-precision registration of multimodal images in complex scenes. The final fused transformation matrix can achieve accurate imaging parameter recovery and pixel-level strict alignment of cross-modal images, providing accurate geometric correspondence for subsequent downstream tasks such as image fusion, object detection, and scene understanding. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0039] Figure 2 This is a schematic diagram of the network structure for the coarse registration part of the common features in the embodiments of the present invention.

[0040] Figure 3 This is a schematic diagram of the network structure of the fine registration part of the imaging parameters in an embodiment of the present invention. Detailed Implementation

[0041] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0042] like Figure 1 As shown, the image registration method based on common features and imaging parameters guided by this embodiment includes the following steps:

[0043] S1, use feature extractors to extract depth feature maps from the input images of the target modality and the reference modality respectively. After projecting the obtained depth feature maps onto a unified structural representation space, establish cross-modal feature association using a bidirectional interactive attention mechanism. Perform geometric alignment on the target modality image to obtain a coarse registration transformation matrix. Use the coarse registration transformation matrix to generate a coarse registration image of the target modality from the input image of the target modality.

[0044] S2, use a depth estimation network to predict dense depth maps for the coarse registration image of the target mode and the input image of the reference mode respectively, and use imaging parameters to construct a three-dimensional point cloud map from the dense depth map. Calculate the fine registration transformation matrix for the three-dimensional point cloud maps of the target mode and the reference mode.

[0045] S3, the coarse registration transformation matrix and the fine registration transformation matrix are fused to obtain the final registration transformation matrix, and the input image of the target mode is used to generate the final registered image of the target mode using the final registration transformation matrix.

[0046] like Figure 2 As shown, in this embodiment, the visible light image is used as the target mode and the infrared image is used as the reference mode. Undoubtedly, the target mode and reference mode in this embodiment can be selected according to actual needs. This embodiment mainly includes two parts: coarse registration of common features in step S1, and fine registration of imaging parameters in steps S2 and S3.

[0047] like Figure 2 As shown, step S1 in this embodiment includes:

[0048] S1.1, extract features from the input images of the target mode and the reference mode using feature extractors with independent parameters, and obtain depth feature maps of the target mode and the reference mode respectively;

[0049] S1.2, the depth feature maps of the target mode and the reference mode are projected onto a unified structural representation space using their respective learnable projection matrices;

[0050] S1.3, the projected target mode and reference mode depth feature maps are enhanced across modes using a bidirectional interactive attention mechanism to obtain the enhanced depth feature maps;

[0051] S1.4, the cosine similarity of each pixel in the enhanced depth feature map of the target mode and the reference mode is calculated by the feature matcher, so as to obtain the coarse registration similarity matrix composed of the cosine similarity of the pixels;

[0052] S1.5, Based on the coarse registration similarity matrix, the transformation matrix estimation algorithm is used to estimate the rotation matrix of the coarse registration. Translation vector And construct the coarse registration transformation matrix:

[0053] ;

[0054] in, This is the coarse registration transformation matrix;

[0055] S1.6, using the coarse registration transformation matrix The target modality image is converted into a coarsely registered target modality image.

[0056] To address the fundamental differences in imaging mechanisms among different modalities, this embodiment proposes a dual-path feature extraction and projection mapping method in step S1.1. For visible light images... With infrared images Since the two modalities have fundamental differences in their imaging mechanisms, features are extracted separately using feature extractors with independent parameters.

[0057] ;

[0058] ;

[0059] in, and These represent the depth feature extraction networks for visible light and infrared modes, respectively. and To extract the feature map, and Where C is the spatial size of the feature map, and C is the number of feature channels. Deep feature extraction networks typically employ convolutional neural network architectures such as the ALIKED model, which can automatically learn hierarchical feature representations.

[0060] Due to the significant differences across modalities, directly sharing parameters often leads to feature collapse, causing features from different modalities to become similar and losing discriminative information. Therefore, a structure of "dual-path network + local parameter sharing + global Transformer fusion" is adopted. Through normalization, attention, and projection operations, the final features are mapped to a unified structural space, eliminating the apparent differences between modalities while preserving geometric structural information. The feature projection process in step S1.2 is defined as follows:

[0061] ;

[0062] ;

[0063] in, and A learnable projection matrix is ​​used to project features from each modality onto a unified structural representation space. The projection matrix is ​​learned through end-to-end training and can adaptively extract cross-modal common features beneficial to the registration task. After projection, and It has comparable semantic representations, and similarity can be calculated directly.

[0064] The structural features of the two modalities are coupled through a bidirectional interactive attention mechanism to enhance their structural consistency. For example... Figure 2 As shown, step S1.3, which involves enhancing the projected depth feature maps of the target and reference modalities across modalities using a bidirectional interactive attention mechanism, includes: using the projected depth feature map of the target modality as a query and the projected depth feature map of the reference modality as a key-value pair, and obtaining the enhanced depth feature map of the target modality through the bidirectional interactive attention mechanism; and using the projected depth feature map of the reference modality as a query and the projected depth feature map of the target modality as a key-value pair, and obtaining the enhanced depth feature map of the reference modality through the bidirectional interactive attention mechanism. This bidirectional interactive design enables the two modalities to complement each other's information, improving the robustness of feature matching. The specific function expression is as follows:

[0065] ;

[0066] ;

[0067] in, , , These represent the query vector for the visible light branch and the key vector and value vector for the infrared branch, respectively. , , These represent the query vector for the infrared branch and the key vector and value vector for the visible light branch, respectively. For feature dimension, This represents the scaling factor. This symmetric cross-modal attention can simultaneously capture the infrared mapping in the visible light spectrum and the visible light mapping in the infrared spectrum, forming a joint constraint and significantly improving the stability of cross-spectral matching.

[0068] like Figure 2As shown, in step S1.4 of this embodiment, the cosine similarity of each pixel in the enhanced depth feature map of the target modality and the reference modality is calculated by the feature matcher. The attention-enhanced features are mapped to a comparable space, and the cross-modal correspondence of each pixel is calculated by the cosine similarity. The calculation method of cosine similarity is as follows:

[0069] ;

[0070] in, Represents the visible light image. The pixel and the infrared image Similarity between pixels and These represent the feature vectors at corresponding locations. For all visible light images, the [missing information - likely a specific image type or feature vector] represents the feature vector at each location. The pixel and the infrared image By combining the similarity between pixels, a coarse registration similarity matrix composed of the cosine similarity of pixels can be obtained. Based on this similarity matrix, a preliminary pixel-level correspondence can be established through the optimal transmission or nearest neighbor matching strategy, and then the global coarse registration transformation matrix can be estimated.

[0071] In step S1.5, the rotation matrix of the coarse registration is estimated using a transformation matrix estimation algorithm based on the coarse registration similarity matrix. Translation vector At this time, the transformation matrix estimation algorithm is a well-known existing algorithm, such as robust estimation methods like RANSAC. Finally, the coarse registration transformation matrix is ​​used... The target modality image is converted into a coarsely registered target modality image.

[0072] like Figure 3 As shown, in step S2, when predicting dense depth maps using a depth estimation network for the coarsely registered image of the target modality and the input image of the reference modality, the depth estimation network can adopt the existing encoder-decoder structured DepthAnything model as needed. Through training on a large-scale unorganized image set with weak geometric supervision, it can stably output dense depth maps in different scenarios. The function expression for predicting dense depth maps using the depth estimation network for the coarsely registered image of the target modality and the input image of the reference modality is as follows:

[0073] ;

[0074] in, For dense depth maps, For deep estimation networks, For the input image, For pixels, For pixels The coordinates. A dense depth map is a depth map whose instantaneous value of pixels exceeds a preset number. Dense depth maps provide a clear mapping from two-dimensional images to three-dimensional space, enabling cross-modal registration to be upgraded to a three-dimensional structural alignment problem, thereby introducing stronger geometric constraints and effectively improving the accuracy and robustness of registration.

[0075] like Figure 3 As shown, in step S2 of this embodiment, the construction of a 3D point cloud map from the dense depth map using imaging parameters (camera intrinsic parameter matrix) is achieved using a 3D reconstruction neural network. The function expression for constructing the 3D point cloud map is as follows:

[0076] ;

[0077] in, This is a 3D point cloud map. For dense depth maps, This is the camera intrinsic parameter matrix (containing parameters such as focal length and principal point coordinates). For pixels, For pixels The coordinates are then determined. By performing this backprojection operation on all pixels, a dense 3D point cloud representation can be obtained, elevating 2D image information to 3D space for processing.

[0078] As an optional implementation, to further improve the stability of the 3D representation, before calculating the fine registration transformation matrix for the 3D point cloud maps of the target modality and the reference modality in step S2 of this embodiment, the following method is used to perform structured feature enhancement on the 3D point cloud maps of the target modality and the reference modality using a Transformer-based multi-scale visual structure encoding network:

[0079] ;

[0080] in, This is a 3D point cloud image with enhanced structured features. This is a 3D point cloud map. For the learned geometric modulation weight matrix, Pixels extracted by a Transformer-based multi-scale visual structure encoding network The surrounding geometric context information, including local curvature and surface normals, is captured. A Transformer-based multi-scale visual structure encoding network captures cross-regional geometric relationships through multiple layers of self-attention modules, and its output features... It can be used to filter out mode-specific noise, making the point cloud distribution more consistent. It should be noted that the Transformer is an existing model, so its implementation details will not be elaborated here. The learned geometric modulation weight matrix... This method is used to fuse high-level structural features with depth estimation results. Weighted fusion can correct local errors in depth prediction, improve the geometric consistency of point clouds, and enhance their robustness to noise.

[0081] In step S2 of this embodiment, when calculating the fine registration transformation matrix for the 3D point cloud maps of the target mode and the reference mode, it includes a least-squares optimization problem of constructing the optimal rigid body transformation between the 3D point cloud maps of the target mode and the reference mode:

[0082] ;

[0083] in, Let be the rotation matrix of the rigid body transformation. Let be the translation vector of the rigid body transformation. For the k-th 3D point in the 3D point cloud map after target modal structured feature enhancement, For the k-th 3D point in the 3D point cloud map after modal structure enhancement, the rotation matrix of the rigid body transformation is obtained by fitting a least-squares optimization problem to each 3D point in the 3D point cloud maps of the target mode and the reference mode. Translation vector And construct the fine registration transformation matrix:

[0084] ;

[0085] in, The transformation matrix for fine registration is determined. The optimal rigid body transformation across modalities, i.e., the rotation matrix and translation vector, is solved by aligning two sets of point clouds. This process is formalized as a least-squares optimization problem, and the transformation matrix for fine imaging parameter estimation is obtained through optimization. Furthermore, if the following equation is not used to perform structured feature enhancement on the 3D point cloud maps of the target and reference modalities using a Transformer-based multi-scale visual structure encoding network, then a least-squares optimization problem for the optimal rigid body transformation between the aligned target and reference modal 3D point cloud maps is constructed:

[0086] ;

[0087] in, Let k be the k-th 3D point in the 3D point cloud of the target modality. The k-th 3D point in the 3D point cloud diagram of the reference modality.

[0088] Finally, the coarse and fine registration transformation matrices are fused using adaptive weight fusion to balance global consistency and local geometric accuracy. In step S3 of this embodiment, the function expression for fusing the coarse and fine registration transformation matrices to obtain the final registration transformation matrix is:

[0089] ;

[0090] in, This is the final registration transformation matrix. This is the coarse registration transformation matrix. For fine registration transformation matrix, This indicates element-wise multiplication. Let be the spatial adaptive weight matrix, where any nth element in the spatial adaptive weight matrix... OK Column elements And its calculation function expression is:

[0091] ;

[0092] ;

[0093] in, The alignment residual of the k-th 3D point in the 3D point cloud image after modal structured feature enhancement is used as a reference. To find the maximum alignment residual among all points in the 3D point cloud image after modal structured feature enhancement, For the k-th 3D point in the 3D point cloud map after target modal structured feature enhancement, Let be the rotation matrix of the rigid body transformation. For the k-th 3D point in the 3D point cloud map enhanced with reference modal structured features, Let be the translation vector of the rigid body transformation. The aforementioned adaptive weight fusion mechanism can be applied in the high-confidence region (spatial adaptive weight matrix). Approaching 1) more fine registration results are used, while in low confidence regions (spatial adaptive weight matrix) If the result approaches 0, then the coarse registration result is used conservatively. The final registration transformation matrix... It can achieve accurate imaging parameter recovery and pixel-level strict alignment of cross-modal images, providing accurate geometric correspondence for subsequent downstream tasks such as image fusion, target detection, and scene understanding.

[0094] To verify the effectiveness of the image registration method in this embodiment, several popular or recently proposed image feature matching methods were compared, covering various technical approaches such as detector-based sparse matching, Transformer-based semi-dense / dense matching, and lightweight matching methods. These included SuperPoint, Ripe, ASpanFormer, DKM, ELoFTR, XoFTR, XFeat, MINIMA, and RoMa. All experiments were performed on a personal computer equipped with an Intel Core i7-14700F CPU, 32 GB of RAM, and an NVIDIA GeForce RTX 3090 GPU (24 GB of VRAM). All models were trained for 50 epochs. In the feature extraction stage, ALIKED was used as the keypoint detection and descriptor extraction network, with a descriptor dimension of 128, a keypoint detection threshold of 0.2, and a maximum number of keypoints detected per image limited to 2048. The initial learning rate was set to 1×10⁻⁶. -4 The experiment employs an exponentially decaying learning rate strategy updated per epoch, where the learning rate remains constant for the first 20 epochs, then decays exponentially from the 20th epoch onwards. Three multimodal association datasets are used for validation: the DroneVehicle dataset, the MultiResSAR dataset, and the Landsat-8 dataset. The pixel-level association mean mapping error (MeanError) is used as the primary evaluation metric to measure the error in pixel-level association accuracy; a lower value indicates better association performance. The calculation formula is as follows:

[0095] ;

[0096] in,( , ) represents the true coordinates of the i-th pixel, , ) represents the predicted coordinates, and N is the total number of pixels involved in the calculation. Table 1 shows a comparison of the registration accuracy of the method in this embodiment and the representative method RoMa on three datasets. Table 2 shows the pixel-level association average mapping error of all considered image registration methods on the three datasets.

[0097]

[0098] As shown in Table 1, the image registration method of this embodiment (hereinafter referred to as "this method") achieves lower registration errors than the representative method on all datasets. On the DroneVehicle dataset, the average mapping error of the image registration method of this embodiment is 3.709 pixels, a decrease of 8.89% compared to the 4.071 pixels of the representative method. On the MultiResSAR dataset, the average mapping error of the image registration method of this embodiment is 8.357 pixels, a decrease of 7.97% compared to the 9.081 pixels of the representative method. On the Landsat-8 dataset, the image registration method of this embodiment achieves the greatest performance improvement, with the average mapping error decreasing from 8.632 pixels to 7.514 pixels, a reduction of 12.95%.

[0099]

[0100] Table 2 shows the pixel-level average mapping error of all considered image registration methods on the three datasets. It can be seen that the image registration method in this embodiment achieves the lowest registration error on all datasets. On the DroneVehicle dataset, the average mapping error of the image registration method in this embodiment is 3.709 pixels, which is better than the RoMa method (4.071 pixels) and the MINIMA method (4.668 pixels). On the MultiResSAR dataset, the average mapping error of the image registration method in this embodiment is 8.357 pixels, which is also better than the RoMa method (9.116 pixels) and the MINIMA method (14.346 pixels). On the Landsat-8 dataset, the average mapping error of the image registration method in this embodiment is 7.514 pixels, which is significantly better than the RoMa method (8.615 pixels) and the MINIMA method (16.122 pixels). Among all the compared methods, detector-based sparse matching methods such as SuperPoint and Ripe perform poorly in cross-modal scenes, mainly because traditional feature point detectors struggle to handle radiometric differences between images of different modalities. While Transformer-based methods such as ASpanFormer and ELoFTR can capture global contextual information, they still suffer from performance bottlenecks in cross-modal matching. The XoFTR method, specifically designed for cross-modal matching, outperforms traditional methods on the MultiResSAR and Landsat-8 datasets, but still falls short of the image registration method presented in this embodiment. The RoMa method, as one of the most advanced dense matching methods currently available, performs best among all compared methods; however, the image registration method in this embodiment further improves registration accuracy by combining common feature extraction and imaging parameter estimation.

[0101] In summary, this embodiment's image registration method, guided by common features and imaging parameters, integrates three key steps: cross-modal common feature extraction, depth-guided imaging parameter estimation, and geometric transformation optimization, achieving high-precision registration of multimodal images in complex scenes. This method first extracts cross-modal common features through a bidirectional interactive attention mechanism to achieve preliminary geometric alignment. Then, it elevates the two-dimensional registration problem to a three-dimensional structural alignment problem using a depth estimation network and a Transformer-based structural encoding network, obtaining accurate imaging geometric parameters. Finally, an adaptive weight fusion strategy significantly improves the overall registration accuracy.

[0102] Those skilled in the art will understand that the technical solutions provided by this invention can take the form of a method, a system, or a computer program product. For example, this invention can provide an image registration system guided by common features and imaging parameters, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the image registration method guided by common features and imaging parameters. This invention can provide a computer-readable storage medium storing a computer program or instructions programmed or configured to execute the image registration method guided by common features and imaging parameters via a processor. This invention can provide a computer program product including a computer program or instructions programmed or configured to execute the image registration method guided by common features and imaging parameters via a processor. Furthermore, this invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this invention can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0103] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. An image registration method based on joint guidance of common features and imaging parameters, characterized in that, The process includes the following steps: S1, extracting depth feature maps from the input images of the target modality and the reference modality using a feature extractor, projecting the obtained depth feature maps onto a unified structural representation space, establishing cross-modal feature associations using a bidirectional interactive attention mechanism, performing geometric alignment on the target modality image to obtain a coarse registration transformation matrix, and generating a coarse registration image of the target modality from the input image of the target modality using the coarse registration transformation matrix; S2, use a depth estimation network to predict dense depth maps for the coarse registration image of the target mode and the input image of the reference mode, respectively. Construct a 3D point cloud map using the dense depth map and imaging parameters. Calculate the fine registration transformation matrix for the 3D point cloud maps of the target mode and the reference mode. S3, fuse the coarse registration transformation matrix and the fine registration transformation matrix to obtain the final registration transformation matrix. Use the final registration transformation matrix to generate the final registered image of the target mode from the input image of the target mode.

2. The image registration method based on common features and imaging parameters jointly guided according to claim 1, characterized in that, Step S1 includes: S1.1, extract features from the input images of the target mode and the reference mode using feature extractors with independent parameters, and obtain depth feature maps of the target mode and the reference mode respectively; S1.2, the depth feature maps of the target mode and the reference mode are projected onto a unified structural representation space using their respective learnable projection matrices; S1.3, the projected target mode and reference mode depth feature maps are enhanced across modes using a bidirectional interactive attention mechanism to obtain the enhanced depth feature maps; S1.4, the cosine similarity of each pixel in the enhanced depth feature map of the target mode and the reference mode is calculated by the feature matcher, so as to obtain the coarse registration similarity matrix composed of the cosine similarity of the pixels; S1.5, Based on the coarse registration similarity matrix, the transformation matrix estimation algorithm is used to estimate the rotation matrix of the coarse registration. Translation vector And construct the coarse registration transformation matrix: ； in, This is the coarse registration transformation matrix; S1.6, using the coarse registration transformation matrix The target modality image is converted into a coarsely registered target modality image.

3. The image registration method based on common features and imaging parameters jointly guided according to claim 2, characterized in that, Step S1.3 involves performing cross-modal enhancement on the projected depth feature maps of the target modality and the reference modality using a bidirectional interactive attention mechanism. This includes: using the projected depth feature map of the target modality as a query and the projected depth feature map of the reference modality as a key-value pair, and obtaining the enhanced depth feature map of the target modality through the bidirectional interactive attention mechanism; and using the projected depth feature map of the reference modality as a query and the projected depth feature map of the target modality as a key-value pair, and obtaining the enhanced depth feature map of the reference modality through the bidirectional interactive attention mechanism.

4. The image registration method based on common features and imaging parameters jointly guided according to claim 1, characterized in that, In step S2, the function expression for constructing a 3D point cloud map from the dense depth map using imaging parameters is: ； in, This is a 3D point cloud map. For dense depth maps, For the camera intrinsic parameter matrix, For pixels, For pixels The coordinates.

5. The image registration method based on common features and imaging parameters jointly guided according to claim 1, characterized in that, Before calculating the fine registration transformation matrix for the 3D point cloud maps of the target and reference modalities in step S2, the following steps are also included: structural feature enhancement of the 3D point cloud maps of the target and reference modalities using a Transformer-based multi-scale visual structure coding network according to the following formula: ； in, This is a 3D point cloud image with enhanced structured features. This is a 3D point cloud map. For the learned geometric modulation weight matrix, Pixels extracted by a Transformer-based multi-scale visual structure encoding network The surrounding geometric context information, including local curvature and surface normals.

6. The image registration method based on common features and imaging parameters jointly guided according to claim 5, characterized in that, In step S2, when calculating the fine registration transformation matrix for the 3D point cloud maps of the target mode and the reference mode, the least squares optimization problem of constructing the optimal rigid body transformation between the 3D point cloud maps of the aligned target mode and the reference mode is included. ； in, Let be the rotation matrix of the rigid body transformation. Let be the translation vector of the rigid body transformation. For the k-th 3D point in the 3D point cloud map after target modal structured feature enhancement, For the k-th 3D point in the 3D point cloud map after modal structure enhancement, the rotation matrix of the rigid body transformation is obtained by fitting a least-squares optimization problem to each 3D point in the 3D point cloud maps of the target mode and the reference mode. Translation vector And construct the fine registration transformation matrix: ； in, This is the fine registration transformation matrix.

7. The image registration method based on common features and imaging parameters jointly guided according to claim 1, characterized in that, The function expression for fusing the coarse registration transformation matrix and the fine registration transformation matrix in step S3 to obtain the final registration transformation matrix is ​​as follows: ； in, This is the final registration transformation matrix. This is an element-wise multiplication operation. This is the coarse registration transformation matrix. For fine registration transformation matrix, Let be the spatial adaptive weight matrix, where any nth element in the spatial adaptive weight matrix... OK Column elements And its calculation function expression is: ； ； in, The alignment residual of the k-th 3D point in the 3D point cloud image after modal structured feature enhancement is used as a reference. To find the maximum alignment residual among all points in the 3D point cloud image after modal structured feature enhancement, For the k-th 3D point in the 3D point cloud map after target modal structured feature enhancement, Let be the rotation matrix of the rigid body transformation. For the k-th 3D point in the 3D point cloud map enhanced with reference modal structured features, Let be the translation vector for the rigid body transformation.

8. An image registration system based on common features and imaging parameters jointly guided, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to execute the image registration method based on common features and imaging parameters as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the image registration method based on common features and imaging parameters jointly guided by any one of claims 1 to 7.

10. A computer program product, comprising a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the image registration method based on common features and imaging parameters jointly guided by any one of claims 1 to 7.

Images