Multimodal image registration method, device, electronic device and storage medium

Abstract

The present invention provides a multimodal image registration method, apparatus, electronic device, and storage medium. The method includes: performing multi-scale edge window filtering on a first image to be registered and a second image to be registered, respectively, to obtain multi-scale edge window filtered result images of the first image to be registered and the second image to be registered; based on the edge window filtered result images, obtaining a first feature point and a feature vector corresponding to the first feature point, as well as a feature vector corresponding to the second feature point; based on the feature vectors, performing feature matching twice on the first feature point and the second feature point to obtain a first pair of feature points with the same name and a second pair of feature points with the same name; and registering the first image to be registered and the second image to be registered based on the first pair of feature points with the same name and the second pair of feature points with the same name. The present invention provides a multimodal image registration method, apparatus, electronic device, and storage medium, which improve the accuracy and efficiency of multimodal image registration and enhance the robustness of multimodal image registration.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multimodal image registration method, device, electronic device and storage medium. Background Art

[0002] Multimodal imagery refers to multiple types of images of the same area or target acquired using different sensors, imaging mechanisms, or wavelengths. Compared to single-modal imagery, multimodal imagery provides more information and has broad application value in areas such as addressing climate change, natural disaster assessment, urban planning, and other global challenges.

[0003] Multimodal image registration is the process of identifying and aligning points between two or more images of different modalities. Multimodal image registration bridges multimodal images, enabling alignment. It is crucial for a variety of remote sensing applications, including image fusion, change detection, image stitching, and 3D reconstruction.

[0004] Traditional multimodal image registration methods in the related art typically include region-based methods, feature-based methods, and deep learning-based methods. However, these traditional multimodal image registration methods are limited by various factors, resulting in low accuracy and efficiency. Therefore, how to perform multimodal image registration more accurately and efficiently is a technical problem that needs to be solved in this field. Summary of the Invention

[0005] The present invention provides a multimodal image registration method, device, electronic device and storage medium to address the defects of low accuracy and efficiency in multimodal image registration in the prior art, and to achieve more accurate and efficient multimodal image registration.

[0006] The present invention provides a multimodal image registration method, comprising the following steps.

[0007] The first image to be registered and the second image to be registered are respectively k The edge window filtering of the first image to be registered is performed to obtain the first k The edge window filtering result image of the scale and the second image to be registered k The first image to be registered and the second image to be registered are images of the same region or the same target but in different modalities. , S represents the number of scales, S is a positive integer greater than 1. When the scale is different, the variance and number of filtering times of the edge window filtering and the radius of the edge window filtering window are different;

[0008] Based on the first image to be registered k The edge window filtering result image of the scale and the second image to be registered k Scaled edge window filtering result image, obtaining first feature points corresponding to the first image to be registered and feature vectors corresponding to the first feature points, as well as second feature points corresponding to the second image to be registered and feature vectors corresponding to the second feature points;

[0009] performing feature matching on the first feature point and the second feature point based on a feature vector corresponding to the first feature point and a feature vector corresponding to the second feature point, and determining the first feature point and the second feature point that are successfully matched as a first feature point pair with the same name;

[0010] Based on the first pair of feature points with the same name, obtaining a first affine transformation matrix between the first image to be registered and the second image to be registered; based on the first affine transformation matrix, performing feature matching on the first feature points and the second feature points again to obtain a second pair of feature points with the same name;

[0011] The first image to be registered and the second image to be registered are registered based on the first pair of feature points with the same name and the second pair of feature points with the same name.

[0012] The present invention also provides a multimodal image registration device, comprising the following modules:

[0013] The edge window filtering module is used to perform the first image to be registered and the second image to be registered. k The edge window filtering of the first image to be registered is performed to obtain the first k The edge window filtering result image of the scale and the second image to be registered k The first image to be registered and the second image to be registered are images of the same region or the same target but in different modalities. , S represents the number of scales, S is a positive integer greater than 1. When the scale is different, the variance of the edge window filtering, the number of filtering times and the radius of the edge window filtering window are different;

[0014] A feature extraction module is used to extract the first image to be registered based on the k The edge window filtering result image of the scale and the second image to be registered k Scaled edge window filtering result image, obtaining first feature points corresponding to the first image to be registered and feature vectors corresponding to the first feature points, as well as second feature points corresponding to the second image to be registered and feature vectors corresponding to the second feature points;

[0015] a first matching module, configured to perform feature matching on the first feature point and the second feature point based on a feature vector corresponding to the first feature point and a feature vector corresponding to the second feature point, and determine the first feature point and the second feature point that are successfully matched as a first feature point pair with the same name;

[0016] a second matching module, configured to obtain a first affine transformation matrix between the first image to be registered and the second image to be registered based on the first pair of feature points of the same name, and perform feature matching on the first feature points and the second feature points again based on the first affine transformation matrix to obtain a second pair of feature points of the same name;

[0017] An image registration module is configured to register the first image to be registered and the second image to be registered based on the first pair of feature points with the same name and the second pair of feature points with the same name.

[0018] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multimodal image registration method as described above is implemented.

[0019] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the multimodal image registration methods described above.

[0020] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the multimodal image registration methods described above.

[0021] The multimodal image registration method, device, electronic device and storage medium provided by the present invention respectively perform multi-scale edge window filtering on the first image to be registered and the second image to be registered, obtain the multi-scale edge window filtering result image of the first image to be registered and the multi-scale edge window filtering result image of the second image to be registered, and then obtain the first feature point corresponding to the first image to be registered and the feature vector corresponding to the first feature point and the second feature point corresponding to the second image to be registered based on the edge window filtering result image, and obtain the first feature point corresponding to the first image to be registered and the feature vector corresponding to the second feature point based on the first feature point and the feature vector corresponding to the first feature point. As well as the second feature point and the feature vector corresponding to the above-mentioned second feature point, the first feature point and the second feature point are feature matched twice to obtain the first feature point pair with the same name and the second feature point pair with the same name, and then based on the above-mentioned first feature point pair with the same name and the second feature point pair with the same name, the first image to be registered and the second image to be registered are registered. Through multi-scale side window filtering, the image edge features are effectively retained, and the robustness of the multimodal image registration is enhanced. Through a two-stage matching strategy, the accuracy and efficiency of the multimodal image registration are improved, providing an effective solution for the automated processing and analysis of remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 This is one of the flow charts of the multimodal image registration method provided by the present invention.

[0024] Figure 2 This is the second flowchart of the multimodal image registration method provided by the present invention.

[0025] Figure 3 This is a schematic diagram of the position of the side window filter window in the multimodal image registration method provided by the present invention.

[0026] Figure 4 The first image to be registered in the multimodal image registration method provided by the present invention is k Schematic diagram of the second-order gradient phase map at different scales.

[0027] Figure 5 This is the first step in the multimodal image registration method provided by the present invention. k The first characteristic point of the scale The corresponding gradient direction histogram.

[0028] Figure 6 This is the first step in the multimodal image registration method provided by the present invention. k Schematic diagram of the feature area corresponding to the first feature point of the scale.

[0029] Figure 7 This is an excerpt of some data sets in the multimodal image registration method provided by the present invention.

[0030] Figure 8 This is an excerpt of the matching results of the multimodal image registration method provided by the present invention on 11 types of multimodal datasets.

[0031] Figure 9 This is an excerpt of the checkerboard-style registration results of the multimodal image registration method provided by the present invention.

[0032] Figure 10 It is a structural schematic diagram of the multimodal image registration device provided by the present invention.

[0033] Figure 11 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0034] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0035] In the description of the invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood broadly. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediary; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0036] In the description of this application, the terms "first", "second", etc. are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of the same type, and do not limit the number of objects. For example, the first object can be one or more. In addition, in the description of this application, "and / or" represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0037] It's important to note that, with the rapid development of multi-sensor information processing technology, satellites equipped with various sensors are providing image data of the Earth's surface with unprecedented frequency and diversity. Compared to single-mode observations, multimodal imagery provides more information. Leveraging these complementary data types, such as optical, synthetic aperture radar, lidar, infrared imagery, vision, and medicine, has broad application value in addressing climate change, natural disaster assessment, urban planning, medical diagnosis, and other global challenges.

[0038] Multimodal image registration and multimodal image registration technology are bridges connecting multimodal data. These multimodal images are usually acquired by different sensors and different imaging mechanisms at different times when observing the same ground object, such as optical-infrared, optical-hyperspectral, spectral-SAR, push-broom thermal infrared-swing-broom thermal infrared, visible light-infrared, etc., so that the obtained images can be accurately aligned. This process is crucial for various remote sensing applications such as image fusion, change detection, image stitching, and three-dimensional reconstruction.

[0039] The main challenges facing multimodal image registration include spatial reference discrepancies and nonlinear radiometric discrepancies. Spatial reference discrepancies are primarily due to different viewpoints and sensor resolutions, resulting in translation, rotation, scale changes, and geometric distortion between images. Nonlinear radiometric discrepancies, on the other hand, are caused by different imaging mechanisms and external environmental conditions, resulting in large grayscale differences between images. These differences increase the difficulty of registration, and traditional methods often fail when processing multimodal images with severe nonlinear radiometric and geometric discrepancies. Improving the accuracy and efficiency of multimodal image registration has become an urgent need.

[0040] Traditional multimodal image registration methods in related technologies generally include region-based methods, feature-based methods, and deep learning-based methods.

[0041] However, the region-based registration method has high computational cost and poor technical accuracy, and requires obtaining the geographic information of the image to eliminate the scale, rotation and translation differences between the multimodal images to be registered in advance, which makes it less applicable.

[0042] Feature-based methods need to deal with both scale and rotation differences. Common registration methods in related technologies, such as the Scale-Invariant Feature Transform (SIFT) method, do not extract enough feature points when processing multimodal images with severe geometric differences and nonlinear radiation differences. The feature description is not robust, making it difficult to accurately reflect the shared information between the two multimodal images, resulting in poor accuracy in multimodal image registration.

[0043] Although deep learning-based methods can improve the accuracy of multimodal image registration to a certain extent, deep learning-based registration methods require the collection of a large number of multimodal images as sample images for model training in advance. Before model training, a large amount of manpower and time costs are required to process and annotate the sample images, resulting in low efficiency of multimodal image registration. In the case of insufficient number of sample images, the accuracy of multimodal image registration is not high. In addition, deep learning-based methods are highly sensitive to rotational differences between images of different modalities, which will further reduce the accuracy of multimodal image registration based on deep learning methods.

[0044] To this end, the present invention provides a multimodal image registration method. The multimodal image provided by the present invention can improve the registration accuracy and efficiency of multimodal images when the multimodal images to be registered have complex geometric differences and nonlinear radiation differences. When faced with multimodal images to be registered with scale differences and rotation differences, it can still match a sufficient number of same-name points while maintaining a high degree of accuracy. During the multimodal image registration process, feature point extraction is effectively performed on images in a multi-scale edge window Gaussian scale space, and a robust descriptor is constructed to extract feature vectors for two matchings, which can increase the number of same-name points while ensuring the accuracy of the registration.

[0045] The following combination Figures 1-10 The multimodal image registration method of the present invention is described.

[0046] Figure 1 This is one of the flow charts of the multimodal image registration method provided by the present invention. Figure 2 This is the second flow chart of the multimodal image registration method provided by the present invention. Figure 1 and Figure 2 As shown, the method includes the following steps: Step 101, performing the first registration image and the second registration image on the first image. k Scaled edge window filtering is used to obtain the first image to be registered. k The edge window filtering result image and the second image to be registered kThe first image to be registered and the second image to be registered are images of the same region or the same target in different modalities. , S represents the number of scales, S It is a positive integer greater than 1. Different scales result in different variances, filtering times, and radius of the edge window filter window.

[0047] It should be noted that the embodiment of the present invention is implemented by a multimodal image registration device, which can be configured in electronic devices such as computers and servers.

[0048] Specifically, the first image to be registered and the second image to be registered are registration objects of the multimodal image registration method provided by the present invention. Based on the multimodal image registration method provided by the present invention, the first image to be registered and the second image to be registered can be registered.

[0049] It is understood that the first image to be registered and the second image to be registered in the embodiments of the present invention can be images of the same region or the same target in different modalities acquired using different sensors, different imaging mechanisms, or different wavelengths. The first image to be registered and the second image to be registered in the embodiments of the present invention can be determined based on actual needs and are not limited in the embodiments of the present invention.

[0050] In an embodiment of the present invention, the first image to be registered and the second image to be registered can be obtained in a variety of ways. For example, in an embodiment of the present invention, the first image to be registered and the second image to be registered can be obtained based on user input; or, in an embodiment of the present invention, the first image to be registered and the second image to be registered can also be received from other electronic devices.

[0051] As an optional embodiment, the first image to be registered and the second image to be registered are respectively k Scaled edge window filtering is used to obtain the first image to be registered. k The edge window filtering result image and the second image to be registered k After obtaining the scaled edge window filtering result image, the method further includes: performing image preprocessing on the first image to be registered and the second image to be registered, wherein the image preprocessing includes at least one of normalization, contrast stretching and Gaussian filtering.

[0052] Specifically, in an embodiment of the present invention, in order to improve the image quality of the first image to be registered and the second image to be registered, reduce noise interference, and thereby improve the accuracy of image registration between the first image to be registered and the second image to be registered, before performing multi-scale edge window filtering on the first image to be registered and the second image to be registered, the first image to be registered and the second image to be registered are preprocessed using normalization, contrast stretching, and Gaussian filtering to obtain the first image to be registered and the second image to be registered after image preprocessing.

[0053] It should be noted that the edge area in the image contains rich visual information. Effectively obtaining the edge features in the image can better complete the registration task in the presence of nonlinear radiation differences. However, whether using isotropic or anisotropic filters, although they can smooth and blur the image to a certain extent, they are not conducive to simultaneously satisfying the retention of image edge features and the removal of image noise. Therefore, it is of great significance to effectively retain the edge feature information of the image and construct a new scale space that can resist nonlinear radiation differences. With this as motivation, the present invention constructs a multi-scale edge window Gaussian scale space.

[0054] Side Window Filtering (SWF) is a filtering technique used in image processing. It aims to minimize blurring effects on image edges while preserving or enhancing edge features. Traditional filtering algorithms (such as Gaussian filtering and mean filtering) typically use a full-window regression approach, where the center of the filter kernel is placed at the pixel to be filtered. When processing image edges, the filter window weights the pixel values ​​on both sides of the edge as it crosses the edge, causing edge information to diffuse along the edge normal, resulting in edge blur.

[0055] Edge window filtering treats each pixel to be filtered as a potential edge and generates multiple local windows around it, with the pixel to be filtered located on one side or aligned with a corner of the window. Based on the filtering results of the neighboring pixels in the side window of the pixel to be filtered, the side window with the smallest L2 distance to the pixel to be filtered is selected as the output. L2 distance is the Euclidean distance, a method for measuring the straight-line distance between two points or two vectors in multidimensional space.

[0056] Therefore, in the embodiment of the present invention, the first image to be registered and the second image to be registered can be subjected to multiple edge window filters of different scales based on edge window filters of different scales to obtain the edge window of the first image to be registered. k The edge window filtering result image and the second image to be registered k Scaled edge window filtering result image.

[0057] As an optional embodiment, the first image to be registered and the second image to be registered are respectively k Scaled edge window filtering is used to obtain the first image to be registered. k The edge window filtering result image and the second image to be registered k The edge window filter result image of the scale includes: initializing the edge window filter parameters, which include the initial variance, the number of scales, the scale factor, the position of each edge window filter window, and the k The radius of the side window filter window, the number of filters and the variance corresponding to the scale, each side window filter window includes a rectangular side window filter window with the long side in the vertical direction and the pixel to be filtered located at the midpoint of the right long side, a rectangular side window filter window with the long side in the vertical direction and the pixel to be filtered located at the midpoint of the left long side, a rectangular side window filter window with the long side in the horizontal direction and the pixel to be filtered located at the midpoint of the upper long side, a rectangular side window filter window with the long side in the horizontal direction and the pixel to be filtered located at the midpoint of the lower long side, and a rectangular side window filter window with the four sides in the horizontal and vertical directions and the pixel to be filtered located at the four sides respectively. A square edge window filtering window with corner points, an isosceles right triangle edge window filtering window with right angles located in the horizontal and vertical directions respectively and the pixel to be filtered is located at the midpoint of the lower right hypotenuse, an isosceles right triangle edge window filtering window with right angles located in the horizontal and vertical directions respectively and the pixel to be filtered is located at the midpoint of the lower left hypotenuse, an isosceles right triangle edge window filtering window with right angles located in the horizontal and vertical directions respectively and the pixel to be filtered is located at the midpoint of the upper right hypotenuse, and an isosceles right triangle edge window filtering window with right angles located in the horizontal and vertical directions respectively and the pixel to be filtered is located at the midpoint of the upper left hypotenuse.

[0058] Specifically, considering that the Gaussian kernel and its derivatives are the only possible smoothing kernels in scale space, embodiments of the present invention use edge window filtering to blur the first and second images to be registered while preserving high-quality edge information. The present invention defines 12 edge window filtering windows in a discrete case to maximize the emphasis on edge features in different directions.

[0059] Figure 3 This is a schematic diagram of the position of the side window filter window in the multimodal image registration method provided by the present invention. Figure 3 In the figure, the blue frame represents the edge window filter window, and the red dots represent the pixels to be filtered. Figure 3As shown, the side window filter window W is a rectangular side window filter window with its long side in the vertical direction and the pixel to be filtered located at the midpoint of the right long side; the side window filter window E is a rectangular side window filter window with its long side in the vertical direction and the pixel to be filtered located at the midpoint of the left long side; the side window filter window S is a rectangular side window filter window with its long side in the horizontal direction and the pixel to be filtered located at the midpoint of the upper long side; the side window filter window N is a rectangular side window filter window with its long side in the horizontal direction and the pixel to be filtered located at the midpoint of the lower long side; the side window filter window NW is a square side window filter window with its four sides in the horizontal and vertical directions and the pixel to be filtered located at the lower right corner; the side window filter window NE is a square side window filter window with its four sides in the horizontal and vertical directions and the pixel to be filtered located at the lower left corner; the side window filter window SW is a square side window filter window with its four sides in the horizontal and vertical directions and the pixel to be filtered located at the lower left corner The filter window DSW is an isosceles right triangle edge window filter window with its right angles in the horizontal and vertical directions and the pixel to be filtered is located at the midpoint of the upper right hypotenuse; the filter window DSE is an isosceles right triangle edge window filter window with its right angles in the horizontal and vertical directions and the pixel to be filtered is located at the midpoint of the upper left hypotenuse; the filter window DSE is an isosceles right triangle edge window filter window with its right angles in the horizontal and vertical directions and the pixel to be filtered is located at the midpoint of the upper left hypotenuse.

[0060] It should be noted that the initial variance, scale number, scale factor and the first k The radius of the edge window filter window corresponding to the scale can be determined based on prior knowledge and / or actual conditions. k The specific value of the edge window filter window radius corresponding to the scale is not limited.

[0061] Optionally, in the embodiment of the present invention, the initial variance of the edge window filter is determined based on prior knowledge and / or actual conditions. is 1.6; the scaling factor is , scale number There are 4 scales, namely the first scale, the second scale, the third scale and the fourth scale; the edge window filter window radius corresponding to the first scale is 3, and the variance corresponding to the first scale is (initial variance), the number of filtering times is 1; the radius of the side window filter window corresponding to the second scale is 5, and the variance corresponding to the second scale is , the number of filtering times is 2; the radius of the edge window filter window corresponding to the third scale is 8, and the variance is , the number of filtering times is 3; the radius of the edge window filter window corresponding to the fourth scale is 11, and the variance is , the number of filtering times is 4. Variance corresponding to scale It can be calculated by the following formula:

[0062]

[0063] It should be noted that, for a square side window filter window, the side window filter window radius refers to the side length of the square side window filter window; for a rectangular side window filter window, the side window filter window radius refers to the length of the short side of the rectangular side window filter window; for an isosceles right triangle side window filter window, the side window filter window radius refers to half of the right angle side of the isosceles right triangle side window filter window.

[0064] Based on the filtering parameters, the first image to be registered and the second image to be registered are respectively k Scaled edge window filtering is used to obtain the first image to be registered. k The edge window filtering result image and the second image to be registered k Scaled edge window filtering result image.

[0065] Each pixel in the image to be registered is taken as a pixel to be filtered. For each pixel to be filtered in the image to be registered, the first k The Gaussian kernels in the horizontal, vertical and diagonal directions of the edge window filtering corresponding to the scale are used, and then the edge window filtering in different directions (including horizontal, vertical and diagonal directions) is performed on each pixel to be filtered to obtain the first k The edge window filtering result of each pixel to be filtered is k The edge window filtering result of the scale is used to obtain the image to be registered. k The images to be registered include a first image to be registered and a second image to be registered.

[0066] The formula for edge window filtering can be defined as:

[0067]

[0068] in, Used to indicate the position of the side window filter window; Represents the pixel to be filtered Centered on Determined edge filter window; Represents pixels; represents the Gaussian kernel weight, which needs to be normalized in each edge window filter window; Indicates that the image to be registered is at position Pixel value of Indicates that the image to be registered is at position The pixel value of .

[0069] Step 102: Based on the first image to be registered k The edge window filtering result image of the scale and the second image to be registered k The edge window filtering result image of the scale is obtained, and the first feature point corresponding to the first image to be registered and the feature vector corresponding to the first feature point are obtained, as well as the second feature point corresponding to the second image to be registered and the feature vector corresponding to the second feature point are obtained.

[0070] Specifically, obtain the first image to be registered k The edge window filtering result image of the scale and the second image to be registered k After filtering the resulting image with the edge window of the scale, the first image to be registered can be k The edge window filtering result image of the scale is obtained by numerical calculation, mathematical statistics and deep learning technology to obtain the first image to be registered corresponding to the second image. k The first characteristic point and the k The feature vector corresponding to the first feature point of the scale can also be based on the first feature point of the second image to be registered. k The edge window filtering result image of the scale is obtained by numerical calculation, mathematical statistics and deep learning technology to obtain the first image corresponding to the second image to be registered. k The second characteristic point and the k The eigenvector corresponding to the second feature point of the scale.

[0071] As an optional embodiment, based on the first image to be registered k The edge window filtering result image of the scale and the second image to be registered k The edge window filtering result image of the scale is obtained, and the first feature point and the feature vector corresponding to the first feature point corresponding to the first image to be registered are obtained, as well as the second feature point and the feature vector corresponding to the second feature point corresponding to the second image to be registered are obtained, including: k The edge window filtering result image of the scale and the second image to be registered k The edge of the image with the edge window filter is extracted to obtain the first image to be registered. k The edge feature map of the second image to be registered k Edge feature map of scale.

[0072] It should be noted that the Sobel algorithm is a gradient-based edge detection algorithm that detects edges by calculating the gradient magnitude and direction of each pixel in the image. The Sobel operator contains two 3x3 matrices, one horizontal (Gx) and one vertical (Gy), which are used to calculate the approximate brightness difference of the image in the horizontal and vertical directions. The Canny algorithm is a multi-stage edge detection algorithm that combines Gaussian smoothing, gradient calculation, non-maximum suppression, dual threshold processing, and edge connection to obtain accurate edge detection results. Edge extraction based on the Canny algorithm can obtain finer edges and obtain more feature points, while edge extraction based on the Canny algorithm can detect the edges with the most significant features.

[0073] Therefore, for each scale feature map corresponding to the first image to be registered and the second image to be registered k In the embodiment of the present invention, the Sobel algorithm or the Canny algorithm can be used to extract the edge of the feature map based on the actual situation to obtain the first image to be registered. k The edge feature map of the second image to be registered k Edge feature map of scale.

[0074] Optionally, in the embodiment of the present invention, the Sobel algorithm is used to extract the edge of the feature map to obtain the first image to be registered. k The edge feature map of the second image to be registered k Edge feature map of scale.

[0075] The Sobel operator is defined as follows:

[0076]

[0077] in, Represents the x-direction component of the Sobel operator; Represents the y-direction component of the Sobel operator.

[0078] Use the above feature map and Convolution can obtain the gradient maps of the above feature maps in the x and y directions respectively, and then obtain the gradient amplitude map b corresponding to the above feature maps.

[0079] Setting the edge threshold ,in represents the average value, Set it to 0.4. The gradient amplitude map b corresponding to the above feature map is greater than the edge threshold The pixels of the first image to be registered are set to 1 and the rest are set to 0. Then non-maximum suppression is performed to retain the local maximum value to obtain finer and clearer edges, thereby obtaining the first image to be registered. k The scaled binary edge feature map and the second image to be registered k Scaled binary edge feature map.

[0080] For the first image to be registered k The edge feature map of the second image to be registered k The edge feature maps of different scales are used to detect corner points respectively, and the first image to be registered corresponding to the first image is obtained. k The first original feature point of the second image to be registered corresponds to the first k The second original feature point of the scale.

[0081] It should be noted that Harris corners are points in an image with dramatic brightness changes or points with maximum curvature on an edge curve. They are one of the most important features of an image. The core idea of ​​the Harris corner detection algorithm is to detect corners by calculating local grayscale changes in the image. The main steps of the algorithm include: Calculating pixel value changes: When a window moves across the image, the change in pixel values ​​within the window is calculated. This is usually achieved by calculating the gradient of the pixels within the window. Constructing a covariance matrix: The covariance matrix is ​​constructed using the gradient information of the pixels within the window. This matrix reflects the second-order statistical characteristics of the grayscale changes of the pixels within the window. Calculating corner response values: The corner response value R is calculated based on the eigenvalues ​​of the covariance matrix. When R is greater than a set threshold, the point is considered a corner. The size and distribution of the eigenvalues ​​determine whether the point is a corner, edge point, or flat area point.

[0082] Harris corner points are less affected by intensity and scale differences and have high computational efficiency. Harris corner points with strong responses can be considered as feature points with obvious structure and stability between multi-source images. Therefore, in the embodiment of the present invention, the first image to be registered is obtained. k The scaled binary edge feature map and the second image to be registered k After the edge feature map is scaled and binarized, the first image to be registered can be detected based on the Harris corner detection algorithm. k The scaled binary edge feature map and the second image to be registered k The scaled binary edge feature map is used for corner detection, and the first image to be registered is k The Harris corner points detected in the scaled binary edge feature map are determined as the first corner points corresponding to the first image to be registered. k The first original feature point of the scale is determined as the Harris corner point detected in the edge feature map of each scale binarization corresponding to the second image to be registered.k The second original feature point of the scale.

[0083] It can be understood that the first image to be registered corresponds to the k The number of the first original feature points of the scale is multiple. k The scale is the number of original feature points.

[0084] Edge feature map The Harris corner response value R of any pixel in can be defined by the following formula:

[0085]

[0086]

[0087] in, and denote the determinant and trace of the matrix respectively; represents the empirical constant; and Respectively represent the edge feature map along x and y Directional gradient; u and v Respectively represent the coordinate values ​​of the above pixel points; The variance is When the Harris corner response value R of any pixel is greater than the set threshold, the pixel is considered to be a Harris corner.

[0088] Based on the non-maximum suppression algorithm, the first image to be registered corresponds to the k The first original feature point of the scale is screened to obtain the first image to be registered corresponding to the first original feature point. k The first feature point of the scale, the first feature point of the second image to be registered k The second original feature points of the scale are screened to obtain the first feature points corresponding to the second image to be registered k The second characteristic point of the scale.

[0089] It should be noted that an important difficulty in multimodal image registration is the diversity of the size and scale relationships between the images to be registered. Therefore, the multimodal image registration method provided by the present invention uses a non-maximum suppression algorithm in the feature point detection stage to detect the first image to be registered. k The first original feature point of the second image to be registered corresponds to the first k The second original feature points of the scale are screened and the first original feature points corresponding to the first image to be registered are eliminated. k The first original feature point of the second image to be registered corresponds to the first kAfter the redundant feature points and noise points are gathered from the second original feature points of the scale, the remaining first image to be registered corresponds to the k The first original feature point of the scale is determined as the first original feature point corresponding to the first image to be registered. k The first feature point of the scale is the first point of the remaining second image to be registered. k The second original feature point of the scale is determined as the first feature point corresponding to the second image to be registered k The second characteristic point of the scale.

[0090] The embodiment of the present invention uses a square cover suppression method to achieve fast and uniform feature point distribution by covering a square area on the original feature points. The core of the method is to use binary search to determine the size of the coverage area and suppress non-maximum points by covering these areas. The original feature points include the first image to be registered corresponding to the first k The first original feature point of the second image to be registered corresponds to the first k The second original feature point of the scale.

[0091] It should be noted that, in the embodiment of the present invention, Indicates the first image to be registered corresponding to the k The first original feature point set of scale is Indicates the first image to be registered corresponding to k The second original feature point set of scale.

[0092] Based on the non-maximum suppression algorithm, the first image to be registered corresponds to the k The first original feature point set of the scale The specific steps of screening include: k The first original feature point set of the scale The pixel values ​​of each first original feature point in the image are ordered from large to small, and the first image to be registered corresponds to the first k The first original feature point set of the scale Sort the first original feature points in .

[0093] In order to accelerate the convergence of the algorithm, according to the size of the first image to be registered, the first image to be registered corresponding to the k The first original feature point set of the scale The number of the first original feature points, the number of the first feature points expected to be output, and the tolerance can be used to pre-calculate the upper and lower boundaries of the binary search using the following formula: 、 :

[0094]

[0095]

[0096]

[0097] in, and Respectively represent the width and height of the first image to be registered; Indicates the first image to be registered that is expected to be output k The number of first characteristic points of the scale; Indicates the first image to be registered corresponding to the k The first original feature point set of the scale The number of the first original feature points in .

[0098] At each step of the binary search, set the grid The resolution is ,in Represents the current suppressed edge length. Then, according to the first image to be registered corresponding to the k The first original feature point set of the scale The arrangement order of the first original feature points in the first image to be registered is traversed. k The first original feature point set of the scale For each original feature point in the first image to be registered, k The first original feature point set of the scale Any first original feature point in , check the first original feature point The grid Is it marked as covered? If the first original feature point The grid If it is not marked as covered, the first original feature point Add to the first image to be registered k The first original feature point set of the scale , and the first original feature point is the center and the side length is The grid in the square area Mark as covered, Take 1, 2, 3, ..., .

[0099] Change Repeat the above process until the first image to be registered corresponds to the k The first original feature point set of the scale The number of the first feature points in Or meet other termination conditions, and finally obtain the uniformly distributed first image to be registered corresponding to the firstk The first feature point set of the scale .

[0100] It should be noted that based on the non-maximum suppression algorithm, the first k The second original feature point set of scale Screening is performed to obtain the first k The first original feature point set of the scale The specific steps are the same as the first image to be registered k The first original feature point set of the scale The specific steps for screening are the same and will not be repeated in the embodiments of the present invention.

[0101] Based on the k The first feature point of the scale and the first image to be registered k The edge window filtering result image of the scale is obtained k The eigenvector corresponding to the first feature point of the scale is based on the k The second feature point of the second image to be registered k The edge window filtering result image of the scale is obtained k The eigenvector corresponding to the second feature point of the scale.

[0102] Specifically, obtain the first image to be registered corresponding to the k The first feature point of the second image to be registered corresponds to the first feature point of the k After the second feature point of the scale, k The first feature point of the scale and the first image to be registered k The edge window filtering result image of the scale is obtained through numerical calculation, mathematical statistics and deep learning technology. k The feature vector corresponding to the first feature point of the scale can also be based on the feature vector corresponding to the second image to be registered. k The second feature point of the second image to be registered k The edge window filtering result image of the scale is obtained through numerical calculation, mathematical statistics and deep learning technology. k The eigenvector corresponding to the second feature point of the scale.

[0103] As an optional embodiment, based on k The first feature point of the scale and the first image to be registered k The edge window filtering result image of the scale is obtained k The eigenvector corresponding to the first feature point of the scale is based on the k The second feature point of the second image to be registered k The edge window filtering result image of the scale is obtainedk The feature vector corresponding to the second feature point of the first image to be registered is: k The edge window filtering result image and the second image to be registered k The edge window filtering result image of the scale is j Directional controllable Gaussian filtering is used to obtain the first image to be registered. k Scale j The controllable Gaussian filter result image and the second image to be registered k Scale j The result image of the controllable Gaussian filter in the direction, , Indicates the number of directions, is a positive integer greater than 1. The rotation angle of the Gaussian filter window for Gaussian filtering is different for different directions.

[0104] It should be noted that, in traditional Gaussian filtering, the Gaussian filter window is usually square, and the four sides of the square Gaussian filter window are respectively located in the horizontal and vertical directions.

[0105] However, in order to capture feature information in more directions in the feature map, the embodiment of the present invention adjusts the rotation angle of the Gaussian filter window to achieve a multi-directional steerable Gaussian filter, so as to obtain feature information in different directions in the feature map.

[0106] In the embodiment of the present invention, the rotation angle of the Gaussian filter window may include but is not limited to 0°, 30°, 60°, 90°, 120° and 150°.

[0107] Optionally, in the embodiment of the present invention, the feature map of each scale corresponding to the first image to be registered can be subjected to controllable Gaussian filtering in six directions (i.e. ), are the first direction, the second direction, the third direction, the fourth direction, the fifth direction and the sixth direction respectively; the rotation angle of the Gaussian filter window corresponding to the first direction is 0°, the rotation angle of the Gaussian filter window corresponding to the second direction The rotation angle of the Gaussian filter window corresponding to the third direction is 30° The rotation angle of the Gaussian filter window corresponding to the fourth direction is 60° The rotation angle of the Gaussian filter window corresponding to the 5th direction is 90° The rotation angle of the Gaussian filter window corresponding to the 6th direction is 120° is 150°.

[0108] The Gaussian function and its higher-order partial derivatives are proved to be directionally controllable, with a variance of Gaussian function And its first-order Gaussian filter kernel at 0° and 90° and They can be expressed as:

[0109]

[0110]

[0111] in, and They represent the coordinates of the pixels to be filtered.

[0112] For the first image to be registered Scaled edge window filtering result image , for the first image to be registered Scaled edge window filtering result image Perform Gaussian filtering in the first direction to obtain the first image to be registered. Scaled edge window filtering result image Gaussian filtering result in the first direction and It can be calculated by the following formula

[0113]

[0114]

[0115] in, Indicates the Scaled edge window filtering result image; Represents convolution calculation; Indicates the Gaussian filter variance corresponding to the first direction.

[0116] Through linear combination, the first image to be registered can be obtained Scale Directional controllable Gaussian filtering results :

[0117]

[0118]

[0119] in, Indicates the number of scales, Indicates the number of directions of Gaussian filtering; represents the scale factor; Indicates the The rotation angle of the Gaussian filter window corresponding to the direction. Set to 6, Set to 0.8.

[0120] Based on the first image to be registered Scale Directional controllable Gaussian filtering results , the first image to be registered can be obtained k Scale j Directionally controllable Gaussian filtering result image

[0121] The first image to be registered k The controllable Gaussian filtering result images in all directions of the scale are superimposed to obtain the first image to be registered. k The first-order gradient map of the scale is used to convert the second image to be registered into k The controllable Gaussian filtering result images in all directions of the scale are superimposed to obtain the second image to be registered. k Scaled first-order gradient map.

[0122] Specifically, obtain the first image to be registered k Scale j After the controllable Gaussian filtering result image of the direction, the first image to be registered can be calculated by the following formula: The controllable Gaussian filtering result images in all directions of the scale are superimposed to obtain the first image to be registered. Scaled first-order gradient map .

[0123]

[0124] Based on the first image to be registered k The first-order gradient map of the scale is obtained to obtain the first image to be registered. k The second-order gradient magnitude map and second-order gradient phase map of the scale are based on the second image to be registered. k The first-order gradient map of the scale is obtained to obtain the second image to be registered. k Scaled second-order gradient magnitude map and second-order gradient phase map.

[0125] Specifically, obtain the first image to be registered k After the first-order gradient map of the scale is obtained, the first image to be registered can be used to k The first-order gradient map of the scale is obtained by Sobel operator to obtain the first image to be registered. k Scaled second-order gradient magnitude map and second-order gradient phase map.

[0126] It should be noted that, considering that multimodal images often have the phenomenon of gradient reversal, in the embodiment of the present invention, the phase of the second-order gradient phase image corresponding to the first image to be registered is unified at Inside.

[0127] Figure 4 The first image to be registered in the multimodal image registration method provided by the present invention is k Schematic diagram of the second-order gradient phase map of the first image to be registered. k The second-order gradient phase image of the scale is as follows Figure 4 shown.

[0128] Based on the first image to be registered k The second-order gradient magnitude map and second-order gradient phase map of the scale are obtained. k The feature vector corresponding to the first feature point of the scale is based on the first feature point of the second image to be registered. k The second-order gradient magnitude map and second-order gradient phase map of the scale are obtained. k The eigenvector corresponding to the second feature point of the scale.

[0129] Specifically, based on the first image to be registered k The second-order gradient amplitude map and second-order gradient phase map of the scale can be obtained by numerical calculation, quantitative statistics, etc. k The feature vector corresponding to the first feature point of the scale is based on the first feature point of the second image to be registered. k The second-order gradient amplitude map and second-order gradient phase map of the scale can also be obtained by numerical calculation, quantitative statistics, etc. k The eigenvector corresponding to the second feature point of the scale.

[0130] As an optional embodiment, based on the first image to be registered k The second-order gradient magnitude map and second-order gradient phase map of the scale are obtained. k The feature vector corresponding to the first feature point of the scale is based on the first feature point of the second image to be registered. k The second-order gradient magnitude map and second-order gradient phase map of the scale are obtained. k The feature vector corresponding to the second feature point of the first image to be registered includes: k In the second-order gradient amplitude map of the scale, determine the k The first target area corresponding to the first feature point of the second image to be registered k In the second-order gradient amplitude map of the scale, determine the k The first target area corresponds to the second feature point of the scale.

[0131] For the first k Gaussian filtering is performed on the first original area corresponding to the first feature point of the scale to obtain the k The first target area corresponding to the first feature point of the scale, k Gaussian filtering is performed on the first original area corresponding to the second feature point of the scale to obtain the kThe first target area corresponds to the second feature point of the scale.

[0132] Based on the first image to be registered k The second-order gradient phase image of the scale is obtained k The gradient amplitude value of the first target area corresponding to the first feature point of the scale in each first phase interval, and then the first phase interval with the largest gradient amplitude value is determined as the first phase interval. k The first phase interval corresponding to the first feature point of the scale is based on the first phase interval of the second image to be registered. k The second-order gradient phase image of the scale is obtained k The gradient amplitude value of the first target area corresponding to the second feature point of the scale in each first phase interval, and then the first phase interval with the largest gradient amplitude value is determined as the first phase interval. k The first phase interval corresponding to the second characteristic point of the scale.

[0133] Based on the k The first phase interval corresponding to the first characteristic point of the scale is determined k The direction of the eigenvector corresponding to the first feature point of the scale is based on the k The first phase interval corresponding to the second characteristic point of the scale is determined k The direction of the eigenvector corresponding to the first feature point of the scale.

[0134] In the first image to be registered k In the second-order gradient amplitude map of the scale, determine the k The feature area corresponding to the first feature point of the second image to be registered is k In the second-order gradient amplitude map of the scale, determine the k The feature area corresponding to the second feature point of the scale.

[0135] Based on the first image to be registered k The second-order gradient phase image of the scale is obtained k The gradient amplitude value of the feature area corresponding to the first feature point of the scale in each second phase interval is used as the first k The eigenvalue of the eigenvector corresponding to the first feature point of the scale is based on the first feature point of the second image to be registered. k The second-order gradient phase image of the scale is obtained k The gradient amplitude value of the feature area corresponding to the second feature point of the scale in each second phase interval is used as the first k The eigenvalue of the eigenvector corresponding to the second feature point of the scale.

[0136] In the first image to be registered k In the second-order gradient amplitude map of the scale, the k The first characteristic point of the scale As the center of the circle, the radius is The circular area is determined as k The first characteristic point of the scale The corresponding first original area.

[0137] Based on the variance Gaussian check of k The first characteristic point of the scale The first original area corresponding to the first is Gaussian filtered to highlight the area closest to the first k The first characteristic point of the scale Gradient direction information, get the k The first characteristic point of the scale The corresponding first target area.

[0138] Wherein, A and B are positive integers greater than 0, and A is greater than B. The specific values ​​of A and B can be determined based on prior knowledge and / or actual conditions. The specific values ​​of A and B are not limited in the embodiments of the present invention.

[0139] Optionally, the value of A is 6; the value of B is 2.

[0140] It should be noted that, in the embodiment of the present invention, Divide equally into First phase intervals, the step size of each first phase interval is .in, It can be a positive integer determined based on prior knowledge and / or actual conditions.

[0141] Optionally, , accordingly, the step length of each first phase interval is 10°.

[0142] Based on the first image to be registered k The second-order gradient phase image of the scale can be obtained k The first characteristic point of the scale The gradient amplitude value of the corresponding first target area in each first phase interval.

[0143] Based on the Gaussian filter k The first characteristic point of the scale The gradient amplitude value of the corresponding first target area in each first phase interval generates the first k The first characteristic point of the scale The corresponding gradient direction histogram.

[0144] Figure 5 This is the first step in the multimodal image registration method provided by the present invention. k The first characteristic point of the scale The corresponding gradient direction histogram.k The first characteristic point of the scale The corresponding gradient direction histogram is as follows Figure 5 shown.

[0145] Based on the k The first characteristic point of the scale The corresponding gradient direction histogram is used to determine the first phase interval with the largest gradient amplitude value as the first k The first characteristic point of the scale The corresponding first phase interval can then be based on the k The first characteristic point of the scale The corresponding first phase interval determines the k The first characteristic point of the scale The descriptor is rotated to the main direction to eliminate the rotation difference.

[0146] In the first image to be registered k In the second-order gradient amplitude map of the scale, the k The first characteristic point of the scale is the center of the circle and the largest radius is , the smaller radius is The circular area is determined as k The first characteristic point of the scale The corresponding second target area will be k The first characteristic point of the scale is the center of the circle and the largest radius is , the smaller radius is E The circular area is determined as k The first characteristic point of the scale The corresponding third target area will be k The first characteristic point of the scale is the center of the circle and the radius is E The circular area is determined as k The first characteristic point of the scale The corresponding fourth target region defines a positive rotation direction with the image coordinate system as a reference, and resists scale differences through descriptors of different radii.

[0147] Wherein, C, D, and E are positive integers greater than 0, and C>D>E. The specific values ​​of C, D, and E can be determined based on prior knowledge and / or actual conditions. The specific values ​​of C, D, and E are not limited in the embodiments of the present invention.

[0148] Optionally, the value of C is 12, the value of D is 9, and the value of E is 2.

[0149] The first k The first characteristic point of the scale The corresponding second target area and the k The first characteristic point of the scale The perpendicular midline of the corresponding third target area is determined as the starting position, and the first k The first characteristic point of the scale The corresponding second target area is divided into The second target sub-region k The first characteristic point of the scale The corresponding third target area is divided into A third target sub-area.

[0150] It should be noted that the preset angle in the embodiment of the present invention can be determined based on prior knowledge and / or actual conditions. The specific value of the preset angle is not limited in the embodiment of the present invention.

[0151] Optionally, the value of the preset angle may be 30°. .

[0152] The first k The first characteristic point of the scale The corresponding second target area The second target sub-area, k The first characteristic point of the scale The corresponding third target area The third target sub-area and k The first characteristic point of the scale The corresponding fourth target area is determined as k The first characteristic point of the scale The corresponding feature area.

[0153] Figure 6 This is the first step in the multimodal image registration method provided by the present invention. k Schematic diagram of the feature area corresponding to the first feature point of the scale. k The first characteristic point of the scale The corresponding feature areas are Figure 6 shown.

[0154] It should be noted that, in the embodiment of the present invention, Divide equally into The second phase intervals, the step size of each first phase interval is .in, It can be a positive integer determined based on prior knowledge and / or actual conditions.

[0155] Optionally, , accordingly, the step length of each first phase interval is 30°.

[0156] Based on the first image to be registered k The second-order gradient phase image of the scale can be obtained k The first characteristic point of the scale The gradient amplitude value of each corresponding feature area in each second phase interval is used as the first k The first characteristic point of the scale The eigenvalues ​​of the corresponding eigenvectors.

[0157] No. k The first characteristic point of the scale The dimension of the corresponding feature vector .

[0158] Optionally, in , In the case of k The first characteristic point of the scale The corresponding feature vector has a dimension of 200.

[0159] It should be noted that based on k The second feature point of the second image to be registered k The edge window filtering result image of the scale is obtained k The specific steps of the feature vector corresponding to the second feature point of the scale are the same as those based on the first k The first feature point of the scale and the first image to be registered k The edge window filtering result image of the scale is obtained k The specific steps of obtaining the feature vector corresponding to the first feature point of the scale are the same as those in the embodiment of the present invention and will not be repeated herein.

[0160] Step 103 : Based on the feature vector corresponding to the first feature point and the feature vector corresponding to the second feature point, feature matching is performed on the first feature point and the second feature point, and the first feature point and the second feature point that are successfully matched are determined as a first feature point pair with the same name.

[0161] Specifically, in an embodiment of the present invention, each first feature point can be traversed, and based on the feature vector corresponding to each first feature point and the feature vector corresponding to each second feature point, the cosine similarity and the nearest neighbor ratio between each first feature point and each second feature point can be calculated.

[0162] For any first feature point, after determining the second feature point with the smallest cosine similarity and nearest neighbor ratio with the first feature point, if the cosine similarity and nearest neighbor ratio between the second feature point and the first feature point are not greater than a predefined threshold, it can be determined that the second feature point and the first feature point are successfully matched, and the first feature point and the second feature point can be determined as a first feature point pair with the same name.

[0163] It can be understood that there are multiple first feature point pairs with the same name.

[0164] After traversing each first feature point, the third feature point pair with the same name can be obtained , Indicates the third pair of feature points with the same name The number of first pairs of feature points with the same name in .

[0165] For the third set of feature points with the same name The first pair of feature points with the same name in is used to remove outliers using the fast sample consistency method (FSC) of the affine transformation model. This time, the error threshold is set to 10 pixels to roughly estimate the transformation model parameters.

[0166] Eliminate the third set of feature point pairs with the same name After removing the outliers in the set, the third set of feature points with the same name can be The first set of feature point pairs with the same name is determined .

[0167] Step 104: Based on the first pair of feature points with the same name, obtain a first affine transformation matrix between the first image to be registered and the second image to be registered; based on the first affine transformation matrix, perform feature matching on the first feature point and the second feature point again to obtain a second pair of feature points with the same name.

[0168] Specifically, after obtaining the first pair of feature points with the same name, the first affine transformation matrix between the first image to be registered and the second image to be registered can be obtained through numerical calculation based on the correspondence between the first feature point and the second feature point in the first pair of feature points with the same name.

[0169] After obtaining the first affine transformation matrix between the first image to be registered and the second image to be registered, feature matching can be performed again on the first feature points and the second feature points based on the first affine transformation matrix through numerical calculation, mathematical statistics, and deep learning technology to obtain a second pair of feature points with the same name.

[0170] As an optional embodiment, based on the first affine transformation matrix, feature matching is performed again on the first feature point and the second feature point to obtain a second pair of feature points with the same name, including: mapping the first feature point to the second image to be registered based on the first affine transformation matrix.

[0171] Specifically, based on the position of the first feature point in the first image to be registered and the first affine transformation matrix between the first image to be registered and the second image to be registered, the first feature point can be mapped to the second image to be registered, thereby obtaining the mapping point of the first feature point in the second image to be registered.

[0172] A target number of second feature points closest to the mapping point of the first feature point in the second image to be registered are determined as candidate points of the same name for the first feature point.

[0173] After obtaining the mapping point of the first feature point in the second image to be registered, a target number of second feature points in the second image to be registered that are closest to the mapping point of the first feature point can be determined as candidate points of the same name for the first feature point.

[0174] It should be noted that the target quantity in the embodiment of the present invention may be determined based on prior knowledge and / or actual conditions.

[0175] Optionally, the target number may be set to 3, that is, the number of candidate points with the same name for the first feature point may be three.

[0176] Based on the target distance between the first feature point and the candidate homonymous point of the first feature point, the first feature point and the candidate homonymous point of the first feature point having the smallest target distance to the first feature point are determined as a second homonymous feature point pair.

[0177] Specifically, considering that the offset directions of adjacent feature points should be substantially the same, in the embodiment of the present invention, the target distance is redefined in the secondary matching stage to perform feature point matching.

[0178] In the embodiment of the present invention, any two points in the image Dot and Target distance between points , defined as the combination of the position distance and feature distance between any two points mentioned above. The specific calculation formula is as follows:

[0179]

[0180] in, represents the cosine similarity, and Respectively Dot and The position of the point in the image, and Respectively Dot and The eigenvector of a point, Represents a predefined transformation model, Represents the parameters of the transformation model.

[0181] Based on the above formula, the target distance between the first feature point and the candidate points with the same name as the first feature point can be calculated.

[0182] Based on the target distance between the first feature point and the candidate homonymous points of the first feature point, the candidate homonymous points of the first feature point having the smallest target distance to the first feature point are determined as a pair of second homonymous feature points.

[0183] Specifically, after calculating the target distance between the first feature point and its candidate homonymous points, the candidate homonymous point with the smallest target distance from the first feature point can be determined together with the first feature point as a second homonymous feature point pair.

[0184] After traversing each first pair of feature points with the same name, the fourth set of feature points with the same name can be obtained , Indicates the fourth pair of feature points with the same name The number of second pairs of feature points with the same name in .

[0185] For the fourth set of feature points with the same name The second pair of feature points with the same name in is used to remove outliers using the fast sample consistency method (FSC) of the affine transformation model. This time, the error threshold is set to 3 pixels to roughly estimate the transformation model parameters.

[0186] Eliminate the fourth feature point pair with the same name After removing the outliers in the , the fourth set of feature point pairs with the same name can be The second set of feature point pairs with the same name is determined .

[0187] Step 105 : Register the first image to be registered and the second image to be registered based on the first pair of feature points with the same name and the second pair of feature points with the same name.

[0188] Specifically, after obtaining the first and second feature point pairs with the same name, the first and second images to be registered can be registered based on the first and second feature point pairs with the same name through numerical calculation, mathematical statistics, etc.

[0189] After the first image to be registered and the second image to be registered are registered, the registration effect of the first image to be registered and the second image to be registered can be quantitatively evaluated based on the root mean square error of the first pair of feature points with the same name and the second pair of feature points with the same name and the number of matches between the first pair of feature points with the same name and the second pair of feature points with the same name. A line diagram of the same name points and a registration diagram in the form of a checkerboard can also be drawn to qualitatively test the registration effect of the first image to be registered and the second image to be registered.

[0190] In an embodiment of the present invention, multi-scale edge window filtering is performed on a first image to be registered and a second image to be registered, respectively, to obtain a multi-scale edge window filtering result image of the first image to be registered and a multi-scale edge window filtering result image of the second image to be registered. Then, based on the edge window filtering result image, a first feature point corresponding to the first image to be registered and a feature vector corresponding to the first feature point, as well as a second feature point corresponding to the second image to be registered and a feature vector corresponding to the second feature point are obtained. Based on the first feature point and the feature vector corresponding to the first feature point, and the second feature point and the feature vector corresponding to the second feature point, feature matching is performed twice on the first feature point and the second feature point to obtain a first pair of feature points with the same name and a second pair of feature points with the same name. Then, based on the first pair of feature points with the same name and the second pair of feature points with the same name, the first image to be registered and the second image to be registered are registered. Multi-scale edge window filtering effectively preserves image edge features, enhances the robustness of multimodal image registration, and improves the accuracy and efficiency of multimodal image registration through a two-stage matching strategy, providing an effective solution for automated processing and analysis of remote sensing images.

[0191] In order to verify the registration effect of the multimodal image registration method provided by the present invention, quantitative and qualitative experiments were conducted on 11 types of multimodal image datasets in three fields.

[0192] Figure 7 This is an excerpt from some datasets used in the multimodal image registration method provided by this invention. For each image pair, quantitative verification is performed using the Success Rate (SR), Root-Mean-Square Error (RMSE), Number of Correct Matching (NCM), and Mean Time (MT). Figure 8 It is one of the registration results of the multimodal image registration method provided by the present invention. Figure 9 This is the second registration result of the multimodal image registration method provided by the present invention. The registration result of the multimodal image registration based on the multimodal image registration method provided by the present invention is as follows: Figure 7 and Figure 9 shown.

[0193] The multimodal image registration method provided by the present invention is named MSG and compared with several state-of-the-art image registration methods (six handcrafted feature-based methods: SIFT, OS-SIFT, RIFT, CoFSM, HOWP, and POS-GIFT; and two deep learning-based methods: SuperPoint+SuperGlue and LoFTR). The comparison results on remote sensing datasets are shown in Table 1. As shown in Table 1, for all image pairs, MSG achieves a higher number of correct homonymous points and a lower RMSE. Experimental results show that, on remote sensing datasets, the proposed method's NCM is more than three times higher than all algorithms except POS-GIFT, with an RMSE of 1.69 pixels. MT ranks third among feature-based algorithms, is scale- and rotation-invariant, and can handle complex geometric and nonlinear radiometric differences.

[0194] Table 1 Quantitative comparison of experimental results of 9 algorithms on remote sensing datasets

[0195]

[0196] The multimodal image registration method provided by the present invention constructs a multi-scale edge window Gaussian filter scale space, retains edge information to the greatest extent while blurring the image, performs corner detection on the binary edge map, increases the repetition rate of feature points, uses the robustness of the second-order gradient enhancement descriptor combined with Gaussian controllable filtering, and fully utilizes densely distributed edge feature points through secondary matching in a limited search space, further improving the registration performance.

[0197] Experimental results demonstrate that the multimodal image registration method proposed in this paper performs exceptionally well in multimodal remote sensing image registration tasks. It is able to obtain a sufficient number of keypoints in the presence of complex edge and texture features, as well as noise, distortion, and other irrational conditions, achieving high registration accuracy while maintaining high time efficiency. Therefore, this paper not only improves the performance of multimodal remote sensing image registration but also demonstrates promising application potential in practice, providing an effective solution for the automated processing and analysis of remote sensing images.

[0198] Figure 10 This is a schematic diagram of the structure of the multimodal image registration device provided by the present invention. Figure 10 The multimodal image registration device provided by the present invention is described. The multimodal image registration device described below and the multimodal image registration method provided by the present invention described above can be referred to each other. Figure 10 As shown, the device includes: an edge window filtering module 1001 , a feature extraction module 1002 , a first matching module 1003 , a second matching module 1004 and an image registration module 1005 .

[0199] The edge window filtering module 1001 is used to perform the first image to be registered and the second image to be registered. k Scaled edge window filtering is used to obtain the first image to be registered. k The edge window filtering result image and the second image to be registered k The first image to be registered and the second image to be registered are images of the same region or the same target in different modalities. , S represents the number of scales, S It is a positive integer greater than 1. Different scales result in different variances, filtering times, and radius of the edge window filter window.

[0200] Feature extraction module 1002, for extracting the first image to be registered based on the k The edge window filtering result image of the scale and the second image to be registered k The edge window filtering result image of the scale is obtained, and the first feature point corresponding to the first image to be registered and the feature vector corresponding to the first feature point are obtained, as well as the second feature point corresponding to the second image to be registered and the feature vector corresponding to the second feature point are obtained.

[0201] The first matching module 1003 is configured to perform feature matching on the first feature point and the second feature point based on the feature vector corresponding to the first feature point and the feature vector corresponding to the second feature point, and determine the successfully matched first feature point and second feature point as a first feature point pair with the same name.

[0202] The second matching module 1004 is used to obtain a first affine transformation matrix between the first image to be registered and the second image to be registered based on the first pair of feature points with the same name, and perform feature matching on the first feature point and the second feature point again based on the first affine transformation matrix to obtain a second pair of feature points with the same name.

[0203] The image registration module 1005 is configured to register the first image to be registered and the second image to be registered based on the first pair of feature points with the same name and the second pair of feature points with the same name.

[0204] Specifically, the edge window filtering module 1001 , the feature extraction module 1002 , the first matching module 1003 , the second matching module 1004 and the image registration module 1005 are electrically connected.

[0205] The multimodal image registration device in the embodiment of the present invention performs multi-scale edge window filtering on the first image to be registered and the second image to be registered respectively, obtains the multi-scale edge window filtering result image of the first image to be registered and the multi-scale edge window filtering result image of the second image to be registered, and then obtains the first feature point corresponding to the first image to be registered and the feature vector corresponding to the first feature point, as well as the second feature point corresponding to the second image to be registered and the feature vector corresponding to the second feature point, the first feature point and the feature vector corresponding to the first feature point and the second feature point based on the edge window filtering result image. The first feature point and the feature vector corresponding to the above-mentioned second feature point are matched twice on the first feature point and the second feature point to obtain the first feature point pair with the same name and the second feature point pair with the same name, and then the first image to be registered and the second image to be registered are registered based on the above-mentioned first feature point pair with the same name and the second feature point pair with the same name. Through multi-scale side window filtering, the image edge features are effectively retained, and the robustness of the multimodal image registration is enhanced. Through a two-stage matching strategy, the accuracy and efficiency of the multimodal image registration are improved, and an effective solution is provided for the automated processing and analysis of remote sensing images.

[0206] Figure 11 An example of a physical structure diagram of an electronic device is shown below. Figure 11 As shown, the electronic device may include: a processor 1110, a communication interface 1120, a memory 1130, and a communication bus 1140, wherein the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other via the communication bus 1140. The processor 1110 may call logic instructions in the memory 1130 to execute the multimodal image registration method.

[0207] Furthermore, the logic instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0208] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal image registration method provided by the above methods.

[0209] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the multimodal image registration method provided by the above methods.

[0210] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0211] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0212] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A multimodal image registration method, characterized in that: include: The first image to be registered and the second image to be registered are respectively k The edge window filtering of the first image to be registered is performed to obtain the first k The edge window filtering result image of the scale and the second image to be registered k The first image to be registered and the second image to be registered are images of the same region or the same target but in different modalities. , S represents the number of scales, S is a positive integer greater than 1. When the scale is different, the variance of the edge window filtering, the number of filtering times and the radius of the edge window filtering window are different; Based on the first image to be registered k The edge window filtering result image of the scale and the second image to be registered k Scaled edge window filtering result image, obtaining first feature points corresponding to the first image to be registered and feature vectors corresponding to the first feature points, as well as second feature points corresponding to the second image to be registered and feature vectors corresponding to the second feature points; performing feature matching on the first feature point and the second feature point based on a feature vector corresponding to the first feature point and a feature vector corresponding to the second feature point, and determining the first feature point and the second feature point that are successfully matched as a first feature point pair with the same name; Based on the first pair of feature points with the same name, obtaining a first affine transformation matrix between the first image to be registered and the second image to be registered; based on the first affine transformation matrix, performing feature matching on the first feature points and the second feature points again to obtain a second pair of feature points with the same name; The first image to be registered and the second image to be registered are registered based on the first pair of feature points with the same name and the second pair of feature points with the same name.

2. The multimodal image registration method according to claim 1, wherein: The first image to be registered k The edge window filtering result image of the scale and the second image to be registered k The method further comprises: obtaining a first feature point corresponding to the first image to be registered and a feature vector corresponding to the first feature point, and a second feature point corresponding to the second image to be registered and a feature vector corresponding to the second feature point, comprising: The first image to be registered is k The edge window filtering result image of the scale and the second image to be registered k The edge window filtering result image of the scale is used to extract the edge, and the first image to be registered is obtained. k The edge feature map of the scale and the second image to be registered k Scaled edge feature map; The first image to be registered k The edge feature map of the scale and the second image to be registered k The edge feature maps of the first scale are respectively used for corner detection to obtain the first image to be registered corresponding to the first image. k The first original feature point of the scale and the second image to be registered corresponding to the first k The second original feature point of the scale; Based on the non-maximum suppression algorithm, the first image to be registered corresponding to the k The first original feature points of the scale are screened to obtain the first original feature points corresponding to the first image to be registered. k The first feature point of the scale, the first feature point of the second image to be registered k The second original feature points of the scale are screened to obtain the first feature points corresponding to the second image to be registered k The second characteristic point of the scale; Based on the k The first feature point of the first image to be registered k The edge window filtering result image of the scale is obtained k The feature vector corresponding to the first feature point of the scale is based on the first k The second feature point of the scale and the second image to be registered k The edge window filtering result image of the scale is obtained k The eigenvector corresponding to the second feature point of the scale.

3. The multimodal image registration method according to claim 2, wherein: The said based on the k The first feature point of the first image to be registered k The edge window filtering result image of the scale is obtained k The feature vector corresponding to the first feature point of the scale is based on the first k The second feature point of the scale and the second image to be registered k The edge window filtering result image of the scale is obtained k The feature vector corresponding to the second feature point of the scale includes: The first image to be registered is k The edge window filtering result image of the scale and the second image to be registered k The edge window filtering result image of the scale is j The controllable Gaussian filter in the direction is used to obtain the first image to be registered. k Scale j The controllable Gaussian filtering result image of the direction and the second image to be registered k Scale j The result image of the controllable Gaussian filter in the direction, , Indicates the number of directions, is a positive integer greater than 1, and the rotation angle of the Gaussian filter window for performing Gaussian filtering is different for different directions; The first image to be registered k The controllable Gaussian filtering result images in all directions of the scale are superimposed to obtain the first image to be registered. k The first-order gradient map of the scale is used to convert the second image to be registered into k The controllable Gaussian filtering result images in all directions of the scale are superimposed to obtain the second image to be registered. k First-order gradient map of scale; Based on the first image to be registered k The first-order gradient map of the scale is obtained by k The second-order gradient magnitude map and the second-order gradient phase map of the scale are based on the second image to be registered. k The first-order gradient map of the scale is obtained by k Scaled second-order gradient magnitude map and second-order gradient phase map; Based on the first image to be registered k The second-order gradient amplitude map and the second-order gradient phase map of the scale are obtained to obtain the second k The feature vector corresponding to the first feature point of the second image to be registered is k The second-order gradient amplitude map and the second-order gradient phase map of the scale are obtained to obtain the second k The eigenvector corresponding to the second feature point of the scale.

4. The multimodal image registration method according to claim 2, wherein: The step of performing feature matching again on the first feature point and the second feature point based on the first affine transformation matrix to obtain a second pair of feature points with the same name includes: Mapping the first feature points to the second image to be registered based on the first affine transformation matrix; Determine a target number of second feature points closest to the mapping point of the first feature point in the second image to be registered as candidate points of the same name for the first feature point; Calculating a target distance between the first feature point and the candidate homonymous point of the first feature point based on a distance between a mapping point of the first feature point in the second image to be registered and the candidate homonymous point of the first feature point, a feature vector of the first feature point, and a feature vector of the candidate homonymous point of the first feature point; Based on the target distance between the first feature point and the candidate homonymous point of the first feature point, the first feature point and the candidate homonymous point of the first feature point having the smallest target distance to the first feature point are determined as a pair of the second homonymous feature points.

5. The multimodal image registration method according to claim 3, wherein: The first image to be registered k The second-order gradient amplitude map and the second-order gradient phase map of the scale are obtained to obtain the second k The feature vector corresponding to the first feature point of the second image to be registered is k The second-order gradient amplitude map and the second-order gradient phase map of the scale are obtained to obtain the second k The feature vector corresponding to the second feature point of the scale includes: In the first image to be registered k In the second-order gradient amplitude map of the scale, determine the k The first target area corresponding to the first feature point of the second image to be registered is k In the second-order gradient amplitude map of the scale, determine the k A first target area corresponding to the second feature point of the scale; Regarding the k Gaussian filtering is performed on the first original area corresponding to the first feature point of the scale to obtain the first k The first target area corresponding to the first feature point of the scale, k Gaussian filtering is performed on the first original area corresponding to the second feature point of the scale to obtain the first k A first target area corresponding to the second feature point of the scale; Based on the first image to be registered k The second-order gradient phase image of the scale is obtained k The gradient amplitude value of the first target area corresponding to the first feature point of the scale in each first phase interval, and then the first phase interval with the largest gradient amplitude value is determined as the first phase interval. k The first phase interval corresponding to the first feature point of the second image to be registered is based on the first phase interval of the first feature point of the second image to be registered. k The second-order gradient phase image of the scale is obtained k The gradient amplitude value of the first target area corresponding to the second feature point of the scale in each first phase interval, and then the first phase interval with the largest gradient amplitude value is determined as the first phase interval. k The first phase interval corresponding to the second characteristic point of the scale; Based on the k The first phase interval corresponding to the first characteristic point of the scale is determined k The direction of the feature vector corresponding to the first feature point of the scale is based on the k The first phase interval corresponding to the second characteristic point of the scale is determined k The direction of the eigenvector corresponding to the first feature point of the scale; In the first image to be registered k In the second-order gradient amplitude map of the scale, determine the k The feature area corresponding to the first feature point of the second image to be registered is k In the second-order gradient amplitude map of the scale, determine the k The feature area corresponding to the second feature point of the scale; Based on the first image to be registered k The second-order gradient phase image of the scale is obtained k The gradient amplitude value of the feature area corresponding to the first feature point of the scale in each second phase interval is used as the first k The eigenvalue of the eigenvector corresponding to the first feature point of the second image to be registered is based on the k The second-order gradient phase image of the scale is obtained k The gradient amplitude value of the feature area corresponding to the second feature point of the scale in each second phase interval is used as the first k The eigenvalue of the eigenvector corresponding to the second feature point of the scale.

6. The multimodal image registration method according to claim 1, wherein: The first image to be registered and the second image to be registered are respectively k The edge window filtering of the first image to be registered is performed to obtain the first k The edge window filtering result image of the scale and the second image to be registered k The resulting image of the edge window filter of the scale includes: Initialize the edge window filter parameters, which include the initial variance, the number of scales, the scale factor, the position of each edge window filter window, and the k The edge window filter window radius, filtering times and variance corresponding to the scale, each of the edge window filter windows includes a rectangular edge window filter window with the long side in the vertical direction and the pixel to be filtered located at the midpoint of the right long side, a rectangular edge window filter window with the long side in the vertical direction and the pixel to be filtered located at the midpoint of the left long side, a rectangular edge window filter window with the long side in the horizontal direction and the pixel to be filtered located at the midpoint of the upper long side, a rectangular edge window filter window with the long side in the horizontal direction and the pixel to be filtered located at the midpoint of the lower long side, and a rectangular edge window filter window with the four sides in the horizontal and vertical directions respectively and the pixel to be filtered located at the four sides respectively. a square side window filtering window with right angles located in the horizontal and vertical directions and the pixel to be filtered is located at the midpoint of the lower right hypotenuse; a square side window filtering window with right angles located in the horizontal and vertical directions and the pixel to be filtered is located at the midpoint of the lower left hypotenuse; a square side window filtering window with right angles located in the horizontal and vertical directions and the pixel to be filtered is located at the midpoint of the upper right hypotenuse; and a square side window filtering window with right angles located in the horizontal and vertical directions and the pixel to be filtered is located at the midpoint of the upper left hypotenuse; Based on the filtering parameters, the first image to be registered and the second image to be registered are respectively subjected to the first k The edge window filtering of the first image to be registered is performed to obtain the first k The edge window filtering result image of the scale and the second image to be registered k Scaled edge window filtering result image.

7. The multimodal image registration method according to any one of claims 1 to 6, characterized in that: The first image to be registered and the second image to be registered are respectively k The edge window filtering of the first image to be registered is performed to obtain the first k The edge window filtering result image of the scale and the second image to be registered k Before filtering the resulting image with an edge window of a certain scale, the method further comprises: Image preprocessing is performed on the first image to be registered and the second image to be registered, where the image preprocessing includes at least one of normalization, contrast stretching, and Gaussian filtering.

8. A multimodal image registration device, characterized in that: include: The edge window filtering module is used to filter the first image to be registered and the second image to be registered. k The edge window filtering of the first image to be registered is performed to obtain the first k The edge window filtering result image of the scale and the second image to be registered k The first image to be registered and the second image to be registered are images of the same region or the same target but in different modalities. , S represents the number of scales, S is a positive integer greater than 1. When the scale is different, the variance of the edge window filtering and the radius of the edge window filtering window are different; A feature extraction module is used to extract the first image to be registered based on the k The edge window filtering result image of the scale and the second image to be registered k Scaled edge window filtering result image, obtaining first feature points corresponding to the first image to be registered and feature vectors corresponding to the first feature points, as well as second feature points corresponding to the second image to be registered and feature vectors corresponding to the second feature points; a first matching module, configured to perform feature matching on the first feature point and the second feature point based on a feature vector corresponding to the first feature point and a feature vector corresponding to the second feature point, and determine the first feature point and the second feature point that are successfully matched as a first feature point pair with the same name; a second matching module, configured to obtain a first affine transformation matrix between the first image to be registered and the second image to be registered based on the first pair of feature points of the same name, and perform feature matching on the first feature points and the second feature points again based on the first affine transformation matrix to obtain a second pair of feature points of the same name; An image registration module is configured to register the first image to be registered and the second image to be registered based on the first pair of feature points with the same name and the second pair of feature points with the same name.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the multimodal image registration method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal image registration method according to any one of claims 1 to 7 is implemented.

Images