Multi-modal image registration method and device, electronic equipment and storage medium

Abstract

The invention provides a multi-modal image registration method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the multi-scale side window filtering of a first to-be-registered image and a second to-be-registered image, and obtaining the multi-scale side window filtering result images of the first to-be-registered image and the second to-be-registered image, based on the side window filtering result image, obtaining a first feature point and a feature vector corresponding to the first feature point and a second feature point and a feature vector corresponding to the second feature point, and based on the feature vectors, performing feature matching on the first feature point and the second feature point twice to obtain a first homonymy feature point pair and a second homonymy feature point pair; and registering the first to-be-registered image and the second to-be-registered image based on the first homonymy feature point pair and the second homonymy feature point pair. According to the multi-modal image registration method and device, the electronic equipment and the storage medium provided by the invention, the accuracy and efficiency of multi-modal image registration are improved, and the robustness of multi-modal image registration is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multimodal image registration method, device, electronic device and storage medium. Background Art

[0002] Multimodal images refer to multiple types of images of the same area or target acquired through different sensors, different imaging mechanisms or different bands. Compared with single-modal images, multimodal images provide more information and have wide application value in many fields such as climate change, natural disaster assessment, urban planning and other global challenges.

[0003] Multimodal image registration refers to the process of identifying the same-name points between two or more images of different modalities and aligning the images. Multimodal image registration is a bridge connecting multimodal images. Through multimodal image registration, multimodal image alignment can be achieved, which is crucial for a variety of remote sensing applications such as image fusion, change detection, image stitching, and 3D reconstruction.

[0004] Traditional multimodal image registration methods in related technologies generally include region-based methods, feature-based methods, and deep learning-based methods. However, the above traditional multimodal image registration methods are limited by various factors, and the accuracy and efficiency of multimodal image registration are not high. Therefore, how to perform multimodal image registration more accurately and efficiently is a technical problem that needs to be solved in this field. Summary of the invention

[0005] The present invention provides a multimodal image registration method, device, electronic device and storage medium, which are used to solve the defects of low accuracy and efficiency of multimodal image registration in the prior art, and realize more accurate and efficient multimodal image registration.

[0006] The present invention provides a multimodal image registration method, comprising the following steps.

[0007] The first image to be registered and the second image to be registered are respectively k The first image to be registered is obtained by filtering the edge window of the first image. k The edge window filtering result image of the scale and the second image to be registered k The first image to be registered and the second image to be registered are images of the same region or the same target but in different modes. , S represents the number of scales, S is a positive integer greater than 1. When the scale is different, the variance and filtering times of the edge window filtering and the radius of the edge window filtering window are different. Based on the first image to be registeredk The edge window filtering result image of the scale and the second image to be registered k Scaled edge window filtering result image, obtaining first feature points corresponding to the first image to be registered and feature vectors corresponding to the first feature points, as well as second feature points corresponding to the second image to be registered and feature vectors corresponding to the second feature points; Based on the feature vector corresponding to the first feature point and the feature vector corresponding to the second feature point, feature matching is performed on the first feature point and the second feature point, and the first feature point and the second feature point that are successfully matched are determined as a first feature point pair with the same name; Based on the first pair of feature points with the same name, a first affine transformation matrix between the first image to be registered and the second image to be registered is obtained, and based on the first affine transformation matrix, feature matching is performed again on the first feature points and the second feature points to obtain a second pair of feature points with the same name; The first image to be registered and the second image to be registered are registered based on the first pair of feature points with the same name and the second pair of feature points with the same name.

[0008] The present invention also provides a multimodal image registration device, comprising the following modules: The edge window filtering module is used to filter the first image to be registered and the second image to be registered respectively. k The first image to be registered is obtained by filtering the edge window of the first image. k The edge window filtering result image of the scale and the second image to be registered k The first image to be registered and the second image to be registered are images of the same region or the same target but in different modes. , S represents the number of scales, S is a positive integer greater than 1. When the scale is different, the variance, number of filtering times and radius of the edge window filtering window for edge window filtering are different. A feature extraction module is used to extract the first image to be registered based on the first image to be registered. k The edge window filtering result image of the scale and the second image to be registered k Scaled edge window filtering result image, obtaining first feature points corresponding to the first image to be registered and feature vectors corresponding to the first feature points, as well as second feature points corresponding to the second image to be registered and feature vectors corresponding to the second feature points; A first matching module, configured to perform feature matching on the first feature point and the second feature point based on a feature vector corresponding to the first feature point and a feature vector corresponding to the second feature point, and determine the first feature point and the second feature point that are successfully matched as a first feature point pair with the same name; A second matching module is used to obtain a first affine transformation matrix between the first image to be registered and the second image to be registered based on the first pair of feature points with the same name, and perform feature matching on the first feature points and the second feature points again based on the first affine transformation matrix to obtain a second pair of feature points with the same name; An image registration module is used to register the first image to be registered and the second image to be registered based on the first pair of feature points with the same name and the second pair of feature points with the same name.

[0009] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multimodal image registration method as described above is implemented.

[0010] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the multimodal image registration method as described in any one of the above is implemented.

[0011] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the multimodal image registration methods described above.

[0012] The multimodal image registration method, device, electronic device and storage medium provided by the present invention respectively perform multi-scale edge window filtering on the first image to be registered and the second image to be registered, obtain the multi-scale edge window filtering result image of the first image to be registered and the multi-scale edge window filtering result image of the second image to be registered, then obtain the first feature point corresponding to the first image to be registered and the feature vector corresponding to the first feature point and the second feature point corresponding to the second image to be registered and the feature vector corresponding to the second feature point based on the edge window filtering result image, and obtain the first feature point corresponding to the first image to be registered and the feature vector corresponding to the second feature point based on the first feature point and the feature vector corresponding to the first feature point The first feature point and the second feature point and the feature vector corresponding to the above-mentioned second feature point are feature matched twice to obtain the first feature point pair with the same name and the second feature point pair with the same name, and then based on the above-mentioned first feature point pair with the same name and the second feature point pair with the same name, the first image to be registered and the second image to be registered are registered. Through multi-scale edge window filtering, the image edge features are effectively retained, and the robustness of the multimodal image registration is enhanced. Through the two-stage matching strategy, the accuracy and efficiency of the multimodal image registration are improved, which provides an effective solution for the automated processing and analysis of remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 This is one of the flow charts of the multimodal image registration method provided by the present invention.

[0015] Figure 2 This is the second flowchart of the multimodal image registration method provided by the present invention.

[0016] Figure 3 It is a schematic diagram of the position of the side window filter window in the multimodal image registration method provided by the present invention.

[0017] Figure 4 The first image to be registered in the multimodal image registration method provided by the present invention is k Schematic diagram of the second-order gradient phase map at scale.

[0018] Figure 5 This is the first method in the multimodal image registration method provided by the present invention. k The first characteristic point of the scale The corresponding gradient direction histogram.

[0019] Figure 6 This is the first method in the multimodal image registration method provided by the present invention. k Schematic diagram of the feature area corresponding to the first feature point of the scale.

[0020] Figure 7 It is an excerpt of some data sets in the multimodal image registration method provided by the present invention.

[0021] Figure 8 This is an excerpt of matching results of the multimodal image registration method provided by the present invention on 11 types of multimodal data sets.

[0022] Fig. 9 This is an excerpt of the registration result in the form of a chessboard of the multimodal image registration method provided by the present invention.

[0023] Fig.10 It is a structural schematic diagram of the multimodal image registration device provided by the present invention.

[0024] Fig.11 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0027] In the description of the present application, the terms "first", "second", etc. are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are usually a class, and the number of objects is not limited. For example, the first object can be one or more. In addition, in the description of the present application, "and / or" represents at least one of the connected objects, and the character " / " generally represents that the front and back associated objects are in an "or" relationship.

[0028] It should be noted that, with the rapid development of multi-sensor information processing technology, satellites equipped with various sensors are providing image data of the Earth's surface with unprecedented frequency and diversity. Compared with single-mode observations, multi-modal images provide more information. Making good use of these complementary data types, such as optics, synthetic aperture radar, lidar, infrared images, vision, medicine, etc., has a wide range of application value in many fields such as addressing climate change, natural disaster assessment, urban planning, medical diagnosis and other global challenges.

[0029] Multimodal image registration and multimodal image registration technology are bridges connecting multimodal data. These multimodal images are usually obtained by different sensors and different imaging mechanisms at different times to observe the same object, such as optical-infrared, optical-hyperspectral, spectral-SAR, push-broom thermal infrared-swing-broom thermal infrared, visible light-infrared, etc., so that the obtained images can be accurately aligned. This process is crucial for a variety of remote sensing applications such as image fusion, change detection, image stitching, and three-dimensional reconstruction.

[0030] The main challenges faced by multimodal image registration include spatial reference differences and nonlinear radiation differences. Spatial reference differences are mainly due to different shooting viewpoints and sensor resolutions, which lead to translation, rotation, scale changes and geometric distortion between images. Nonlinear radiation differences are caused by different imaging mechanisms and external environmental conditions, resulting in large grayscale differences between images. These differences increase the difficulty of registration. Traditional methods often fail when dealing with multimodal images with severe nonlinear radiation differences and geometric differences. Improving the accuracy and efficiency of multimodal image registration has become an urgent need.

[0031] Traditional multimodal image registration methods in related technologies generally include region-based methods, feature-based methods, and deep learning-based methods.

[0032] However, the region-based registration method has high computational cost and poor technical accuracy, and requires the acquisition of geographic information of the image to eliminate the scale, rotation and translation differences between the multimodal images to be registered in advance, and has poor applicability.

[0033] Feature-based methods need to deal with both scale differences and rotation differences. Commonly used registration methods in related technologies, such as the Scale-Invariant Feature Transform (SIFT) method, have an insufficient number of feature points to extract and weak robustness in feature description when processing multimodal images with severe geometric differences and nonlinear radiation differences. It is difficult to accurately reflect the shared information between two multimodal images, resulting in poor accuracy in multimodal image registration.

[0034] Although the deep learning-based method can improve the accuracy of multimodal image registration to a certain extent, the deep learning-based registration method requires the collection of a large number of multimodal images as sample images for model training in advance. Before model training, a large amount of manpower and time costs are required to process and annotate the sample images, resulting in low efficiency of multimodal image registration. When the number of sample images is insufficient, the accuracy of multimodal image registration is not high. In addition, the deep learning-based method is highly sensitive to the rotation differences between images of different modalities, which will further reduce the accuracy of multimodal image registration based on deep learning.

[0035] In this regard, the present invention provides a multimodal image registration method. The multimodal image provided by the present invention can improve the registration accuracy and efficiency of the multimodal image when the multimodal image to be registered has complex geometric differences and nonlinear radiation differences. When facing the multimodal images to be registered with scale differences and rotation differences, it can still match enough points of the same name and maintain a high accuracy. In the process of multimodal image registration, feature point extraction is effectively performed on the image of the multi-scale edge window Gaussian scale space, and a robust descriptor is constructed to extract feature vectors for two matchings, which can increase the number of points of the same name while ensuring the accuracy of the registration.

[0036] Combine the following Figure 1-Figure 10 The multimodal image registration method of the present invention is described.

[0037] Figure 1 This is one of the flow charts of the multimodal image registration method provided by the present invention. Figure 2 FIG. 2 is a flow chart of the multimodal image registration method provided by the present invention. Figure 1 and Figure 2 As shown, the method comprises the following steps: Step 101, performing the first registration image and the second registration image on the first image. k The edge window filter of the scale is used to obtain the first image to be registered. k The edge window filtering result image and the second image to be registered k The first image to be registered and the second image to be registered are images of the same region or the same target in different modes. , S represents the number of scales, S It is a positive integer greater than 1. Different scales result in different variances, filtering times and radius of the edge window filter window.

[0038] It should be noted that the embodiment of the present invention is implemented by a multimodal image registration device, which can be configured in electronic devices such as computers and servers.

[0039] Specifically, the first image to be registered and the second image to be registered are the registration objects of the multimodal image registration method provided by the present invention. Based on the multimodal image registration method provided by the present invention, the first image to be registered and the second image to be registered can be registered.

[0040] It is understandable that the first image to be registered and the second image to be registered in the embodiment of the present invention may be images of the same area or the same target in different modes acquired by different sensors, different imaging mechanisms or different bands. The first image to be registered and the second image to be registered in the embodiment of the present invention may be determined based on actual needs, and the first image to be registered and the second image to be registered are not limited in the embodiment of the present invention.

[0041] In an embodiment of the present invention, the first image to be registered and the second image to be registered can be obtained in a variety of ways. For example, in an embodiment of the present invention, the first image to be registered and the second image to be registered can be obtained based on user input; or, in an embodiment of the present invention, the first image to be registered and the second image to be registered sent by other electronic devices can also be received.

[0042] As an optional embodiment, the first image to be registered and the second image to be registered are respectively k The edge window filter of the scale is used to obtain the first image to be registered. k The edge window filtering result image and the second image to be registered k After obtaining the scaled edge window filtering result image, the method further comprises: performing image preprocessing on the first image to be registered and the second image to be registered, wherein the image preprocessing comprises at least one of normalization, contrast stretching and Gaussian filtering.

[0043] Specifically, in an embodiment of the present invention, in order to improve the image quality of the first image to be registered and the second image to be registered, reduce noise interference, and thereby improve the accuracy of image registration between the first image to be registered and the second image to be registered, before performing multi-scale edge window filtering on the first image to be registered and the second image to be registered, image preprocessing is performed on the first image to be registered and the second image to be registered using normalization, contrast stretching, and Gaussian filtering to obtain the first image to be registered and the second image to be registered after image preprocessing.

[0044] It should be noted that the edge area in the image contains rich visual information, and effectively obtaining the edge features in the image can better complete the registration task in the case of nonlinear radiation differences. However, whether using isotropic or anisotropic filters, although they can smooth and blur the image to a certain extent, they are not conducive to simultaneously satisfying the retention of image edge features and the removal of image noise. Therefore, it is of great significance to effectively retain the image edge feature information and construct a new scale space that can resist nonlinear radiation differences. With this as motivation, the present invention constructs a multi-scale edge window Gaussian scale space.

[0045] Side Window Filtering (SWF) is a filtering technique in image processing. It aims to minimize the blurring effect on the edge of the image while maintaining or enhancing the edge features. Traditional filtering algorithms (such as Gaussian filtering and mean filtering) usually use the full window regression method, that is, the center of the filter kernel window is placed at the position of the pixel to be filtered. In this way, when processing the edge of the image, the filter window will perform weighted calculations on the pixel values ​​on both sides of the edge when crossing the edge, causing the edge information to diffuse in the direction of the edge normal, thereby causing edge blur.

[0046] The side window filter regards each pixel to be filtered as a potential edge and generates multiple local windows around it, where the pixel to be filtered is located on one side of the above window or aligned with a corner point. According to the filtering results of the neighboring pixels in the side window of the pixel to be filtered, the side window with the smallest L2 distance of the filtering result relative to the pixel to be filtered is selected as the output result. Among them, the L2 distance is the Euclidean distance, which is a method of measuring the straight-line distance between two points or two vectors in multidimensional space.

[0047] Therefore, in the embodiment of the present invention, based on the edge window filters of different scales, the first image to be registered and the second image to be registered can be respectively subjected to edge window filtering of different scales to obtain the first image to be registered. k The edge window filtering result image and the second image to be registered k Scaled edge window filtering result image.

[0048] As an optional embodiment, the first image to be registered and the second image to be registered are respectively k The edge window filter of the scale is used to obtain the first image to be registered. k The edge window filtering result image and the second image to be registered k The edge window filter result image of the scale includes: initializing edge window filter parameters, the edge window filter parameters include initial variance, scale number, scale factor, position of each edge window filter window and the kThe radius of the side window filter window, the number of filtering times and the variance corresponding to the scale, each side window filter window includes a rectangular side window filter window with a long side in the vertical direction and a pixel point to be filtered at the midpoint of the right long side, a rectangular side window filter window with a long side in the vertical direction and a pixel point to be filtered at the midpoint of the left long side, a rectangular side window filter window with a long side in the horizontal direction and a pixel point to be filtered at the midpoint of the upper long side, a rectangular side window filter window with a long side in the horizontal direction and a pixel point to be filtered at the midpoint of the lower long side, and a rectangular side window filter window with four sides in the horizontal and vertical directions and the pixel points to be filtered are located at four A square edge window filter window for corner points, an isosceles right triangle edge window filter window with right angles located in the horizontal and vertical directions respectively and the pixel to be filtered is located at the midpoint of the lower right hypotenuse, an isosceles right triangle edge window filter window with right angles located in the horizontal and vertical directions respectively and the pixel to be filtered is located at the midpoint of the lower left hypotenuse, an isosceles right triangle edge window filter window with right angles located in the horizontal and vertical directions respectively and the pixel to be filtered is located at the midpoint of the upper right hypotenuse, and an isosceles right triangle edge window filter window with right angles located in the horizontal and vertical directions respectively and the pixel to be filtered is located at the midpoint of the upper left hypotenuse.

[0049] Specifically, considering that the Gaussian kernel and its derivative are the only possible smoothing kernels in the scale space, the embodiment of the present invention blurs the first image to be registered and the second image to be registered by edge window filtering while maintaining high-quality edge information. The present invention defines 12 edge window filtering windows in a discrete case to maximize the edge features in different directions.

[0050] Figure 3 It is a schematic diagram of the position of the side window filter window in the multimodal image registration method provided by the present invention. Figure 3 In the figure, the blue frame represents the edge window filter window, and the red dots represent the pixels to be filtered. Figure 3As shown, the side window filter window W is a rectangular side window filter window with its long side in the vertical direction and the pixel to be filtered located at the midpoint of the right long side; the side window filter window E is a rectangular side window filter window with its long side in the vertical direction and the pixel to be filtered located at the midpoint of the left long side; the side window filter window S is a rectangular side window filter window with its long side in the horizontal direction and the pixel to be filtered located at the midpoint of the upper long side; the side window filter window N is a rectangular side window filter window with its long side in the horizontal direction and the pixel to be filtered located at the midpoint of the lower long side; the side window filter window NW is a square side window filter window with its four sides in the horizontal and vertical directions respectively and the pixel to be filtered located at the lower right corner; the side window filter window NE is a square side window filter window with its four sides in the horizontal and vertical directions respectively and the pixel to be filtered located at the lower left corner; the side window filter window SW is a square side window filter window with its four sides in the horizontal and vertical directions respectively and the pixel to be filtered located at the lower left corner The filter window DSW is an isosceles right triangle side window filter window with its right angles located in the horizontal and vertical directions and the pixel to be filtered is located at the midpoint of the upper right hypotenuse; the filter window SE is a square side window filter window with its four sides in the horizontal and vertical directions and the pixel to be filtered is located at the upper left corner; the filter window DNW is an isosceles right triangle side window filter window with its right angles located in the horizontal and vertical directions and the pixel to be filtered is located at the midpoint of the lower right hypotenuse; the filter window DNE is an isosceles right triangle side window filter window with its right angles located in the horizontal and vertical directions and the pixel to be filtered is located at the midpoint of the lower left hypotenuse; the filter window DSW is an isosceles right triangle side window filter window with its right angles located in the horizontal and vertical directions and the pixel to be filtered is located at the midpoint of the upper right hypotenuse; the filter window DSE is an isosceles right triangle side window filter window with its right angles located in the horizontal and vertical directions and the pixel to be filtered is located at the midpoint of the upper left hypotenuse.

[0051] It should be noted that the initial variance, scale number, proportional factor and the first k The radius of the edge window filter window corresponding to the scale can be determined based on prior knowledge and / or actual conditions. k The specific value of the edge window filter window radius corresponding to the scale is not limited.

[0052] Optionally, in the embodiment of the present invention, the initial variance of the edge window filter is determined based on prior knowledge and / or actual conditions. is 1.6; the scaling factor is , the number of scales There are 4 scales, namely the first scale, the second scale, the third scale and the fourth scale; the edge window filter window radius corresponding to the first scale is 3, and the variance corresponding to the first scale is (initial variance), the number of filtering times is 1; the radius of the side window filter window corresponding to the second scale is 5, and the variance corresponding to the second scale is , the number of filtering times is 2; the edge window filter window radius corresponding to the third scale is 8, and the variance is , the number of filtering times is 3; the radius of the edge window filter window corresponding to the fourth scale is 11, and the variance is , the number of filtering times is 4. Variance corresponding to the scale It can be calculated by the following formula:

[0053] It should be noted that, for a square side window filter window, the side window filter window radius refers to the side length of the square side window filter window; for a rectangular side window filter window, the side window filter window radius refers to the short side length of the rectangular side window filter window; for an isosceles right triangle side window filter window, the side window filter window radius refers to half of the right angle side of the isosceles right triangle side window filter window.

[0054] Based on the filtering parameters, the first image to be registered and the second image to be registered are respectively k The edge window filter of the scale is used to obtain the first image to be registered. k The edge window filtering result image and the second image to be registered k Scaled edge window filtering result image.

[0055] Each pixel in the image to be registered is used as a pixel to be filtered. For each pixel to be filtered in the image to be registered, the first k The Gaussian kernels in the horizontal, vertical and diagonal directions of the edge window filtering corresponding to the scale are used, and then the edge window filtering in different directions (including horizontal, vertical and diagonal directions) is performed on each pixel to be filtered to obtain the first k The edge window filtering result of each pixel to be filtered is k The edge window filtering result of the scale is used to obtain the image to be registered. k The image to be registered includes a first image to be registered and a second image to be registered.

[0056] The formula for edge window filtering can be defined as:

[0057] in, Used to indicate the position of the edge window filter window; Represents the pixel to be filtered Centered on Determine the edge filter window; Represents pixels; represents the Gaussian kernel weight, which needs to be normalized in each edge window filter window; Indicates that the image to be registered is at position The pixel value of Indicates that the image to be registered is at position The pixel value of .

[0058] Step 102: based on the first image to be registered k The edge window filtering result image of the scale and the second image to be registered k The edge window filtering result image of the scale is obtained, and the first feature point corresponding to the first image to be registered and the feature vector corresponding to the first feature point are obtained, as well as the second feature point corresponding to the second image to be registered and the feature vector corresponding to the second feature point are obtained.

[0059] Specifically, obtain the first image to be registered k The edge window filtering result image of the scale and the second image to be registered k After filtering the resulting image with a small edge window, the first image to be registered can be k The edge window filtering result image of the scale is used to obtain the first image to be registered corresponding to the first image through numerical calculation, mathematical statistics and deep learning technology. k The first characteristic point and the k The feature vector corresponding to the first feature point of the second image to be registered can also be calculated based on the first feature point of the second image to be registered. k The edge window filtering result image of the scale is used to obtain the first image corresponding to the second image to be registered through numerical calculation, mathematical statistics and deep learning technology. k The second characteristic point and the k The eigenvector corresponding to the second feature point of the scale.

[0060] As an optional embodiment, based on the first image to be registered k The edge window filtering result image of the scale and the second image to be registered k The first feature point and the feature vector corresponding to the first feature point corresponding to the first image to be registered are obtained, as well as the second feature point and the feature vector corresponding to the second feature point corresponding to the second image to be registered are obtained, including: respectively performing the first feature point and the feature vector corresponding to the second feature point of the first image to be registered. k The edge window filtering result image of the scale and the second image to be registered k The edge window filtering result image of the scale is used for edge extraction to obtain the first image to be registered. k The edge feature map of the second image to be registered k Edge feature map of scale.

[0061] It should be noted that the Sobel algorithm is a gradient-based edge detection algorithm that detects edges by calculating the gradient amplitude and direction of each pixel in the image. The Sobel operator contains two sets of 3x3 matrices, namely horizontal (Gx) and vertical (Gy), which are used to calculate the approximate brightness difference of the image in the horizontal and vertical directions. The Canny algorithm is a multi-stage edge detection algorithm that combines multiple steps such as Gaussian smoothing, gradient calculation, non-maximum suppression, dual threshold processing and edge connection to obtain accurate edge detection results. Edge extraction based on the Canny algorithm can obtain more refined edges to obtain more feature points, while edge extraction based on the Canny algorithm can detect the edges with the most significant features.

[0062] Therefore, for each scale feature map corresponding to the first image to be registered and the second image to be registered k In the embodiment of the present invention, based on the actual situation, the Sobel algorithm or the Canny algorithm can be used to extract the edge of the above feature map to obtain the first image to be registered. k The edge feature map of the second image to be registered k Edge feature map of scale.

[0063] Optionally, in the embodiment of the present invention, the Sobel algorithm is used to extract the edge of the feature map to obtain the first image to be registered. k The edge feature map of the second image to be registered k Edge feature map of scale.

[0064] The definition of the Sobel operator is as follows:

[0065] in, Represents the x-direction component of the Sobel operator; Represents the y-direction component of the Sobel operator.

[0066] Use the above feature map and Convolution can obtain the gradient maps of the above feature maps in the x and y directions respectively, and then obtain the gradient amplitude map b corresponding to the above feature maps.

[0067] Setting the edge threshold ,in represents the average value, Set it to 0.4. The gradient amplitude map b corresponding to the above feature map is greater than the edge threshold The pixels of are set to 1 and the rest are set to 0, and then non-maximum suppression is performed to retain the local maximum value to obtain finer and clearer edges, thereby obtaining the first image to be registered.k The scaled binary edge feature map and the second image to be registered k Scaled binary edge feature map.

[0068] For the first image to be registered k The edge feature map of the second image to be registered k The edge feature maps of different scales are used to detect corner points respectively, and the first image to be registered corresponding to the first image is obtained. k The first original feature point of the second image to be registered corresponds to the first k The second original feature point of the scale.

[0069] It should be noted that Harris corners refer to points where the brightness of an image changes dramatically or points with the maximum curvature on an edge curve, and are one of the important features of an image. The core idea of ​​the Harris corner detection algorithm is to detect corners by calculating the local grayscale changes of an image. The main steps of the algorithm include: Calculating the change in pixel value: When the window moves on the image, calculate the change in the pixel value in the window. This is usually achieved by calculating the gradient of the pixels in the window. Constructing a covariance matrix: Constructing a covariance matrix using the gradient information of the pixels in the window. This matrix reflects the second-order statistical characteristics of the grayscale changes of the pixels in the window. Calculating the corner response value: Calculate the corner response value R based on the eigenvalue of the covariance matrix. When R is greater than the set threshold, the point is considered to be a corner. The size and distribution of the eigenvalue determine whether the point is a corner, edge point, or flat area point.

[0070] Harris corner points are less affected by intensity and scale differences and have high computational efficiency. Harris corner points with strong responses can be considered as feature points with obvious structure and stability between multi-source images. Therefore, in the embodiment of the present invention, the first image to be registered is obtained. k The scaled binary edge feature map and the second image to be registered k After the edge feature map is scaled and binarized, the first image to be registered can be detected based on the Harris corner detection algorithm. k The scaled binary edge feature map and the second image to be registered k The scaled binary edge feature map is used for corner detection, and the first image to be registered is k The Harris corner point detected in the scaled binary edge feature map is determined as the first image to be registered corresponding to the k The first original feature point of the scale is determined as the Harris corner point detected in the edge feature map of each scale binarization corresponding to the second image to be registered. k The second original feature point of the scale.

[0071] It can be understood that the first image to be registered corresponds to the k The number of the first original feature points of the scale is multiple. k The scale is the number of original feature points.

[0072] Edge feature map The Harris corner response value R of any pixel in can be defined by the following formula:

[0073] in, and denote the determinant and trace of the matrix respectively; represents an empirical constant; and Respectively represent the edge feature map along x and y Directional gradient; u and v Respectively represent the coordinate values ​​of the above pixel points; The variance is When the Harris corner point response value R of any pixel point is greater than the set threshold, the pixel point is considered to be a Harris corner point.

[0074] Based on the non-maximum suppression algorithm, the first image to be registered corresponds to the k The first original feature points of the scale are screened to obtain the first image to be registered corresponding to the first k The first feature point of the scale corresponds to the first k The second original feature points of the scale are screened to obtain the first k The second characteristic point of the scale.

[0075] It should be noted that an important difficulty in multimodal image registration is that the size and scale relationship between the images to be registered are diverse. Therefore, in the feature point detection link, the multimodal image registration method provided by the present invention uses a non-maximum suppression algorithm to detect the first image to be registered. k The first original feature point of the second image to be registered corresponds to the first k The second original feature points of the scale are screened and the first original feature points corresponding to the first image to be registered are removed. k The first original feature point of the second image to be registered corresponds to the first k After the redundant feature points and noise points are gathered from the second original feature points of the scale, the remaining first image to be registered corresponding to the kThe first original feature point of the scale is determined as the first image to be registered corresponding to the k The first feature point of the scale is the first feature point of the remaining second image to be registered. k The second original feature point of the scale is determined as the first feature point corresponding to the second image to be registered k The second characteristic point of the scale.

[0076] The embodiment of the present invention uses a square coverage suppression method to achieve fast and uniform feature point distribution by covering a square area on the original feature points. The core of this method is to use binary search to determine the size of the coverage area and suppress non-maximum points by covering these areas. The original feature points include the first image to be registered corresponding to the first image. k The first original feature point of the second image to be registered corresponds to the first k The second original feature point of the scale.

[0077] It should be noted that, in the embodiments of the present invention, Indicates the first image to be registered corresponding to the k The first original feature point set of scale is Indicates the first image to be registered corresponding to k The second original feature point set of scale.

[0078] Based on the non-maximum suppression algorithm, the first image to be registered corresponds to the k The first original feature point set of scale The specific steps of screening include: k The first original feature point set of scale The pixel values ​​of each first original feature point in the image are ordered from large to small, and the first image to be registered corresponds to the first k The first original feature point set of scale Sort the first original feature points in .

[0079] In order to accelerate the convergence of the algorithm, according to the size of the first image to be registered, the first image to be registered corresponding to the k The first original feature point set of scale The number of the first original feature points, the number of the first feature points expected to be output, and the tolerance can be used to pre-calculate the upper and lower boundaries of the binary search using the following formula: , :

[0080] in, and Respectively represent the width and height of the first image to be registered; Indicates the first image to be registered corresponding to the desired output k The number of first characteristic points of the scale; Indicates the first image to be registered corresponding to the k The first original feature point set of scale The number of the first original feature points in .

[0081] At each step of the binary search, set the grid The resolution is ,in represents the current suppressed edge length. Then, according to the first image to be registered corresponding to the k The first original feature point set of scale The arrangement order of the first original feature points in the first image to be registered is traversed. k The first original feature point set of scale For each original feature point in the first image to be registered, k The first original feature point set of scale Any first original feature point in , check the first original feature point The grid Is it marked as covered? If the first original feature point The grid is not marked as covered, then the first original feature point Add to the first image to be registered corresponding to the k The first original feature point set of scale , and the first original feature point is the center and the side length is The grid in the square area Mark as covered, Take 1, 2, 3, ..., .

[0082] Change The above process is repeated until the first image to be registered corresponds to the k The first original feature point set of scale The number of the first feature points in Or meet other termination conditions, and finally obtain the uniformly distributed first image to be registered corresponding to the first k The first feature point set of the scale .

[0083] It should be noted that based on the non-maximum suppression algorithm, the firstk The second original feature point set of scale Screening is performed to obtain the first k The first original feature point set of scale The specific steps are the same as the first image to be registered corresponding to the first image k The first original feature point set of scale The specific steps for screening are the same and will not be repeated in the embodiments of the present invention.

[0084] Based on k The first feature point of the scale and the first image to be registered k The edge window filtering result image of the scale is obtained k The feature vector corresponding to the first feature point of the scale is based on the k The second feature point of the second scale and the second image to be registered k The edge window filtering result image of the scale is obtained k The eigenvector corresponding to the second feature point of the scale.

[0085] Specifically, obtain the first image to be registered corresponding to the first k The first feature point of the scale and the first feature point of the second image to be registered k After the second feature point of the scale, we can k The first feature point of the scale and the first image to be registered k The edge window filtering result image of the scale is obtained through numerical calculation, mathematical statistics and deep learning technology. k The feature vector corresponding to the first feature point of the scale can also be based on the first feature point corresponding to the second image to be registered. k The second feature point of the second scale and the second image to be registered k The edge window filtering result image of the scale is obtained through numerical calculation, mathematical statistics and deep learning technology. k The eigenvector corresponding to the second feature point of the scale.

[0086] As an optional embodiment, based on k The first feature point of the scale and the first image to be registered k The edge window filtering result image of the scale is obtained k The feature vector corresponding to the first feature point of the scale is based on the k The second feature point of the second scale and the second image to be registered k The edge window filtering result image of the scale is obtained k The feature vector corresponding to the second feature point of the first image to be registered is k The edge window filtering result image and the second image to be registered kThe edge window filtering result image of the scale is j The controllable Gaussian filter in the direction is used to obtain the first image to be registered. k Scale j The controllable Gaussian filter result image and the second image to be registered k Scale j The result image of the controllable Gaussian filter in the direction, , Indicates the number of directions, is a positive integer greater than 1. The rotation angle of the Gaussian filter window for Gaussian filtering is different for different directions.

[0087] It should be noted that, in traditional Gaussian filtering, the Gaussian filter window is usually square, and the four sides of the square Gaussian filter window are respectively located in the horizontal and vertical directions.

[0088] However, in order to capture feature information in more directions in the feature map, in an embodiment of the present invention, a multi-directional controllable Gaussian filter (steerable filter) is implemented by adjusting the rotation angle of the Gaussian filter window to obtain feature information in different directions in the feature map.

[0089] In the embodiment of the present invention, the rotation angle of the Gaussian filter window may include but is not limited to 0°, 30°, 60°, 90°, 120° and 150°.

[0090] Optionally, in the embodiment of the present invention, the feature map of each scale corresponding to the first image to be registered can be subjected to controllable Gaussian filtering in six directions (i.e. ), are the first direction, the second direction, the third direction, the fourth direction, the fifth direction and the sixth direction respectively; the rotation angle of the Gaussian filter window corresponding to the first direction is is 0°, the rotation angle of the Gaussian filter window corresponding to the second direction The rotation angle of the Gaussian filter window corresponding to the third direction is 30° The rotation angle of the Gaussian filter window corresponding to the fourth direction is 60° The rotation angle of the Gaussian filter window corresponding to the 5th direction is 90° The rotation angle of the Gaussian filter window corresponding to the sixth direction is 120° is 150°.

[0091] The Gaussian function and its higher-order partial derivatives are proven to be directionally controllable, with a variance of Gaussian function And its first-order Gaussian filter kernel at 0° and 90° and Can be expressed as:

[0092] in, and They respectively represent the coordinates of the pixels to be filtered.

[0093] For the first image to be registered Scaled edge window filtering result image , for the first image to be registered Scaled edge window filtering result image By performing Gaussian filtering in the first direction, the first image to be registered can be obtained. Scaled edge window filtering result image Gaussian filtering result in the first direction and It can be calculated by the following formula

[0094] in, Indicates Scaled edge window filtering result image; Represents convolution calculation; Represents the Gaussian filter variance corresponding to the first direction.

[0095] Through linear combination, the first image to be registered can be obtained Scale Directional controllable Gaussian filtering results :

[0096] in, represents the number of scales, Indicates the number of directions of Gaussian filtering; represents the scale factor; Indicates The rotation angle of the Gaussian filter window corresponding to the direction. Set to 6, Set to 0.8.

[0097] Based on the first image to be registered Scale Directional controllable Gaussian filtering results , the first image to be registered can be obtained k Scale j Directionally controllable Gaussian filtering result image

[0098] The first image to be registered k The controllable Gaussian filtering result images in all directions of the scale are superimposed to obtain the first image to be registered. k The first-order gradient map of the scale is k The controllable Gaussian filtering result images in all directions of the scale are superimposed to obtain the second image to be registered. k First-order gradient map of the scale.

[0099] Specifically, obtain the first image to be registered k Scale j After the controllable Gaussian filtering result image is obtained, the first image to be registered can be obtained by the following formula: The controllable Gaussian filtering result images in all directions of the scale are superimposed to obtain the first image to be registered. First-order gradient map of scale .

[0100]

[0101] Based on the first image to be registered k The first-order gradient map of the scale is obtained to obtain the first image to be registered. k The second-order gradient amplitude map and the second-order gradient phase map of the second image to be registered are k The first-order gradient map of the scale is obtained to obtain the second image to be registered. k Scaled second-order gradient magnitude map and second-order gradient phase map.

[0102] Specifically, obtain the first image to be registered k After the first-order gradient map of the scale is obtained, the first image to be registered can be k The first-order gradient map of the scale is obtained by Sobel operator to obtain the first image to be registered. k Scaled second-order gradient magnitude map and second-order gradient phase map.

[0103] It should be noted that, considering that multimodal images often have a gradient reversal phenomenon, in the embodiment of the present invention, the phase of the second-order gradient phase image corresponding to the first image to be registered is unified at Inside.

[0104] Figure 4 The first image to be registered in the multimodal image registration method provided by the present invention is k Schematic diagram of the second-order gradient phase image of the first image to be registered. k The second-order gradient phase image of the scale is as follows Figure 4 shown.

[0105] Based on the first image to be registeredk The second-order gradient amplitude map and second-order gradient phase map of the scale are obtained. k The feature vector corresponding to the first feature point of the second image to be registered is k The second-order gradient amplitude map and second-order gradient phase map of the scale are obtained. k The eigenvector corresponding to the second feature point of the scale.

[0106] Specifically, based on the first image to be registered k The second-order gradient amplitude map and second-order gradient phase map of the scale can be obtained by numerical calculation, quantitative statistics, etc. k The feature vector corresponding to the first feature point of the second image to be registered is k The second-order gradient amplitude map and second-order gradient phase map of the scale can also be obtained by numerical calculation, quantitative statistics, etc. k The eigenvector corresponding to the second feature point of the scale.

[0107] As an optional embodiment, based on the first image to be registered k The second-order gradient amplitude map and second-order gradient phase map of the scale are obtained. k The feature vector corresponding to the first feature point of the second image to be registered is k The second-order gradient amplitude map and second-order gradient phase map of the scale are obtained. k The feature vector corresponding to the second feature point of the first image to be registered includes: k In the second-order gradient amplitude map of the scale, determine the k The first target area corresponding to the first feature point of the second image to be registered k In the second-order gradient amplitude map of the scale, determine the k The first target area corresponds to the second feature point of the scale.

[0108] For k Gaussian filtering is performed on the first original area corresponding to the first feature point of the scale to obtain the k The first target area corresponding to the first feature point of the scale k The first original area corresponding to the second feature point of the scale is Gaussian filtered to obtain the k The first target area corresponds to the second feature point of the scale.

[0109] Based on the first image to be registered k The second-order gradient phase image of the scale is obtained k The gradient amplitude value of the first target area corresponding to the first feature point of the scale in each first phase interval, and then the first phase interval with the largest gradient amplitude value is determined as the first phase interval. kThe first phase interval corresponding to the first feature point of the scale is based on the first phase interval of the second image to be registered. k The second-order gradient phase image of the scale is obtained k The gradient amplitude value of the first target area corresponding to the second feature point of the scale in each first phase interval, and then the first phase interval with the largest gradient amplitude value is determined as the first phase interval. k The first phase interval corresponding to the second characteristic point of the scale.

[0110] Based on k The first phase interval corresponding to the first characteristic point of the scale is determined k The direction of the eigenvector corresponding to the first feature point of the scale is based on the k The first phase interval corresponding to the second characteristic point of the scale is determined k The direction of the eigenvector corresponding to the first feature point of the scale.

[0111] In the first image to be registered k In the second-order gradient amplitude map of the scale, determine the k The feature area corresponding to the first feature point of the second image to be registered k In the second-order gradient amplitude map of the scale, determine the k The feature area corresponding to the second feature point of the scale.

[0112] Based on the first image to be registered k The second-order gradient phase image of the scale is obtained k The gradient amplitude value of the feature area corresponding to the first feature point of the scale in each second phase interval is used as the first k The eigenvalue of the eigenvector corresponding to the first feature point of the scale is calculated based on the first feature point of the second image to be registered. k The second-order gradient phase image of the scale is obtained k The gradient amplitude value of the feature area corresponding to the second feature point of the scale in each second phase interval is used as the first k The eigenvalue of the eigenvector corresponding to the second feature point of the scale.

[0113] In the first image to be registered k In the second-order gradient amplitude map of the scale, the k The first characteristic point of the scale As the center of the circle, the radius is The circular area is determined as k The first characteristic point of the scale The corresponding first original area.

[0114] Based on the variance Gaussian check of k The first characteristic point of the scale The first original area corresponding to the first k The first characteristic point of the scale Gradient direction information of k The first characteristic point of the scale The corresponding first target area.

[0115] Wherein, A and B are positive integers greater than 0, and A is greater than B. The specific values ​​of A and B can be determined based on prior knowledge and / or actual conditions. The specific values ​​of A and B are not limited in the embodiments of the present invention.

[0116] Optionally, the value of A is 6; the value of B is 2.

[0117] It should be noted that in the embodiments of the present invention, Divide into The first phase intervals, the step length of each first phase interval is .in, It can be a positive integer determined based on prior knowledge and / or actual conditions.

[0118] Optionally, , accordingly, the step length of each first phase interval is 10°.

[0119] Based on the first image to be registered k The second-order gradient phase image of the scale can be obtained k The first characteristic point of the scale The gradient amplitude value of the corresponding first target area in each first phase interval.

[0120] Based on the Gaussian filter k The first characteristic point of the scale The corresponding first target area in each first phase interval has a gradient amplitude value, and generates the first k The first characteristic point of the scale The corresponding gradient direction histogram.

[0121] Figure 5 This is the first method in the multimodal image registration method provided by the present invention. k The first characteristic point of the scale The corresponding gradient direction histogram. k The first characteristic point of the scale The corresponding gradient direction histogram is as follows Figure 5 shown.

[0122] Based on k The first characteristic point of the scale The corresponding gradient direction histogram is used to determine the first phase interval with the largest gradient amplitude value as the first kThe first characteristic point of the scale The corresponding first phase interval can then be based on the k The first characteristic point of the scale The first phase interval corresponding to the k The first characteristic point of the scale The corresponding feature vector direction, the descriptor is rotated to the main direction to eliminate the rotation difference.

[0123] In the first image to be registered k In the second-order gradient amplitude map of the scale, the k The first characteristic point of the scale is the center of the circle and the larger radius is The smaller radius is The circular area is determined as k The first characteristic point of the scale The corresponding second target area will be k The first characteristic point of the scale is the center of the circle and the largest radius is , the smaller radius is E The circular area is determined as k The first characteristic point of the scale The corresponding third target area will be k The first characteristic point of the scale is the center of the circle and the radius is E The circular area is determined as k The first characteristic point of the scale The corresponding fourth target region defines the positive rotation direction with the image coordinate system as reference, and resists scale differences through descriptors of different radii.

[0124] Wherein, C, D and E are positive integers greater than 0, and C>D>E. The specific values ​​of C, D and E can be determined based on prior knowledge and / or actual conditions. The specific values ​​of C, D and E are not limited in the embodiments of the present invention.

[0125] Optionally, the value of C is 12, the value of D is 9, and the value of E is 2.

[0126] The first k The first characteristic point of the scale The corresponding second target area and the k The first characteristic point of the scale The perpendicular midline of the corresponding third target area is determined as the starting position, and the first k The first characteristic point of the scale The corresponding second target area is divided into The second target sub-region kThe first characteristic point of the scale The corresponding third target area is divided into a third target sub-region.

[0127] It should be noted that the preset angle in the embodiment of the present invention may be determined based on prior knowledge and / or actual conditions. The specific value of the preset angle is not limited in the embodiment of the present invention.

[0128] Optionally, the value of the preset angle may be 30°. .

[0129] The first k The first characteristic point of the scale The corresponding second target area The second target sub-area, k The first characteristic point of the scale The corresponding third target area The third target sub-area and k The first characteristic point of the scale The corresponding fourth target area is determined as k The first characteristic point of the scale The corresponding feature area.

[0130] Figure 6 This is the first method in the multimodal image registration method provided by the present invention. k Schematic diagram of the feature area corresponding to the first feature point of the scale. k The first characteristic point of the scale The corresponding feature areas are Figure 6 shown.

[0131] It should be noted that in the embodiments of the present invention, Divide into The second phase intervals, the step length of each first phase interval is .in, It can be a positive integer determined based on prior knowledge and / or actual conditions.

[0132] Optionally, , accordingly, the step length of each first phase interval is 30°.

[0133] Based on the first image to be registered k The second-order gradient phase image of the scale can be obtained k The first characteristic point of the scale The corresponding gradient amplitude value of each feature area in each second phase interval is taken as the first k The first characteristic point of the scale The eigenvalues ​​of the corresponding eigenvectors.

[0134] No. k The first characteristic point of the scale The dimension of the corresponding feature vector .

[0135] Optionally, in , In the case of k The first characteristic point of the scale The corresponding feature vector has a dimension of 200.

[0136] It should be noted that based on k The second feature point of the second scale and the second image to be registered k The edge window filtering result image of the scale is obtained k The specific steps of the feature vector corresponding to the second feature point of the scale are similar to those based on the k The first feature point of the scale and the first image to be registered k The edge window filtering result image of the scale is obtained k The specific steps of obtaining the feature vector corresponding to the first feature point of the scale are the same as those in the embodiment of the present invention and will not be repeated herein.

[0137] Step 103: Based on the feature vector corresponding to the first feature point and the feature vector corresponding to the second feature point, feature matching is performed on the first feature point and the second feature point, and the first feature point and the second feature point that are successfully matched are determined as a first feature point pair with the same name.

[0138] Specifically, in the embodiment of the present invention, each first feature point can be traversed, and based on the feature vector corresponding to each first feature point and the feature vector corresponding to each second feature point, the cosine similarity and the nearest neighbor ratio between each first feature point and each second feature point can be calculated.

[0139] For any first feature point, after determining the second feature point with the smallest cosine similarity and nearest neighbor ratio with the first feature point, if the cosine similarity and nearest neighbor ratio between the second feature point and the first feature point are not greater than a predefined threshold, it can be determined that the second feature point and the first feature point are successfully matched, and the first feature point and the second feature point can be determined as a first feature point pair with the same name.

[0140] It can be understood that there are multiple first feature point pairs with the same name.

[0141] After traversing each first feature point, the third set of feature points with the same name can be obtained. , Indicates the third pair of feature points with the same name The number of first pairs of feature points with the same name in .

[0142] For the third set of feature points with the same name The first pair of feature points with the same name in is used to remove outliers using the fast sample consistency method (FSC) of the affine transformation model. This time, the error threshold is set to 10 pixels to roughly estimate the transformation model parameters.

[0143] Eliminate the third feature point pair with the same name After removing the outliers in the third feature point pair set with the same name, The first set of feature point pairs with the same name is determined in .

[0144] Step 104: based on the first pair of feature points with the same name, obtain a first affine transformation matrix between the first image to be registered and the second image to be registered; based on the first affine transformation matrix, perform feature matching on the first feature point and the second feature point again to obtain a second pair of feature points with the same name.

[0145] Specifically, after obtaining the first pair of feature points with the same name, the first affine transformation matrix between the first image to be registered and the second image to be registered can be obtained by numerical calculation based on the correspondence between the first feature point and the second feature point in the first pair of feature points with the same name.

[0146] After obtaining the first affine transformation matrix between the first image to be registered and the second image to be registered, feature matching can be performed again on the first feature points and the second feature points based on the first affine transformation matrix through numerical calculation, mathematical statistics, and deep learning technology to obtain a second pair of feature points with the same name.

[0147] As an optional embodiment, based on the first affine transformation matrix, feature matching is performed again on the first feature point and the second feature point to obtain a second pair of feature points with the same name, including: based on the first affine transformation matrix, mapping the first feature point to the second image to be registered.

[0148] Specifically, based on the position of the first feature point in the first image to be registered and the first affine transformation matrix between the first image to be registered and the second image to be registered, the first feature point can be mapped to the second image to be registered, thereby obtaining the mapping point of the first feature point in the second image to be registered.

[0149] A target number of second feature points closest to the mapping point of the first feature point in the second image to be registered are determined as candidate points of the same name for the first feature point.

[0150] After obtaining the mapping point of the first feature point in the second image to be registered, a target number of second feature points in the second image to be registered that are closest to the mapping point of the first feature point can be determined as candidate points of the same name for the first feature point.

[0151] It should be noted that the target quantity in the embodiment of the present invention may be determined based on prior knowledge and / or actual conditions.

[0152] Optionally, the target number may be set to 3, that is, the number of candidate points of the same name for the first feature point may be three.

[0153] Based on the target distance between the first feature point and the candidate homonymous point of the first feature point, the first feature point and the candidate homonymous point of the first feature point having the smallest target distance to the first feature point are determined as a pair of second homonymous feature points.

[0154] Specifically, considering that the offset directions of adjacent feature points should be substantially the same, in the embodiment of the present invention, the target distance is redefined in the secondary matching stage to perform feature point matching.

[0155] In the embodiment of the present invention, any two points in the image Point and Target distance between points , defined as the combination of the position distance and feature distance between any two points mentioned above, the specific calculation formula is as follows:

[0156] in, represents the cosine similarity, and Respectively Point and The location of the point in the image, and Respectively Point and The eigenvector of a point, represents a predefined transformation model, Represents the parameters of the transformation model.

[0157] Based on the above formula, the target distance between the first feature point and the candidate same-name point of the first feature point can be calculated.

[0158] Based on the target distance between the first feature point and the candidate points with the same name as the first feature point, the candidate points with the same name as the first feature point having the smallest target distance to the first feature point are determined as a pair of second feature points with the same name.

[0159] Specifically, after calculating the target distance between the first feature point and the candidate homonymous points of the first feature point, the candidate homonymous point with the smallest target distance to the first feature point among the candidate homonymous points of the first feature point can be determined together with the first feature point as a pair of second homonymous feature points.

[0160] After traversing each first pair of feature points with the same name, the fourth set of feature points with the same name can be obtained. , Indicates the fourth pair of feature points with the same name The number of second pairs of feature points with the same name in .

[0161] For the fourth set of feature points with the same name The second pair of feature points with the same name in is used to remove outliers using the fast sample consistency method (FSC) of the affine transformation model. This time the error threshold is set to 3 pixels to roughly estimate the transformation model parameters.

[0162] Eliminate the fourth set of feature point pairs with the same name After removing the outliers in the fourth feature point pair set with the same name, The second set of feature point pairs with the same name is determined .

[0163] Step 105 : Based on the first feature point pair with the same name and the second feature point pair with the same name, align the first image to be registered and the second image to be registered.

[0164] Specifically, after obtaining the first feature point pair with the same name and the second feature point pair with the same name, the first image to be registered and the second image to be registered can be registered based on the first feature point pair with the same name and the second feature point pair with the same name through numerical calculation, mathematical statistics and the like.

[0165] After the first image to be registered and the second image to be registered are registered, the registration effect of the first image to be registered and the second image to be registered can be quantitatively evaluated based on the root mean square error of the first pair of feature points with the same name and the second pair of feature points with the same name and the number of matches between the first pair of feature points with the same name and the second pair of feature points with the same name. A line diagram of the points with the same name and a registration diagram in the form of a checkerboard can also be drawn to qualitatively test the registration effect of the first image to be registered and the second image to be registered.

[0166] In the embodiment of the present invention, after performing multi-scale edge window filtering on the first image to be registered and the second image to be registered respectively to obtain the multi-scale edge window filtering result image of the first image to be registered and the multi-scale edge window filtering result image of the second image to be registered, based on the edge window filtering result image, the first feature point corresponding to the first image to be registered and the feature vector corresponding to the first feature point and the second feature point corresponding to the second image to be registered and the feature vector corresponding to the second feature point are obtained; based on the first feature point and the feature vector corresponding to the first feature point and the feature vector corresponding to the second feature point, the first feature point and the second feature point are feature matched twice to obtain the first feature point pair with the same name and the second feature point pair with the same name; then, based on the first feature point pair with the same name and the second feature point pair with the same name, the first image to be registered and the second image to be registered are registered; through the multi-scale edge window filtering, the image edge features are effectively retained, the robustness of the multimodal image registration is enhanced; through the two-stage matching strategy, the accuracy and efficiency of the multimodal image registration are improved, and an effective solution is provided for the automatic processing and analysis of remote sensing images.

[0167] In order to verify the registration effect of the multimodal image registration method provided by the present invention, the present invention selected quantitative and qualitative experiments on 11 types of multimodal image datasets in three fields.

[0168] Figure 7 It is an excerpt of some data sets in the multimodal image registration method provided by the present invention. For each image pair, the success rate (SR), the root mean square error (RMSE), the number of correctly matched pairs of homonymous points (NCM) and the average running time (MT) are used to perform quantitative tests. Figure 8 It is one of the registration results of the multimodal image registration method provided by the present invention. Fig. 9 This is the second registration result of the multimodal image registration method provided by the present invention. The registration result of the multimodal image registration based on the multimodal image registration method provided by the present invention is as follows: Figure 7 and Fig. 9 shown.

[0169] The multimodal image registration method provided by the present invention is named MSG, and is compared with several optimal image registration methods in the related art (6 methods based on manual features, SIFT, OS-SIFT, RIFT, CoFSM, HOWP, POS-GIFT, and 2 methods based on deep learning, SuperPoint+SuperGlue, LoFTR), wherein the comparison results on the remote sensing data set are shown in Table 1. As can be seen from Table 1, for all image pairs, MSG can obtain more correct same-name points and achieve a smaller RMSE. The experimental results show that on the remote sensing data set, the NCM of the present invention is more than 3 times higher than the algorithms except POS-GIFT, with an RMSE of 1.69 pixels. MT ranks third among the feature-based algorithms, and has scale and rotation invariance, and can handle complex geometric differences and nonlinear radiation differences.

[0170] Table 1 Quantitative comparison of experimental results of nine algorithms on remote sensing datasets

[0171] The multimodal image registration method provided by the present invention constructs a multi-scale edge window Gaussian filter scale space, retains edge information to a great extent while blurring the image, performs corner point detection on a binary edge map, increases the repetition rate of feature points, uses the robustness of the second-order gradient enhancement descriptor combined with Gaussian controllable filtering, fully utilizes densely distributed edge feature points by limiting the secondary matching of the search space, and further improves the registration performance.

[0172] Experimental results show that the multimodal image registration method provided by the present invention performs well in processing multimodal remote sensing image registration tasks, and can obtain enough same-name points under complex edge features, texture features, noise, distortion or other irrational conditions, achieving high registration accuracy and high time efficiency. Therefore, the present invention not only improves the performance of multimodal remote sensing image registration, but also shows good application potential in practice, providing an effective solution for the automated processing and analysis of remote sensing images.

[0173] Fig.10 Schematic diagram of the structure of the multimodal image registration device provided by the present invention. Fig.10 The multimodal image registration device provided by the present invention is described. The multimodal image registration device described below and the multimodal image registration method provided by the present invention described above can be referred to each other. Fig.10 As shown, the device includes: an edge window filtering module 1001, a feature extraction module 1002, a first matching module 1003, a second matching module 1004 and an image registration module 1005.

[0174] The edge window filtering module 1001 is used to perform the first image to be registered and the second image to be registered. k The edge window filter of the scale is used to obtain the first image to be registered. k The edge window filtering result image and the second image to be registered k The first image to be registered and the second image to be registered are images of the same region or the same target in different modes. , S represents the number of scales, S It is a positive integer greater than 1. Different scales result in different variances, filtering times and radius of the edge window filter window.

[0175] The feature extraction module 1002 is used to extract the first image to be registered based on the first image to be registered. k The edge window filtering result image of the scale and the second image to be registered k The edge window filtering result image of the scale is obtained, and the first feature point corresponding to the first image to be registered and the feature vector corresponding to the first feature point are obtained, as well as the second feature point corresponding to the second image to be registered and the feature vector corresponding to the second feature point are obtained.

[0176] The first matching module 1003 is used to perform feature matching on the first feature point and the second feature point based on the feature vector corresponding to the first feature point and the feature vector corresponding to the second feature point, and determine the first feature point and the second feature point that are successfully matched as a first feature point pair with the same name.

[0177] The second matching module 1004 is used to obtain a first affine transformation matrix between the first image to be registered and the second image to be registered based on the first pair of feature points with the same name, and perform feature matching on the first feature point and the second feature point again based on the first affine transformation matrix to obtain a second pair of feature points with the same name.

[0178] The image registration module 1005 is used to register the first image to be registered and the second image to be registered based on the first feature point pair with the same name and the second feature point pair with the same name.

[0179] Specifically, the edge window filtering module 1001, the feature extraction module 1002, the first matching module 1003, the second matching module 1004 and the image registration module 1005 are electrically connected.

[0180] The multimodal image registration device in the embodiment of the present invention performs multi-scale edge window filtering on the first image to be registered and the second image to be registered respectively, obtains the multi-scale edge window filtering result image of the first image to be registered and the multi-scale edge window filtering result image of the second image to be registered, and then obtains the first feature point corresponding to the first image to be registered and the feature vector corresponding to the first feature point, as well as the second feature point corresponding to the second image to be registered and the feature vector corresponding to the second feature point, based on the edge window filtering result image. The first feature point and the second feature point corresponding to the feature vector are matched twice to obtain the first feature point pair with the same name and the second feature point pair with the same name, and then the first image to be registered and the second image to be registered are registered based on the first feature point pair with the same name and the second feature point pair with the same name. Through multi-scale edge window filtering, the image edge features are effectively retained, and the robustness of the multimodal image registration is enhanced. Through the two-stage matching strategy, the accuracy and efficiency of the multimodal image registration are improved, which provides an effective solution for the automatic processing and analysis of remote sensing images.

[0181] Fig.11 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.11 As shown, the electronic device may include: a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120 and the memory 1130 communicate with each other through the communication bus 1140. The processor 1110 may call the logic instructions in the memory 1130 to execute the multimodal image registration method.

[0182] In addition, the logic instructions in the above-mentioned memory 1130 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0183] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal image registration method provided by the above-mentioned methods.

[0184] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multimodal image registration method provided by the above-mentioned methods.

[0185] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0186] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal image registration method, characterized in that: include: The first image to be registered and the second image to be registered are respectively k The first image to be registered is obtained by filtering the edge window of the first image. k The edge window filtering result image of the scale and the second image to be registered k The first image to be registered and the second image to be registered are images of the same region or the same target but in different modes. , S represents the number of scales, S is a positive integer greater than 1. When the scale is different, the variance, number of filtering times and radius of the edge window filtering window for edge window filtering are different. Based on the first image to be registered k The edge window filtering result image of the scale and the second image to be registered k Scaled edge window filtering result image, obtaining first feature points corresponding to the first image to be registered and feature vectors corresponding to the first feature points, as well as second feature points corresponding to the second image to be registered and feature vectors corresponding to the second feature points; Based on the feature vector corresponding to the first feature point and the feature vector corresponding to the second feature point, feature matching is performed on the first feature point and the second feature point, and the first feature point and the second feature point that are successfully matched are determined as a first feature point pair with the same name; Based on the first pair of feature points with the same name, a first affine transformation matrix between the first image to be registered and the second image to be registered is obtained, and based on the first affine transformation matrix, feature matching is performed again on the first feature points and the second feature points to obtain a second pair of feature points with the same name; The first image to be registered and the second image to be registered are registered based on the first pair of feature points with the same name and the second pair of feature points with the same name.

2. The multimodal image registration method according to claim 1, characterized in that: The first image to be registered k The edge window filtering result image of the scale and the second image to be registered k The method further comprises: obtaining a first feature point corresponding to the first image to be registered and a feature vector corresponding to the first feature point, and a second feature point corresponding to the second image to be registered and a feature vector corresponding to the second feature point, including: The first image to be registered is respectively k The edge window filtering result image of the scale and the second image to be registered k The edge window filtering result image of the scale is used for edge extraction to obtain the first image to be registered. k The edge feature map of the scale and the second image to be registered k Scaled edge feature map; The first image to be registered k The edge feature map of the scale and the second image to be registered k The edge feature maps of the first to-be-registered image are respectively subjected to corner point detection to obtain the first k The first original feature point of the scale and the second image to be registered corresponding to the k The second original feature point of the scale; Based on the non-maximum suppression algorithm, the first image to be registered corresponding to the k The first original feature points of the first to-be-registered image are screened to obtain the first k The first feature point of the scale is the first feature point of the second image to be registered k The second original feature points of the scale are screened to obtain the first k The second characteristic point of the scale; Based on the k The first feature point of the first image to be registered k The edge window filtering result image of the scale is obtained k The feature vector corresponding to the first feature point of the scale is based on the first k The second feature point of the second image to be registered k The edge window filtering result image of the scale is obtained k The eigenvector corresponding to the second feature point of the scale.

3. The multimodal image registration method according to claim 2, characterized in that: The k The first feature point of the first image to be registered k The edge window filtering result image of the scale is obtained k The feature vector corresponding to the first feature point of the scale is based on the first k The second feature point of the second image to be registered k The edge window filtering result image of the scale is obtained k The feature vector corresponding to the second feature point of the scale includes: The first image to be registered is respectively k The edge window filtering result image of the scale and the second image to be registered k The edge window filtering result image of the scale is j The controllable Gaussian filter in the direction is used to obtain the first image to be registered. k Scale j The controllable Gaussian filtering result image and the second image to be registered are k Scale j The result image of the controllable Gaussian filter in the direction, , Indicates the number of directions, is a positive integer greater than 1, and the rotation angle of the Gaussian filter window for Gaussian filtering is different for different directions; The first image to be registered k The controllable Gaussian filtering result images in all directions of the scale are superimposed to obtain the first image to be registered. k The first-order gradient map of the scale is k The controllable Gaussian filtering result images in all directions of the scale are superimposed to obtain the second image to be registered. k First-order gradient map of scale; Based on the first image to be registered k The first-order gradient map of the first image to be registered is obtained. k The second-order gradient magnitude map and the second-order gradient phase map of the second to-be-registered image are k The first-order gradient map of the scale is obtained by k Scaled second-order gradient magnitude map and second-order gradient phase map; Based on the first image to be registered k The second-order gradient amplitude map and the second-order gradient phase map of the scale are obtained. k The feature vector corresponding to the first feature point of the second image to be registered is k The second-order gradient amplitude map and the second-order gradient phase map of the scale are obtained. k The eigenvector corresponding to the second feature point of the scale.

4. The multimodal image registration method according to claim 2, characterized in that: The step of performing feature matching again on the first feature point and the second feature point based on the first affine transformation matrix to obtain a second pair of feature points with the same name includes: Based on the first affine transformation matrix, mapping the first feature point into the second image to be registered; Determine a target number of second feature points in the second image to be registered that are closest to the mapping point of the first feature point as candidate points of the same name for the first feature point; Calculate a target distance between the first feature point and the candidate homonymous point of the first feature point based on the distance between the mapping point of the first feature point in the second image to be registered and the candidate homonymous point of the first feature point, the feature vector of the first feature point, and the feature vector of the candidate homonymous point of the first feature point; Based on the target distance between the first feature point and the candidate homonymous point of the first feature point, the first feature point and the candidate homonymous point of the first feature point having the smallest target distance to the first feature point are determined as a pair of the second homonymous feature points.

5. The multimodal image registration method according to claim 3, characterized in that: based on the first image to be registered k The second-order gradient amplitude map and the second-order gradient phase map of the scale are obtained. k The feature vector corresponding to the first feature point of the second image to be registered is k The second-order gradient amplitude map and the second-order gradient phase map of the scale are obtained. k The feature vector corresponding to the second feature point of the scale includes: In the first image to be registered k In the second-order gradient amplitude map of the scale, determine the k The first target area corresponding to the first feature point of the second to-be-registered image is k In the second-order gradient amplitude map of the scale, determine the k A first target area corresponding to a second feature point of the scale; For the said k The first original area corresponding to the first feature point of the scale is Gaussian filtered to obtain the first k The first target area corresponding to the first feature point of the scale, k The first original area corresponding to the second feature point of the scale is Gaussian filtered to obtain the first k A first target area corresponding to a second feature point of the scale; Based on the first image to be registered k The second-order gradient phase image of the scale is obtained k The gradient amplitude value of the first target area corresponding to the first feature point of the scale in each first phase interval, and then the first phase interval with the largest gradient amplitude value is determined as the first phase interval. k The first phase interval corresponding to the first feature point of the second image to be registered is based on the first phase interval of the first feature point of the second image to be registered. k The second-order gradient phase image of the scale is obtained k The gradient amplitude value of the first target area corresponding to the second feature point of the scale in each first phase interval, and then the first phase interval with the largest gradient amplitude value is determined as the first phase interval. k A first phase interval corresponding to a second characteristic point of the scale; Based on the k The first phase interval corresponding to the first characteristic point of the scale is determined k The direction of the feature vector corresponding to the first feature point of the scale is based on the first k The first phase interval corresponding to the second characteristic point of the scale is determined k The direction of the eigenvector corresponding to the first eigenpoint of the scale; In the first image to be registered k In the second-order gradient amplitude map of the scale, determine the k The feature area corresponding to the first feature point of the second image to be registered is k In the second-order gradient amplitude map of the scale, determine the k The feature area corresponding to the second feature point of the scale; Based on the first image to be registered k The second-order gradient phase image of the scale is obtained k The gradient amplitude value of the feature area corresponding to the first feature point of the scale in each second phase interval is used as the first k The eigenvalue of the eigenvector corresponding to the first feature point of the second image to be registered is based on the k The second-order gradient phase image of the scale is obtained k The gradient amplitude value of the feature area corresponding to the second feature point of the scale in each second phase interval is used as the first k The eigenvalue of the eigenvector corresponding to the second feature point of the scale.

6. The multimodal image registration method according to claim 1, characterized in that: The first image to be registered and the second image to be registered are respectively k The first image to be registered is obtained by filtering the edge window of the first image. k The edge window filtering result image of the scale and the second image to be registered k The resulting image of the edge window filter of the scale includes: Initialize the edge window filter parameters, which include the initial variance, the number of scales, the scale factor, the position of each edge window filter window, and the k The edge window filter window radius, filtering times and variance corresponding to the scale, each of the edge window filter windows includes a rectangular edge window filter window with a long side in the vertical direction and a pixel point to be filtered at the midpoint of the right long side, a rectangular edge window filter window with a long side in the vertical direction and a pixel point to be filtered at the midpoint of the left long side, a rectangular edge window filter window with a long side in the horizontal direction and a pixel point to be filtered at the midpoint of the upper long side, a rectangular edge window filter window with a long side in the horizontal direction and a pixel point to be filtered at the midpoint of the lower long side, and a rectangular edge window filter window with four sides in the horizontal and vertical directions respectively and the pixel points to be filtered are located at the four sides respectively. A square side window filtering window with right angles located in the horizontal and vertical directions and the pixel to be filtered is located at the midpoint of the lower right hypotenuse, an isosceles right triangle side window filtering window with right angles located in the horizontal and vertical directions and the pixel to be filtered is located at the midpoint of the lower left hypotenuse, an isosceles right triangle side window filtering window with right angles located in the horizontal and vertical directions and the pixel to be filtered is located at the midpoint of the upper right hypotenuse, and an isosceles right triangle side window filtering window with right angles located in the horizontal and vertical directions and the pixel to be filtered is located at the midpoint of the upper left hypotenuse; Based on the filtering parameters, the first image to be registered and the second image to be registered are respectively subjected to the first k The first image to be registered is obtained by filtering the edge window of the first image. k The edge window filtering result image of the scale and the second image to be registered k Scaled edge window filtering result image.

7. The multimodal image registration method according to any one of claims 1 to 6, characterized in that: The first image to be registered and the second image to be registered are respectively k The first image to be registered is obtained by filtering the edge window of the first image. k The edge window filtering result image of the scale and the second image to be registered k Before filtering the resulting image with a side window of a certain scale, the method further comprises: Image preprocessing is performed on the first image to be registered and the second image to be registered, wherein the image preprocessing includes at least one of normalization, contrast stretching and Gaussian filtering.

8. A multimodal image registration device, characterized in that: include: The edge window filtering module is used to filter the first image to be registered and the second image to be registered respectively. k The first image to be registered is obtained by filtering the edge window of the first image. k The edge window filtering result image of the scale and the second image to be registered k The first image to be registered and the second image to be registered are images of the same region or the same target but in different modes. , S represents the number of scales, S is a positive integer greater than 1. When the scale is different, the variance of the edge window filtering and the radius of the edge window filtering window are different. A feature extraction module is used to extract the first image to be registered based on the first image to be registered. k The edge window filtering result image of the scale and the second image to be registered k Scaled edge window filtering result image, obtaining first feature points corresponding to the first image to be registered and feature vectors corresponding to the first feature points, as well as second feature points corresponding to the second image to be registered and feature vectors corresponding to the second feature points; A first matching module, configured to perform feature matching on the first feature point and the second feature point based on a feature vector corresponding to the first feature point and a feature vector corresponding to the second feature point, and determine the first feature point and the second feature point that are successfully matched as a first feature point pair with the same name; A second matching module is used to obtain a first affine transformation matrix between the first image to be registered and the second image to be registered based on the first pair of feature points with the same name, and perform feature matching on the first feature points and the second feature points again based on the first affine transformation matrix to obtain a second pair of feature points with the same name; An image registration module is used to register the first image to be registered and the second image to be registered based on the first pair of feature points with the same name and the second pair of feature points with the same name.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the multimodal image registration method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal image registration method according to any one of claims 1 to 7 is implemented.

Images