Two-stage different-source remote sensing image registration method with radiation and rotation invariance

Abstract

The invention discloses a two-stage different-source remote sensing image registration method with radiation and rotation invariance. The method comprises the following steps: performing WLD filtering on a to-be-registered image group to obtain a WLD structure saliency map and a WLD directional diagram sequence of the to-be-registered image group; constructing a feature description vector of the to-be-registered image group based on the WLD structure saliency map and the WLD directional diagram sequence; matching feature points among the to-be-registered image groups according to the feature description vectors to obtain rotation reference point pairs; extracting WLD structural feature description vectors of the image groups to be registered according to the rotating reference point pairs, and matching feature points among the image groups to be registered again according to the WLD structural feature description vectors to obtain multiple pairs of final matching point pairs; and registering the to-be-registered image group according to the final matching point pair. According to the invention, the registration precision and effect can be enhanced.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a two-stage heterogeneous remote sensing image registration method with radiation and rotation invariance. Background Art

[0002] The types of imaging sensors currently used in remote sensing technology include visible light, infrared, synthetic aperture radar (SAR), hyperspectral, etc. There is complementarity between high-quality remote sensing images of different perspectives and resolutions captured by multiple sensors under different conditions, providing sufficient data support for applications such as natural environment monitoring, electronic maps and navigation.

[0003] The comprehensive application of multi-sensor data first needs to deal with the spatial position differences of heterogeneous remote sensing images due to differences in imaging sensors, platforms, etc. Image registration, a technology that aligns the spatial position of two or more images obtained under different conditions of the same scene, is an important means to eliminate the spatial position differences between heterogeneous remote sensing images. The key to image registration is to overcome the geometric and radiometric differences between heterogeneous remote sensing images and to explore the common structural features of heterogeneous remote sensing images.

[0004] Most of the currently popular heterogeneous remote sensing image registration methods are based on direct matching in the feature space. In the process of heterogeneous remote sensing image registration, this method only uses the local neighborhood information of key points for feature description; however, in the presence of radiation differences, the local neighborhood information is not robust to rotation, resulting in poor heterogeneous remote sensing image registration in the presence of rotation.

[0005] Therefore, the currently popular heterogeneous remote sensing image registration method cannot construct robust rotation invariant features due to the influence of radiation differences, and the registration effect is poor. Summary of the invention

[0006] The embodiment of the present invention provides a two-stage heterogeneous remote sensing image registration method with radiation and rotation invariance, which can solve the problem that the currently popular heterogeneous remote sensing image registration method cannot construct robust rotation invariant features due to the influence of radiation differences, resulting in poor registration effect.

[0007] In a first aspect, an embodiment of the present invention provides a two-stage heterogeneous remote sensing image registration method with radiation and rotation invariance, the method comprising:

[0008] Performing enhanced Weber local descriptor WLD filtering on the image group to be registered to obtain a WLD structural saliency map and a WLD direction map sequence of the image group to be registered, wherein the image group to be registered includes a reference image and an image to be registered with the same shooting scene, and the WLD direction map sequence includes WLD direction maps corresponding to multiple directions one by one;

[0009] Based on the WLD structural saliency map and WLD direction map sequence, a feature description vector of the image group to be registered is constructed;

[0010] Matching feature points between the image groups to be registered according to the feature description vectors to obtain a rotation reference point pair of the image groups to be registered, wherein the rotation reference point pair includes a pair of matching points from the reference image and the image to be registered respectively;

[0011] Extracting the WLD structural feature description vector of the image group to be registered according to the rotation reference point pair, and matching the feature points between the image groups to be registered again according to the WLD structural feature description vector to obtain multiple pairs of final matching point pairs of the image groups to be registered;

[0012] The image group to be registered is registered according to the final matching point pairs.

[0013] In a second aspect, an embodiment of the present invention provides a two-stage heterogeneous remote sensing image registration device with radiation and rotation invariance, comprising:

[0014] A filtering module, the filtering module is used to perform enhanced Weber local descriptor WLD filtering on the image group to be registered to obtain a WLD structural saliency map and a WLD direction map sequence of the image group to be registered, wherein the image group to be registered includes a reference image and an image to be registered with the same shooting scene, and the WLD direction map sequence includes WLD direction maps corresponding to multiple directions one by one;

[0015] A feature extraction module, which is used to construct a feature description vector of the image group to be registered based on the WLD structural saliency map and the WLD direction map sequence;

[0016] A registration module, the registration module is used to match feature points between the image groups to be registered according to the feature description vector to obtain a rotation reference point pair of the image groups to be registered, wherein the rotation reference point pair includes a pair of matching points from the reference image and the image to be registered respectively;

[0017] The feature extraction module is also used to extract the WLD structural feature description vector of the image group to be registered according to the rotation reference point pair, and match the feature points between the image groups to be registered again according to the WLD structural feature description vector to obtain multiple pairs of final matching point pairs of the image groups to be registered;

[0018] The registration module is also used to register the image group to be registered according to the final matching point pair.

[0019] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory is used to store computer programs; the processor can be used to execute the computer program (instructions) stored in the memory to implement the method of the first aspect above.

[0020] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed, the method of the first aspect described above can be implemented.

[0021] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0022] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: according to the method provided by the present invention, the structural features of the image group to be registered, namely, the feature description vector and the WLD structural feature description vector, are extracted through the WLD algorithm, and registration is performed based on these features, which can avoid the error caused by the radiation difference between the heterogeneous reference image and the image to be registered and improve the registration accuracy; the image group to be registered is matched twice based on the WLD algorithm, which can increase the robustness of the rotation invariance feature of the WLD structural feature description vector, thereby further enhancing the registration effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flowchart for implementing a two-stage heterogeneous remote sensing image registration method with radiation and rotation invariance provided by an embodiment of the present invention;

[0024] Figure 2 A schematic diagram of a scenario for generating a WLD directional pattern sequence provided by an embodiment of the present invention;

[0025] Figure 3 A schematic diagram of a first WLD histogram provided by an embodiment of the present invention;

[0026] Figure 4 A schematic diagram of a sub-region provided in an embodiment of the present invention;

[0027] Figure 5 A schematic diagram of the structure of a two-stage heterogeneous remote sensing image registration device with radiation and rotation invariance provided by an embodiment of the present invention;

[0028] Figure 6 A comparative schematic diagram of a registration effect provided by an embodiment of the present invention;

[0029] Figure 7 A schematic diagram of a registration effect provided by the present invention;

[0030] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0032] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0033] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0034] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]", depending on the context.

[0035] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0036] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0037] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0038] The two-stage heterogeneous remote sensing image registration method with radiation and rotation invariance provided by the embodiment of the present invention can be applied to electronic devices such as mobile terminals, personal laptops, supercomputers, etc. The embodiment of the present invention does not impose any restrictions on the specific type of electronic devices.

[0039] Figure 1 The flowchart shown is a two-stage heterogeneous remote sensing image registration method with radiation and rotation invariance provided by an embodiment of the present invention. As an example but not a limitation, the method can be applied to the above electronic device and includes steps S101-S106. Each step is described below:

[0040] S101, performing WLD filtering on the image group to be registered to obtain a WLD structural saliency map and a WLD direction map sequence of the image group to be registered.

[0041] Exemplarily, the group of images to be registered may include a reference image and an image to be registered.

[0042] In a possible implementation, the WLD structural saliency map of the to-be-registered image group may be equivalent to a differential excitation image in a conventional enhanced Weber Local Descriptor (Weber Local Descriptor) filtering algorithm.

[0043] In one example, a traditional WLD algorithm may be used to calculate the WLD structural saliency map of the image group to be registered.

[0044] In another example, if at least one of the reference image and the image to be registered is a SAR image, the pixel value of each pixel in the image can be weighted, and the WLD structural saliency map of the image group to be registered can be calculated using the weighted pixel value, so that the WLD structural saliency map of the image group to be registered can more accurately characterize the structural features of the image group to be registered.

[0045] Exemplarily, the WLD structural saliency map may satisfy the following formula:

[0046]

[0047] in:

[0048]

[0049] ξ(p) is the WLD structural saliency map of a pixel point p in the image to be registered or the reference image, N(p) is the set of pixels in the rectangular neighborhood centered on pixel point p, q∈N(p) means that pixel point q is a pixel point in N(p), M(q) is the set of pixels in the rectangular neighborhood centered on pixel point q, m∈M(q) means that pixel point m is a pixel point in M(q), I(m) is the pixel value of pixel point m, σ is the standard deviation parameter, mx 、m y are the x-axis coordinate value and y-axis coordinate value of pixel point m, respectively, x ,q y are the x-axis coordinate value and y-axis coordinate value of pixel point q respectively.

[0050] For example, the size of the rectangular neighborhood may be 7×7.

[0051] In a possible implementation, in order to cope with the impact of rotation differences in remote sensing images, the WLD directional map sequence may include WLD directional maps corresponding to multiple directions one by one established by convolution, and the WLD directional map may be equivalent to the directional map in the WLD algorithm.

[0052] In one example, see Figure 2 , the vertical convolution kernel and the parallel convolution kernel can be determined according to the size of the rectangular neighborhood. Then these two convolution kernels are rotated clockwise at the i-th preset angle and convoluted on the original image group to be registered to obtain the convolution map D v and D h , the i-th WLD direction map can be determined based on these two convolution maps.

[0053] For example, see Figure 2 , the vertical convolution kernel can be a matrix of size 7×7, with the element value of the center column being 0, the element value on the left being -1, and the element value on the right being 1.

[0054] Similarly, see Figure 2 The horizontal convolution kernel is also a 7×7 matrix, with the element value of the center row being 0, the element value below being -1, and the element value above being 1.

[0055] Exemplarily, the i-th preset angle may be Wherein L is the total number of WLD directional patterns, i=0,1,...L-1.

[0056] Exemplarily, the WLD pattern may satisfy the following formula:

[0057]

[0058] θ i (p) is the i-th WLD pattern of pixel p, θ i (p)∈[0,2π], mod is the mapping function, D v (p) is the convolution image processed by the vertical convolution kernel, D h (p) is the convolution image processed by the horizontal convolution kernel, ε is a constant used to prevent division by zero, which can be equal to 10 -5 .

[0059] S102, constructing a feature description vector of the image group to be registered based on the WLD structural saliency map and the WLD direction map sequence.

[0060] In some embodiments, feature points of the image to be registered may be extracted based on the WLD structural saliency map, and the first main direction of each feature point may be determined; and then a feature description vector of each feature point may be determined according to the first main direction of each feature point.

[0061] Exemplarily, the feature description vectors of all feature points constitute the feature description vector of the image group to be registered.

[0062] In one possible implementation, the points of interest (pixels) of the WLD structural saliency map can be extracted by using a fast corner detection operator (Features from Accelerated Segment Test, FAST), and then unstable points whose response values ​​are lower than a response value threshold and points with insufficient information in the image edge neighborhood are eliminated to obtain multiple feature points of the image group to be registered.

[0063] Exemplarily, the number of feature points may be as high as 5,000.

[0064] In a possible implementation, based on the WLD algorithm, the WLD structural saliency map of each feature point can be used as a differential excitation image in the WLD algorithm and a WLD direction map of the feature point can be used as a direction image in the WLD algorithm to calculate the one-dimensional WLD histogram of the two images to obtain multiple first WLD histograms of the feature point. Then, all the first WLD histograms of the feature point are superimposed, and the direction corresponding to the highest column in the superimposed first WLD histogram is used to determine the first main direction of the feature point.

[0065] For example, see Figure 3 The first WLD histogram may include graph columns corresponding to multiple directions one by one, wherein the i-th graph column may correspond to the i-th preset angle.

[0066] In one example, a circular neighborhood with a radius of R1 can be intercepted from the WLD significant structure map of the feature point and each WLD directional map in the WLD directional map sequence, with the feature point as the center, to obtain the third circular area of ​​the feature point. Then, based on the WLD algorithm, the i-th first WLD histogram is determined according to the WLD significant structure map and the i-th WLD directional map in the third circular area of ​​the feature point.

[0067] In one example, see Figure 3 , the columns of all the first WLD histograms of the same feature point can be added together to obtain the superimposed first WLD histogram, and then the highest column (e.g. Figure 3The direction corresponding to the column x5 in the figure (for example, 120°) is taken as the first main direction of the feature point.

[0068] In a possible implementation, a circular neighborhood with a radius of R2 can be intercepted with the feature point as the center on the WLD structural saliency map and the WLD first main direction map to obtain the first circular area of ​​the feature point. Then, the first circular area is divided according to different angles and radii to obtain the following: Figure 4 Finally, based on the WLD algorithm, the two-dimensional second WLD histogram in each first sub-region is counted according to the WLD structural saliency map and the WLD first main direction map of the feature point; the second WLD histogram in each sub-region is vectorized and then spliced ​​and normalized in turn to obtain the feature description vector of the feature point.

[0069] Exemplarily, the WLD first main direction pattern may be a WLD direction pattern corresponding to the first main direction of the feature point.

[0070] Exemplarily, the number of WLD structure significance columns S in the second WLD histogram coarse Number of columns in the WLD structure direction T coarse They are 14 and 7 respectively.

[0071] Optionally, before dividing the first sub-region, the first circular region may be rotated to the first main direction of the feature point.

[0072] In one example, see Figure 4 , we can first divide the first circular area into N equal parts according to different radii coarse Concentric circles, and then divide the ring except the central circular area into M according to the angle coarse sector rings, we get (N coarse -1)·M coarse +1 first sub-region.

[0073] For example, the radius R2 may be equal to 70 N coarse It can be 6, the number of sectors is M coarse It can be 12.

[0074] S103, matching feature points between the image groups to be registered according to the feature description vectors to obtain a pair of rotation reference points.

[0075] In a possible implementation, the Euclidean distance between the feature description vector of the reference image and the feature description vector of the image to be registered can be calculated, and the feature points corresponding to the closest pair of feature description vectors are taken as a pair of first matching point pairs; then the errors of all first matching point pairs are estimated, and the first matching point pair with the smallest error is taken as the rotation reference point pair (P ref ,P sen), used for subsequent fine registration adjustment; where P ref is the feature point in the reference image, P sen are feature points in the image to be registered.

[0076] For example, if the feature description vector of a feature point in the reference image is x1, the feature description vectors of all feature points in the image to be registered are y1-y 5000 , we can calculate x1 and y1, x1 and y2, ... x1 and y 5000 If the Euclidean distance between x1 and y1 is the shortest, the feature point corresponding to x1 and the feature point corresponding to y1 can be taken as a pair of first matching points.

[0077] In one example, before estimating the error of the first matching point pair, the first distance threshold can be used to eliminate unstable matching points in the first matching point pair, such as the first matching point pair whose Euclidean distance is greater than the first distance threshold; and then, based on the affine transformation matrix between the remaining first matching point pairs, the random sampling consistency (RANSAC) algorithm is used to further eliminate gross errors.

[0078] In one example, an affine transformation matrix between the first matching point pairs after the gross errors are eliminated may be estimated, and then the error between the first matching point pairs after the gross errors are eliminated may be estimated using the estimated affine transformation matrix.

[0079] S104, extracting a WLD structural feature description vector of the image group to be registered according to the rotation reference point pair.

[0080] In a possible implementation, it is possible to calculate the direction of each feature point in the reference image to the rotation reference point P ref The second main direction of the feature point is obtained by the relative direction of the feature point. Similar to the above step S102, a circular neighborhood with a radius of R3 can be intercepted with each feature point as the center on the WLD structural saliency map and the WLD second main direction map to obtain a second circular area. The second circular area is rotated to the second main direction, and the rotated second circular area is divided according to different angles and radii to obtain a second sub-area (see Figure 4 ). According to the WLD structural saliency map and the WLD second main direction map of the feature point, the two-dimensional third WLD structural saliency map of each second sub-region of the feature point is counted. The third WLD histogram of each second sub-region is vectorized and then spliced ​​and normalized in sequence to obtain the WLD structural feature description vector of each feature point.

[0081] Exemplarily, the WLD second main direction pattern is a WLD direction pattern corresponding to the second main direction of the feature point.

[0082] In one example, the radius R3 may be 60, and the number of concentric circles N fine It can be 8, the number of sectors is M fine It can be 8. The number of WLD structure significance columns S of the third two-dimensional WLD histogram fine Number of columns in the WLD structure direction T fine They can be 14 and 7 respectively.

[0083] S105, matching feature points between the image groups to be registered again according to the WLD structural feature description vector to obtain multiple pairs of final matching point pairs.

[0084] In one example, similar to step S103 above, the Euclidean distance between the WLD structural feature description vector of the reference image and the WLD structural feature description vector of the image to be registered can be calculated, and then the second matching point pair of the image group to be registered is constructed according to the nearest neighbor Euclidean distance; the matching points in the second matching point pair that do not meet the preset conditions are eliminated according to the nearest neighbor Euclidean distance and the next nearest neighbor Euclidean distance to obtain the final matching point pair.

[0085] Exemplarily, the preset condition may be less than a second distance threshold.

[0086] Similarly, the RANSAC algorithm can be used to eliminate gross errors from the final matching point pairs based on the affine transformation matrix between the final matching point pairs.

[0087] Specifically, the RANSAC algorithm can divide inliers and outliers according to the estimated error, and obtain the best model in the iterative process, and finally obtain the corresponding inlier set, which is the final matching point pair after gross error elimination.

[0088] S106, registering the image group to be registered according to the final matching point pair.

[0089] In one example, the coordinates of the final matching point pair / the final matching point pair after gross error elimination can be substituted into the affine transformation model to fit the optimal affine transformation matrix; according to the affine transformation matrix, the resampling of the image to be registered is completed by bilinear interpolation so that the spatial positions of all pixels of the image to be registered are consistent with those of the reference image.

[0090] According to the method provided by the present invention, the structural features of the image group to be registered, namely the feature description vector and the WLD structural feature description vector, are extracted through the WLD algorithm, and registration is performed based on these features, which can avoid the error caused by the radiation difference between the heterogeneous reference image and the image to be registered and improve the registration accuracy; matching the image group to be registered twice based on the WLD algorithm can increase the robustness of the rotation invariance feature of the WLD structural feature description vector, thereby further enhancing the registration effect.

[0091] Figure 5The structure diagram of a two-stage heterogeneous remote sensing image registration device with radiation and rotation invariance provided by an embodiment of the present invention is shown. As an example but not a limitation, the device 500 may include a filtering module 510 , a feature extraction module 520 , and a registration module 530 .

[0092] Exemplarily, the filtering module 510 is used to perform enhanced Weber local descriptor WLD filtering on the image group to be registered to obtain a WLD structural saliency map and a WLD direction map sequence of the image group to be registered, wherein the image group to be registered includes a reference image and an image to be registered with the same shooting scene, and the WLD direction map sequence includes WLD direction maps corresponding to multiple directions one by one; the feature extraction module 520 is used to construct a feature description vector of the image group to be registered based on the WLD structural saliency map and the WLD direction map sequence; the registration module 530 is used to match feature points between the image groups to be registered according to the feature description vector to obtain a rotation reference point pair of the image group to be registered, wherein the rotation reference point pair includes a pair of matching points from the reference image and the image to be registered respectively; the feature extraction module 520 is also used to extract the WLD structural feature description vector of the image group to be registered according to the rotation reference point pair, and match the feature points between the image groups to be registered again according to the WLD structural feature description vector to obtain multiple pairs of final matching point pairs of the image group to be registered; the registration module 530 is also used to register the image group to be registered according to the final matching point pairs.

[0093] In order to better illustrate the beneficial effects of the present invention, the following simulation experiments were conducted:

[0094] Exemplarily, the simulation experiment can be simulated on a 13th Gen Intel(R) Core(TM) i7-13700 2.10GHz CPU and Windows 11 operating system using MATLAB 2022b developed by Mathworks, USA. The test data is 20 pairs of optical and SAR remote sensing images, which are remote sensing images from Sentinel-2 and Sentinel-1 satellites, and the images are cropped to 800×800 pixels. To simulate the rotation difference in real situations, each SAR image is rotated 30°×i, (i=0,1,...,6), and finally 20×7 pairs of images are obtained for testing.

[0095] Specifically, the simulation experiment can compare the scale-invariant feature description method (SAR-SIFT) from the paper: SAR-SIFT: A SIFT-Like Algorithm for SAR Images. IEEE Transactions on Geoscience and Remote Sensing, 33(1): 453-466, 2015, the radiation-invariant feature transform algorithm (RIFT) from the paper: Multi-Modal Image Matching Based on Radiation-Variation Insensitive Feature Transform. IEEE Transactions on Image Processing, 29: 3296-3310, 2020, and the registration effect of the present invention.

[0096] Figure 6 A comparative schematic diagram of a registration effect provided by an embodiment of the present invention is shown.

[0097] In one example, the optical image can be used as a reference image and the SAR image can be used as an image to be registered. After the registration is completed, the registration effect is measured by two indicators, namely, the correct matching point pair ratio and the root mean square error of the matching point pairs.

[0098] Exemplarily, the root mean square error of the matching point pair may satisfy the following formula:

[0099]

[0100] Among them, RMSE is the root mean square error of the matching point pairs, CM represents the number of point pairs whose error calculated according to the true transformation matrix T is less than 3 pixels, and P s ' en is the pixel from the image to be registered in the final matching point pair, P r ' ef is the pixel from the reference image in the final matching point pair.

[0101] For details, see Figure 6 , Figure 6 (a) is a schematic diagram showing the comparison of the correct matching point ratios of the SAR-SIFT method, the RIFT method and the method provided by the present invention. Figure 6 (b) is a schematic diagram comparing the root mean square error of matching point pairs of the AR-SIFT method, the RIFT method and the method provided by the present invention.

[0102] See also Figure 6It can be seen that the method provided by the present invention can cope with significant radiation differences in heterogeneous remote sensing images, and by using a two-stage image registration framework based on a global reference center, robust heterogeneous remote sensing image registration can be achieved under different levels of rotational differences, achieving more accurate registration effects than the comparative method.

[0103] Figure 7 Shown is a schematic diagram of a registration effect provided by the present invention.

[0104] See also Figure 7 , Figure 7 (a) is a schematic diagram of the final matching point pairs obtained when the image to be registered is rotated at different angles relative to the reference image according to the method provided by the present invention. Figure 7 (b) is a schematic diagram of a group of images to be registered after being registered obtained when the images to be registered are rotated at different angles relative to the reference image according to the method of the present invention.

[0105] See also Figure 7 It can be seen that the method provided by the present invention can obtain a large number of final matching point pairs under different rotation differences, and realizes robust registration of the images to be registered under different rotation angles.

[0106] Therefore, according to the method provided by the present invention, the structural features of the image group to be registered, that is, the feature description vector and the WLD structural feature description vector, are extracted through the WLD algorithm, and registration is performed based on these features, which can avoid the error caused by the radiation difference between the heterogeneous reference image and the image to be registered and improve the registration accuracy; matching the image group to be registered twice based on the WLD algorithm can increase the robustness of the rotation invariance feature of the WLD structural feature description vector, thereby further enhancing the registration effect.

[0107] Figure 8 FIG. 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 8 The electronic device 800 shown may include: at least one processor 810 ( Figure 8 Only one processor is shown in the figure), a memory 820, and a computer program 830 stored in the memory 820 and executable on the at least one processor 810, wherein the processor 810 implements the steps of any of the above-mentioned method embodiments when executing the computer program 830.

[0108] The electronic device 800 may be a processing device such as a robot that can implement the above method. The embodiment of the present invention does not impose any limitation on the specific type of the electronic device.

[0109] Those skilled in the art will understand that Figure 8The electronic device 800 is merely an example and does not constitute a limitation on the electronic device. The electronic device 800 may include more or fewer components than shown in the figure, or may combine certain components, or may include different components. For example, the electronic device 800 may also include an input and output interface.

[0110] The processor 810 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASTC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0111] The memory 820 may be an internal storage unit in some embodiments, such as a hard disk or a memory. The memory 820 may also be an external storage device in other embodiments, such as a plug-in hard disk, a smart memory card (SmartMemory Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 820 may also include both an internal storage unit and an external storage device. The memory 820 is used to store an operating system, an application program, a boot loader (Boot Loader), data, and other programs, such as the program code of the computer program, etc. The memory 820 may also be used to temporarily store data that has been output or is to be output.

[0112] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0113] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0114] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0115] An embodiment of the present invention provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0116] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0117] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0118] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

Claims

1. A two-stage heterogeneous remote sensing image registration method with radiation and rotation invariance, characterized in that: include: Performing enhanced Weber local descriptor (WLD) filtering on the image group to be registered to obtain a WLD structural saliency map and a WLD direction map sequence of the image group to be registered, wherein the image group to be registered includes a reference image and an image to be registered having the same shooting scene, and the WLD direction map sequence includes WLD direction maps corresponding to multiple directions one by one; Constructing a feature description vector of the image group to be registered based on the WLD structural saliency map and the WLD direction map sequence; Matching feature points between the image groups to be registered according to the feature description vector to obtain a rotation reference point pair of the image groups to be registered, wherein the rotation reference point pair includes a pair of matching points from the reference image and the image to be registered respectively; Extracting a WLD structural feature description vector of the image group to be registered according to the rotation reference point pair, and matching feature points between the image groups to be registered again according to the WLD structural feature description vector to obtain multiple pairs of final matching point pairs of the image groups to be registered; The group of images to be registered is registered according to the final matching point pair.

2. The method according to claim 1, characterized in that At least one image in the group of images to be registered is a synthetic aperture radar image; The WLD structure saliency map satisfies the following formula: in: ξ(p) is the WLD structural saliency map of a pixel point p in the image to be registered or the reference image, N(p) is the set of pixels in the rectangular neighborhood centered on pixel point p, q∈N(p) means that pixel point q is a pixel point in N(p), M(q) is the set of pixels in the rectangular neighborhood centered on pixel point q, m∈M(q) means that pixel point m is a pixel point in M(q), I(m) is the pixel value of pixel point m, σ is the standard deviation parameter, m x 、m y are the x-axis coordinate value and y-axis coordinate value of pixel point m, respectively, x ,q y are the x-axis coordinate value and y-axis coordinate value of pixel point q respectively.

3. The method according to claim 1, characterized in that The step of constructing the feature description vector of the image group to be registered based on the WLD structural saliency map and the WLD direction map sequence includes: Extracting feature points of the image group to be registered based on the WLD structural saliency map; Determine a plurality of one-dimensional first WLD histograms of each feature point according to the WLD structural saliency map and the WLD direction map sequence, wherein the first WLD histogram includes graph columns corresponding to the plurality of directions one by one; Superimposing the first WLD histograms of the same feature point to obtain a superimposed first WLD histogram of the feature point; The direction corresponding to the highest column in the superimposed first WLD histogram is used as the first main direction of the feature point; The feature description vector of the feature point is determined according to the WLD first main direction map and the WLD structural saliency map, wherein the WLD first main direction map is the WLD direction map corresponding to the first main direction, and the feature description vectors of all feature points constitute the feature description vector of the image group to be registered.

4. The method according to claim 3, characterized in that The step of determining the feature description vector of the feature point according to the WLD first main direction map and the WLD structural saliency map comprises: On the WLD structure saliency map and the WLD first main direction map, a circular neighborhood is intercepted with the feature point as the center to obtain a first circular area; Divide the first circular area according to different angles and radii to obtain first sub-areas; Counting a two-dimensional second WLD histogram in each of the first sub-areas according to the WLD structure saliency map and the WLD first main direction map; The second WLD histogram of each of the first sub-regions is vectorized and then concatenated and normalized in sequence to obtain a feature description vector of the feature point.

5. The method according to claim 3, characterized in that: The step of matching the feature points between the image groups to be registered according to the feature description vector to obtain a rotation reference point pair of the image groups to be registered includes: Calculating the Euclidean distance between the feature description vector of the reference image and the feature description vector of the image to be registered; The feature points corresponding to a pair of feature description vectors with the closest Euclidean distance are taken as a pair of first matching point pairs; The error of each pair of the first matching points is estimated, and the first matching point pair with the smallest estimated error is used as the rotation reference point pair.

6. The method according to claim 3, characterized in that: The step of extracting the WLD structural feature description vector of the image group to be registered according to the rotation reference point pair comprises: Calculate the relative direction from the feature point in the reference image to the rotation reference point to obtain a second main direction of the feature point; A second circular area is obtained by intercepting a circular neighborhood with the feature point as the center on the WLD structure saliency map and the WLD second main direction map, wherein the WLD second main direction map is a WLD direction map corresponding to the second main direction; Rotating the second circular area to the second main direction and dividing the rotated second circular area according to different angles and radii to obtain second sub-areas; Counting a third two-dimensional WLD histogram of the second sub-region according to the WLD structure saliency map and the WLD second main direction map; The third WLD histogram of each second sub-region is vectorized and then concatenated and normalized in sequence to obtain a WLD structural feature description vector of the feature point.

7. The method according to claim 3, characterized in that The step of matching the feature points between the image groups to be registered again according to the WLD structural feature description vector to obtain multiple pairs of final matching point pairs of the image groups to be registered includes: Calculating the Euclidean distance between the WLD structural feature description vector of the reference image and the WLD structural feature description vector of the image to be registered; A second matching point pair of the image group to be registered is constructed according to the nearest neighbor Euclidean distance, and matching points in the second matching point pair that do not meet preset conditions are eliminated according to the nearest neighbor Euclidean distance and the next nearest neighbor Euclidean distance to obtain the final matching point pair.

8. A two-stage heterogeneous remote sensing image registration device with radiation and rotation invariance, characterized in that: include: A filtering module, wherein the filtering module is used to perform enhanced Weber local descriptor (WLD) filtering on the image group to be registered to obtain a WLD structural saliency map and a WLD direction map sequence of the image group to be registered, wherein the image group to be registered includes a reference image and an image to be registered having the same shooting scene, and the WLD direction map sequence includes WLD direction maps corresponding to multiple directions one by one; A feature extraction module, the feature extraction module is used to construct a feature description vector of the image group to be registered based on the WLD structural saliency map and the WLD direction map sequence; A registration module, the registration module is used to match the feature points between the image groups to be registered according to the feature description vector to obtain a rotation reference point pair of the image groups to be registered, wherein the rotation reference point pair includes a pair of matching points from the reference image and the image to be registered respectively; The feature extraction module is also used to extract the WLD structural feature description vector of the image group to be registered according to the rotation reference point pair, and match the feature points between the image groups to be registered again according to the WLD structural feature description vector to obtain multiple pairs of final matching point pairs of the image groups to be registered; The registration module is further used to register the group of images to be registered according to the final matching point pairs.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the electronic device, the method according to any one of claims 1 to 7 is implemented.

Images