Two-stage heterogeneous remote sensing image registration method with radiation and rotational invariance

Abstract

The application discloses a two-stage heterogeneous remote sensing image registration method with radiation and rotation invariance, comprising the following steps: performing WLD filtering on a group of images to be registered to obtain a WLD structure saliency map and a WLD direction map sequence of the group of images to be registered; constructing a feature description vector of the group of images to be registered based on the WLD structure saliency map and the WLD direction map sequence; matching feature points between the group of images to be registered according to the feature description vector to obtain a rotation reference point pair; extracting a WLD structure feature description vector of the group of images to be registered according to the rotation reference point pair, and matching feature points between the group of images to be registered again according to the WLD structure feature description vector to obtain a plurality of pairs of final matching point pairs; and registering the group of images to be registered according to the final matching point pairs. The application can enhance registration accuracy and effect.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a two-stage heterogeneous remote sensing image registration method with radiation and rotation invariance. BACKGROUND

[0002] At present, the imaging sensor types used in remote sensing technology include visible light, infrared, synthetic aperture radar (SAR), hyperspectrum, etc. The different views and different resolutions of high-quality remote sensing images captured by multi-sensor under different conditions have complementarity, which provides sufficient data guarantee for natural environment monitoring, electronic map and navigation applications.

[0003] The comprehensive application of multi-sensor data first needs to process the spatial position difference of heterogeneous remote sensing images due to different imaging sensors, platforms, etc. Image registration, a technology of aligning the spatial positions of two images or multiple images of the same scene obtained under different conditions, is an important means to eliminate the spatial position difference between heterogeneous remote sensing images. The key of image registration is to overcome the geometric and radiation differences between heterogeneous remote sensing images and to mine the structural features common to heterogeneous remote sensing images.

[0004] Most of the current popular heterogeneous remote sensing image registration methods are direct matching in feature space. In the process of heterogeneous remote sensing image registration, only the local neighborhood information of key points is used for feature description. However, the local neighborhood information is not robust to rotation, so that the registration effect of heterogeneous remote sensing images with rotation is not good.

[0005] Therefore, the current popular heterogeneous remote sensing image registration method cannot construct a robust rotation-invariant feature due to the influence of radiation difference, and the registration effect is poor. SUMMARY

[0006] The embodiment of the present application provides a two-stage heterogeneous remote sensing image registration method with radiation and rotation invariance, which can solve the problem that the current popular heterogeneous remote sensing image registration method cannot construct a robust rotation-invariant feature due to the influence of radiation difference, and the registration effect is poor.

[0007] In a first aspect, the embodiment of the present application provides a two-stage heterogeneous remote sensing image registration method with radiation and rotation invariance, which comprises:

[0008] performing WLD filter enhancement on the to-be-registered image group to obtain a WLD structure saliency map and a WLD direction map sequence of the to-be-registered image group, wherein the to-be-registered image group comprises a reference image and a to-be-registered image of the same scene, and the WLD direction map sequence comprises a WLD direction map corresponding to a plurality of directions one by one;

[0009] constructing a feature description vector of the image group to be registered based on the WLD structure saliency map and the WLD orientation map sequence;

[0010] matching feature points between the image group to be registered according to the feature description vector, to obtain a rotation reference point pair of the image group to be registered, wherein the rotation reference point pair includes a pair of matching points respectively from the reference image and the image to be registered;

[0011] extracting a WLD structure feature description vector of the image group to be registered according to the rotation reference point pair, and matching feature points between the image group to be registered again according to the WLD structure feature description vector, to obtain a plurality of pairs of final matching point pairs of the image group to be registered;

[0012] registering the image group to be registered according to the final matching point pairs.

[0013] In a second aspect, an embodiment of the present application provides a two-stage heterogeneous remote sensing image registration device with radiation and rotation invariance, comprising:

[0014] a filtering module, configured to perform WLD filtering on the image group to be registered to obtain a WLD structure saliency map and a WLD orientation map sequence of the image group to be registered, wherein the image group to be registered includes a reference image and an image to be registered of the same scene, and the WLD orientation map sequence includes a WLD orientation map corresponding to each of a plurality of directions;

[0015] a feature extraction module, configured to construct a feature description vector of the image group to be registered based on the WLD structure saliency map and the WLD orientation map sequence;

[0016] a registration module, configured to match feature points between the image group to be registered according to the feature description vector, to obtain a rotation reference point pair of the image group to be registered, wherein the rotation reference point pair includes a pair of matching points respectively from the reference image and the image to be registered;

[0017] The feature extraction module is further configured to extract a WLD structure feature description vector of the image group to be registered according to the rotation reference point pair, and match feature points between the image group to be registered again according to the WLD structure feature description vector, to obtain a plurality of pairs of final matching point pairs of the image group to be registered;

[0018] The registration module is further configured to register the image group to be registered according to the final matching point pairs.

[0019] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory is configured to store a computer program; and the processor is configured to execute the computer program (instructions) stored in the memory to implement the method of the first aspect.

[0020] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, in which a computer program is stored, and when the computer program is executed, the method of the first aspect can be implemented.

[0021] It can be understood that the beneficial effects of the second aspect to the fourth aspect can be referred to the related description of the first aspect, and will not be repeated here.

[0022] Compared with the prior art, the beneficial effects of the embodiment of the present application are: according to the method provided by the present application, the structural features of the to-be-registered image group are extracted by the WLD algorithm, that is, the feature description vector and the WLD structural feature description vector, and registration is performed according to these features, which can avoid errors caused by radiation differences between the heterogeneous reference image and the to-be-registered image, and improve the registration accuracy; based on the WLD algorithm, the to-be-registered image group is matched twice, which can increase the robustness of the WLD structural feature description vector, which is a rotation invariant feature, thereby further enhancing the registration effect. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 An implementation flowchart of the two-stage heterogeneous remote sensing image registration method with radiation and rotation invariance provided by the embodiment of the present application is provided.

[0024] Figure 2 A scene schematic diagram of generating a WLD direction map sequence provided by the embodiment of the present application is provided.

[0025] Figure 3 A schematic diagram of a first WLD histogram provided by the embodiment of the present application is provided.

[0026] Figure 4 A schematic diagram of a sub-region provided by the embodiment of the present application is provided.

[0027] Figure 5 A structural schematic diagram of a two-stage heterogeneous remote sensing image registration device with radiation and rotation invariance provided by the embodiment of the present application is provided.

[0028] Figure 6 A comparative schematic diagram of registration effects provided by the embodiment of the present application is provided.

[0029] Figure 7 A schematic diagram of registration effects provided by the present application is provided.

[0030] Figure 8 A structural schematic diagram of an electronic device provided by the embodiment of the present application is provided. DETAILED DESCRIPTION

[0031] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, technologies, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0032] It is to be understood that the terminology "includes", "has", "holds", "contains" and / or "comprising", "including", "having" and their conjugates, when used in the present specification and in the following claims, means "including but not limited to", and not to the exclusion of any other features, integers, steps, operations, elements, components, and / or groups thereof.

[0033] It is also to be understood that the terminology "and / or" when used in the present specification and in the following claims, means "and / or", both inclusive and exclusive, and that the conjunction "or" when used in the present specification and in the following claims, means "and / or", both inclusive and exclusive.

[0034] As used in the present specification and in the following claims, the term "if' can be interpreted as meaning "when", or "once", or "in response to a determination", or "in response to a detection" depending on the context. Similarly, the phrase "if determined", or "if detected [the recited condition or event]" can be interpreted as meaning "once determined", or "in response to a determination", or "once detected [the recited condition or event]", or "in response to a detection [the recited condition or event]", depending on the context.

[0035] In addition, the terms "first", "second", "third", etc. in the description of the present specification and in the following claims are only used to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0036] Reference in the specification to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" or "in some embodiments" in various places in the specification are not necessarily all referring to the same embodiment, although it can. The terms "including", "containing", "having" and variations thereof mean "including but not limited to", unless expressly specified otherwise.

[0037] The present application will be further described with reference to the following specific examples, but the embodiments of the present application are not limited thereto.

[0038] The two-stage heterogeneous remote sensing image registration method with radiation and rotation invariance provided by the embodiment of the present application can be applied to electronic devices such as mobile terminals, personal notebook computers, supercomputers and the like, and the embodiment of the present application does not limit the specific type of the electronic device.

[0039] Figure 1 An implementation flowchart of the two-stage heterogeneous remote sensing image registration method with radiation and rotation invariance provided by the embodiment of the present application is shown. As an example but not limitation, the method can be applied to the above-mentioned electronic device and includes steps S101-S106, which are described as follows:

[0040] S101, performing WLD filtering on the to-be-registered image set to obtain a WLD structure saliency map and a WLD direction map sequence of the to-be-registered image set.

[0041] For example, the to-be-registered image set can include a reference image and a to-be-registered image.

[0042] In a possible implementation, the WLD structure saliency map of the to-be-registered image set can be equivalent to a differential excitation image in a conventional enhanced Weber Local Descriptor filtering algorithm.

[0043] In an example, the WLD structure saliency map of the to-be-registered image set can be calculated using a conventional WLD algorithm.

[0044] In another example, if at least one of the reference image and the to-be-registered image is a SAR image, the pixel value of each pixel in the image can be weighted, and the WLD structure saliency map of the to-be-registered image set is calculated using the weighted pixel value, so that the WLD structure saliency map of the to-be-registered image set can more accurately represent the structural features of the to-be-registered image set.

[0045] For example, the WLD structure saliency map can satisfy the following formula:

[0046]

[0047] Wherein:

[0048]

[0049]

[0050] The WLD structure saliency map of a pixel point p in the to-be-registered image or the reference image, is a set of pixel points in a rectangular neighborhood centered on the pixel point p , and represents the pixel pointq for One pixel in the image. In pixels q The set of pixels in the rectangular neighborhood centered on the center. Represents pixels m for One pixel in the image. For pixels m pixel values, The standard deviation parameter, , Each pixel m of x axis coordinate values y axis coordinate values, , Each pixel q of x axis coordinate values y Axis coordinate values.

[0051] For example, the size of the rectangular neighborhood can be .

[0052] In one possible implementation, to address the impact of rotational differences in remote sensing images, the WLD pattern sequence may include WLD patterns that correspond one-to-one with multiple directions, established through convolution. The WLD pattern can be equivalent to the pattern in the WLD algorithm.

[0053] In one example, see Figure 2 The vertical and horizontal convolution kernels can be determined based on the size of the rectangular neighborhood. Then, these two kernels are used in sequence... i After rotating the original image group to be registered clockwise by a preset angle, a convolution is performed to obtain the convolutional image. and Based on these two convolutional graphs, the first... i One WLD orientation pattern.

[0054] For example, see Figure 2 The vertical convolution kernel can be a kernel of size 1. The matrix has elements with a value of 0 in the middle column, -1 on the left, and 1 on the right.

[0055] Similarly, see Figure 2 The horizontal convolution kernel is also a kernel of size 1. The matrix has elements with a value of 0 in the middle row, -1 in the bottom row, and 1 in the top row.

[0056] For example, the first i Each preset angle can be ,inL the total number of WLD direction patterns, i = 0, 1,... L - 1.

[0057] Exemplarily, the WLD direction pattern can satisfy the following formula:

[0058]

[0059] is the i-th WLD direction pattern of the pixel point p, i , is a mapping function, is a convolution pattern processed by a vertical direction convolution kernel, is a convolution pattern processed by a horizontal direction convolution kernel, is a constant for preventing division by zero, and can be equal to .

[0060] S102, constructing a feature description vector of the image group to be registered based on the WLD structure saliency map and the WLD direction pattern sequence.

[0061] In some embodiments, the feature points of the image to be registered can be extracted based on the WLD structure saliency map first, and the first principal direction of each feature point is determined; then the feature description vector of each feature point is determined according to the first principal direction of each feature point.

[0062] Exemplarily, the feature description vectors of all the feature points constitute the feature description vector of the image group to be registered.

[0063] In a possible implementation, the feature points of the image group to be registered can be obtained by extracting the interest points (pixels) of the WLD structure saliency map using the Features from Accelerated Segment Test (FAST), and then eliminating unstable points with a response value lower than a response value threshold and points with insufficient image edge neighborhood information.

[0064] Exemplarily, the number of feature points can be up to 5000.

[0065] In a possible implementation, based on the WLD algorithm, the WLD structure saliency map of each feature point is taken as the difference image in the WLD algorithm, and one WLD direction pattern of the feature point is taken as the direction image in the WLD algorithm, and one-dimensional WLD histograms of the two images are calculated to obtain a plurality of first WLD histograms of the feature point. Then, all the first WLD histograms of the feature points are superimposed, and the direction corresponding to the highest histogram column in the superimposed first WLD histogram is determined as the first principal direction of the feature point.

[0066] ​For example, refer to Figure 3 The first WLD histogram can include a plurality of histogram columns corresponding to a plurality of directions. The first histogram column can correspond to the first preset angle. i The second histogram column can correspond to the second preset angle. i

[0067] In one example, a third circular region of the feature point can be obtained by intercepting a circular neighborhood with a radius of centered at the feature point from the WLD structure saliency map of the feature point and each WLD direction map in the WLD direction map sequence. Then, the first WLD histogram of the feature point can be determined based on the WLD algorithm according to the WLD structure saliency map and the first WLD direction map in the third circular region of the feature point. i i

[0068] In one example, refer to Figure 3 The first WLD histogram of the feature point can be obtained by adding the histogram columns corresponding to the same feature point of all the first WLD histograms. Then, the direction corresponding to the highest histogram column (for example, the histogram column x5 in Figure 3 ) can be taken as the first principal direction of the feature point.

[0069] In one possible implementation, a first circular region of the feature point can be obtained by intercepting a circular neighborhood with a radius of centered at the feature point from the WLD structure saliency map of the feature point and the WLD first principal direction map. Then, the first circular region can be divided according to different angles and radii to obtain a plurality of first sub-regions as shown in Figure 4 Finally, the feature description vector of the feature point can be obtained by vectorizing the second WLD histogram in each first sub-region, and then sequentially splicing and normalizing the second WLD histogram.

[0070] For example, the WLD first principal direction map can be the WLD direction map corresponding to the first principal direction of the feature point.

[0071] For example, the number of WLD structure saliency columns and the number of WLD structure direction columns in the second WLD histogram are 14 and 7, respectively.

[0072] Optionally, the first circular region can be rotated to the first principal direction of the feature point before the first sub-regions are divided.

[0073] In one example, refer to Figure 4 The first circular region can be divided into a plurality of first sub-regions according to different radii.​​​ The concentric circles are divided into

[0074] The radius may be equal to 70, may be 6, and the number of sectors may be 12.

[0075] S103, matching feature points between the image groups to be registered according to the feature description vectors, to obtain a rotation reference point pair.

[0076] In a possible implementation, the Euclidean distance between the feature description vector of the reference image and the feature description vector of the image to be registered can be calculated, and the feature points corresponding to the nearest pair of feature description vectors are taken as a pair of first matching points; then the errors of all the first matching point pairs are estimated, and the first matching point pair with the minimum error is taken as the rotation reference point pair for subsequent fine registration adjustment; wherein is a feature point in the reference image, is a feature point in the image to be registered.

[0077] For example, if the feature description vector of a feature point in the reference image is x1, and the feature description vectors of all the feature points in the image to be registered are y1-y 5000 , the Euclidean distances between x1 and y1, x1 and y2, …, x1 and y 5000 may be calculated, and if the Euclidean distance between x1 and y1 is the nearest, the feature points corresponding to x1 and y1 are taken as a pair of first matching points.

[0078] In an example, before estimating the errors of the first matching point pairs, the unstable matching points in the first matching point pairs, for example, the first matching point pairs with a Euclidean distance greater than a first distance threshold, can be removed; then the affine transformation matrix between the remaining first matching point pairs is used to perform further gross error removal by using the Random Sample Consensus (RANSAC) algorithm.

[0079] In an example, the affine transformation matrix between the first matching point pairs after gross error removal can be estimated, and then the errors between the first matching point pairs after gross error removal are estimated by using the estimated affine transformation matrix.

[0080] S104, extracting the WLD structural feature description vector of the image group to be registered according to the rotation reference point pair.

[0081] ​​In one possible implementation, the second principal direction of each feature point in the reference image can be calculated by the relative direction of the feature point pointing to the rotation reference point . Similar to step S102, a second circular region can be obtained by taking a circular neighborhood with a radius of centered at each feature point on the WLD structure saliency map and the WLD second principal direction map. The second circular region is rotated to the second principal direction, and the rotated second circular region is divided into second sub-regions according to different angles and radii (see Figure 4 ). A third WLD structure saliency map of each second sub-region of the feature point is obtained according to the WLD structure saliency map and the WLD second principal direction map of the feature point. The third WLD histogram of each second sub-region is vectorized and then sequentially spliced and normalized to obtain a WLD structure feature description vector of each feature point.

[0082] For example, the WLD second principal direction map is a WLD direction map corresponding to the second principal direction of the feature point.

[0083] In one example, the radius may be 60, the number of concentric circles may be 8, and the number of sectors may be 8. The number of WLD structure saliency columns of the third WLD histogram in two dimensions may be 14, and the number of WLD structure direction columns may be 7.

[0084] S105, the feature points between the to-be-registered image groups are matched again according to the WLD structure feature description vectors to obtain a plurality of pairs of final matching points.

[0085] In one example, similar to step S103, the Euclidean distance between the WLD structure feature description vector of the reference image and the WLD structure feature description vector of the to-be-registered image can be calculated, and then a second matching point pair of the to-be-registered image group is constructed according to the nearest neighbor Euclidean distance; the matching points in the second matching point pair that do not satisfy a preset condition are removed according to the nearest neighbor Euclidean distance and the second nearest neighbor Euclidean distance to obtain the final matching point pair.

[0086] For example, the preset condition can be less than a second distance threshold.

[0087] Similarly, the RANSAC algorithm can also be used to remove gross errors from the final matching point pair based on the affine transformation matrix between the final matching point pairs.

[0088] Specifically, the RANSAC algorithm can divide the inliers and outliers according to the estimation error, and obtain the best model in the iteration process, and finally obtain the corresponding inlier set as the final matching point pair after removing the gross errors.

[0089] S106, registering the to-be-registered image group according to the final matching point pairs.

[0090] In one example, the coordinates of the final matching point pairs after the elimination of the gross errors can be substituted into an affine transformation model to fit an optimal affine transformation matrix; and according to the affine transformation matrix, the resampling of the to-be-registered image is completed by means of bilinear interpolation, so that all pixel space positions of the to-be-registered image and the reference image are consistent.

[0091] According to the method provided in the application, the structural features of the to-be-registered image group, i.e., the feature description vectors and the WLD structural feature description vectors, are extracted by the WLD algorithm, and the registration according to these features can avoid errors caused by the radiation difference between the heterogeneous reference image and the to-be-registered image, and improve the registration accuracy; the twice matching of the to-be-registered image group based on the WLD algorithm can increase the robustness of the WLD structural feature description vectors, which are rotation-invariant features, thereby further enhancing the registration effect.

[0092] Figure 5 Fig. 1 shows a structural schematic diagram of a two-stage heterogeneous remote sensing image registration device with radiation and rotation invariance provided by an embodiment of the application. By way of example but not limitation, the device 500 can include a filtering module 510, a feature extraction module 520, and a registration module 530.

[0093] By way of example, the filtering module 510 is configured to perform WLD filtering on the to-be-registered image group to obtain a WLD structural saliency map and a WLD direction map sequence of the to-be-registered image group, wherein the to-be-registered image group includes a reference image and a to-be-registered image of the same scene, and the WLD direction map sequence includes WLD direction maps corresponding to a plurality of directions one by one; the feature extraction module 520 is configured to construct a feature description vector of the to-be-registered image group based on the WLD structural saliency map and the WLD direction map sequence; and the registration module 530 is configured to match feature points between the to-be-registered image group according to the feature description vector to obtain a rotation reference point pair of the to-be-registered image group, wherein the rotation reference point pair includes a pair of matching points from the reference image and the to-be-registered image, respectively; the feature extraction module 520 is further configured to extract a WLD structural feature description vector of the to-be-registered image group according to the rotation reference point pair, and match feature points between the to-be-registered image group again according to the WLD structural feature description vector to obtain a plurality of pairs of final matching point pairs of the to-be-registered image group; and the registration module 530 is further configured to register the to-be-registered image group according to the final matching point pairs.

[0094] In order to better illustrate the beneficial effects of the application, the following simulation experiments are performed:

[0095] For example, the simulation experiment can be simulated on a central processing unit of 13th Gen Intel(R) Core(TM) i7-13700 2.10 GHz CPU, a Windows 11 operating system, and MATLAB 2022b developed by Mathworks, USA. The test data is 20 pairs of optical and SAR remote sensing image pairs, respectively from Sentinel-2 and Sentinel-1 satellite remote sensing images, and the images are cropped to 800x800 pixels. To simulate the rotation difference in real situations, each SAR image is rotated by 0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, and 95 degrees, respectively, to obtain 20x7 pairs of image pairs for testing.

[0096] Specifically, the simulation experiment can compare the registration effect of the scale-invariant feature description method (SAR-SIFT) from the paper: SAR-SIFT: A SIFT-Like Algorithm for SAR Images. IEEE Transactions on Geoscience and Remote Sensing, 33(1): 453-466, 2015, the radiation-invariant feature transformation algorithm (RIFT) from the paper: Multi-Modal Image Matching Based on Radiation-Variation Insensitive Feature Transform. IEEE Transactions on Image Processing, 29: 3296-3310, 2020, and the present application.

[0097] Figure 6 A comparison diagram of the registration effect provided by an embodiment of the present application is shown.

[0098] In one example, the optical image can be used as a reference image, and the SAR image can be used as a to-be-registered image. After registration, the registration effect is measured by two indicators, and the measurement indicators are the correct matching point ratio and the root mean square error of the matching point pair.

[0099] For example, the root mean square error of the matching point pair can satisfy the following formula:

[0100]

[0101] wherein, is the root mean square error of the matching point pair, represents the error calculated according to the true transformation matrix the number of point pairs with an error less than 3 pixels, is the pixel from the to-be-registered image in the final matching point pair,​ to match the pixel from the reference image in the final matching point pair.

[0102] Specifically, referring to Figure 6 , Figure 6 (a) in the figure is a comparison diagram of the correct matching point pair ratio of the SAR-SIFT method, the RIFT method and the method provided by the application, Figure 6 (b) in the figure is a comparison diagram of the root mean square error of the matching point pair of the AR-SIFT method, the RIFT method and the method provided by the application.

[0103] Referring to Figure 6 It can be seen that the method provided by the application can cope with the significant radiation difference in the heterogeneous remote sensing images, and through the use of the two-stage image registration framework based on the global reference center, the robust registration of the heterogeneous remote sensing images can be realized under different levels of rotation difference, and more accurate registration effect can be achieved compared with the comparison method.

[0104] Figure 7 The figure shows a schematic diagram of the registration effect provided by the application.

[0105] Referring to Figure 7 , Figure 7 (a) in the figure is a schematic diagram of the final matching point pair obtained by the method provided by the application under the condition that the rotation angle of the image to be registered relative to the reference image is different, Figure 7 (b) in the figure is a schematic diagram of the registered image group to be registered obtained by the method provided by the application under the condition that the rotation angle of the image to be registered relative to the reference image is different.

[0106] Referring to Figure 7 It can be seen that the method provided by the application can obtain a large number of final matching point pairs under different rotation differences, and realizes the robust registration of the image to be registered under different rotation angles.

[0107] Therefore, according to the method provided by the application, the structural features of the image group to be registered, i.e. the feature description vector and the WLD structural feature description vector, are extracted by the WLD algorithm, and the registration according to these features can avoid the error caused by the radiation difference between the reference image and the image to be registered, and improve the registration accuracy; the WLD algorithm is used to match the image group to be registered twice, which can increase the robustness of the WLD structural feature description vector which is a rotation invariant feature, thereby further enhancing the registration effect.

[0108] Figure 8 The figure shows a structural schematic diagram of an electronic device provided by an embodiment of the application. As Figure 8 The electronic device 800 shown can include at least one processor 810 Figure 8The electronic device 800 can be a processing device such as a robot, which can implement the above method. The embodiments of the present application do not limit the specific type of the electronic device.

[0109] The electronic device 800 can be a processing device such as a robot, which can implement the above method. The embodiments of the present application do not limit the specific type of the electronic device.

[0110] Those skilled in the art can understand that Figure 8 The electronic device 800 is only an example and does not limit the electronic device. The electronic device 800 can include more or fewer components than shown, or can combine some components, or have different components. For example, the electronic device 800 can also include an input / output interface.

[0111] The processor 810 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic components, discrete hardware components, or the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0112] The memory 820 can be an internal storage unit, such as a hard disk or a memory, in some embodiments. The memory 820 can also be an external storage device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, or the like, in other embodiments. Further, the memory 820 can include both an internal storage unit and an external storage device. The memory 820 is used to store an operating system, an application program, a boot loader, data, and other programs, such as program codes of the computer program. The memory 820 can also be used to temporarily store data that has been output or will be output.

[0113] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0114] It can be clearly understood by those skilled in the art that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for convenient distinction, and do not limit the protection scope of the present application. The specific working process of the unit and module in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0115] The embodiment of the present application also provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps in each of the above method embodiments.

[0116] The embodiment of the present application provides a computer program product, when the computer program product runs on an electronic device, so that the electronic device executes to realize the steps in each of the above method embodiments.

[0117] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the present application realizes all or part of the processes in the above embodiment methods, which can be completed by instructing related hardware through a computer program. The computer program can be stored in a computer readable storage medium. The computer program, when executed by a processor, can realize the steps of each of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium at least includes any entity or device capable of carrying the computer program code to the photographing device / terminal equipment, recording medium, computer memory, read-only memory (ROM, Read Only Memory), etc. Only Memory), random access memory (RAM), electrical carrier signals, telecommunication signals and software distribution media. For example, U-disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, computer readable media can not be electrical carrier signals and telecommunication signals.

[0118] In the above embodiments, the description of each embodiment is focused on, and the part not described or recorded in a certain embodiment can be referred to the relevant description of other embodiments.

[0119] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software mode depends on the specific application and design constraints of the technical solution. The professional technicians can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

Claims

1. A two-stage heterogeneous remote sensing image registration method with radiometric and rotational invariance, characterized in that, include: Enhanced Weber Local Descriptor (WLD) filtering is applied to the image group to be registered to obtain the WLD structure saliency map and WLD orientation map sequence of the image group to be registered. The image group to be registered includes a reference image and the image to be registered with the same shooting scene. The WLD orientation map sequence includes WLD orientation maps that correspond one-to-one with multiple directions. The feature description vector of the image group to be registered is constructed based on the WLD structure saliency map and the WLD orientation map sequence. Based on the feature description vector, feature points between the groups of images to be registered are matched to obtain rotation reference point pairs of the groups of images to be registered, wherein the rotation reference point pairs include a pair of matching points respectively from the reference image and the image to be registered; Based on the rotation reference point pair, extract the WLD structural feature description vector of the image group to be registered, and match the feature points between the image groups to be registered again based on the WLD structural feature description vector to obtain multiple pairs of final matching point pairs of the image group to be registered. The image group to be registered is registered according to the final matching point pair; The step of constructing the feature description vector of the image group to be registered based on the WLD structure saliency map and the WLD orientation map sequence includes: Feature points of the image group to be registered are extracted based on the WLD structure saliency map. Based on the WLD structure saliency map and the WLD direction map sequence, multiple one-dimensional first WLD histograms are determined for each feature point, wherein the first WLD histogram includes bars that correspond one-to-one with the multiple directions; The first WLD histogram of the same feature point is obtained by superimposing the first WLD histogram of the feature point; The direction corresponding to the highest bar in the superimposed first WLD histogram is taken as the first principal direction of the feature point. The feature description vectors of the feature points are determined based on the first principal orientation map of WLD and the WLD structure saliency map, wherein the first principal orientation map of WLD is the WLD orientation map corresponding to the first principal orientation, and the feature description vectors of all feature points constitute the feature description vectors of the image group to be registered. The step of matching feature points between the image groups to be registered based on the feature description vector to obtain rotation reference point pairs for the image groups to be registered includes: Calculate the Euclidean distance between the feature description vector of the reference image and the feature description vector of the image to be registered; The feature points corresponding to the pair of feature description vectors that are closest in Euclidean distance are taken as the first matching point pair; Estimate the error of each pair of the first matching points, and take the first matching point pair with the smallest estimated error as the rotation reference point pair.

2. The method according to claim 1, characterized in that, At least one image in the group of images to be registered is a synthetic aperture radar image; The WLD structure saliency map satisfies the following formula: in: A pixel in the image to be registered or the reference image. p The WLD structure saliency diagram, In pixels p The set of pixels in the rectangular neighborhood centered on the center. Represents pixels q for One pixel in the image. In pixels q The set of pixels in the rectangular neighborhood centered on the center. Represents pixels m for One pixel in the image. For pixels m pixel values, The standard deviation parameter, , Each pixel m of x axis coordinate values y axis coordinate values, , Each pixel q of x axis coordinate values y Axis coordinate values.

3. The method according to claim 1, characterized in that, The step of determining the feature description vector of the feature point based on the first principal orientation map of the WLD and the WLD structural saliency map includes: A first circular region is obtained by extracting a circular neighborhood centered on the feature point on the WLD structure saliency map and the WLD first main orientation map. The first circular region is divided into a first sub-region based on different angles and radii; Based on the WLD structure saliency map and the WLD first principal orientation map, a two-dimensional second WLD histogram is calculated for each of the first sub-regions; The second WLD histogram of each of the first sub-regions is vectorized and then sequentially concatenated and normalized to obtain the feature description vector of the feature point.

4. The method according to claim 1, characterized in that, The step of extracting the WLD structural feature description vector of the image group to be registered based on the rotation reference point pair includes: The second principal direction of the feature point is obtained by calculating the relative direction from the feature point in the reference image to the rotation reference point; A second circular region is obtained by extracting a circular neighborhood centered on the feature point on the WLD structure saliency map and the WLD second main direction map, wherein the WLD second main direction map is the WLD direction map corresponding to the second main direction; The second circular region is rotated to the second main direction and the rotated second circular region is divided according to different angles and radii to obtain the second sub-region; Based on the WLD structure saliency map and the WLD second principal orientation map, a two-dimensional third WLD histogram of the second sub-region is calculated; After vectorizing the third WLD histogram of each second sub-region, the vectors are sequentially concatenated and normalized to obtain the WLD structural feature description vector of the feature point.

5. The method according to claim 1, characterized in that, The step of matching feature points between the groups of images to be registered again based on the WLD structural feature description vector to obtain multiple pairs of final matching point pairs for the groups of images to be registered includes: Calculate the Euclidean distance between the WLD structural feature description vector of the reference image and the WLD structural feature description vector of the image to be registered; The second matching point pair of the image group to be registered is constructed based on the nearest neighbor Euclidean distance, and the matching points in the second matching point pair that do not meet the preset conditions are removed based on the nearest neighbor Euclidean distance and the second nearest neighbor Euclidean distance, so as to obtain the final matching point pair.

6. A two-stage heterogeneous remote sensing image registration device with radiometric and rotational invariance, characterized in that, include: The filtering module is used to perform enhanced Weber Local Descriptor (WLD) filtering on the image group to be registered to obtain the WLD structure saliency map and WLD orientation map sequence of the image group to be registered. The image group to be registered includes a reference image and an image to be registered with the same shooting scene. The WLD orientation map sequence includes WLD orientation maps that correspond one-to-one with multiple directions. The feature extraction module is used to construct a feature description vector for the image group to be registered based on the WLD structure saliency map and the WLD orientation map sequence. A registration module is used to match feature points between the image groups to be registered according to the feature description vector to obtain rotation reference point pairs of the image groups to be registered, wherein the rotation reference point pairs include a pair of matching points respectively from the reference image and the image to be registered; The feature extraction module is further configured to extract the WLD structure feature description vector of the image group to be registered based on the rotation reference point pair, and match the feature points between the image groups to be registered again based on the WLD structure feature description vector to obtain multiple pairs of final matching point pairs of the image group to be registered. The registration module is also used to register the image group to be registered according to the final matching point pair; The feature extraction module is specifically used to: extract feature points of the image group to be registered based on the WLD structure saliency map; Based on the WLD structure saliency map and the WLD direction map sequence, multiple one-dimensional first WLD histograms are determined for each feature point, wherein the first WLD histogram includes bars that correspond one-to-one with the multiple directions; The first WLD histogram of the same feature point is obtained by superimposing the first WLD histogram of the feature point; The direction corresponding to the highest bar in the superimposed first WLD histogram is taken as the first principal direction of the feature point. The feature description vectors of the feature points are determined based on the first principal orientation map of WLD and the WLD structure saliency map, wherein the first principal orientation map of WLD is the WLD orientation map corresponding to the first principal orientation, and the feature description vectors of all feature points constitute the feature description vectors of the image group to be registered. The registration module is specifically used to: calculate the Euclidean distance between the feature description vector of the reference image and the feature description vector of the image to be registered; The feature points corresponding to the pair of feature description vectors that are closest in Euclidean distance are taken as the first matching point pair; Estimate the error of each pair of the first matching points, and take the first matching point pair with the smallest estimated error as the rotation reference point pair.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by an electronic device, it implements the method as described in any one of claims 1-5.

Images