Optical and sar image registration method based on self-similarity structure difference information

Abstract

The application discloses an optical and SAR image registration method based on self-similar structure difference information, which comprises the following steps: S1, constructing a nonlinear scale space, and then constructing a nonlinear Harris scale space based on the nonlinear scale space to extract feature points; S2, selecting a logarithmic polar coordinate descriptor to match the feature points; S3, removing the false matching point pairs by using geometric probability information to obtain the coarsely corrected image point pairs; S4, screening the initial optical feature points based on structure information and mapping to the SAR image; screening the feature points on the obtained SAR image again, removing the corresponding matching point pairs at the same time, and forming an initial matching point pair set; and S5, constructing a self-similar structure difference descriptor and adjusting the positions of the feature points on the SAR image. The method can effectively describe the neighborhood information of the feature points, has good distinguishability, and realizes accurate registration of the images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to an optical and SAR image registration method based on self-similar structure difference information. BACKGROUND

[0002] Image registration refers to a process of finding a spatial transformation that maps a to-be-registered image to a reference image, so that points in the image corresponding to the same position in space are one-to-one corresponding. With the extensive research and development of deep learning, it plays an increasingly important role in image registration, and gradually becomes a hot spot in image matching. However, the dependence of deep learning model on a large amount of annotated data is still a major obstacle. The high cost of SAR image acquisition and the scarcity of annotated data limit the amount and diversity of available data, hindering the wide practical application of deep learning model in the field. For a long time, feature-based methods have been the focus of research in the field of optical and SAR image registration. These methods aim to identify features that can be clearly distinguished from other areas and are repeatable, so as to achieve reliable feature matching.

[0003] Optical images can provide very high spatial resolution and texture details, have strong intuitiveness, and can have very high measurement accuracy, but are easily affected by light and atmospheric factors. SAR can work normally in bad environments such as deep night and rainy days, and has all-weather and all-day imaging capability. The combination of optical images and SAR images can achieve complementary information and form a more comprehensive and accurate information source, thereby improving the accuracy of subsequent image processing tasks. However, due to the differences in imaging mechanisms between optical images and SAR images, SAR images are affected by speckle noise, and there are obvious radiation differences and local geometric distortions between the two kinds of images, which leads to great challenges in achieving high-precision registration of SAR images and optical images.

[0004] The complementary imaging characteristics of SAR images and optical images make the integration of these two data types have wide application and great significance in image processing tasks such as change detection, image fusion and 3D reconstruction. Image registration is a crucial step in these tasks, and the accuracy of registration directly affects the accuracy and reliability of their results. SAR image and optical image registration has wide application prospects in city planning and management, environmental monitoring, disaster response and agricultural monitoring, so studying SAR image and optical image registration has important practical significance. SUMMARY

[0005] The present application aims to solve the problems that optical image and SAR image registration algorithms are difficult to meet the robustness and precision at the same time, and proposes an optical and SAR image registration method based on self-similar structure difference information.

[0006] In order to solve the above technical problems, the technical solution of the present invention is:

[0007] A method for optical and SAR image registration based on self-similar structural difference information includes the following steps:

[0008] S1. Construct a nonlinear scale space on the optical image and the SAR image, then construct a nonlinear Harris scale space on the nonlinear scale space, extract extreme points in the nonlinear Harris scale space and use them as feature points;

[0009] S2. Selecting a logarithmic polar coordinate descriptor to describe the feature points, and then matching the feature points using the logarithmic polar coordinate descriptor to obtain matching point pairs;

[0010] S3. Use geometric probability information to eliminate mismatched point pairs to obtain roughly corrected image point pairs. Perform FSC processing on the roughly corrected image point pairs to obtain an initial transformation matrix and perform preliminary image registration.

[0011] S4. The feature points of the extracted optical image are screened based on the structural information and mapped to the SAR image. The feature points on the SAR image obtained after mapping are screened again, and the corresponding matching point pairs are removed while screening to form a preliminary set of matching point pairs;

[0012] S5. Construct a self-similar structure difference descriptor, adjust the position of feature points on the SAR image, and obtain accurate matching point pairs to achieve the final image registration.

[0013] Preferably, step S3 specifically includes the following processing steps:

[0014] S3-1: Give an evaluation criterion based on probability information to determine whether it is a correct matching point pair. Ns represents the number of successes, Nl represents the number of failures, and when Ns ≥ 2 and Ns > Nl, the judgment is true. When Nl = 3, the judgment is false.

[0015] S3-2: Traverse the obtained matching point pairs and determine whether they are correct matching point pairs. For each set of point pairs to be determined, randomly select two other sets of matching point pairs. On the optical image, connect the remaining two randomly selected feature points with the feature point to be measured as the vertex, and calculate the angle θ formed by them. o Similarly, the feature point on the SAR image corresponding to the feature point to be measured on the optical image is connected to the other two corresponding randomly selected feature points, and the angle θ formed by them can be calculated. s . Set a threshold T a , when θ o -θ s |≤T aWhen , it is recorded as a success, and the matching point pair is judged as a wrong matching point pair according to the evaluation criteria and is removed;

[0016] S3-3: Randomly select two sets of matching point pairs from the new matching point set, and obtain the distance between the two feature points on their respective images. and Calculate the ratio between them Repeat the selection multiple times and calculate the ratio σ obtained each time i Average value As a reference scale ratio between two images;

[0017] S3-4: After obtaining the new matching point set, traverse each set of feature point pairs again. For each set of point pairs to be judged, randomly select another set of matching point pairs. According to S3-3, the scale ratio σ between them can be obtained, and the threshold T is set. r ,when When , it is recorded as a success, and the matching point pair is judged to be a correct matching point pair according to the evaluation criteria. FSC is then performed on the final matching point set to obtain the initial transformation matrix;

[0018] Preferably, step S4 specifically includes the following processing steps:

[0019] S4-1: Calculate the phase consistency response amplitude of the initial feature points of the optical image, and remove 20% of the feature points in ascending order of response value, while retaining 80% of the feature points;

[0020] S4-2: The Canny algorithm obtains a binary edge map of the optical image. In the binary edge map, the feature points on the optical image are traversed and the ratio T of the pixels occupied by the true value in the n×n area centered on the feature point is calculated. e , set a threshold d ed , when T e <d ed , remove the feature point.

[0021] S4-3: The feature points after optical image screening are mapped to the SAR image, and steps S4-1 and S4-2 are repeated on the SAR image. At the same time, the feature points on the corresponding optical image are deleted to obtain the final initial matching point pairs;

[0022] Preferably, step S5 specifically includes the following processing steps:

[0023] S5-1: Divide the area where the optical image feature points are located into several blocks. In each block, use the logarithmic Gabor odd-symmetric wavelet to obtain its direction information, and construct a phase information histogram based on the phase response value.

[0024] S5-2: Label each area from left to right and then from top to bottom.

[0025] S5-3: Compare each block with each other in pairs, and complete the comparison of each block with other uncompared blocks in the order of labeling. Based on the phase amplitude direction histogram of S5-1, a descriptor that can represent the structural difference between blocks is constructed, and its correlation is mapped into a binary form to obtain the descriptor of each feature point on the optical image;

[0026] S5-4: For each feature point on the optical image, traverse the m×m region centered on the corresponding feature point on the SAR image, and construct a descriptor for each pixel in the region according to S5-2 and S5-3. Based on the number N of identical values ​​between the optical image feature point descriptor and the SAR image pixel descriptor at the same position, select the pixel in the region on the SAR image that maximizes N as the new feature point corresponding to the optical image feature point to achieve more accurate matching.

[0027] The present invention has the following characteristics and beneficial effects:

[0028] The present invention proposes an improved FSC method that combines probability and geometric information, significantly speeds up the time required when FSC is dominated by outliers, and enhances the robustness of the algorithm.

[0029] The present invention adopts a feature point screening method based on phase consistency information and feature point neighborhood structure information to eliminate feature points with poor domain structure information and retain feature points with strong robustness.

[0030] The present invention proposes a descriptor constructed by utilizing the self-similar structural difference information in the field of image feature points. The descriptor can effectively describe the neighborhood information of the feature points, has good distinguishability, and realizes accurate image registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 It is the main process of the coarse-to-fine optical and SAR image registration method based on the self-similarity structure and self-difference information of the images.

[0033] Figure 2 This is the first set of experimental simulation diagrams in this embodiment.

[0034] Figure 3 yes Figure 2 After the key points of the two images are matched using the above matching method, a one-to-one corresponding key point pair between the two images is obtained.

[0035] Figure 4 It is a chessboard fusion image of the registration results before and after fine registration. Figure 4 (a) is the fusion image after rough registration. Figure 4 (b) is the fusion image after precise registration. DETAILED DESCRIPTION

[0036] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0037] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings.

[0038] On the contrary, the present invention covers any alternatives, modifications, equivalents, and solutions that fall within the spirit and scope of the present invention as defined by the claims. Furthermore, to facilitate a better understanding of the present invention, certain specific details are described in detail below in the detailed description of the present invention. Those skilled in the art will be able to fully understand the present invention without these details.

[0039] The present invention provides an optical and SAR image registration method based on self-similar structure difference information, such as Figure 1 As shown, the following steps are included:

[0040] Step 1: First, obtain the optical image and SAR image. In this embodiment, Figure 2 As shown in the figure, optical images and SAR images taken at different times in the same area are selected, and the size is 628*618. Figure 2 (a) was filmed in March 2009. Figure 2 (b) was taken in January 2008. A nonlinear scale space is constructed on the optical image and the SAR image, and then a nonlinear Harris scale space is constructed on the nonlinear scale space. Extreme points are extracted from the nonlinear Harris scale space and used as feature points.

[0041] Step 2: Select the logarithmic polar coordinate descriptor to describe the feature points, and then match the feature points using the logarithmic polar coordinate descriptor to obtain matching point pairs.

[0042] It can be understood that the matching point pair refers to the corresponding feature points in the optical image and the SAR image, such as Figure 3 As shown in FIG, after the two images are matched with key points by the above matching method, a one-to-one corresponding key point pair between the two images is obtained.

[0043] Step 3: Use geometric probability information to remove the wrong matching point pairs to obtain the roughly corrected image point pairs. Perform FSC processing on the roughly corrected image point pairs and calculate the initial transformation matrix. The fusion image result of the roughly registered image is as follows: Figure 4 As shown in (a).

[0044] Specifically, in this embodiment, a criterion for determining whether a point pair is correctly matched is provided by combining probability information. Ns represents the number of successes, Nl represents the number of failures, and the judgment rule is defined as follows:

[0045]

[0046] When Ns≥2 and Ns>Nl, the judgment is true. When Nl=3, the judgment is false. Wherein, Q is a state variable indicating whether the matching point pair is a correct match.

[0047] It should be noted that the determination of the state variables depends not only on the final number of successful and failed trials, but also on the variables involved in the process.

[0048] Furthermore, the erroneous matching point pairs are screened out by the following two erroneous matching point pair screening methods.

[0049] First, traverse the obtained matching point pairs. For each set of point pairs to be judged, randomly select any two other sets of matching point pairs. On the optical image, connect the remaining two randomly selected feature points with the feature point to be measured as the vertex, and calculate the angle θ formed by them. o Similarly, the feature point on the SAR image corresponding to the feature point to be measured on the optical image is connected to the other two corresponding randomly selected feature points, and the angle θ formed by them can be calculated. s In this embodiment, the threshold T is set a =π / 18, when |θ o -θ s |≤T a , it is recorded as a success, otherwise it is recorded as a failure, and then the matching point pair is judged as an incorrect matching point pair according to the evaluation criteria and is eliminated.

[0050] Then, a new set of matching point pairs is obtained after the first screening, and then each set of matching point pairs in the new set is traversed again. For each set of point pairs to be judged, another set of matching point pairs is randomly selected. From these two sets of matching point pairs, the distance between the two feature points on each image is obtained. and Calculate the scale ratio of the two sets of matching point pairs In this embodiment, the threshold T is set r =1 / 3, when When , it is recorded as a success, and the matching point pair is judged according to the evaluation criteria to determine whether it is a correct matching point pair, and then the wrong matching point pairs are eliminated.

[0051] in, As a reference scale ratio, it is obtained as follows:

[0052] Randomly select two sets of matching point pairs from the new matching point pair set obtained after the first screening, and obtain the distance between the two feature points on their respective images. and Calculate the ratio between them Repeat N times and calculate the ratio σ obtained each time i Average value The average value is then used as the reference scale ratio between the two images.

[0053] After the above two wrong matching points are screened out, the roughly registered image point pairs are obtained.

[0054] The FSC processing is performed on the coarsely registered image point pairs to obtain the initial transformation matrix, and the initial transformation matrix is ​​used to perform affine transformation on the SAR image. The affine transformed SAR image is preliminarily registered with the optical image, and the affine transformed SAR image is applied in subsequent steps.

[0055] It should be noted that the FSC algorithm is a conventional technical means, which is mainly used in image registration to screen matching results to improve the accuracy and efficiency of registration. The FSC algorithm, or Fast Sample Consensus algorithm, is an iterative algorithm used to estimate mathematical model parameters from a set of data while robustly handling outliers and noise. In the context of image registration, the FSC algorithm is used to optimize and verify the corresponding point set obtained by feature matching, thereby ensuring the accuracy and reliability of registration.

[0056] Step 4: The feature points of the optical image extracted in step 1 are screened based on the structural information and mapped to the affine transformed SAR image. The feature points on the mapped SAR image are screened again, and the corresponding matching point pairs are removed during the screening to form an intermediate matching point pair set.

[0057] Specifically, the following sub-steps are included:

[0058] Step 4-1: Calculate the phase consistency response amplitude of the initial feature point of the optical image, where the phase consistency calculation formula is as follows:

[0059]

[0060] Among them, (x, y) represents the coordinates of the feature point, the subscript n represents the scale of the filter, and An (x,y) and j n (x,y) represents the amplitude and phase of the filter response, W(x,y) is the weight coefficient of frequency expansion, is the weighted average phase, T is the noise threshold, and ε is a very small constant.

[0061] The response values ​​of all initial feature points in the optical image are obtained by the above algorithm. In this embodiment, 20% of the feature points are eliminated in ascending order of response values, and 80% of the feature points are retained.

[0062] Step 4-2: Use the Canny algorithm to obtain a binary edge map of the optical image. In the binary edge map, traverse the feature points on the optical image and calculate the ratio T of the pixels occupied by the true value in the n×n area centered on the feature point. e =N p / N all , set a threshold d ed , when T e <d ed , remove the feature point.

[0063] Specifically, the formula is as follows:

[0064]

[0065] Among them, g(x,y) represents the response value of the edge map at the (x,y) position, N p Indicates the number of pixels with a value of 1 in the area, N all Indicates the total number of points in the area, d ed The empirical value of the ratio threshold is 0.4 and n is 10.

[0066] Step 4-3: Map the feature points after filtering the optical image to the SAR image after affine transformation, repeat steps 4-1 and 4-2 on the SAR image after affine transformation, and also delete the corresponding feature points on the optical image to obtain a preliminary set of matching point pairs.

[0067] Step 5: Construct a self-similar structure difference descriptor, adjust the position of the filtered feature points on the SAR image, and obtain accurate matching point pairs to achieve the final image registration.

[0068] Specifically, for each feature point retained after screening in step 4 on the optical image, a self-similar structure difference descriptor is constructed. All pixels within the m×m area centered on the feature point on the SAR image corresponding to the feature point are traversed, where m is 20, and a self-similar structure difference descriptor is constructed for each pixel in this area.

[0069] Among them, the method of constructing the self-similar structure difference descriptor is as follows:

[0070] The area where the optical image feature points are located is divided into several blocks. In each block, the direction information is obtained using the logarithmic Gabor odd-symmetric wavelet, and the phase information histogram is constructed by combining the phase response value.

[0071] Label each area from left to right and then from top to bottom;

[0072] The blocks are compared with each other in pairs, and each block is compared with other uncompared blocks in the order of the labels. A descriptor that can represent the structural difference between blocks is constructed based on the phase information histogram:

[0073] D asdpc ={Δd 1,2 ,Δd 1,3 ,...,Δd 1,8 ,Δd 2,3 ,...,

[0074] Δd 2,8 ,...,Δd 6，8 ,Δd 7,8}

[0075] Where Δd i,j Indicates the result of comparing the i-th block and the j-th block. The calculation formula is as follows:

[0076]

[0077] Where k represents the total number of bins. In the process of constructing the histogram, in order to overcome the gradient reversal phenomenon, the direction information is limited to 0-180 degrees and divided into k direction intervals. In the formula, and They represent the values ​​of the kth bin in the i-th and j-th region blocks respectively, and sgn(x) is a sign function.

[0078] The formula for sgn(x) is as follows:

[0079]

[0080] And map its correlation into binary form to obtain the descriptor of each feature point on the optical image.

[0081] Finally, by comparing with the descriptor constructed by the optical feature point, according to the number N of the same values ​​of the descriptor constructed by the optical image feature point and the pixel point descriptor on the SAR image at the same position, the pixel point in the area on the SAR image that makes N the largest is selected as the new feature point corresponding to the optical image feature point to achieve more accurate matching, such as Figure 4 (b) shown.

[0082] By Figure 4 As can be seen from the above (a) and (b), the fusion effect of fine registration is better than that of coarse registration, and after fine registration, good registration accuracy is achieved, and the algorithm can achieve good registration.

[0083] The embodiments of the present application are described in detail above with reference to the drawings, but the present application is not limited to the described embodiments. For those skilled in the art, various changes, modifications, replacements and variations of the embodiments including components are made without departing from the principles and spirits of the present application, and still fall within the protection scope of the present application.

Claims

1. A method for optical and SAR image registration based on self-similar structural difference information, characterized in that: The steps include: Step 1: first obtain an optical image and a SAR image, construct a nonlinear scale space on the optical image and the SAR image, then construct a nonlinear Harris scale space on the nonlinear scale space, extract extreme points on the nonlinear Harris scale space and use them as feature points; Step 2: Select the logarithmic polar coordinate descriptor to describe the feature points, and then match the feature points using the logarithmic polar coordinate descriptor to obtain matching point pairs; Step 3: Use geometric probability information to eliminate incorrect matching point pairs to obtain roughly corrected image point pairs. Perform FSC processing on the roughly corrected image point pairs to obtain an initial transformation matrix. Use the initial transformation matrix to perform affine transformation on the SAR image. The affine transformed SAR image is preliminarily aligned with the optical image. The affine transformed SAR image is used in all subsequent steps. Methods for removing incorrect matching point pairs include: Elimination of the first incorrect matching point pair: Determine whether the matching point pair is an incorrect matching point pair based on the evaluation criteria and eliminate it; The second screening of incorrect matching point pairs is as follows: a new set of matching point pairs is obtained after the first screening, and then each set of matching point pairs in the new set is traversed again. For each set of point pairs to be judged, any other set of matching point pairs is randomly selected, and the threshold T is set according to the scale ratio σ. r ,when When , it is recorded as a success, and the matching point pair is judged as a correct matching point pair according to the evaluation criteria, and then the wrong matching point pairs are eliminated; Step 4: The feature points of the optical image extracted in step 1 are screened based on the structural information and mapped to the SAR image. The feature points on the SAR image obtained after mapping are screened again, and the corresponding matching point pairs are removed during the screening to form a preliminary matching point pair set; Step 5: Construct a self-similar structure difference descriptor, adjust the position of the filtered feature points on the SAR image, and obtain accurate matching point pairs to achieve the final image registration.

2. The optical and SAR image registration method based on self-similar structural difference information according to claim 1, characterized in that In step 3, the evaluation criteria for filtering out incorrect matching point pairs are: Ns represents the number of successful verifications, Nl represents the number of failed verifications, and the judgment rules are defined as follows: Q is a state variable indicating whether the matching point pair is a correct match.

3. The optical and SAR image registration method based on self-similar structural difference information according to claim 2, characterized in that: In step 3, the method for filtering out incorrect matching point pairs is as follows: Traverse the obtained matching point pairs, and for each set of point pairs to be judged, randomly select any two other sets of matching point pairs, and calculate the angle θ formed by the feature point to be measured as the vertex on the optical image and SAR image respectively o and θ s , set the threshold T a , when |θ o -θ s |≤T a If the matching point pair is correct, it is recorded as a success, otherwise it is recorded as a failure. Then, the matching point pair is judged as an incorrect matching point pair according to the evaluation criteria and is eliminated.

4. The optical and SAR image registration method based on self-similar structural difference information according to claim 3, characterized in that: The method for obtaining the scale ratio σ is as follows: Randomly select two sets of matching point pairs from the new matching point pair set obtained after the first screening, and obtain the distance between the two feature points on their respective images. and Calculate the ratio between them Repeat the selection multiple times and calculate the ratio σ obtained each time i Average value As the reference scale ratio between the two images.

5. The optical and SAR image registration method based on self-similar structural difference information according to claim 1, characterized in that: Step 4 includes the following sub-steps: Step 4-1, calculate the phase consistency response amplitude of the initial feature points of the optical image, and remove 20% of the feature points in ascending order of response value, and retain 80% of the feature points; Step 4-2: Use the Canny algorithm to obtain a binary edge map of the optical image. In the binary edge map, traverse the feature points on the optical image and calculate the ratio T of the pixels occupied by the true value in the n×n area centered on the feature point. e , set a threshold d ed , when T e <d ed , remove the feature point.

6. The optical and SAR image registration method based on self-similar structural difference information according to claim 5, characterized in that: The step 4 further includes step 4-3: Map the feature points filtered from the optical image to the SAR image, repeat steps 4-1 and 4-2 on the SAR image, and also delete the corresponding feature points on the optical image to obtain a preliminary set of matching point pairs.

7. The optical and SAR image registration method based on self-similar structural difference information according to claim 1, characterized in that: The self-similar structure difference descriptor is expressed as follows: D asdpc ={Δd 1,2 ,Δd 1,3 ,...,Δd 1,8 ,Δd 2,3 ,..., Δd 2,8 ,...,Δd 6，8 ,Δd 7,8 } Where Δd i,j Indicates the result of comparing the i-th block and the j-th block, Δd i,j The calculation formula is as follows: Where k represents the total number of bins, and They represent the values ​​of the kth bin in the i-th and j-th region blocks respectively, and sgn(x) is a sign function.

8. The optical and SAR image registration method based on self-similar structural difference information according to claim 7, characterized in that: In step 5, for each feature point retained after screening in step 4 on the optical image, a self-similar structure difference descriptor is constructed, and all pixels within an m×m area centered on the feature point on the SAR image corresponding to the feature point are traversed. A self-similar structure difference descriptor is constructed for each pixel in the area, and compared with the descriptor constructed for the feature point on the optical image. Based on the number N of identical values ​​between the optical image feature point descriptor and the SAR image pixel descriptor at the same position, the pixel in the area on the SAR image with the largest N is selected as the new feature point corresponding to the optical image feature point to achieve more accurate matching.

Images