Multi-source image sub-pixel registration method based on fusion feedback

Abstract

The invention discloses a multi-source image sub-pixel registration method based on fusion feedback, and the method comprises the steps: multi-modal image coarse registration based on high-precision feature points, Poisson registration image fusion based on a gradient domain, global registration fusion coefficient calculation and evaluation, and multi-modal image fine registration based on fusion feedback. And iteratively optimizing registration parameters based on fusion feedback. According to the method, firstly, a deformation field is adjusted by fusing feedback information, so that the nonlinear geometric deviation among multi-modal images can be effectively solved; and secondly, visual registration quality evaluation is provided by fusing feedback results, and the robustness and precision of registration can be improved by guiding parameter adjustment.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal image registration, and in particular relates to a multi-source image sub-pixel registration method based on fusion feedback. Background Art

[0002] High-precision registration of multi-source images is one of the key technologies in the fields of change detection, target recognition and data fusion, and is also an important research direction in image processing. For example, by registering visible light and infrared images, their respective imaging characteristics can be fully utilized to generate enhanced images with clear texture details and not affected by lighting, which can be effectively applied to scenes such as concealed security deployment, traffic supervision, and detection of special targets (such as drones and ships).

[0003] For example, a Chinese patent document with publication number CN115018892A discloses a method and device for automatic registration of remote sensing images, including: preprocessing incompletely overlapping remote sensing image pairs to be registered to obtain corresponding multiple image block pairs, and performing affine transformation on the multiple image block pairs; constructing and training an image registration deep neural network model; inputting the multiple image block pairs after affine transformation into the trained image registration deep neural network model, and outputting registered matching feature point pairs; reprojecting the registered matching feature point pairs to the corresponding original remote sensing image pairs to be registered, and obtaining the final matching results of the remote sensing image pairs to be registered.

[0004] The Chinese patent document with publication number CN104794678A discloses a high spatial resolution remote sensing image automatic registration method based on SIFT feature points. The method includes the following steps: step 1, preprocessing; step 2, extracting SIFT feature points; step 3, purifying SIFT feature points; step 4, establishing a KD tree; step 5, searching for the nearest neighbor and the next nearest neighbor nodes; step 6, extracting feature matching pairs; step 7, calculating the affine transformation matrix; step 8, performing affine transformation.

[0005] However, traditional image registration methods have the following shortcomings: First, affine transformation and other global geometric transformation models have limited capabilities in dealing with complex nonlinear geometric deviations. Multi-source images often produce local nonlinear deformations due to factors such as atmospheric refraction and sensor distortion, and traditional methods are difficult to effectively characterize and correct these nonlinear geometric deviations.

[0006] Secondly, the traditional registration method based on feature points is highly dependent on high-precision feature point extraction and matching. However, for multimodal images (such as visible light images and infrared images), due to the differences in sensor imaging mechanisms, the radiation characteristics and texture features are significantly different, resulting in insufficient feature point extraction accuracy, and matching points are easily disturbed by noise or texture differences, which ultimately affects the accuracy and robustness of registration.

[0007] Finally, faced with the complexity of multimodal images, it is difficult for traditional methods to intuitively judge the registration effect. It is difficult to fully reflect the degree of global alignment between images only through point-to-point matching or optimizing the convergence of the objective function. Summary of the invention

[0008] The present invention provides a multi-source image sub-pixel registration method based on fusion feedback, which can achieve sub-pixel high-precision registration of multi-modal images and provide strong support for tasks such as change detection, target recognition and multi-source data fusion.

[0009] A multi-source image sub-pixel registration method based on fusion feedback comprises the following steps: (1) Divide multi-source images into reference images and the image to be registered , extract the reference image and the image to be registered After matching the feature points, a coarse registration image is obtained. ; (2) The reference image and coarsely registered images Convert to the same gradient domain and use the Poisson fusion principle to align the reference image and coarsely registered images Fusion is performed to obtain the initial fusion image ; (3) Perform quality assessment on the artifacts of the fused image to obtain the fusion coefficient, including artifact-based fusion loss estimation , edge strength used to characterize the degree of artifacts , Artifact Position Mask ; (4) Based on edge strength , Artifact Position Mask Construct or update the local deformation field, determine the horizontal and vertical offsets, and use the latest local deformation field to transform the reference image and coarsely registered images Perform precise registration to obtain a precise registration image ; Then the reference image and finely registered images Fusion is performed to obtain a secondary fusion image , and further obtain the secondary fusion image Artifact-based fusion loss estimation ; (5) Determine the artifact-based fusion loss estimate Whether it satisfies the fusion loss constraint threshold If it is not satisfied, the image will be precisely aligned As a coarse registration image , return to step (2), if satisfied, stop the iteration and output the image after precise registration and the fused image.

[0010] The present invention can effectively solve the nonlinear geometric deviation between multimodal images by adjusting the deformation field by fusing feedback information; secondly, the fused feedback results provide an intuitive registration quality assessment, and improve the robustness and accuracy of the registration by guiding parameter adjustment (such as deformation direction and amplitude).

[0011] In step (1), the reference image is transformed using an affine transformation and the image to be registered Perform feature point matching.

[0012] In step (3), artifacts exist at the edges of the reference image and the coarsely registered image, and the quality of the artifacts after image fusion needs to be evaluated in combination with the edge information transfer amount.

[0013] In step (4), the local deformation field is constructed. The specific process is as follows: Artifact location mask Detect the area where the artifacts are concentrated, combine the information of the area where the artifacts are concentrated, and use the gradient direction Indicates the direction of change of the artifact at the edge; By edge strength Adjust the offset size, use edge strength As a weight to offset the local deformation field, the horizontal offset is obtained and vertical offset .

[0014] Gradient direction Indicates the direction of change of the artifact at the edge. The formula is as follows: ; in, and are respectively formed by the Sobel operator convolution kernel at point The images obtained by calculation in horizontal and vertical directions; The main direction of the offset is determined according to the gradient direction. If , the main offset direction is horizontal, if , the main offset direction is the vertical direction; if it is other angles, it is decomposed into horizontal and vertical directions to form an oblique adjustment.

[0015] The formula for constructing the local deformation field is as follows: ; In the formula, is the horizontal offset, is the vertical offset, is the learning rate, which is used to control the step size of the offset update. The pixel coordinates representing the artifact part.

[0016] The formula for updating the local formation field is as follows: ; In the formula, is the updated horizontal offset, is the updated vertical offset, express Neighborhood of is the weight for smoothing the local deformation field, where Neighborhood The pixel coordinates in , Respectively represent the neighborhood Each pixel in and The offset of the direction.

[0017] After updating the local deformation field, the local deformation field is smoothed and the missing pixel values ​​are supplemented by the nearest neighbor interpolation or bilinear interpolation method.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention combines artifact evaluation indicators through a fusion feedback mechanism to dynamically optimize registration parameters to solve the nonlinear geometric deviation and registration quality problems between multimodal images. By using the fusion results to directly display the registration effect, combined with artifact detection and quantitative evaluation, the direction and amplitude of the local deformation field are adjusted in iterative optimization, and finally sub-pixel high-precision registration of multimodal images can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The present invention is a flowchart of a multi-source image sub-pixel registration method based on fusion feedback according to an embodiment of the present invention.

[0020] Figure 2 Schematic diagram of image coarse matching based on feature points in an embodiment of the present invention.

[0021] Figure 3 Schematic diagram of image fusion based on gradient domain in an embodiment of the present invention.

[0022] Figure 4 Schematic diagram of precise image registration based on fusion feedback in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0024] A multi-source image sub-pixel registration method based on fusion feedback performs rough matching of images through feature point pairs obtained by traditional registration methods; uses a Poisson image fusion model based on gradient domain to fuse images, and measures the artifact impact of poorly registered areas through artifact evaluation coefficients; constructs a dynamic deformation field based on artifact-based fusion loss estimation constraint relationship and combines iterative optimization algorithm to achieve high-precision registration of multi-modal images.

[0025] Specifically, Figure 1 As shown, the following steps are included: Step 1: Coarse registration of multimodal images based on high-precision feature points.

[0026] For high-precision feature point detection, the scale-invariant feature transform (SIFT) is used to construct detectors at different scales of the image based on the Gaussian difference (DoG) operator. A 16×16 neighborhood is extracted around each detected feature point, and finally the information of the neighborhoods at different scales is constructed into a total of 128-dimensional vectors for gradient information statistics. The DoG operator formula is as follows: ; ; in, is the image to be extracted feature points, is the scale factor, which indicates the blurriness of the image.

[0027] In view of the influence of intensity difference in the original multimodal image on gradient calculation, a secondary gradient calculation is added on the basis of the original gradient information statistics to make feature extraction more robust. The calculation formula of gradient amplitude and direction is as follows: ; ; in, and The calculations for the horizontal and vertical directions of the secondary gradient are respectively: Direction information for quadratic gradient calculation.

[0028] According to the above gradient information, the feature descriptor is constructed , detect the feature points between images, and then remove the mismatched points according to the RANSAC algorithm to obtain a more accurate reference image feature point pair ... And the feature point pairs of the image to be registered ... , the extracted feature points can be used for coarse registration of multimodal images by affine transformation, the formula is as follows: ; in, , , , , , is the affine matrix parameter to be solved.

[0029] The solution of the matrix M parameters is obtained by the least squares method, and the formula is as follows: ; like Figure 2 As shown, the multi-source images are divided into reference images and images to be registered, feature points of the reference images and the images to be registered are extracted, and a coarse registered image is obtained after feature point matching.

[0030] Step 2: Image fusion based on Poisson registration in gradient domain.

[0031] Since there are intensity mapping grayscale differences in multimodal images, in order to make image fusion transition more naturally and maintain the inherent texture information and detail features of the image, the images are converted to the same gradient domain for fusion, such as Figure 3 shown.

[0032] According to the Poisson fusion principle, the image is described as a Posson equation solution problem, and its expression is: ; in, represents the fusion area, is the region boundary, is the fused image representation function, is the image representation function before fusion, is the fused image gradient field, is the gradient field of the fused image, Represents the image representation function outside the fusion region.

[0033] Let the integral function , and then substitute it into the Euler-Lagrange equation to solve the minimum value, the expression is: ; For two-dimensional space, is a function of x and y, so the above formula can be written in the form of the following Poisson equation: ; in, represents the second-order differential value, is the Laplace operator, represents the divergence solving operation, is the gradient field of the image region before fusion.

[0034] Since the image is discrete, the pixel value of the fused image is solved according to the discrete formula of the Laplace operator and the above formula: ; Substitute the boundary constraints of the fusion area into the constructed Poisson's equations are solved to obtain x That is, the pixel value of the fused image.

[0035] Step 3: Calculate and evaluate the global registration fusion coefficients.

[0036] Poor registration accuracy between multimodal images will lead to artifacts in the image fusion process. Artifacts mainly exist at the edges of the source images. Therefore, it is necessary to evaluate the quality of image fusion artifacts in combination with the edge information transfer to provide feedback on the registration parameters.

[0037] The fusion coefficients include artifact-based fusion loss estimation, edge strength, and artifact position mask; artifact-based fusion loss estimation ( )as follows: ; in, Indicates the importance of each image to the fused image, represents the edge information in the source image X preserved in the fused image, and , and Respectively The edge strength and direction retained at the location are usually calculated using the Sobel edge operator in the index measurement. It is used to determine the location of the artifact. If the edge strength of a certain part of the fused image is stronger than the corresponding position in the source image, it is judged to be an artifact. The judgment relationship is as follows: ; in, , , and is a constant, It is related to the edge strength of the image in the horizontal and vertical directions, and its formula is as follows: ; in, and are respectively formed by the Sobel operator convolution kernel at point The image obtained is calculated in horizontal and vertical directions.

[0038] Step 4: Multimodal image precise registration based on fusion feedback.

[0039] Based on the edge strength and artifact position mask evaluation in the above fusion coefficients, the local deformation field is constructed. First, AM Detect the area where artifacts are concentrated, and then adjust the local deformation field by edge strength and edge direction, such as Figure 4 As shown, the specific implementation is as follows: (1) In order to adjust the direction of the local deformation field, the gradient direction is used in combination with the artifact area information. Indicates the direction of change of the artifact at the edge. The formula is as follows: ; in, and are respectively formed by the Sobel operator convolution kernel at point The image obtained is calculated in horizontal and vertical directions.

[0040] The main direction of the shift can be determined based on the gradient direction. , the main offset direction is horizontal, , the main offset direction is the vertical direction. If it is other angles, it can be decomposed into horizontal and vertical directions to form an oblique adjustment.

[0041] (2) Adjust the offset size by edge strength, using edge strength As the weight for the offset of the local deformation field, the adjustment parameters are as follows: ; in, is the horizontal offset, is the vertical offset, is the learning rate, which is used to control the step size of the offset update.

[0042] When applying the modified local deformation field to correct and re-integrate the image, in order to avoid image distortion caused by excessive deformation, the local deformation field needs to be properly smoothed. In addition, since some pixels may be mapped to non-integer coordinates after deformation, it is necessary to use methods such as nearest neighbor interpolation or bilinear interpolation to supplement the missing pixel values ​​to ensure the integrity and continuity of the image.

[0043] Step 5: Iterative optimization of registration parameters based on fusion feedback.

[0044] According to the above change relationship, the image registration effect can be optimized by the following iterative method: First, construct the initial local deformation field When the input image to be registered and After rough registration, After coarse registration Fusion is performed to obtain the initial fused image ,use Get initial indicators , , , and , so the updated local deformation field can be obtained and used for The updated deformation field expression is as follows: ; in, express Neighborhood of is the weight for smoothing the local deformation field, where Neighborhood The pixel coordinates in , Respectively represent the neighborhood The offset of each pixel in the x and y directions.

[0045] After precise alignment, we get ,use With reference image Fusion is performed to obtain a secondary fusion image , then we can calculate ,judge Is it true? If so, stop the iteration and output the image and fused image after precise registration. Otherwise, continue to return to step 2 for iterative fusion calculation until the condition is met. is the fusion loss constraint threshold.

[0046] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-source image sub-pixel registration method based on fusion feedback, characterized in that: The following steps are involved: (1) Divide multi-source images into reference images and the image to be registered , extract the reference image and the image to be registered After matching the feature points, a coarse registration image is obtained. ; (2) The reference image and coarsely registered images Convert to the same gradient domain and use the Poisson fusion principle to align the reference image and coarsely registered images Fusion is performed to obtain the initial fusion image ; (3) Perform quality assessment on the artifacts of the fused image to obtain the fusion coefficient, including artifact-based fusion loss estimation , edge strength used to characterize the degree of artifact , Artifact Location Mask ; (4) Based on edge strength , Artifact Location Mask Construct or update the local deformation field, determine the horizontal and vertical offsets, and use the latest local deformation field to transform the reference image and coarsely registered images Perform precise registration to obtain a precise registration image ; Then the reference image and finely registered images Fusion is performed to obtain a secondary fusion image , and further obtain the secondary fusion image Artifact-based fusion loss estimation ; (5) Determine the artifact-based fusion loss estimate Whether it satisfies the fusion loss constraint threshold If it is not satisfied, the image will be precisely aligned As a coarse registration image , return to step (2), if satisfied, stop the iteration and output the image after precise registration and the fused image.

2. The multi-source image sub-pixel registration method based on fusion feedback according to claim 1 is characterized in that: In step (1), the reference image is transformed using an affine transformation and the image to be registered Perform feature point matching.

3. The multi-source image sub-pixel registration method based on fusion feedback according to claim 1 is characterized in that: In step (3), artifacts exist at the edges of the reference image and the coarsely registered image, and the quality of the artifacts after image fusion needs to be evaluated in combination with the edge information transfer amount.

4. The multi-source image sub-pixel registration method based on fusion feedback according to claim 1 is characterized in that: In step (4), the local deformation field is constructed. The specific process is as follows: Artifact location mask Detect the area where the artifacts are concentrated, combine the information of the area where the artifacts are concentrated, and use the gradient direction Indicates the direction in which the artifact changes at the edge; By edge strength Adjust the offset size, use edge strength As a weight to offset the local deformation field, the horizontal offset is obtained and vertical offset .

5. The multi-source image sub-pixel registration method based on fusion feedback according to claim 4 is characterized in that: Gradient direction Indicates the direction of change of the artifact at the edge. The formula is as follows: ； in, and are respectively formed by the Sobel operator convolution kernel at point The images obtained by calculation in horizontal and vertical directions; The main direction of the offset is determined according to the gradient direction. If , the main offset direction is horizontal, if , the main offset direction is the vertical direction; if it is other angles, it is decomposed into horizontal and vertical directions to form an oblique adjustment.

6. The multi-source image sub-pixel registration method based on fusion feedback according to claim 4 is characterized in that: The formula for constructing the local deformation field is as follows: ； In the formula, is the horizontal offset, is the vertical offset, is the learning rate, which is used to control the step size of the offset update. The pixel coordinates representing the artifact part.

7. The multi-source image sub-pixel registration method based on fusion feedback according to claim 6 is characterized in that: The formula for updating the local formation field is as follows: ； In the formula, is the updated horizontal offset, is the updated vertical offset, express Neighborhood of is the weight for smoothing the local deformation field, where Neighborhood The pixel coordinates in , Respectively represent the neighborhood Each pixel in and The offset of the direction.

8. The multi-source image sub-pixel registration method based on fusion feedback according to claim 7 is characterized in that: In step (4), after the local deformation field is updated, the local deformation field is smoothed, and the missing pixel values ​​are supplemented by the nearest neighbor interpolation or bilinear interpolation method.

Images