A Multi-Source Image Sub-Pixel Registration Method Based on Fusion Feedback

Abstract

The present invention discloses a multi-source image sub-pixel registration method based on fusion feedback, including: rough registration of multi-modal images based on high-precision feature points, Poisson registration image fusion based on the gradient domain, calculation and evaluation of global registration fusion coefficients, fine registration of multi-modal images based on fusion feedback, and iterative optimization of registration parameters based on fusion feedback. First, the present invention can effectively solve the non-linear geometric deviation between multi-modal images by adjusting the deformation field through fusion feedback information; secondly, the fusion feedback result provides an intuitive registration quality assessment, and by guiding parameter adjustment, the robustness and accuracy of registration can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal image registration, and in particular relates to a multi-source image sub-pixel registration method based on fusion feedback. Background Art

[0002] High-precision registration of multi-source images is one of the key technologies in the fields of change detection, target recognition, and data fusion, and is also an important research direction in image processing. For example, by registering visible light and infrared images, an enhanced image with clear texture details and not affected by illumination can be generated by making full use of their respective imaging characteristics, which can be effectively applied to scenarios such as covert security layout, traffic supervision, and detection of special targets (such as drones and ships).

[0003] For example, the Chinese patent document with the publication number CN115018892A discloses an automatic registration method and device for remote sensing images, including: preprocessing an incompletely overlapping pair of remote sensing images to be registered to obtain corresponding multiple pairs of image blocks, and performing affine transformation on the multiple pairs of image blocks; constructing an image registration deep neural network model and training it; inputting the multiple pairs of image blocks after affine transformation into the trained image registration deep neural network model, and outputting registered matching feature point pairs; re-projecting the registered matching feature point pairs into the corresponding original remote sensing images to be registered to obtain the final matching result of the remote sensing images to be registered.

[0004] The Chinese patent document with the publication number CN104794678A discloses an automatic registration method for high-spatial-resolution remote sensing images based on SIFT feature points. The method includes the following steps: Step 1, preprocessing; Step 2, extracting SIFT feature points; Step 3, purifying the SIFT feature points; Step 4, building a KD tree; Step 5, searching for the nearest neighbor and the next-nearest neighbor nodes; Step 6, extracting feature matching pairs; Step 7, calculating an affine transformation matrix; Step 8, performing affine transformation.

[0005] However, traditional image registration methods have the following deficiencies:

[0006] First of all, affine transformation and other global geometric transformation models have limited capabilities in dealing with complex non-linear geometric deviations. Multi-source images often produce local non-linear deformations due to factors such as atmospheric refraction and sensor distortion, and traditional methods are difficult to effectively characterize and correct these non-linear geometric deviations.

[0007] Secondly, traditional registration methods based on feature points highly rely on high-precision feature point extraction and matching. However, for multi-modal images (such as visible light images and infrared images), due to the differences in sensor imaging mechanisms, the radiation characteristics and texture features are significantly different, resulting in insufficient accuracy of feature point extraction. Matching points are easily interfered by noise or texture differences, ultimately affecting the accuracy and robustness of registration.

[0008] Finally, in the face of the complexity of multi-modal images, traditional methods are difficult to intuitively judge the registration effect. It is difficult to comprehensively reflect the global alignment degree between images only through point-to-point matching or the convergence of the optimization objective function. Summary of the Invention

[0009] The present invention provides a multi-source image sub-pixel registration method based on fusion feedback, which can achieve high-precision sub-pixel registration of multi-modal images and provide strong support for tasks such as change detection, target recognition, and multi-source data fusion.

[0010] A multi-source image sub-pixel registration method based on fusion feedback includes the following steps:

[0011] (1) Divide the multi-source images into a reference image and the image to be registered , extract the feature points of the reference image and the image to be registered , and obtain a roughly registered image after feature point matching ;

[0012] (2) Transform the reference image and the roughly registered image into the same gradient domain, and use the Poisson fusion principle to fuse the reference image and the roughly registered image to obtain a primary fused image ;

[0013] (3) Evaluate the quality of the artifacts in the fused image to obtain a fusion coefficient, including the fusion loss estimation based on artifacts , the edge intensity used to characterize the degree of artifacts , and the artifact position mask ;

[0014] (4) Construct or update the local deformation field based on the edge intensity and the artifact position mask , determine the horizontal direction offset and the vertical direction offset; and use the latest local deformation field to perform fine registration on the reference image and the roughly registered image to obtain a finely registered image ; Then, the reference image And the finely registered image are fused to obtain a secondary fused image , and further obtain the secondary fused image artifact-based fusion loss estimation ;

[0015] (5) Determine the artifact-based fusion loss estimation whether it satisfies being less than or equal to the fusion loss constraint threshold . If not satisfied, the finely registered image is used as the coarsely registered image , and return to step (2) again. If satisfied, stop the iteration and output the image after fine registration and the fused image.

[0016] The present invention can effectively solve the non-linear geometric deviation between multi-modal images by adjusting the deformation field through fusion feedback information; secondly, the fusion feedback result provides an intuitive registration quality assessment, and improves the robustness and accuracy of registration by guiding parameter adjustment (such as deformation direction and amplitude).

[0017] In step (1), the reference image and the image to be registered are subjected to feature point matching using affine transformation.

[0018] In step (3), artifacts exist at the edges of the reference image and the coarsely registered image, and it is necessary to combine the edge information transfer amount to evaluate the quality of the artifacts after image fusion.

[0019] In step (4), a local deformation field is constructed, and the specific process is as follows:

[0020] Based on the artifact position mask the artifact concentration area is detected, and combining the information of the artifact concentration area, the gradient direction is used to represent the change direction of the artifact at the edge;

[0021] The offset amount is adjusted by the edge intensity , and the edge intensity is used as the weight to offset the local deformation field, and the horizontal direction offset and the vertical direction offset are obtained.

[0022] The gradient direction represents the change direction of the artifact at the edge, and the formula is as follows:

[0023] ;

[0024] Among them, and are respectively obtained by convolving the Sobel operator kernel at the point Images obtained by horizontal and vertical direction calculations;

[0025] Determine the main direction of the offset according to the gradient direction. If , the main offset direction is the horizontal direction. If , the main offset direction is the vertical direction; if it is other angles, it is decomposed into horizontal and vertical directions to form an oblique adjustment.

[0026] The formula for constructing the local deformation field is as follows:

[0027] ;

[0028] In the formula, is the offset in the horizontal direction, is the offset in the vertical direction, is the learning rate, which is used to control the step size of the offset update, represents the pixel coordinates of the artifact part.

[0029] The formula for updating the local formation field is as follows:

[0030] ;

[0031] In the formula, is the updated offset in the horizontal direction, is the updated offset in the vertical direction, represents 's neighborhood, is the weight for smoothing the local deformation field, where is the neighborhood 's pixel coordinates, , respectively represent the offsets of each pixel in the neighborhood in the and directions.

[0032] After updating the local deformation field, smooth the local deformation field and use the nearest neighbor interpolation or bilinear interpolation method to supplement the missing pixel values.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention dynamically optimizes the registration parameters by integrating the feedback mechanism and the artifact evaluation index to solve the non - linear geometric deviation and registration quality problems between multi - modal images. By directly displaying the registration effect using the fusion result, combining artifact detection and quantitative evaluation, adjusting the direction and amplitude of the local deformation field in the iterative optimization, finally, sub - pixel - level high - precision registration of multi - modal images can be achieved. Brief Description of the Drawings

[0035] Figure 1 This is a flowchart of a multi-source image sub-pixel registration method based on fusion feedback according to an embodiment of the present invention.

[0036] Figure 2 This is a schematic diagram of rough matching of images based on feature points in an embodiment of the present invention.

[0037] Figure 3 This is a schematic diagram of image fusion based on the gradient domain in an embodiment of the present invention.

[0038] Figure 4 This is a schematic diagram of fine registration of images based on fusion feedback in an embodiment of the present invention. Detailed implementation manners

[0039] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the following embodiments are intended to facilitate the understanding of the present invention, but do not limit it in any way.

[0040] A multi-source image sub-pixel registration method based on fusion feedback performs rough matching of images through feature point pairs obtained by traditional registration methods; uses a Poisson image fusion model based on the gradient domain to fuse images, and measures the artifact influence of misregistered regions through an artifact evaluation coefficient; constructs a dynamic deformation field according to the constraint relationship of artifact-based fusion loss estimation and combines an iterative optimization algorithm to achieve high-precision registration of multi-modal images.

[0041] Specifically, as Figure 1 shown, it includes the following steps:

[0042] Step 1, rough registration of multi-modal images based on high-precision feature points.

[0043] For high-precision feature point detection, the scale-invariant feature transform (SIFT) is used to construct detectors at different scales of the image based on the difference-of-Gaussians (DoG) operator. A 16×16 neighborhood is extracted around each detected feature point, and finally, the information of the neighborhoods at different scales is constructed into a total of 128-dimensional vectors for gradient information statistics. The formula of the DoG operator is as follows:

[0044] ;

[0045] ;

[0046] where is the image from which feature points are to be extracted, is the scale factor, indicating the blurriness of the image.

[0047] Regarding the influence of intensity differences in the original multi-modal image on gradient calculation, quadratic gradient calculation is added on the basis of the original gradient information statistics to make feature extraction more robust. The calculation formulas for the gradient magnitude and direction are as follows:

[0048] ;

[0049] ;

[0050] Among them, and are the horizontal and vertical direction calculations of the quadratic gradient respectively, is the direction information of the quadratic gradient calculation.

[0051] According to the above gradient information, a feature descriptor is constructed to detect the feature points between images, and then the mismatched points are removed according to the RANSAC algorithm to obtain more accurate reference image feature point pairs ... and the feature point pairs of the image to be registered ... The extracted feature points can be used for rough registration of multi-modal images by affine transformation. The formula is as follows:

[0052] ;

[0053] Among them, , , , , , are the affine matrix parameters to be solved.

[0054] The solution of the matrix M parameters is obtained by the least squares method. The formula is as follows:

[0055] ;

[0056] As Figure 2 shown, the multi-source images are divided into a reference image and an image to be registered, the feature points of the reference image and the image to be registered are extracted, and the rough registration image is obtained after feature point matching.

[0057] Step 2, Poisson registration image fusion based on the gradient domain.

[0058] Due to the intensity mapping gray difference in the multi-modal images, in order to make the image fusion transition more naturally and maintain the inherent texture information and detail features of the images, the images are converted to the same gradient domain for fusion, as Figure 3 shown.

[0059] According to the Poisson fusion principle, the image is described as a problem of solving the Possion equation, and its expression is:

[0060] ;

[0061] Among them, represents the fusion region, is the region boundary, is the image representation function after fusion, is the image representation function before fusion, is the image gradient field after fusion, is the gradient field of the fused image, represents the image representation function outside the fusion region.

[0062] Let the integral function , substitute it into the Euler-Lagrange equation and solve for the minimum value, and its expression is:

[0063] ;

[0064] For the two-dimensional space, is a function of x and y, so the above formula can be written in the form of the following Poisson equation:

[0065] ;

[0066] Among them, represents the second-order differential value, is the Laplace operator, represents the divergence solving operation, is the gradient field of the image region before fusion.

[0067] Since the image is discrete, according to the discrete formula of the Laplace operator and the above formula, the pixel values of the fused image are solved:

[0068] ;

[0069] Substitute the boundary constraints of the fusion region into the constructed Poisson equation system, and solve to obtain x That is, the pixel values of the fused image.

[0070] Step 3, Global registration fusion coefficient calculation and evaluation.

[0071] Since poor registration accuracy between multi-modal images will lead to artifacts in the image fusion process, and the artifacts mainly exist at the edges of the source images, it is necessary to evaluate the quality of the image fusion artifacts in combination with the edge information transfer amount to feedback the registration parameters.

[0072] The fusion coefficient includes artifact-based fusion loss estimation, edge intensity, and artifact position mask; the artifact-based fusion loss estimation ( ) is as follows:

[0073] ;

[0074] Among them, represents the importance of each image to the fused image, represents the edge information of the source image X retained in the fused image, and , and respectively represent the retained edge intensity and direction at is used to judge the position of the artifact. If the edge intensity at a certain position in the fused image is stronger than the corresponding position in the source image, it is judged as an artifact. The judgment relationship is as follows:

[0075] ;

[0076] Among them, , , and are constants, is related to the edge intensity of the image in the horizontal and vertical directions. The formula is as follows:

[0077] ;

[0078] Among them, and are the images obtained by convolving the Sobel operator kernel in the horizontal and vertical directions at the point respectively.

[0079] Step 4, multi-modal image fine registration based on fusion feedback.

[0080] Based on the edge intensity and artifact position mask evaluation in the above fusion coefficient, a local deformation field is constructed. First, the artifact concentration area is detected based on AM , and then the local deformation field is adjusted through the edge intensity and edge direction. As shown in Figure 4 , the specific implementation method is as follows:

[0081] (1) To adjust the direction of the local deformation field, combined with the artifact area information, the gradient direction represents the change direction of the artifact at the edge. The formula is as follows:

[0082] ;

[0083] Among them, and are images obtained by convolving the Sobel operator kernels at points in the horizontal and vertical directions respectively.

[0084] Based on the gradient direction, the main direction of the offset can be determined. If , the main offset direction is the horizontal direction. , the main offset direction is the vertical direction. If it is other angles, it can be decomposed into the horizontal and vertical directions to form an oblique adjustment.

[0085] (2)Adjust the offset amount through the edge intensity, and use the edge intensity as the weight to offset the local deformation field. The adjustment parameters are as follows:

[0086] ;

[0087] where, is the offset in the horizontal direction, is the offset in the vertical direction, is the learning rate, which is used to control the step size of the offset update.

[0088] When applying the modified local deformation field to deform and re - fuse the image, in order to avoid image distortion caused by excessive deformation, it is necessary to perform appropriate smoothing on the local deformation field. In addition, since some pixel points may be mapped to non - integer coordinates after deformation, methods such as nearest - neighbor interpolation or bilinear interpolation need to be used to supplement the missing pixel values to ensure the integrity and continuity of the image.

[0089] Step 5, Iterative optimization of the registration parameters based on the fusion feedback.

[0090] According to the above change relationship, the registration effect of the image can be optimized by the following iterative method:

[0091] First, construct the initial local deformation field When the input image to be registered and are coarsely registered, then and the coarsely registered are fused to obtain the initial fused image . Using to obtain the initial indicators , , , and , thus an updated local deformation field can be obtained and used for fine registration. The expression of the updated deformation field is as follows:

[0092] ;

[0093] Among them, represents the neighborhood of is the weight for smoothing the local deformation field, where is the neighborhood the pixel coordinates in , respectively represent the offsets of each pixel in the neighborhood in the x and y directions.

[0094] After fine registration, is obtained. Using and the reference image for fusion, the secondary fusion image is obtained. At this time, is calculated and it is judged whether holds. If it holds, the iteration stops and the image after fine registration and the fusion image are output. Otherwise, return to step 2 to continue the iterative fusion calculation until the condition is met, where is the fusion loss constraint threshold.

[0095] The above-described embodiments have elaborated on the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the present invention. Any modification, supplement, and equivalent replacement made within the scope of the principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A multi-source image sub-pixel registration method based on fusion feedback, characterized in that: The following steps are involved: (1) Divide multi-source images into reference images and the image to be registered , extract the reference image and the image to be registered After matching the feature points, a coarse registration image is obtained. ; (2) The reference image and coarsely registered images Convert to the same gradient domain and use the Poisson fusion principle to align the reference image and coarsely registered images Fusion is performed to obtain the initial fusion image ; (3) Perform quality assessment on the artifacts of the fused image to obtain the fusion coefficient, including artifact-based fusion loss estimation , edge strength used to characterize the degree of artifact , Artifact Location Mask ; (4) Based on edge strength , Artifact Location Mask Construct or update the local deformation field, determine the horizontal and vertical offsets, and use the latest local deformation field to transform the reference image and coarsely registered images Perform precise registration to obtain a precise registration image ; Then the reference image and finely registered images Fusion is performed to obtain a secondary fusion image , and further obtain the secondary fusion image Artifact-based fusion loss estimation ; Construct a local deformation field. The specific process is as follows: Artifact location mask Detect the area where the artifacts are concentrated, combine the information of the area where the artifacts are concentrated, and use the gradient direction Indicates the direction in which the artifact changes at the edge; By edge strength Adjust the offset size and use edge strength As a weight to offset the local deformation field, the horizontal offset is obtained and vertical offset ; The formula for constructing the local deformation field is as follows: ； In the formula, is the horizontal offset, is the vertical offset, is the learning rate, which is used to control the step size of the offset update. Represents the pixel coordinates of the artifact part; The formula for updating the local deformation field is as follows: ； In the formula, is the updated horizontal offset, is the updated vertical offset, express Neighborhood of is the weight for smoothing the local deformation field, where Neighborhood The pixel coordinates in , Respectively represent the neighborhood Each pixel in and Direction offset; (5) Determine the artifact-based fusion loss estimate Whether it satisfies the fusion loss constraint threshold If it is not satisfied, the image will be precisely aligned As a coarse registration image , return to step (2), if satisfied, stop the iteration and output the image after precise registration and the fused image.

2. The multi-source image sub-pixel registration method based on fusion feedback according to claim 1 is characterized in that: In step (1), the reference image is transformed using an affine transformation and the image to be registered Perform feature point matching.

3. The multi-source image sub-pixel registration method based on fusion feedback according to claim 1 is characterized in that: In step (3), artifacts exist at the edges of the reference image and the coarsely registered image, and the quality of the artifacts after image fusion needs to be evaluated in combination with the edge information transfer amount.

4. The multi-source image sub-pixel registration method based on fusion feedback according to claim 1 is characterized in that: Gradient direction Indicates the direction of change of the artifact at the edge. The formula is as follows: ； in, and are respectively formed by the Sobel operator convolution kernel at point The images obtained by calculation in horizontal and vertical directions; The main direction of the offset is determined according to the gradient direction. If , the main offset direction is horizontal, if , the main offset direction is the vertical direction; if it is other angles, it is decomposed into horizontal and vertical directions to form an oblique adjustment.

5. The multi-source image sub-pixel registration method based on fusion feedback according to claim 1 is characterized in that: In step (4), after the local deformation field is updated, the local deformation field is smoothed, and the missing pixel values ​​are supplemented by the nearest neighbor interpolation or bilinear interpolation method.

Images