Image registration method and system based on fractional order joint spectrum and directional consistency
By employing a fractional-order joint spectrum and orientation-consistent image registration method, and utilizing fractional-order cumulative spectrum and fast cross-correlation algorithm, the problem of insufficient accuracy in visible light-SAR image registration is solved, achieving accurate registration in low-texture and high-noise scenarios. This method is suitable for embedded devices and small ground observation terminals.
Patent Information
- Application Number
- CN202611024525.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-10
- Publication Date
- 2026-11-06
AI Technical Summary
Existing visible light-SAR image registration methods are not accurate enough in low-texture and high-noise remote sensing scenes, and deep learning methods rely on labeled data and have high hardware costs, which cannot meet the requirements of real-time registration.
An image registration method based on fractional-order joint spectrum and directional consistency is adopted. By calculating the fractional-order cumulative spectrum of single-modality and cross-modality images, combined with fast cross-correlation algorithm and multi-directional gradient statistics, coarse registration and accurate matching of images are achieved, false matching pairs are eliminated, and cross-modal image registration is completed.
Without relying on labeled data and noise preprocessing, this method solves the problem of numerous false points and insufficient accuracy in matching visible light images and SAR images due to differences in imaging mechanisms. It achieves accurate registration of cross-modal images and is suitable for embedded devices and small ground observation terminals.
Smart Images
Figure CN123052007A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image registration technology, and particularly relates to an image registration method based on fractional-order joint spectrum and orientation consistency. Background Technology
[0002] Visible light images and synthetic aperture radar (SAR) images are two mainstream data sources in remote sensing observation, and their imaging mechanisms differ significantly: visible light images have rich textures and intuitive visual features, but are easily affected by lighting, clouds, fog, and weather; SAR images have all-day, all-weather observation capabilities, unaffected by day / night cycles or cloud / rain interference, but suffer from severe speckle noise, and their radiometric characteristics differ greatly from visible light images. Visible light-SAR cross-modal image registration is a core prerequisite step for remote sensing applications such as image fusion, change detection, ground feature identification, and disaster monitoring, and the registration accuracy directly determines the effectiveness of subsequent tasks.
[0003] Existing visible light-SAR image registration methods are mainly divided into two categories: traditional feature methods and deep learning methods, both of which have obvious shortcomings:
[0004] Traditional feature extraction methods, represented by algorithms such as SIFT, SAR-SIFT, CFOG, and HOPC, generally rely on local texture features described by gradient statistics. However, in low-texture remote sensing scenes such as deserts, sea surfaces, and snow-covered landscapes, feature discriminative power drops significantly. Furthermore, the strong speckle noise in SAR images and the nonlinear radiometric distortion of both types of images severely interfere with feature extraction and matching. In addition, traditional methods typically employ a cascaded process of denoising first and then feature extraction; the pre-filtering operation easily weakens the effective information in weak-texture regions, causing errors to propagate step by step.
[0005] Deep learning registration methods, such as ME-SSM, F3Net, and AESF, rely on massive amounts of labeled samples to learn cross-modal feature mapping relationships, resulting in relatively higher registration accuracy. However, these methods have three major drawbacks: First, they are highly dependent on labeled datasets, and their generalization ability drops sharply in scenarios lacking labeled samples, such as polar regions, uncontrolled overseas areas, and sudden disasters. Second, model inference depends on GPU hardware, making it difficult to deploy on embedded remote sensing devices and small ground observation terminals, resulting in high application costs. Third, the network structure is complex, and the inference latency is relatively large, failing to meet the requirements for real-time registration.
[0006] Meanwhile, existing studies on fractional-order statistical features only apply fractional-order cumulants (FOC) and fractional-order cumulants spectra (FOSC) to denoising single SAR images, without extending to cross-modal joint feature modeling of visible light and SAR. This fails to simultaneously address the problems of numerous false matches and insufficient registration accuracy caused by nonlinear radiometric differences and noise interference. Therefore, to address the shortcomings of existing technologies, there is an urgent need to design a training-free, low-computing-power, and robust cross-modal image registration method to adapt to extreme remote sensing scenarios such as low-texture and high-noise environments. Summary of the Invention
[0007] The purpose of this invention is to provide an image registration method and system based on fractional-order joint spectrum and orientation consistency, aiming to solve the problems mentioned in the background art.
[0008] This invention is implemented as follows: an image registration method based on fractional-order joint spectrum and orientation consistency, comprising the following steps:
[0009] Acquire multimodal images, perform sliding block division on the multimodal images, calculate the fractional-order cumulative spectrum of single modes, and generate various single-mode feature maps; the multimodal images include visible light images and SAR images;
[0010] Based on the fast cross-correlation algorithm, global coarse matching is performed on multiple single-modal feature maps to obtain an initial coarse matching point matrix;
[0011] Based on the initial coarse matching point matrix, coarse registration is performed on the multimodal images, and the single-mode fractional-order cumulative spectrum and cross-mode fractional-order joint cumulative spectrum of the SAR images are recalculated to obtain new SAR single-mode feature maps and joint feature maps.
[0012] Multi-directional gradients are extracted from the new SAR single-mode feature map and joint feature map, and multi-frequency, multi-scale feature vectors are constructed.
[0013] Based on multi-frequency, multi-scale feature vectors, precise matching is performed using multi-directional consistency and image features to obtain a precise matching set.
[0014] Based on the precise matching set, the affine transformation matrix is solved, and geometric transformation and subpixel interpolation are performed on the SAR image to obtain the registered SAR image, thus completing the registration of multimodal images.
[0015] Furthermore, the steps of acquiring multimodal images, performing sliding block division on the multimodal images, calculating the fractional-order cumulant spectrum of a single modality, and generating multiple single-modal feature maps specifically include:
[0016] Visible light images SAR images Perform sliding block division separately, defining the visible light image block as... SAR image patch ;
[0017] Calculate the first, second, and third order single-modal fractional moments of a visible light image patch, expressed as follows: , , Its mathematical expression is:
[0018] ;
[0019] ;
[0020] ;
[0021] In the formula, The center position of the image patch; denoted as the two-dimensional spatial interval; M and N represent the number of rows and columns of pixels in the image patch; 1b, 2b, and 3b represent the fractional order. ;
[0022] Single-modal fractional moments based on visible light image patches , , Calculate its 3b-order single-mode fractional cumulant, its mathematical expression is:
[0023] ;
[0024] Calculate the Mittag-Leffler function for different spatial intervals. The value of is expressed mathematically as follows:
[0025] ;
[0026] In the formula, Spatial interval; Γ represents the chosen k-th angular frequency; Γ is the gamma function, a core function of fractional calculus theory. The imaginary unit;
[0027] Single-modal fractional-order cumulants based on visible light image patches Calculate its 3b-order single-mode fractional-order cumulant spectrum to generate a visible light single-mode feature map. Its mathematical expression is:
[0028] ;
[0029] In the formula, Angular frequency;
[0030] The first, second, and third order single-mode fractional moments of a SAR image patch are calculated, and are expressed as follows: , , Its mathematical expression is:
[0031] ;
[0032] ;
[0033] ;
[0034] In the formula, The center position of the image patch; denoted as the two-dimensional spatial interval; M and N represent the number of rows and columns of pixels in the image patch; 1b, 2b, and 3b represent the fractional order. ;
[0035] Single-mode fractional moments based on SAR image patches , , Calculate its 3b-order single-mode fractional cumulant, its mathematical expression is:
[0036] ;
[0037] Single-modal fractional-order cumulants based on SAR image patches Calculate the fractional-order cumulative spectrum of a 3b-order SAR single-mode to generate a SAR single-mode feature map. Its mathematical expression is:
[0038] ;
[0039] Visible light single-mode feature map and SAR single-mode feature map Perform global normalization and then map it to a preset pixel range to obtain a normalized SAR single-modality feature map. Normalized visible light single-mode feature map .
[0040] Furthermore, based on the fast cross-correlation algorithm, the steps of performing global coarse matching on multiple single-modal feature maps to obtain the initial coarse matching point matrix specifically include:
[0041] Normalized SAR single-mode feature map Normalized visible light single-mode feature map Perform two-dimensional fast cross-correlation calculations to generate a global similarity heatmap. Its mathematical expression is:
[0042] ;
[0043] In the formula, i and j represent the row and column coordinates of the pixel in the feature map, u and v represent the row and column offsets; M1 and N1 are the number of rows and columns of the feature map.
[0044] Based on the image size M2×N2, and with a preset threshold K for the number of coarse matching points, the coordinates of the top K peaks in the global similarity heatmap are extracted and used to form an initial coarse matching point matrix using a matrix stacking method. and .
[0045] Furthermore, based on the initial coarse matching point matrix, coarse registration is performed on the multimodal images, and the single-mode fractional-order cumulative spectrum and cross-mode fractional-order joint cumulative spectrum of the SAR images are recalculated to obtain new SAR single-mode feature maps and joint feature maps. The specific steps include:
[0046] Based on the initial coarse matching point matrix and Calculate the coarse registration affine transformation matrix Its mathematical expression is:
[0047] ;
[0048] In the formula, for The transpose of the matrix;
[0049] Based on the coarse registration affine transformation matrix For the original SAR image Perform a global geometric transformation to obtain the transformed SAR image. Its mathematical expression is:
[0050] ;
[0051] Based on the transformed SAR image Recalculate and generate new SAR single-mode feature maps Then, global normalization is performed, and the data is mapped to a preset pixel range to generate a new normalized SAR single-modality feature map. ;
[0052] The mathematical expression for calculating the fractional joint moments is as follows:
[0053] ;
[0054] In the formula, For visible light image patches, For SAR image patches; The center position of the image patch; denoted as the two-dimensional spatial interval; M and N represent the number of rows and columns of pixels in the image patch; 1b, 2b, and 3b represent the fractional order. ;
[0055] Based on the single-mode fractional moment of the visible light image patch and Single-mode fractional moments of SAR image patches and Fractional joint moments Calculate the joint cumulant of the 3b-order cross-modal fractional order. Its mathematical expression is:
[0056] ;
[0057] According to the Mittag-Leffler function Calculate the two-dimensional fractional-order joint cumulant spectrum and generate the joint feature map. Its mathematical expression is:
[0058] ;
[0059] joint feature map Perform global normalization, then map to a preset pixel range to obtain a normalized joint feature map. .
[0060] Furthermore, the steps of extracting multi-directional gradients from the new SAR single-mode feature map and joint feature map, and constructing multi-frequency, multi-scale feature vectors, specifically include:
[0061] For the new normalized SAR single-mode feature map and normalized joint feature map Create angular frequencies respectively , , , The Gaussian pyramid at that time, and detect key points;
[0062] According to the Gaussian pyramid, and simultaneously within the circular regions of all key points of the Gaussian pyramid based on the four angular frequency combinations, according to the direction... and angular frequency Calculate gradient magnitude Its mathematical expression is:
[0063] ;
[0064] In the formula, This represents the feature map in the Gaussian pyramid, where u and v are the pixel coordinates of the feature map; and These are the partial derivatives in the horizontal and vertical directions, respectively;
[0065] Gradient statistics of key points detected at multiple scales are grouped by angular frequency, sorted by angle from smallest to largest within each group, and sorted by angular frequency from smallest to largest between groups to generate SAR feature vectors. Joint eigenvectors .
[0066] Furthermore, the steps for obtaining a precise matching set based on multi-frequency, multi-scale feature vectors, multi-directional consistency, and image features include:
[0067] Calculating feature vectors using cosine similarity and The matching degree, its mathematical expression is:
[0068] ;
[0069] In the formula, For L2 norm operations;
[0070] When the matching degree of the feature vector is greater than a preset threshold, the matching pair is considered a valid match. False matching pairs are eliminated, and valid matching points are retained to form a matrix of precise matching points. and This yields an exact matching set.
[0071] Furthermore, based on the precise matching set, the affine transformation matrix is solved, and geometric transformation and sub-pixel interpolation are performed on the SAR image to obtain the registered SAR image. This completes the registration of multimodal images, specifically including:
[0072] Based on the precise matching point-pair matrix and Calculate the affine transformation matrix for accurate registration. Its mathematical expression is:
[0073] ;
[0074] According to the affine transformation matrix Perform a global geometric transformation to obtain the transformed SAR image. Its mathematical expression is:
[0075] ;
[0076] Based on the transformed SAR image Select floating-point coordinates one by one The four integer neighboring pixels around the sub-pixel Transformed SAR image Subpixel interpolation is performed to obtain the registered SAR image. Its mathematical expression is:
[0077] ;
[0078] In the formula, .
[0079] Based on the initial visible light image and the registered SAR image Output the final registered image The registration of visible light images and SAR images is completed, and its mathematical expression is:
[0080] ;
[0081] In the formula, This is an element-wise matrix multiplication operation.
[0082] Another objective of this invention is to provide an image registration system based on fractional-order joint spectrum and orientation consistency, for implementing the aforementioned image registration method based on fractional-order joint spectrum and orientation consistency, comprising:
[0083] The feature map generation module is used to acquire multimodal images, perform sliding block division on the multimodal images, calculate the fractional-order cumulative spectrum of a single mode, and generate various single-mode feature maps; the multimodal images include visible light images and SAR images;
[0084] The feature map coarse matching module is used to perform global coarse matching on multiple single-modal feature maps based on a fast cross-correlation algorithm to obtain an initial coarse matching point matrix.
[0085] The image coarse registration module is used to perform coarse registration of multimodal images based on the initial coarse matching point matrix, recalculate the single-mode fractional-order cumulative spectrum and cross-mode fractional-order joint cumulative spectrum of SAR images, and obtain new SAR single-mode feature maps and joint feature maps.
[0086] The feature vector construction module is used to extract multi-directional gradients from new SAR single-mode feature maps and joint feature maps, and construct multi-frequency, multi-scale feature vectors.
[0087] The precise matching module is used to perform precise matching based on multi-frequency, multi-scale feature vectors, multi-directional consistency, and image features to obtain a precise matching set.
[0088] The image registration output module is used to solve the affine transformation matrix based on the precise matching set, perform geometric transformation and sub-pixel interpolation on the SAR image, obtain the registered SAR image, and complete the registration of multimodal images.
[0089] This invention provides an image registration method based on fractional-order joint spectrum and directional consistency. It utilizes the noise suppression characteristics of the single-modal fractional-order cumulant spectrum of an image, combined with a fast cross-correlation algorithm, to coarsely register visible light and SAR images. Then, the single-modal fractional-order cumulant spectrum is extended to a cross-modal fractional-order joint cumulant spectrum. Through joint statistical modeling of the visible light and SAR images, the nonlinear radiometric differences between the two types of images are weakened. Subsequently, a multi-frequency pyramid and gradient statistics are used to construct multi-directional consistent fractional-order image features. Finally, multi-directional consistency constraints are used to eliminate false matches, achieving cross-modal image registration. This invention can solve the problems of numerous false matches and insufficient registration accuracy caused by the large differences in imaging mechanisms between visible light and SAR images, without relying on labeled data or noise preprocessing, thus achieving accurate cross-modal image registration. Attached Figure Description
[0090] Figure 1 This is a flowchart illustrating the image registration method based on fractional-order joint spectrum and orientation consistency provided in an embodiment of the present invention.
[0091] Figure 2 A visual comparison of fractional-order joint cumulant spectrum and traditional fractional-order features;
[0092] Figure 3 This is a schematic diagram of the Gaussian pyramid.
[0093] Figure 4 This is a schematic diagram of the gradient statistics region of the feature map;
[0094] Figure 5 This is a diagram showing the registration results between the visible light image and the SAR image. Detailed Implementation
[0095] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0096] To overcome the shortcomings of existing visible light-SAR image registration methods, such as severe effects from speckle noise and cross-modal nonlinear radiation differences, and the heavy data dependence and high deployment costs of deep learning methods, one embodiment of this invention provides an image registration method based on fractional-order joint spectrum and orientation consistency. This method relies on fractional-order statistical features to suppress noise at its source, combines multi-directional structural constraints to select effective matching pairs, and requires no model training throughout the process, relying solely on the CPU. Specifically, as shown... Figure 1 As shown, the method includes the following steps:
[0097] S1. Acquire multimodal images, perform sliding block division on the multimodal images, calculate the fractional-order cumulative spectrum of single modes, and generate multiple single-mode feature maps; the multimodal images include visible light images and SAR images; in this embodiment of the invention, the registration of visible light images and SAR images is used as an example, but it is not limited thereto; step S1 specifically includes:
[0098] S11, Visible light image SAR images Perform 3×3 sliding blocks respectively, and define the visible light image block as... SAR image patch Among them, the original SAR image like Figure 2 As shown in (a);
[0099] S12. Calculate the first, second, and third order single-modal fractional moments (FOMs) of the visible light image patch, expressed as follows: , , Its mathematical expression is:
[0100] ;
[0101] ;
[0102] ;
[0103] In the formula, The center position of the image patch; is the two-dimensional spatial interval; M and N are the number of rows and columns of pixels in the image patch, both set to 3; 1b, 2b, and 3b are the fractional orders. ;
[0104] S13, Single-modal fractional moment based on visible light image patches , , The mathematical expression for calculating its 3b-order single-mode fractional cumulant (FOC) is as follows:
[0105] ;
[0106] S14. Calculate the Mittag-Leffler function for different spatial intervals. The value of is expressed mathematically as follows:
[0107] ;
[0108] In the formula, Spatial interval; Γ represents the chosen k-th angular frequency; Γ is the gamma function, a core function of fractional calculus theory. The imaginary unit;
[0109] S15. Single-modal fractional-order cumulant based on visible light image patches Calculate its 3b-order single-mode fractional cumulant spectrum (FOSC) to generate a visible light single-mode feature map. Its mathematical expression is:
[0110] ;
[0111] In the formula, Angular frequency;
[0112] S16. Calculate the first, second, and third order single-mode fractional moments (FOMs) of the SAR image patch, denoted as follows: , , Its mathematical expression is:
[0113] ;
[0114] ;
[0115] ;
[0116] In the formula, The center position of the image patch; is the two-dimensional spatial interval; M and N are the number of rows and columns of pixels in the image patch, both set to 3; 1b, 2b, and 3b are the fractional orders. ;
[0117] S17. Single-mode fractional moments based on SAR image patches , , Calculate its 3b-order single-mode fractional-order cumulant, and its corresponding feature map is as follows. Figure 2 As shown in (b), its mathematical expression is:
[0118] ;
[0119] S18. Single-modal fractional-order cumulant based on SAR image patches Calculate the fractional-order cumulant spectrum of a 3b-order SAR single-mode, and generate, as shown below. Figure 2 The SAR single-mode feature map shown in (c) Its mathematical expression is:
[0120] ;
[0121] S19. Visible light single-mode feature map and SAR single-mode feature map Perform global normalization, then map to 0-255 to eliminate pixel amplitude bias and obtain normalized SAR single-mode feature maps. Normalized visible light single-mode feature map Its mathematical expression is:
[0122] ;
[0123] In the formula, max() is the maximum value function.
[0124] S2. Based on the fast cross-correlation algorithm, perform global coarse matching on multiple single-modal feature maps to obtain an initial coarse matching point matrix; Step S2 specifically includes:
[0125] S21. Normalized SAR single-mode feature map Normalized visible light single-mode feature map Perform two-dimensional fast cross-correlation calculations to generate a global similarity heatmap. Its mathematical expression is:
[0126] ;
[0127] In the formula, i and j represent the row and column coordinates of the pixel in the feature map, u and v represent the row and column offsets; M1 and N1 are the number of rows and columns of the feature map.
[0128] S22. Based on the image size M2×N2, set the threshold for the number of coarse matching points as K=50M2N2 / 512 / 512. Extract the peak coordinates of the top K values in the global similarity heatmap, and form an initial coarse matching point matrix using the matrix stacking method. and .
[0129] S3. Based on the initial coarse matching point matrix, perform coarse registration on the multimodal images, and recalculate the single-mode fractional-order cumulative spectrum and cross-mode fractional-order joint cumulative spectrum (FOSJC) of the SAR images to obtain new SAR single-mode feature maps and joint feature maps; Step S3 specifically includes:
[0130] S31. Based on the initial coarse matching point matrix and Calculate the coarse registration affine transformation matrix Its mathematical expression is:
[0131] ;
[0132] In the formula, for The transpose of the matrix;
[0133] S32. Based on the coarse registration affine transformation matrix For the original SAR image Perform a global geometric transformation to obtain the transformed SAR image. Its mathematical expression is:
[0134] ;
[0135] S33. Based on the transformed SAR image Following the steps S16-S18 above, recalculate and generate new SAR single-mode feature maps. Then, global normalization is performed, and the result is mapped to 0-255 to generate a new normalized SAR single-mode feature map. ;
[0136] S34. Calculate the fractional joint moment (FOJM), its mathematical expression is:
[0137] ;
[0138] In the formula, For visible light image patches, For SAR image patches; The center position of the image patch; is the two-dimensional spatial interval; M and N are the number of rows and columns of pixels in the image patch, both set to 3; 1b, 2b, and 3b are the fractional orders. ;
[0139] S35. Based on the single-modal fractional moment of the visible light image patch. and Single-mode fractional moments of SAR image patches and Fractional joint moments Calculate the joint cumulant of the 3b-order cross-modal fractional order. Its mathematical expression is:
[0140] ;
[0141] S36. Based on the Mittag-Leffler function obtained in step S14 Calculate the two-dimensional fractional-order joint cumulant spectrum and generate, as follows: Figure 2 The joint feature map shown in (d) Its mathematical expression is:
[0142] ;
[0143] S37, Joint Feature Map Perform global normalization, then map to 0-255 to eliminate pixel amplitude bias and obtain a normalized joint feature map. Its mathematical expression is:
[0144] ;
[0145] In the formula, max() is the maximum value function.
[0146] S4. Extract multi-directional gradients from the new SAR single-mode feature map and joint feature map respectively, and construct multi-frequency, multi-scale feature vectors; Step S4 specifically includes:
[0147] S41. Following the traditional SIFT method, generate the new normalized SAR single-modality feature map. and normalized joint feature map Create respectively as follows Figure 3 The angular frequency shown is , , , The Gaussian pyramid at that time, and detect key points;
[0148] S42, such as Figure 4 As shown, according to the Gaussian pyramid, simultaneously within the circular regions of all key points of the Gaussian pyramid based on the four angular frequency combinations, according to the direction... and angular frequency Calculate gradient magnitude Its mathematical expression is:
[0149] ;
[0150] In the formula, This represents the feature map in the Gaussian pyramid, where u and v are the pixel coordinates of the feature map; and These are the partial derivatives in the horizontal and vertical directions, respectively;
[0151] S43. Group the gradient statistics of key points detected at multiple scales by angular frequency, sort within each group by angle from smallest to largest, and sort between groups by angular frequency from smallest to largest to generate SAR feature vectors. Joint eigenvectors .
[0152] S5. Based on the multi-frequency, multi-scale feature vectors, perform precise matching based on multi-directional consistency and image features to obtain a precise matching set; Step S5 specifically includes:
[0153] S51. The feature vector obtained by using cosine similarity calculation step S43 and The matching degree, its mathematical expression is:
[0154] ;
[0155] In the formula, For L2 norm operations;
[0156] S52. When the matching degree of the feature vector is greater than the preset threshold If the match is valid, then false matches are discarded, and valid matches are retained to form a matrix of exact matching pairs. and This yields an exact matching set.
[0157] S6. Based on the precise matching set, solve the affine transformation matrix, perform geometric transformation and sub-pixel interpolation on the SAR image to obtain the registered SAR image, and complete the registration of multimodal images; Step S6 specifically includes:
[0158] S61. Based on the exact matching point-to-point matrix and Calculate the affine transformation matrix for accurate registration. Its mathematical expression is:
[0159] ;
[0160] S62. Based on the affine transformation matrix Perform a global geometric transformation to obtain the transformed SAR image. Its mathematical expression is:
[0161] ;
[0162] S63. Based on the transformed SAR image Select floating-point coordinates one by one The four integer neighboring pixels around the sub-pixel Transformed SAR image Subpixel interpolation is performed to obtain the registered SAR image. Its mathematical expression is:
[0163] ;
[0164] In the formula, .
[0165] S64. Based on the initial visible light image and the registered SAR image Output the final registered image The registration of visible light images and SAR images is completed, and its mathematical expression is:
[0166] ;
[0167] In the formula, This is an element-wise matrix multiplication operation.
[0168] The feasibility of the method provided in the embodiments of the present invention will be verified by specific tests below.
[0169] 1. Working conditions: This experiment uses an Intel(R) Core(TM) i5-9400F CPU @ 2.90GHz processor, a PC running Windows 10, a GeForce GTX 1060 graphics card, and Python as the programming language.
[0170] 2. Experimental Content and Result Analysis: The registration and splicing results of this experiment are as follows: Figure 5 As shown, the matching of the present invention (such as...) can be seen. Figure 5 The image stitching method described in this paper is relatively accurate. After mapping, the texture at the edges of the SAR image has significant continuity with the texture of the visible light image. In contrast, existing OS-SIFT algorithms used for visible light-SAR image registration (such as...) Figure 5 The images stitched together by OS-SIFT in the image are misaligned.
[0171] The experimental results above show that the method of the present invention first uses single-modal features to perform preliminary image registration, making joint feature calculation possible. Then, based on the cross-modal fractional-order joint cumulant spectrum, it reduces the nonlinear radiation differences caused by different imaging mechanisms, and achieves relatively accurate cross-modal image registration and stitching with clear and obvious results.
[0172] In another embodiment of the present invention, an image registration system based on fractional-order joint spectrum and orientation consistency is also provided to implement the above-described image registration method based on fractional-order joint spectrum and orientation consistency, specifically including:
[0173] The feature map generation module is used to acquire multimodal images, perform sliding block division on the multimodal images, calculate the fractional-order cumulative spectrum of a single mode, and generate various single-mode feature maps; the multimodal images include visible light images and SAR images;
[0174] The feature map coarse matching module is used to perform global coarse matching on multiple single-modal feature maps based on a fast cross-correlation algorithm to obtain an initial coarse matching point matrix.
[0175] The image coarse registration module is used to perform coarse registration of multimodal images based on the initial coarse matching point matrix, recalculate the single-mode fractional-order cumulative spectrum and cross-mode fractional-order joint cumulative spectrum of SAR images, and obtain new SAR single-mode feature maps and joint feature maps.
[0176] The feature vector construction module is used to extract multi-directional gradients from new SAR single-mode feature maps and joint feature maps, and construct multi-frequency, multi-scale feature vectors.
[0177] The precise matching module is used to perform precise matching based on multi-frequency, multi-scale feature vectors, multi-directional consistency, and image features to obtain a precise matching set.
[0178] The image registration output module is used to solve the affine transformation matrix based on the precise matching set, perform geometric transformation and sub-pixel interpolation on the SAR image, obtain the registered SAR image, and complete the registration of multimodal images.
[0179] It should be understood that although the steps in the flowcharts of the embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in each embodiment may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.
[0180] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods.
[0181] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.
Claims
1. An image registration method based on fractional-order joint spectrum and orientation consistency, characterized in that, Includes the following steps: Acquire multimodal images, perform sliding block division on the multimodal images, calculate the fractional-order cumulative spectrum of single modes, and generate various single-mode feature maps; the multimodal images include visible light images and SAR images; Based on the fast cross-correlation algorithm, global coarse matching is performed on multiple single-modal feature maps to obtain an initial coarse matching point matrix; Based on the initial coarse matching point matrix, coarse registration is performed on the multimodal images, and the single-mode fractional-order cumulative spectrum and cross-mode fractional-order joint cumulative spectrum of the SAR images are recalculated to obtain new SAR single-mode feature maps and joint feature maps. Multi-directional gradients are extracted from the new SAR single-mode feature map and joint feature map, and multi-frequency, multi-scale feature vectors are constructed. Based on multi-frequency, multi-scale feature vectors, precise matching is performed using multi-directional consistency and image features to obtain a precise matching set. Based on the precise matching set, the affine transformation matrix is solved, and geometric transformation and subpixel interpolation are performed on the SAR image to obtain the registered SAR image, thus completing the registration of multimodal images.
2. The image registration method based on fractional-order joint spectrum and orientation consistency according to claim 1, characterized in that, The steps of acquiring multimodal images, performing sliding block division on the multimodal images, calculating the fractional-order cumulant spectrum of a single modality, and generating multiple single-modal feature maps specifically include: Visible light images SAR images Perform sliding block division separately, defining the visible light image block as... SAR image patch ; Calculate the first, second, and third order single-modal fractional moments of a visible light image patch, expressed as follows: , , Its mathematical expression is: ； ； ； In the formula, The center position of the image patch; denoted as the two-dimensional spatial interval; M and N represent the number of rows and columns of pixels in the image patch; 1b, 2b, and 3b represent the fractional order. ; Single-modal fractional moments based on visible light image patches , , Calculate its 3b-order single-mode fractional cumulant, its mathematical expression is: ； Calculate the Mittag-Leffler function for different spatial intervals. The value of is expressed mathematically as follows: ； In the formula, Spatial interval; Γ represents the chosen k-th angular frequency; Γ is the gamma function, a core function of fractional calculus theory. The imaginary unit; Single-modal fractional-order cumulants based on visible light image patches Calculate its 3b-order single-mode fractional-order cumulant spectrum to generate a visible light single-mode feature map. Its mathematical expression is: ； In the formula, Angular frequency; The first, second, and third order single-mode fractional moments of a SAR image patch are calculated, and are expressed as follows: , , Its mathematical expression is: ； ； ； In the formula, The center position of the image patch; denoted as the two-dimensional spatial interval; M and N represent the number of rows and columns of pixels in the image patch; 1b, 2b, and 3b represent the fractional order. ; Single-mode fractional moments based on SAR image patches , , Calculate its 3b-order single-mode fractional cumulant, its mathematical expression is: ； Single-modal fractional-order cumulants based on SAR image patches Calculate the fractional-order cumulative spectrum of a 3b-order SAR single-mode to generate a SAR single-mode feature map. Its mathematical expression is: ； Visible light single-mode feature map and SAR single-mode feature map Perform global normalization and then map it to a preset pixel range to obtain a normalized SAR single-modality feature map. Normalized visible light single-mode feature map .
3. The image registration method based on fractional-order joint spectrum and orientation consistency according to claim 2, characterized in that, Based on the fast cross-correlation algorithm, the steps for performing global coarse matching on multiple single-modal feature maps to obtain the initial coarse matching point matrix include: Normalized SAR single-mode feature map Normalized visible light single-mode feature map Perform two-dimensional fast cross-correlation calculations to generate a global similarity heatmap. Its mathematical expression is: ； In the formula, i and j represent the row and column coordinates of the pixel in the feature map, u and v represent the row and column offsets; M1 and N1 are the number of rows and columns of the feature map. Based on the image size M2×N2, and with a preset threshold K for the number of coarse matching points, the coordinates of the top K peaks in the global similarity heatmap are extracted and used to form an initial coarse matching point matrix using a matrix stacking method. and .
4. The image registration method based on fractional-order joint spectrum and orientation consistency according to claim 3, characterized in that, Based on the initial coarse matching point matrix, the steps of coarse registration of multimodal images, recalculation of the single-modal fractional-order cumulant spectrum and cross-modal fractional-order joint cumulant spectrum of SAR images, and obtaining new SAR single-modal feature maps and joint feature maps specifically include: Based on the initial coarse matching point matrix and Calculate the coarse registration affine transformation matrix Its mathematical expression is: ； In the formula, for The transpose of the matrix; Based on the coarse registration affine transformation matrix For the original SAR image Perform a global geometric transformation to obtain the transformed SAR image. Its mathematical expression is: ； Based on the transformed SAR image Recalculate and generate new SAR single-mode feature maps Then, global normalization is performed, and the data is mapped to a preset pixel range to generate a new normalized SAR single-modality feature map. ; The mathematical expression for calculating the fractional joint moments is as follows: ； In the formula, For visible light image patches, For SAR image patches; The center position of the image patch; denoted as the two-dimensional spatial interval; M and N represent the number of rows and columns of pixels in the image patch; 1b, 2b, and 3b represent the fractional order. ; Based on the single-mode fractional moment of the visible light image patch and Single-mode fractional moments of SAR image patches and Fractional joint moments Calculate the joint cumulant of the 3b-order cross-modal fractional order. Its mathematical expression is: ； According to the Mittag-Leffler function Calculate the two-dimensional fractional-order joint cumulant spectrum and generate the joint feature map. Its mathematical expression is: ； joint feature map Perform global normalization, then map to a preset pixel range to obtain a normalized joint feature map. .
5. The image registration method based on fractional-order joint spectrum and orientation consistency according to claim 4, characterized in that, The steps for extracting multi-directional gradients from the new SAR single-mode feature map and joint feature map, and constructing multi-frequency, multi-scale feature vectors, specifically include: For the new normalized SAR single-mode feature map and normalized joint feature map Create angular frequencies respectively , , , The Gaussian pyramid at that time, and detect key points; According to the Gaussian pyramid, and simultaneously within the circular regions of all key points of the Gaussian pyramid based on the four angular frequency combinations, according to the direction... and angular frequency Calculate gradient magnitude Its mathematical expression is: ； In the formula, This represents the feature map in the Gaussian pyramid, where u and v are the pixel coordinates of the feature map; and These are the partial derivatives in the horizontal and vertical directions, respectively; Gradient statistics of key points detected at multiple scales are grouped by angular frequency, sorted by angle from smallest to largest within each group, and sorted by angular frequency from smallest to largest between groups to generate SAR feature vectors. Joint eigenvectors .
6. The image registration method based on fractional-order joint spectrum and orientation consistency according to claim 5, characterized in that, The steps for obtaining a precise matching set based on multi-frequency, multi-scale feature vectors, multi-directional consistency, and image features include: Calculating feature vectors using cosine similarity and The matching degree, its mathematical expression is: ； In the formula, For L2 norm operations; When the matching degree of the feature vector is greater than a preset threshold, the matching pair is considered a valid match. False matching pairs are eliminated, and valid matching points are retained to form a matrix of precise matching points. and This yields an exact matching set.
7. The image registration method based on fractional-order joint spectrum and orientation consistency according to claim 6, characterized in that, Based on the precise matching set, the affine transformation matrix is solved, and geometric transformation and sub-pixel interpolation are performed on the SAR image to obtain the registered SAR image. The steps to complete the registration of multimodal images specifically include: Based on the precise matching point-pair matrix and Calculate the affine transformation matrix for accurate registration. Its mathematical expression is: ； According to the affine transformation matrix Perform a global geometric transformation to obtain the transformed SAR image. Its mathematical expression is: ； Based on the transformed SAR image Select floating-point coordinates one by one The four integer neighboring pixels around the sub-pixel Transformed SAR image Subpixel interpolation is performed to obtain the registered SAR image. Its mathematical expression is: ； In the formula, . Based on the initial visible light image and the registered SAR image Output the final registered image The registration of visible light images and SAR images is completed, and its mathematical expression is: ； In the formula, This is an element-wise matrix multiplication operation.
8. An image registration system based on fractional-order joint spectrum and orientation consistency, used to implement the image registration method based on fractional-order joint spectrum and orientation consistency as described in any one of claims 1-7, characterized in that, include: The feature map generation module is used to acquire multimodal images, perform sliding block division on the multimodal images, calculate the fractional-order cumulative spectrum of a single mode, and generate various single-mode feature maps; the multimodal images include visible light images and SAR images; The feature map coarse matching module is used to perform global coarse matching on multiple single-modal feature maps based on a fast cross-correlation algorithm to obtain an initial coarse matching point matrix. The image coarse registration module is used to perform coarse registration of multimodal images based on the initial coarse matching point matrix, recalculate the single-mode fractional-order cumulative spectrum and cross-mode fractional-order joint cumulative spectrum of SAR images, and obtain new SAR single-mode feature maps and joint feature maps. The feature vector construction module is used to extract multi-directional gradients from new SAR single-mode feature maps and joint feature maps, and construct multi-frequency, multi-scale feature vectors. The precise matching module is used to perform precise matching based on multi-frequency, multi-scale feature vectors, multi-directional consistency, and image features to obtain a precise matching set. The image registration output module is used to solve the affine transformation matrix based on the precise matching set, perform geometric transformation and sub-pixel interpolation on the SAR image, obtain the registered SAR image, and complete the registration of multimodal images.