A method and system for registration of infrared and visible light image fusion

Abstract

The application discloses an infrared and visible light image fusion registration method and system, and relates to the field of image registration. V ' and an infrared video image are collected, data preprocessing is performed, and a preprocessed visible light image and a preprocessed infrared image are outputted. I ' and an infrared video image are collected, data preprocessing is performed, and a preprocessed visible light image and a preprocessed infrared image are outputted. V ' and an infrared video image are collected, data preprocessing is performed, and a preprocessed visible light image and a preprocessed infrared image are outputted. I ' and an infrared video image are collected, data preprocessing is performed, and a preprocessed visible light image and a preprocessed infrared image are outputted. The application realizes accurate optimization of real-time translation parameters from stability guarantee of feature extraction, reliability screening of matching pairs, to efficient solution of initial parameters.

Description

Technical Field

[0001] This invention relates to the field of image registration technology, and more specifically, to a registration method and system for infrared and visible light image fusion. Background Technology

[0002] Visible light images are acquired using conventional optical sensors and contain rich texture details and scene contour information; infrared images capture the thermal radiation characteristics of targets, are not limited by lighting conditions, and can penetrate interference such as smoke and fog. The fusion of the two can achieve complementary advantages and improve the comprehensiveness and accuracy of scene perception.

[0003] Accurate registration of the two types of images is a prerequisite for effective fusion, but existing technologies face significant challenges in the registration process:

[0004] On the one hand, visible light and infrared images have significant modal differences—the former focuses on optical texture, while the latter focuses on thermal distribution. Furthermore, infrared images often suffer from low resolution and a lack of texture information, making it difficult for traditional feature extraction methods to capture stable and discriminative feature points. In existing technologies, feature point extraction often relies on single-scale or simple features, resulting in low-dimensional descriptors with insufficient robustness. These descriptors cannot effectively resist the effects of illumination fluctuations, noise, and modal differences, and are prone to feature mismatch or omission.

[0005] On the other hand, feature matching does not classify the reliability of matching pairs. Existing techniques treat all matching pairs as equally important, directly using scale (s), rotation (s), and other parameters. The calculation of registration parameters, such as outliers, leads to low-confidence matching and estimation of interference parameters for outliers. To eliminate outliers, traditional random sampling algorithms require a large number of iterative samplings, which not only increases the computational load but may also cause parameters to converge to local optima due to invalid iterations, making it difficult to balance accuracy and efficiency.

[0006] Therefore, how to dynamically adjust the sampling strategy to reduce iterations and accurately solve s and θ based on the modal differences between the two types of images has become a key technical problem in improving the registration performance of visible light and infrared images. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies, this invention provides a registration method and system for infrared and visible light image fusion.

[0008] The technical solution of this invention is as follows: a registration optimization method for infrared and visible light image fusion, characterized by comprising the following steps:

[0009] Image acquisition and preprocessing: Acquiring visible light video images With infrared video images ,right and Perform data preprocessing and output the preprocessed visible light image. 'Compared with preprocessed infrared images ';

[0010] Calculate registration parameters: for 'and Perform registration parameter calculations to obtain relative registration parameters, which include the scale parameter s and the rotation parameter. Translation parameters (dx, dy);

[0011] Image fusion: based on Preprocessed visible light images and Perform transformation alignment, use a fusion algorithm to fuse the aligned image, and output the fused image. The fusion algorithm includes a grayscale fusion algorithm or a pseudo-color fusion algorithm.

[0012] The calculation of the relative registration parameters adopts a method combining initial registration and real-time registration, specifically including:

[0013] Initial registration: A multi-scale feature descriptor extraction algorithm is used to obtain the registration information from the preprocessed visible light image. Compared with preprocessed infrared images Extracting feature points and corresponding high-dimensional descriptors , ;calculate and similarity distance ,based on Feature matching pairs are weighted for partitioning The weighted random sampling consensus algorithm is adopted, according to... Adjusting the sampling probability k, s is obtained through iterative calculation. ,Right now:

[0014]

[0015] in For weighted sampling subsets, For matching pairs in The following error;

[0016] Real-time registration: combining scale parameter s with Solve for the preprocessed visible light image 'Compared with preprocessed infrared images The maximum mutual information (dx, dy) is:

[0017] ; in, For mutual information functions, For scale-rotation transformation function, This is for transforming compound operations.

[0018] In this invention, a multi-scale feature descriptor extraction algorithm is used to extract feature points and corresponding high-dimensional descriptors from visible light and infrared video images, respectively. This effectively captures the essential features of visible light and infrared images at different scales, enhancing adaptability to image detail differences, illumination changes, and noise interference. Then, the similarity distance between feature point descriptors of different images is calculated, and feature matching pairs are weighted based on this similarity distance to obtain hierarchical matching pairs. Subsequently, a weighted random sampling consensus algorithm is used to dynamically adjust the sampling probability according to the matching pair weights, prioritizing the selection of high-level matching pairs to construct a sampling subset. The scale and rotation parameters between the two images are obtained through iterative calculation, directly linking the contribution of the feature matching pair to its reliability, thus obtaining more reliable matching pairs. The impact of high-weight, low-confidence, or mismatched pairs is minimized, reducing interference from external factors in the calculation of registration parameters. This significantly improves the accuracy of the initial model estimation, reduces invalid iterations, and enhances the algorithm's convergence speed while maintaining the accuracy of scale and rotation parameters. The maximum mutual information algorithm, combined with the scale and rotation parameters obtained from the initial registration, optimizes the search strategy to find the translation parameters that maximize the mutual information between the preprocessed visible light and infrared images. This enables real-time registration parameter calculation for both types of images, focusing on fine-tuning the translation deviation. It fully utilizes the statistical correlation between images to improve registration accuracy while balancing real-time requirements by constraining the search range, ensuring the registration process remains efficient and stable even in dynamic scenarios.

[0019] As a preferred embodiment, the method further includes:

[0020] Multi-scale feature descriptor extraction algorithms were used to extract features from preprocessed visible light images. Compared with preprocessed infrared images Extracting feature points This includes the following steps:

[0021] Multi-scale convolutional feature extraction: Using a pre-defined convolutional kernel, for... Perform multi-scale convolution operations separately, and combine them with non-linear activation functions to generate feature maps at least two scales, i.e.:

[0022] ; in, K represents the preset number of scales. , For the first The scale is preset for the convolution kernel size; * indicates the convolution operation. For the preset nonlinear activation function, For the l-th scale bias term;

[0023] in, It could also be:

[0024] ;

[0025] Indicates the first The feature map pixel values ​​at the scale (l≥1, where l is the preset scale number); * represents the convolution operation; ReLU() is the activation function; This is a preprocessed visible light / infrared image. For the first The convolution kernel of the preset size of the scale, (Fl−1(i,j)) is the first... Scale-based feature map. For the first Preset biases for the scale;

[0026] Pixel feature response values ​​are calculated based on multi-scale feature maps. Candidate feature points are selected according to a preset threshold, and effective feature points are obtained by performing neighborhood non-maximum suppression. ,Right now:

[0027] ; in, For pixels Characteristic response value, For multi-scale eigenvalue maximization calculation, This is for calculating the mean of multi-scale eigenvalues.

[0028] Filter out ( Candidate feature points (based on a preset response value threshold) are output after non-maximum suppression. Effective feature point set Effective feature point set .

[0029] As a preferred embodiment, the method further includes:

[0030] The generation and effective feature points Corresponding high-dimensional descriptor This includes the following steps:

[0031] Local feature region extraction: Each feature point Centered on the feature map at the corresponding scale, extract... (S is the preset size) Local feature area ;

[0032] High-dimensional vector transformation and normalization: Expanding into a one-dimensional vector and performing L2 normalization yields a high-dimensional descriptor, namely: ;

[0033] in For feature points High-dimensional descriptors For matrix expansion operations, It is an L2 norm;

[0034] Descriptor output: Summary All feature points have to Summary All feature points , and Used for the similarity distance The calculation.

[0035] As a preferred embodiment, the method further includes:

[0036] calculate and similarity distance ,Right now:

[0037] right Arbitrary descriptor and Arbitrary descriptor Calculate the Euclidean distance between the two as the similarity distance. ,Right now:

[0038] ; Where N is the preset dimension of the high-dimensional descriptor. for The k-th eigenvalue, for The k-th eigenvalue;

[0039] Will The feature matching pairs are weighted according to the comparison results after being compared with a preset distance threshold. To support the sampling probability adjustment and s of the weighted random sampling consensus algorithm, Iterative calculation.

[0040] As a preferred embodiment, the method further includes:

[0041] Based on similarity distance Weighting is performed on the feature matching pairs, i.e.:

[0042] Set at least two preset distance thresholds Based on the similarity distance The comparison results with the threshold are used to assign corresponding weights to each visible light feature point-infrared feature point matching pair. :

[0043] ; For high weighting, for example, 1.8-2.5; For medium weighting and preset weight values, such as 0.9-1.2, A high confidence threshold, for example, 0.3-0.5; The thresholds are set to a medium confidence level and a preset distance threshold, for example, 0.7-0.9. If the matched pair is located in an occluded area, A value of 0.85-0.95 can be used to avoid misjudgment;

[0044] Calculate the sum of all non-zero weights, and divide the weight value of each non-zero weight pair by the sum to obtain the normalized weights.

[0045] ; in, For the non-zero weight value, The normalized weights Used for sampling probability allocation in subsequent weighted random sampling consensus algorithms.

[0046] As a preferred embodiment, the method further includes:

[0047] Based on normalized weights By using random number generation and interval matching methods, a preset number of matching points are selected from non-zero weight matching pairs to construct a sampling subset. The preset number is not less than the minimum number of points required to solve the affine transformation model, for example, four or more. Each matching point pair includes one corresponding valid feature point. ;

[0048] Iterative calculation yields s and ,include:

[0049] Based on sampling subsets Calculate the initial scale parameters Initial rotation parameters And count all feature matching pairs in , Reprojection error ;

[0050] Repeat the above loop, with the number of iterations K dynamically calculated:

[0051] ; in, To set a constant for the reliability, To match the estimated proportion of outliers in the sampled subset, n is the minimum number of matching points required to construct the sampled subset. This is the weighting adjustment factor. It ranges from 0.6 to 0.8.

[0052] Until the model parameters with the largest number of interior points and an error less than a preset threshold are found, the optimal scaling parameter s and rotation parameter are output. ,Right now:

[0053] ; Furthermore, the proportion of external points The interior points are:

[0054] Based on sampling subsets Calculate the initial scale parameters Initial rotation parameters And count all feature matching pairs in , Reprojection error ;

[0055] Interior point: reprojection error Feature matching pairs with a preset error threshold of 1.5 pixels or less are considered valid matching points that conform to the current model.

[0056] Outer point: reprojection error Feature matching pairs with a preset error threshold are judged as invalid matching points that deviate from the current model;

[0057] Outer point scale The proportion of outliers to total feature matching pairs:

[0058]

[0059] As a preferred embodiment, the method further includes:

[0060] The translation parameters are solved using the maximum mutual information algorithm. ,include:

[0061] Based on preprocessed visible light images 'or infrared image' The temporal continuity of ' is achieved by calculating the global motion vector through inter-frame feature tracking. Based on the global motion vector Define translation parameters The search range is determined, and the goal is to find the range that makes the search range more comprehensive. 'and The (dx,dy) pairs with the largest mutual information include:

[0062] Extract stable feature points from the previous frame of the preprocessed image. By tracking features between frames, the correspondence between valid feature points in the current frame and the previous frame is determined, thus obtaining feature point pairs. Based on the coordinate difference of the corresponding feature points, calculate the local displacement vector for each matching pair:

[0063] ; in, The coordinates of the feature points in the previous frame. Match the coordinates of the point in the current frame;

[0064] Abnormal displacement vectors are filtered based on statistical characteristics, and the global motion vector is calculated using a statistical aggregation method for the valid displacement vectors. :

[0065] ;

[0066] ;

[0067] Weights are based on the stability of feature points;

[0068] And mutual information function include:

[0069] ;

[0070] in, , The normalized grayscale distributions of images A and B are given. It is a joint distribution.

[0071] As a preferred embodiment, the method further includes:

[0072] The preprocessed image and ,include:

[0073] based on right and Perform transformation alignment.

[0074] ;

[0075] in, For aligned visible light images, used for s and Achieve pixel-level alignment.

[0076] Furthermore, based on global motion vectors Divide translation parameters The search range, the search range is based on Centered on the preset pixel offset range: ,in The preset base offset coefficient is set to 2-5 pixels; if the global motion vector is a reused value, Adaptability increases, such as expanding to 1.5 to 2 times the base value;

[0077] Within the search range, calculate the intermediate visible light image. Compared with preprocessed infrared images Different Choose the mutual information value that maximizes the mutual information. As the optimal translation parameter, that is:

[0078]

[0079] Maximize mutual information Used to quantify the statistical correlation between two images.

[0080] As a preferred embodiment, the method further includes:

[0081] The timing consistency verification method monitors registration errors in real time and adjusts them dynamically, including:

[0082] Collect the registration error between the current frame and the previous 3 frames. And calculate the mean error.

[0083] ;

[0084] Calculate the standard deviation of the error. The formula is:

[0085] ;

[0086] when Maintain current registration parameters;

[0087] when Expand the search range for real-time registration, enabling Solve again This ensures that the error regression is within a stable range of ≤1.0 pixels.

[0088] The present invention also includes a registration system for infrared / visible light image fusion, the system employing the registration optimization method for infrared / visible light image fusion as described in any one of claims 1-9, comprising:

[0089] Preprocessing module: used for acquiring visible light video images With infrared video images ,right and Perform data preprocessing and output the preprocessed visible light image. 'Compared with preprocessed infrared images ';

[0090] Registration parameter module: used for... 'and Perform registration parameter calculations to obtain relative registration parameters, which include the scale parameter s and the rotation parameter. Translation parameters (dx, dy);

[0091] Image fusion module: used for image fusion based on Preprocessed visible light images and Perform transformation alignment, use a fusion algorithm to fuse the aligned image, and output the fused image. The fusion algorithm includes a grayscale fusion algorithm or a pseudo-color fusion algorithm.

[0092] The registration parameter module further includes:

[0093] Initial registration module: used to extract features from various scales using a multi-scale feature descriptor extraction algorithm. Extracting feature points and corresponding high-dimensional descriptors , ;calculate and similarity distance ,based on Feature matching pairs are weighted for partitioning The weighted random sampling consensus algorithm is adopted, according to... Adjusting the sampling probability k, s is obtained through iterative calculation. ,Right now:

[0094] ;

[0095] in For weighted sampling subsets, For matching pairs in The following error;

[0096] Real-time registration module: used to combine scale parameter s with Solve for the preprocessed visible light image 'Compared with preprocessed infrared images The maximum mutual information (dx, dy) is:

[0097] ;

[0098] in, For mutual information functions, For scale-rotation transformation function, This is for transforming compound operations.

[0099] According to the above-described solution, the beneficial effects of this invention are as follows:

[0100] This invention improves the accuracy and efficiency of infrared and visible light image registration by ensuring the stability of feature extraction, screening the reliability of matching pairs, and efficiently solving the initial parameters, ultimately achieving accurate optimization of real-time translation parameters and enhancing the consistency of fused images. It is especially suitable for dynamic scene registration needs in complex environments. Attached Figure Description

[0101] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0102] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0103] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0104] It should be noted that when a component is referred to as "fixed," "set," or "connected" to another component, it may be located directly or indirectly on that other component. The terms "upper," "lower," "left," "right," "front," "rear," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or position based on the accompanying drawings, and are for ease of description only, and should not be construed as limiting the technical solution. The terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features. "Many" means two or more, unless otherwise explicitly specified. "Several" means one or more, unless otherwise explicitly specified.

[0105] Example 1

[0106] This invention is applicable to various scenarios of infrared and visible light image fusion, such as security monitoring for all-weather day and night surveillance, vehicle-mounted assisted driving in complex road conditions, forest fire prevention for long-distance fire source detection, industrial equipment detection, and military reconnaissance. By accurately selecting the proportion of outliers in feature matching, the iterative computation is optimized, thus solving the registration deviation problem caused by modal differences and environmental interference in different scenarios.

[0107] An infrared-visible light image fusion registration optimization method is disclosed. This registration method, as an application carrier, can be flexibly configured in programs, hardware devices, computer storage media, or cloud algorithm interfaces based on actual scenario requirements. At the program level, it can be compiled across operating systems (Linux, Android, Windows); at the hardware level, it is compatible with processors of different computing power (from low-power embedded chips such as FPGAs and DSPs to high-performance cloud CPUs / GPUs); and at the storage level, it supports various computer-readable media. The deployment form can be flexibly selected according to the hardware resources and functional requirements of the actual scenario, effectively reducing the development and adaptation costs of multi-scenario applications, improving the practicality and promotional value of the method, and enabling rapid deployment and application in multiple scenarios.

[0108] This embodiment includes a registration system for infrared / visible light image fusion, wherein the system employs the registration optimization method for infrared / visible light image fusion as described in any one of claims 1-9, including:

[0109] Preprocessing module: used for acquiring visible light video images With infrared video images ,right and Perform data preprocessing and output the preprocessed visible light image. 'Compared with preprocessed infrared images ';

[0110] Registration parameter module: used for... 'and Perform registration parameter calculations to obtain relative registration parameters, which include the scale parameter s and the rotation parameter. Translation parameters (dx, dy);

[0111] Image fusion module: used for image fusion based on Preprocessed visible light images and Perform transformation alignment, use a fusion algorithm to fuse the aligned image, and output the fused image. The fusion algorithm includes a grayscale fusion algorithm or a pseudo-color fusion algorithm.

[0112] The registration parameter module further includes:

[0113] Initial registration module: used to extract features from various scales using a multi-scale feature descriptor extraction algorithm. Extracting feature points and corresponding high-dimensional descriptors , ;calculate and similarity distance ,based on Feature matching pairs are weighted for partitioning The weighted random sampling consensus algorithm is adopted, according to... Adjusting the sampling probability k, s is obtained through iterative calculation. ,Right now:

[0114] ; in For weighted sampling subsets, For matching pairs in The following error;

[0115] Real-time registration module: used to combine scale parameter s with Solve for the preprocessed visible light image 'Compared with preprocessed infrared images The maximum mutual information (dx, dy) is:

[0116] ;

[0117] in, For mutual information functions, For scale-rotation transformation function, This is for transforming compound operations.

[0118] The system also includes integrated processing using FPGA and DSP modules. The FPGA performs pre-processing, the DSP handles core computation, and a suitable data interaction interface is constructed.

[0119] FPGA module: responsible for image acquisition synchronization, parallel preprocessing, feature extraction, and geometric transformation tasks, relying on its hardware parallel logic unit—LUT, registers to realize multi-channel data parallel processing;

[0120] DSP module: Responsible for tasks such as weight partitioning, weighted random sampling consensus algorithm (weighted RANSAC) iteration, and mutual information calculation. Relying on its dedicated floating-point unit (FPU) and Harvard architecture, it improves the iteration efficiency of complex algorithms.

[0121] Data interaction interface: High-speed data interaction between FPGA and DSP is achieved through EMIF (external memory interface) or AXI4 bus, with a data transmission rate of ≥1GB / s, ensuring real-time synchronization of feature data and registration parameters between the two modules.

[0122] Example 2

[0123] like Figure 1 As shown, an infrared-visible light image fusion registration optimization method includes four main steps: image acquisition and preprocessing, initial registration, real-time registration, and image fusion. The specific applications of each step are as follows:

[0124] This security monitoring device employs a dual-sensor system integrating a 1080P visible light camera and a 640×512 resolution infrared thermal imager. A synchronous trigger module ensures that both images are captured simultaneously in the same scene, and each outputs a visible light video image. With infrared video images .

[0125] I. Perform data preprocessing on two types of raw images. The specific preprocessing includes:

[0126] right A 5×5 Gaussian filter is used to suppress random noise under low illumination. The standard deviation of the Gaussian kernel is set to 1.2. Then, the grayscale distribution is adjusted by adaptive histogram equalization to improve the texture contrast of static targets such as walls and equipment brackets.

[0127] right A 3×3 median filter is used to eliminate thermal noise interference and avoid artifacts at high temperatures. Then, grayscale normalization is performed to map the infrared image grayscale values ​​to the [0, 255] range, consistent with the grayscale range of the visible light image. After preprocessing, a preprocessed visible light image with a uniform resolution of 640×512 is output. 'Compared with preprocessed infrared images The data is temporarily stored in the device's built-in DDR4 memory to provide high-quality image data for subsequent feature extraction.

[0128] II. Initial Registration: Solving for the Scale Parameter s and Rotation Parameter

[0129] Initial registration achieves scale and rotation alignment between the two types of images through multi-scale feature extraction, similarity calculation, and a weighted random sampling consistency algorithm. The specific steps are as follows:

[0130] 2.1 Initial Registration: Through multi-scale feature extraction, similarity calculation, and a weighted random sampling consistency algorithm, scale and rotation alignment of the two types of images is achieved. The specific steps are as follows:

[0131] Using pre-sized convolution kernels 'and Perform multi-scale convolution operations separately, with a preset number of scales K=3 (corresponding to 1×, 1.5×, and 2× scales), and the convolution kernel at scale l. The size is 3×3, and feature maps at various scales are generated using the ReLU nonlinear activation function. The formula is as follows:

[0132] ;

[0133] Indicates the first The feature map pixel values ​​at the scale (l≥1, where l is the preset scale number); * represents the convolution operation; ReLU() is the activation function; This is a preprocessed visible light / infrared image. For the first The convolution kernel of the preset size of the scale, (Fl−1(i,j)) is the first... Scale-based feature map. For the first Preset biases for the scale;

[0134] Pixel feature response value calculated based on multi-scale feature map The formula is:

[0135] ;

[0136] in, For pixels Characteristic response value, For multi-scale eigenvalue maximization calculation, This is for calculating the mean of multi-scale eigenvalues.

[0137] Preset =25, filter out Candidate feature points; after 3×3 neighborhood nonmaximum suppression, points with non-maximum response values ​​in the local neighborhood are removed, resulting in... Effective feature point set Effective feature point set .

[0138] For effective feature point set Effective feature point set Each feature point in Generate the corresponding Gaussian descriptor :

[0139] 2.2 High-dimensional descriptor generation

[0140] Local feature region extraction: Centered on the feature map at its corresponding scale, a 16×16 local feature region is extracted. ;

[0141] Vector transformation and normalization: through matrix expansion operations, Will Convert to a 256-dimensional one-dimensional vector, perform L2 normalization on this vector to obtain the Gaussian descriptor, as shown in the formula:

[0142] ;

[0143] in For feature points High-dimensional descriptors For matrix expansion operations, It is an L2 norm.

[0144] calculate Examples of similarity between features are given, and feature matching pairs are weighted based on these similarity examples:

[0145] 2.3. Similarity Calculation and Weighting:

[0146] right Arbitrary descriptor and Arbitrary descriptor Calculate the Euclidean distance between the two as the similarity distance. ,Right now:

[0147] ;

[0148] in, for The k-th eigenvalue, for The k-th eigenvalue;

[0149] Preset distance threshold =0.4, =0.8, based on similarity distance Weighting the comparison results with the distance threshold

[0150] ;

[0151] Normalization is performed on the matching pairs with non-zero weights to obtain normalized weights. ,Right now

[0152] ;

[0153] In this embodiment, M is 80-120, ensuring... .

[0154] 2.4 Weighted Random Sampling Consensus Algorithm for Solving s and

[0155] Based on normalized weights The scaling parameter s and rotation parameter are solved using a weighted random sampling consensus algorithm. ,Right now

[0156] Sampling subset construction: based on Adjust the sampling probability of each matching pair; the higher the weight, the greater the sampling probability. Select four matching pairs from the non-zero weight pairs, solve for the minimum number of points in the affine transformation model, and construct the sampling subset. ;

[0157] Initial parameter estimation: based on Establish and affine transformation relationship, calculate initial scale parameters With initial rotation parameters And calculate the reprojection error of all matching pairs under this parameter. ,

[0158] ;

[0159] in, for Coordinates of feature points in the middle, for The coordinates of the corresponding matching point;

[0160] Iterative convergence solution: Assume a pre-set confidence level Weighting adjustment coefficient Based on the current external point ratio =Number of outliers / Total number of matched pairs. In this embodiment, the initial proportion of outliers is approximately 25%. The number of iterations K is dynamically calculated using the following formula:

[0161] ;

[0162] Set reprojection error threshold Pixels, will Matching pairs are identified as inliers. The sampling subset construction and parameter estimation are repeated until the number of iterations reaches K or the model parameter with the largest number of inliers is found. Finally, the optimal scaling parameters are output. With optimal rotation parameters In this embodiment, s is approximately 1.02-1.05. The absolute value is ≤1.5°, that is:

[0163] ;

[0164] III. Real-time Registration: Solving for Translation Parameters (dx, dy)

[0165] Real-time registration combined with initial registration By using global motion vector constraints and the maximum mutual information algorithm, the translation parameters can be accurately solved. Specific steps are as follows:

[0166] 3.1 Global Motion Vector Calculation

[0167] based on Temporal continuity—calculating the global motion vector between the previous frame at time t-1 and the current frame at time t. :

[0168] Effective feature point tracking: from time t-1 Extract stable feature points from the initial registration and reuse them. Feature points obscured by trees or temporary obstacles are removed, retaining 60-80 valid points. The correspondence between valid feature points at time t and time t-1 is determined using the LK optical flow tracing algorithm, resulting in feature point pairs. ;

[0169] Local Displacement and Outlier Filtering: Calculating Local Displacement Vectors Based on Coordinate Differences Between Feature Point Pairs The formula is:

[0170]

[0171] The 3σ criterion is used to filter out abnormal displacement vectors, retaining valid vectors within the 99.7% confidence interval;

[0172] Global motion aggregation: The median aggregation method is used to calculate the global motion vector from the effective displacement vectors. The formula is as follows:

[0173] ;

[0174] In this embodiment, The range of values ​​is If the number of effective displacement vectors is less than 30, then reuse the pixel. Global motion vector at time t .

[0175] 3.2 Search Space Constraints and Mutual Information Solving

[0176] Preprocessing transformation: 'Based on optimal parameter s and Perform a scale-rotation transformation to obtain an intermediate visible light image with scale and rotation aligned. ;

[0177] Search scope definition: based on Define translation parameters The search range is set with a preset offset coefficient. pixels, search range:

[0178]

[0179] If the global motion vector is a reused value, then \(\Delta\) is expanded to 4 pixels to ensure that the actual translation deviation is covered;

[0180] Mutual information maximization solution: Within the above search range, calculate and Different The mutual information value MI is given below, and the formula for calculating mutual information is:

[0181] ;

[0182] in, , They are respectively , The normalized gray distribution The grayscale distribution is a joint distribution of the two; the (dx,dy) that maximizes MI is chosen as the optimal translation parameter, i.e.:

[0183] ;

[0184] In this embodiment, the value range of (dx,dy) is [-2,2] pixels, ensuring... and Achieve pixel-level alignment.

[0185] IV. Image Fusion

[0186] Based on the optimal registration parameter s, With (dx,dy), for and Perform a transformation alignment; the alignment formula is:

[0187] ;

[0188] in, To align the visible light image, with Achieving a one-to-one coordinate correspondence. A grayscale fusion algorithm is used to fuse the aligned images: [The algorithm is then applied to the image]. Areas with clear textures, such as wall edges and equipment brackets, are given blending weight. ,right Regions with distinct thermal radiation characteristics, such as human body outlines and heating devices, are assigned a fusion weight of 1- Generate a fused image.

[0189] The fused image retains the detailed textures of visible light images, such as screw holes in equipment and brick seams in walls, while also including thermal target information from infrared images, such as the clear outlines of pedestrians at night. This effectively supports the dual recognition requirements of static environment and dynamic targets in security monitoring.

[0190] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A registration optimization method for infrared-visible light image fusion, characterized in that, Includes the following steps: Image acquisition and preprocessing: Acquiring visible light video images With infrared video images ,right and Perform data preprocessing and output the preprocessed visible light image. 'Compared with preprocessed infrared images '; Calculate registration parameters: for 'and Perform registration parameter calculations to obtain relative registration parameters, which include the scale parameter s and the rotation parameter. Translation parameters (dx, dy); Image fusion: based on Preprocessed visible light images and Perform transformation alignment, use a fusion algorithm to fuse the aligned image, and output the fused image. The fusion algorithm includes a grayscale fusion algorithm or a pseudo-color fusion algorithm. The calculation of the relative registration parameters adopts a method combining initial registration and real-time registration, specifically including: Initial registration: Using a multi-scale feature descriptor extraction algorithm, from... Extracting feature points and corresponding high-dimensional descriptors , ;calculate and similarity distance ,based on Feature matching pairs are weighted for partitioning The weighted random sampling consensus algorithm is adopted, according to... Adjusting the sampling probability k, s is obtained through iterative calculation. ,Right now: ； in For weighted sampling subsets, For matching pairs in The following error; Real-time registration: combining scale parameter s with Solve for the preprocessed visible light image 'Compared with preprocessed infrared images The maximum mutual information (dx, dy) is: ； in, For mutual information functions, For scale-rotation transformation function, This is for transforming compound operations.

2. The registration optimization method for infrared and visible light image fusion according to claim 1, characterized in that, Multi-scale feature descriptor extraction algorithms were used to extract features from the preprocessed visible light images. Compared with preprocessed infrared images Extracting feature points This includes the following steps: Using a pre-defined convolution kernel, for Perform multi-scale convolution operations separately, and combine them with non-linear activation functions to generate feature maps at least two scales, i.e.: ； in, K represents the preset number of scales. , For the first The scale is preset for the convolution kernel size; * indicates the convolution operation. For the preset nonlinear activation function, For the l-th scale bias term; Pixel feature response values ​​are calculated based on multi-scale feature maps. Candidate feature points are selected according to a preset threshold, and effective feature points are obtained by performing neighborhood non-maximum suppression. ,Right now: ； in, For pixels Characteristic response value, For multi-scale eigenvalue maximization calculation, This is for calculating the mean of multi-scale eigenvalues. Filter out ( Candidate feature points (based on a preset response value threshold) are output after non-maximum suppression. Effective feature point set Effective feature point set .

3. The registration optimization method for infrared and visible light image fusion according to claim 2, characterized in that, The generation and effective feature points Corresponding high-dimensional descriptor This includes the following steps: by Each feature point Centered on the feature map at the corresponding scale, extract... (S is the preset size) Local feature area ; Will Expand into a one-dimensional vector, perform L2 normalization to obtain a high-dimensional descriptor, and output it, i.e.: ； in For feature points High-dimensional descriptors For matrix expansion operations, It is an L2 norm.

4. The registration optimization method for infrared and visible light image fusion according to claim 3, characterized in that, Also includes right Arbitrary descriptor and Arbitrary descriptor Calculate the Euclidean distance between the two as the similarity distance. ,Right now: ； Where N is the preset dimension of the high-dimensional descriptor. for The k-th eigenvalue, for The k-th eigenvalue; Will The feature matching pairs are weighted according to the comparison results after being compared with a preset distance threshold. .

5. The registration optimization method for infrared and visible light image fusion according to claim 4, characterized in that, It also includes similarity distance. Weighting is performed on the feature matching pairs, i.e.: Set at least two preset distance thresholds Based on the similarity distance The comparison results with the threshold are used to assign corresponding weights to each visible light feature point-infrared feature point matching pair. : ； High weight; For medium weight and preset weight values, The high confidence threshold; The thresholds are set as medium confidence level and preset distance threshold; Calculate normalized weights , ； in, For the non-zero weight value, .

6. The registration optimization method for infrared-visible light image fusion according to claim 5, characterized in that, Based on normalized weights By using random number generation and interval matching methods, a preset number of matching points are selected from non-zero weight matching pairs to construct a sampling subset. The preset number is not less than the minimum number of points required to solve the affine transformation model; Iterative calculation yields s and ,include: Based on sampling subsets Calculate the initial scale parameters Initial rotation parameters And count all feature matching pairs in , Reprojection error ; Repeat the above loop, with the number of iterations K dynamically calculated: ； in, To set a constant for the reliability, To match the estimated proportion of outliers in the sampled subset, n is the minimum number of matching points required to construct the sampled subset. This is the weighting adjustment factor. It is 0.6~0.8; Until the model parameters with the largest number of interior points and an error less than a preset threshold are found, the optimal scaling parameter s and rotation parameter are output. ,Right now: 。 7. The registration optimization method for infrared and visible light image fusion according to claim 6, characterized in that, The translation parameters are solved using the maximum mutual information algorithm. ,include: Based on the temporal continuity of the preprocessed visible light or infrared images, global motion vectors are calculated through inter-frame feature tracking. Based on the global motion vector Fixed translation parameters The search range is determined, and the goal is to find the range that makes the search range more comprehensive. 'and The (dx, dy) pairs with the largest mutual information include: Extract stable feature points from the previous frame of the preprocessed image. By tracking inter-frame features, the correspondence between valid feature points in the current frame and the previous frame is determined, thus obtaining feature point pairs. ; Based on the coordinate difference of the corresponding feature points, calculate the local displacement vector of each matching pair: ； in, The coordinates of the feature points in the previous frame. Match the coordinates of the point in the current frame; Abnormal displacement vectors are filtered based on statistical characteristics, and the global motion vector is calculated using a statistical aggregation method for the valid displacement vectors. : ； ； The weights are based on the stability of feature points; and Mutual information functions include: ； in, , The normalized grayscale distribution of images A and B. It is a joint distribution.

8. The registration optimization method for infrared-visible image fusion according to any one of claims 1-7, characterized in that, The preprocessed image and ,include: based on right and Perform transformation alignment. ； in, For aligned visible light images, used for s and Achieve pixel-level alignment.

9. The registration optimization method for infrared and visible light image fusion according to claim 7, characterized in that, Timing consistency verification, real-time monitoring of registration errors and dynamic adjustment, including: Collect the registration error between the current frame and the previous 3 frames. And calculate the mean error. ； Calculate the standard deviation of the error. The formula is: ； when Maintain current registration parameters; when Expand the search range for real-time registration, enabling Solve again This ensures that the error regression is within a stable range of ≤1.0 pixels.

10. A registration system for infrared-visible light image fusion, characterized in that, The system employs the infrared / visible image fusion registration optimization method as described in any one of claims 1-9, including: Preprocessing module: used for acquiring visible light video images With infrared video images ,right and Perform data preprocessing and output the preprocessed visible light image. 'Compared with preprocessed infrared images '; Registration parameter module: used for... 'and Perform registration parameter calculations to obtain relative registration parameters, which include the scale parameter s and the rotation parameter. Translation parameters (dx, dy); Image fusion module: used for image fusion based on Preprocessed visible light images and Perform transformation alignment, use a fusion algorithm to fuse the aligned image, and output the fused image. The fusion algorithm includes a grayscale fusion algorithm or a pseudo-color fusion algorithm. The registration parameter module further includes: Initial registration module: Used to extract features from the preprocessed visible light image using a multi-scale feature descriptor extraction algorithm. Compared with preprocessed infrared images Extracting feature points and corresponding high-dimensional descriptors , ;calculate and similarity distance ,based on Feature matching pairs are weighted for partitioning The weighted random sampling consensus algorithm is adopted, according to... Adjusting the sampling probability k, s is obtained through iterative calculation. , ； in For weighted sampling subsets, For matching pairs in The following error; Real-time registration module: used to combine scale parameter s with Solve for the preprocessed visible light image 'Compared with preprocessed infrared images The maximum mutual information (dx, dy) is: ； in, For mutual information functions, For scale-rotation transformation function, This is for transforming compound operations.

Images