A method, system, device and storage medium for bifocal camera image registration and fusion based on binocular ranging

Abstract

The present invention discloses a method, system, device, and storage medium for bifocal camera image registration and fusion based on binocular ranging, which is applied to the field of image processing technology. The method comprises: performing distortion correction on the bifocal image to obtain the intrinsic and extrinsic parameters of the bifocal camera, and performing stereo correction and mapping transformation on the bifocal image based on the intrinsic and extrinsic parameters; performing parallax estimation on the corrected image using the SGBM stereo matching method, calculating the distance from the target to the bifocal camera based on the value within the center range of the image, extracting the corresponding preliminary registration parameters and affine transformation homography matrix based on the registration parameter index table for image registration; performing pixel value ratio conversion on the bifocal image after image registration, extracting high-frequency and low-frequency information of the infrared image, and fusing the infrared image with the visible light image based on the high-frequency and low-frequency information. The present invention effectively improves the real-time and practicality of image registration and fusion, and further improves the accuracy of image registration and fusion.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, system, device and storage medium for bifocal camera image registration and fusion based on binocular ranging. Background Art

[0002] With increasing demand for information, many scenarios are no longer satisfied with the image information provided by single-source cameras. The development and application of multi-source cameras has become mainstream. Multi-source cameras can analyze a specific scene from multiple dimensions. Visible light and infrared dual-light cameras are one such example. Infrared cameras rely on differences in target thermal radiation for detection. They are unaffected by light intensity and can effectively detect and identify objects in all light intensity levels. However, infrared cameras lack image resolution and color saturation, so they need to be combined with visible light cameras to fully capture scene information.

[0003] In recent years, with the rapid development of optoelectronic technology, bi-optic cameras have been widely used in various fields, such as power inspection, agricultural monitoring, security surveillance, and environmental monitoring. In these fields, bi-optic cameras can capture multiple types of information simultaneously, avoiding the tedious process of multiple captures and processing required by a single imaging system, thereby improving work efficiency and reducing labor and time costs. However, the differences in the wavelengths of infrared and visible light, as well as the hardware properties of the cameras used, lead to differences in the imaging targets, increasing the difficulty of image registration and fusion of infrared and visible light images.

[0004] Currently, there are two approaches to image registration and fusion: traditional and deep learning. Traditional image registration and fusion methods process images through geometric operations. These methods require few resources, offer high computational efficiency, and offer strong real-time performance, making them ideal for resource-constrained devices. Deep learning methods, on the other hand, rely on network-based self-learning for registration and fusion, eliminating the need for human intervention and offering excellent scalability and adaptability. However, they require significant resources and cannot achieve real-time performance on embedded devices.

[0005] Currently, the related invention patents include: 1. Application (Patent) Number: CN202410759303.5, Patent Name: Image Registration and Fusion Method, Device, Equipment, and Medium. This patent uses a trained registration model to perform secondary registration on the preliminarily registered infrared image and visible light image, and a fusion model to fuse the secondary registered infrared image and visible light image. Although this method simplifies the registration process and reduces the difficulty of the registration task, it uses a deep learning model for registration and fusion, and cannot achieve real-time performance on low-computing terminal devices. 2. Application (Patent) Number: CN202311029104.0, Patent Name: A Method for Infrared-Visible Heterogeneous Image Registration for Photovoltaic Defect Detection. This patent removes redundant feature points, classifies and matches the feature points, and then obtains a spatial transformation affine transformation homography matrix to align infrared and visible light images. This method first uses the trained Mask R-CNN to segment the image to obtain a background image, then uses the SIFT algorithm to extract features from the background image, then uses a combination of PSO (particle swarm optimization) and K-means algorithms for feature screening, and finally uses the RANSAC method for feature point matching and registration to calculate the affine transformation homography matrix to align the images. The calculation process is complex and time-consuming, and it cannot be run on terminal devices. 3. Application (patent) number: CN202310792919.8, patent name: A method, device and registration method for infrared and visible light image fusion. This method is to pre-calibrate and register the infrared image and the visible light image by affine transformation at different distances to obtain the affine matrices at different distances. Then, a laser ranging module is used to measure the distance from the target to the lens to select different affine matrices, thereby achieving registration and fusion of the infrared image and the visible light image. It has a simple and efficient function and is very suitable for low-computing-power, low-cost terminals. However, the laser ranging module it uses has a small ranging range, is not effective for targets at a slightly longer distance, and lacks robustness. In addition, the use of a long-distance laser ranging module will lead to uncontrollable costs.

[0006] From the above, it can be concluded that the existing image registration and fusion technology of bifocal camera images has high computational complexity and requires a large amount of computing resources during image registration and fusion, which affects the real-time and practicality of the algorithm. In addition, the fusion effect is not ideal, and there are shortcomings such as blurred edges, loss of details, and desaturated texture and color.

[0007] Therefore, how to provide a method, system, device and storage medium for bifocal camera image registration and fusion based on binocular ranging that can effectively reduce the computational complexity in the image registration and fusion process of bifocal camera images, reduce the large amount of computing resources required for image registration and fusion, improve the real-time and practicality of image registration and fusion, and further improve the accuracy of image registration and fusion is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0008] In view of this, the present invention proposes a method, system, device and storage medium for bifocal camera image registration and fusion based on binocular ranging.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] A bifocal camera image registration and fusion method based on binocular ranging includes:

[0011] Step 1: Perform distortion correction on the bifocal image to obtain the intrinsic and extrinsic parameters of the bifocal camera. Based on the intrinsic and extrinsic parameters, perform stereo correction and mapping transformation on the bifocal image.

[0012] Step 2: Use the SGBM stereo matching method to calculate the disparity of the corrected image obtained in step 1, and calculate the distance from the target to the bifocal camera based on the value in the center range of the image. Extract the corresponding preliminary registration parameters and affine transformation homography matrix according to the registration parameter index table for image registration; the registration parameter index table is constructed based on the manual preliminary registration and the affine transformation homography matrix calculated by matching feature points of the preliminary registration image extracted based on LightGlue;

[0013] Step 3: Calculate the pixel value ratio of the registered bi-optical image to extract the high-frequency and low-frequency information of the infrared image, and fuse the infrared image and the visible light image based on the high-frequency and low-frequency information.

[0014] Optionally, in step 1, the bi-optical image is calibrated and distortion corrected using an infrared checkerboard calibration plate.

[0015] Optionally, in step 1, based on intrinsic and extrinsic parameters, stereo rectification and mapping transformation are performed on the bi-focal image using the stereoRectify function, the initUndistortRectifyMap function, and the remap function of OpenCV.

[0016] Optionally, in step 2, the distance from the target to the bifocal camera is calculated based on the value within the center range of the image, specifically:

[0017] ;in, is the center point within the center range; is the width within the center range; is the highest within the central range;

[0018] Average the values in the center range is the distance from the target to the bifocal camera.

[0019] Optionally, in step 3, high-frequency information and low-frequency information of the infrared image are extracted, specifically:

[0020] Scale the pixel values of the infrared image so that the pixel values are within The high-frequency information and low-frequency information are extracted respectively with 0.5 as the boundary.

[0021] Optionally, in step 3, the infrared image and the visible light image are fused based on the high-frequency information and the low-frequency information, specifically as follows:

[0022] For high-frequency information, give weight , for low-frequency information, give weight , ;

[0023] The high-frequency information and the low-frequency information are combined based on the weight to obtain a fusion coefficient. Each pixel in the visible light image is multiplied by the fusion coefficient to obtain a fused image of the infrared image and the visible light image.

[0024] Optionally, high-frequency information and low-frequency information are combined based on weights to obtain a fusion coefficient as follows:

[0025]

[0026] in, is the fusion coefficient; For high-frequency information; is low-frequency information; is the weight.

[0027] The present invention also provides a binocular ranging-based bifocal camera image registration and fusion system using a binocular ranging-based bifocal camera image registration and fusion method, comprising:

[0028] Bi-optical image preprocessing module: used to perform distortion correction on bi-optical images, obtain the intrinsic and extrinsic parameters of the bi-optical camera, and perform stereo correction and mapping transformation on the bi-optical images based on the intrinsic and extrinsic parameters;

[0029] Bifocal image registration module: This module uses the SGBM stereo matching method to perform parallax estimation on the corrected image obtained in step 1, calculates the distance from the target to the bifocal camera based on the values within the center range of the image, and extracts the corresponding preliminary registration parameters and affine transformation homography matrix from the registration parameter index table for image registration. The registration parameter index table is constructed based on the manual preliminary registration and the affine transformation homography matrix calculated using the matching feature points of the preliminary registration image extracted using LightGlue.

[0030] Bi-optical image fusion module: used to convert the pixel value ratio of the bi-optical image after image registration, extract the high-frequency information and low-frequency information of the infrared image, and fuse the infrared image and visible light image based on the high-frequency information and low-frequency information.

[0031] The present invention further provides an electronic device, comprising:

[0032] memory for storing computer programs;

[0033] The processor is used to implement the steps of a bifocal camera image registration and fusion method based on binocular ranging when executing a computer program.

[0034] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the steps of a method for image registration and fusion of bifocal cameras based on binocular ranging.

[0035] As can be seen from the above technical solutions, compared to the prior art, the present invention proposes a method, system, device, and storage medium for bifocal camera image registration and fusion based on binocular ranging. Distortion correction and stereo calibration are performed on bifocal images, and the distance from the target to the bifocal camera is calculated using the SGBM stereo matching method. Dynamic selection of registration parameters is performed based on a registration parameter index table constructed from manual preliminary registration and an affine transformation homography matrix calculated using LightGlue extraction of matching feature points. This not only improves the real-time performance of image registration and fusion, but also reduces registration complexity, avoiding the computational resources and time required by the use of multi-layer deep learning models or complex feature screening algorithms in the prior art. Furthermore, efficient fusion of the infrared image and the visible light image is achieved based on the extracted high-frequency and low-frequency information of the infrared image, enhancing the contrast and detail of the final output image and enabling users to better obtain the required information. Compared to the prior art, this method has the advantage of being able to more fully utilize multi-source information in image fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0037] Figure 1 Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0039] Example 1:

[0040] Embodiment 1 of the present invention discloses a method for image registration and fusion of bifocal cameras based on binocular ranging, such as Figure 1 As shown, including:

[0041] Step 1: Perform distortion correction on the bifocal image to obtain the intrinsic and extrinsic parameters of the bifocal camera. Based on the intrinsic and extrinsic parameters, perform stereo correction and mapping transformation on the bifocal image.

[0042] Due to lens distortion in visible-light and infrared cameras, the resulting image deviates from the real-world image. Therefore, camera calibration is necessary to eliminate this distortion and achieve clearer and more accurate images. Therefore, an infrared checkerboard calibration plate is used to calibrate and correct bifocal image distortion. First, the distance between the bifocal camera and the calibration plate is fixed so that the checkerboard is centered in the bifocal image and the image is clear. The checkerboard is then rotated, and 20 bifocal images of the checkerboard are collected at different angles. Finally, the collected bifocal images are calibrated and corrected using the OpenCV calibration program to obtain the intrinsic and extrinsic parameters corresponding to the bifocal camera, thereby achieving distortion correction.

[0043] Because bifocal cameras have different focal lengths, fields of view, and resolutions, real-world objects appear at different positions and sizes in bifocal images. Therefore, stereo matching correction is required to ensure consistent resolution and horizontal pixel alignment in bifocal images, which also supports subsequent ranging. Therefore, based on intrinsic and extrinsic parameters, OpenCV's stereoRectify, initUndistortRectifyMap, and remap functions are used to perform stereo correction and mapping on the bifocal images, resulting in a stereo-corrected bifocal image.

[0044] Step 2: Use the SGBM stereo matching method to calculate the disparity of the rectified image obtained in Step 1. This method maintains the quality of the disparity map while reducing computational complexity, making it ideal for embedded devices. After calculating the disparity map, the distance from the target to the bifocal camera is calculated based on the values within the center range of the map. Image registration is then performed by extracting the corresponding preliminary registration parameters and affine transformation homography matrix from the registration parameter index table. The registration parameter index table is constructed based on the affine transformation homography matrix calculated from the manual preliminary registration and the feature points of the preliminary registration images extracted using LightGlue.

[0045] The distance from the target to the bifocal camera is calculated based on the values in the center range of the image. Specifically:

[0046] ;in, is the center point within the center range; is the width within the center range; is the highest within the central range;

[0047] Average the values in the center range is the distance from the target to the bifocal camera.

[0048] The construction process of the registration parameter index table is as follows:

[0049] At 1-meter intervals, a bi-optical camera was used to capture 50 pairs of distortion-corrected infrared checkerboard calibration images. Then, the checkerboard calibration images in the visible and infrared images were manually aligned by scaling, resulting in overlap. The scaling range parameters for the overlapping images were obtained, completing the initial registration. However, this initial registration only ensures that most visible and infrared image targets within the overlapping scaling range are aligned, which can result in ghosting when the images are fused, affecting the image fusion quality. Therefore, a secondary registration is performed on the overlapping images to ensure precise alignment of infrared and visible image targets. LightGlue is a deep learning-based local feature matching method that rapidly matches feature points extracted by SuperPoint, resulting in better correspondences and more accurate relative poses, resulting in more accurate matched feature points. Therefore, for each pair of initially registered visible and infrared overlapping images, LightGlue is used to extract matching feature points and calculate the affine transformation homography matrix. Finally, the distance values, the corresponding preliminary registration parameters and the affine transformation homography matrix are made into a registration parameter index table.

[0050] Step 3: To ensure that the fused image maintains the color and texture information of the visible light image and the edge brightness of the infrared image, the pixel value ratio of the registered bi-optical image is converted to extract the high-frequency and low-frequency information of the infrared image. The infrared image and the visible light image are fused based on the high-frequency and low-frequency information.

[0051] Extract high-frequency information and low-frequency information of infrared images, specifically:

[0052] Scale the pixel values of the infrared image so that the pixel values are within The high-frequency information and low-frequency information are extracted respectively with 0.5 as the boundary.

[0053] The fusion of infrared images and visible light images is performed based on high-frequency information and low-frequency information, specifically:

[0054] For high-frequency information, give weight , for low-frequency information, give weight , ;

[0055] The high-frequency information and the low-frequency information are combined based on the weight to obtain a fusion coefficient. Each pixel in the visible light image is multiplied by the fusion coefficient to obtain a fused image of the infrared image and the visible light image.

[0056] The high-frequency information and low-frequency information are combined based on the weights to obtain the fusion coefficient, as follows:

[0057]

[0058] in, is the fusion coefficient; For high-frequency information; is low-frequency information; is the weight.

[0059] Example 2:

[0060] Embodiment 2 of the present invention discloses a binocular ranging-based bifocal camera image registration and fusion system using a binocular ranging-based bifocal camera image registration and fusion method, comprising:

[0061] Bi-optical image preprocessing module: used to perform distortion correction on bi-optical images, obtain the intrinsic and extrinsic parameters of the bi-optical camera, and perform stereo correction and mapping transformation on the bi-optical images based on the intrinsic and extrinsic parameters;

[0062] Bifocal image registration module: This module uses the SGBM stereo matching method to perform parallax estimation on the corrected image obtained in step 1, calculates the distance from the target to the bifocal camera based on the values within the center range of the image, and extracts the corresponding preliminary registration parameters and affine transformation homography matrix from the registration parameter index table for image registration. The registration parameter index table is constructed based on the manual preliminary registration and the affine transformation homography matrix calculated using the matching feature points of the preliminary registration image extracted using LightGlue.

[0063] Bi-optical image fusion module: used to convert the pixel value ratio of the bi-optical image after image registration, extract the high-frequency information and low-frequency information of the infrared image, and fuse the infrared image and visible light image based on the high-frequency information and low-frequency information.

[0064] Example 3:

[0065] Embodiment 3 of the present invention discloses an electronic device, including:

[0066] memory for storing computer programs;

[0067] The processor is used to implement the steps of a bifocal camera image registration and fusion method based on binocular ranging when executing a computer program.

[0068] Example 4:

[0069] Embodiment 4 of the present invention discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the steps of a method for image registration and fusion of bifocal cameras based on binocular ranging.

[0070] The embodiments of the present invention disclose a method, system, device, and storage medium for bifocal camera image registration and fusion based on binocular ranging. Distortion correction and stereo calibration are performed on bifocal images, and the distance from the target to the bifocal camera is calculated using the SGBM stereo matching method. Registration parameters are dynamically selected based on a registration parameter index table constructed from manual preliminary registration and an affine transformation homography matrix calculated based on LightGlue extraction of matching feature points. This not only improves the real-time performance of image registration and fusion, but also reduces the complexity of registration, avoiding the computational resources and time required by the use of multi-layer deep learning models or complex feature screening algorithms in the prior art. Simultaneously, the infrared image and visible light image are fused based on the extracted high-frequency and low-frequency information of the infrared image, achieving efficient fusion of the visible light and infrared images, enhancing the contrast and detail of the final output image, and enabling users to better obtain the required information. Compared to the prior art, this method has the advantage of being able to more fully utilize multi-source information in image fusion.

[0071] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0072] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A bifocal camera image registration and fusion method based on binocular ranging, characterized in that: include: Step 1: performing distortion correction on the bifocal image to obtain the intrinsic and extrinsic parameters of the bifocal camera, and performing stereo correction and mapping transformation on the bifocal image based on the intrinsic and extrinsic parameters; Step 2: Use the SGBM stereo matching method to perform disparity estimation on the corrected image obtained in step 1, and calculate the distance from the target to the bifocal camera based on the value within the center range of the image. Extract the corresponding preliminary registration parameters and affine transformation homography matrix according to the registration parameter index table for image registration; wherein the registration parameter index table is constructed based on manual preliminary registration and the affine transformation homography matrix calculated by matching feature points of the preliminary registration image extracted based on LightGlue; Step 3: Calculate the pixel value ratio of the registered bi-optical image to extract the high-frequency information and low-frequency information of the infrared image, and fuse the infrared image and the visible light image based on the high-frequency information and low-frequency information; In step 3, the high-frequency information and low-frequency information of the infrared image are extracted, specifically: Scaling the pixel values of the infrared image so that the pixel values are between [0, 1], and extracting high-frequency information and low-frequency information respectively with 0.5 as the boundary; In step 3, the infrared image and the visible light image are fused based on the high-frequency information and the low-frequency information, specifically: For the high-frequency information, a weight θ is given, and for the low-frequency information, a weight 1-θ is given, θ∈[0,1]; Combining the high-frequency information and the low-frequency information based on the weight to obtain a fusion coefficient, and multiplying each pixel in the visible light image by the fusion coefficient to obtain a fused image of the infrared image and the visible light image; The high-frequency information and the low-frequency information are combined based on the weight to obtain a fusion coefficient as follows: k=P*θ+Q*(1-θ)+0.5; Among them, k is the fusion coefficient; P is the high-frequency information; Q is the low-frequency information; θ is the weight.

2. The method for image registration and fusion of a bifocal camera based on binocular ranging according to claim 1, characterized in that: In step 1, the bi-optical image is calibrated and distortion corrected using an infrared checkerboard calibration plate.

3. The method for bifocal camera image registration and fusion based on binocular ranging according to claim 1, characterized in that: In step 1, based on the intrinsic and extrinsic parameters, stereo rectification and mapping transformation are performed on the bifocal image using the stereoRectify function, the initUndistortRectifyMap function, and the remap function of Opencv.

4. The method for bifocal camera image registration and fusion based on binocular ranging according to claim 1, characterized in that: In step 2, the distance from the target to the bifocal camera is calculated based on the value in the center range of the image, specifically: {(x,y),(w,h)}; (x,y) is the center point of the center range; w is the width of the center range; h is the height of the center range; The average value mean of the values within the center range is taken as the distance from the target to the bifocal camera.

5. A binocular ranging-based bifocal camera image registration and fusion system using the binocular ranging-based bifocal camera image registration and fusion method according to any one of claims 1 to 4, characterized in that: include: Bifocal image preprocessing module: used to perform distortion correction on the bifocal image, obtain the intrinsic and extrinsic parameters of the bifocal camera, and perform stereo correction and mapping transformation on the bifocal image based on the intrinsic and extrinsic parameters; Bifocal image registration module: This module is used to calculate the disparity of the corrected image obtained in step 1 using the SGBM stereo matching method, calculate the distance between the target and the bifocal camera based on the value within the center range of the image, and extract the corresponding preliminary registration parameters and affine transformation homography matrix from the registration parameter index table for image registration. The registration parameter index table is constructed based on the manual preliminary registration and the affine transformation homography matrix calculated using the preliminary registration image matching feature points extracted using LightGlue. Bi-optical image fusion module: used to convert the pixel value ratio of the bi-optical image after image registration, extract the high-frequency information and low-frequency information of the infrared image, and fuse the infrared image and the visible light image based on the high-frequency information and low-frequency information.

6. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of a method for bifocal camera image registration and fusion based on binocular ranging as described in any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the method for bifocal camera image registration and fusion based on binocular ranging are implemented as described in any one of claims 1 to 4.

Images