Binocular distance measurement-based dual-light camera image registration fusion method, system and device, and storage medium

Abstract

The invention discloses a dual-light camera image registration fusion method, system and device based on binocular distance measurement and a storage medium, which are applied to the technical field of image processing, and the method comprises the steps: carrying out the distortion correction of a dual-light image, obtaining the internal and external parameters of a dual-light camera, and carrying out the stereo correction and mapping transformation of the dual-light image based on the internal and external parameters; performing parallax estimation calculation on the corrected image by using an SGBM stereo matching method, calculating a distance from a target to the dual-light camera according to a value in a center range in the image, and extracting corresponding preliminary registration parameters and an affine transformation homography matrix according to a registration parameter index table to perform image registration; and performing pixel value proportion conversion on the dual-light image after image registration, extracting high-frequency information and low-frequency information of the infrared image, and performing fusion of the infrared image and the visible light image based on the high-frequency information and the low-frequency information. According to the method, the real-time performance and practicability of image registration fusion are effectively improved, and the image registration fusion precision is further improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, system, device and storage medium for bifocal camera image registration and fusion based on binocular ranging. Background Art

[0002] With the increase in information demand, many scenes are no longer satisfied with the image information provided by single-source cameras, and the development and application of multi-source cameras have become mainstream. Multi-source cameras can analyze the same specific scene from multiple dimensions, and visible light and infrared dual-light cameras are one of them. Infrared cameras rely on the difference in target thermal radiation for detection and are not affected by the intensity of light. They can achieve effective detection and recognition in scenes of any light intensity. However, the image resolution and color level of infrared cameras are insufficient, so they need to be combined with visible light cameras to fully capture scene information.

[0003] In recent years, with the rapid development of optoelectronic technology, dual-light cameras have been widely used in many fields, such as power inspection, agricultural monitoring, security monitoring, environmental monitoring, etc. In these fields, dual-light cameras can obtain multiple types of information at one time, avoiding the tedious process of multiple shooting and processing required by a single imaging system, thereby improving work efficiency and reducing manpower and time costs. However, the differences in the wavelength bands of infrared and visible light and the hardware properties of the two cameras lead to differences between imaging targets, increasing the difficulty of image registration and fusion of infrared images and visible light images.

[0004] At present, there are two methods for image registration and fusion: traditional and deep learning methods. Traditional image registration and fusion methods process images through geometric operations, which require few resources, have high computational efficiency, and strong real-time performance, making them very suitable for use on resource-constrained terminals. Deep learning methods, on the other hand, use network self-learning to perform registration and fusion, without manual intervention, and have good scalability and adaptability, but they require large resources and cannot achieve real-time performance on embedded terminals.

[0005] At present, the invention patents related to it are: 1. Application (patent) number: CN202410759303.5, patent name: image registration and fusion method, device, equipment and medium. This patent uses a trained registration model to perform secondary registration on the infrared image and visible light image that have been preliminarily registered, and a fusion model to fuse the infrared image and visible light image that have been secondary registered. Although this method simplifies the registration process and reduces the difficulty of the registration task, it uses a deep learning model for registration and fusion, and cannot achieve real-time performance on low-computing terminal devices. 2. Application (patent) number: CN202311029104.0, patent name: A method for infrared-visible light heterogeneous image registration for photovoltaic defect detection. This patent removes redundant feature points, classifies and matches feature points, and then obtains a spatial transformation affine transformation homography matrix to register infrared-visible light images. This method first uses the trained Mask R-CNN to segment the image to obtain the background image, then uses the SIFT algorithm to extract features from the background image, then uses the PSO (particle swarm) and K-means algorithms to perform feature screening, and finally uses the RANSAC method to match feature points and calculate the affine transformation homography matrix to align the image. The calculation process is complicated and time-consuming, and it cannot be run on the terminal device. 3. Application (patent) number: CN202310792919.8, patent name: A method, device and registration method for infrared and visible light image fusion. This method is to perform affine transformation calibration and registration on infrared images and visible light images at different distances in advance to obtain affine matrices at different distances, and then use a laser ranging module to measure the distance from the target to the lens to select different affine matrices, thereby realizing the registration and fusion of infrared images and visible light images. It has a simple and efficient function and is very suitable for low-computing-power, low-cost terminals. However, the laser ranging module it uses has a small ranging range, is not effective for targets at a slightly longer distance, and lacks robustness. In addition, the use of a long-distance laser ranging module will result in uncontrollable costs.

[0006] From the above, it can be concluded that the existing image registration and fusion technology of bi-focal camera images has high computational complexity and requires a large amount of computing resources during image registration and fusion, which affects the real-time and practicality of the algorithm. In addition, the fusion effect is not ideal, and there are shortcomings such as blurred edges, lost details, and unsaturated texture colors.

[0007] Therefore, how to provide a method, system, device and storage medium for bi-focal camera image registration and fusion based on binocular ranging that can effectively reduce the computational complexity in the image registration and fusion process of bi-focal camera images, reduce the large amount of computing resources required for image registration and fusion, improve the real-time and practicality of image registration and fusion, and further improve the image registration and fusion accuracy is a problem that technicians in this field urgently need to solve. Summary of the invention

[0008] In view of this, the present invention proposes a method, system, device and storage medium for bi-optical camera image registration and fusion based on binocular ranging.

[0009] In order to achieve the above object, the present invention adopts the following technical solution: A bifocal camera image registration and fusion method based on binocular ranging, comprising: Step 1: Perform distortion correction on the bi-focal image to obtain the intrinsic and extrinsic parameters of the bi-focal camera, and perform stereo correction and mapping transformation on the bi-focal image based on the intrinsic and extrinsic parameters; Step 2: Use the SGBM stereo matching method to calculate the disparity of the corrected image obtained in step 1, and calculate the distance from the target to the bifocal camera based on the value in the center range of the image. Extract the corresponding preliminary registration parameters and affine transformation homography matrix according to the registration parameter index table for image registration; the registration parameter index table is constructed based on manual preliminary registration and the affine transformation homography matrix calculated by matching feature points of preliminary registration images extracted based on LightGlue; Step 3: Calculate the pixel value ratio of the registered bi-optical image, extract the high-frequency information and low-frequency information of the infrared image, and fuse the infrared image and the visible light image based on the high-frequency information and the low-frequency information.

[0010] Optionally, in step 1, an infrared checkerboard calibration plate is used to calibrate and correct the bi-optical image for distortion.

[0011] Optionally, in step 1, based on the intrinsic parameters and the extrinsic parameters, stereoRectify function, initUndistortRectifyMap function and remap function of Opencv are used to perform stereo correction and mapping transformation on the bi-focal image.

[0012] Optionally, in step 2, the distance from the target to the bifocal camera is calculated according to the value in the center range of the graph, specifically: ;in, is the center point within the center range; is the width within the center range; is the height within the central range; Average the values ​​in the center range is the distance from the target to the bifocal camera.

[0013] Optionally, in step 3, high-frequency information and low-frequency information of the infrared image are extracted, specifically: The pixel values ​​of the infrared image are scaled so that the pixel values ​​are within The high-frequency information and low-frequency information are extracted respectively with 0.5 as the boundary.

[0014] Optionally, in step 3, the infrared image and the visible light image are fused based on the high-frequency information and the low-frequency information, specifically: For high-frequency information, give weight , for low-frequency information, give weight , ; The high-frequency information and the low-frequency information are combined based on the weight to obtain a fusion coefficient, and each pixel in the visible light image is multiplied by the fusion coefficient to obtain a fused image of the infrared image and the visible light image.

[0015] Optionally, high-frequency information and low-frequency information are combined based on weights to obtain a fusion coefficient, as follows: in, is the fusion coefficient; For high-frequency information; It is low frequency information; is the weight.

[0016] The present invention also provides a binocular ranging-based bifocal camera image registration and fusion system using a binocular ranging-based bifocal camera image registration and fusion method, comprising: Bi-optical image preprocessing module: used to perform distortion correction on the bi-optical image, obtain the internal and external parameters of the bi-optical camera, and perform stereo correction and mapping transformation on the bi-optical image based on the internal and external parameters; Bifocal image registration module: used to calculate the disparity estimation of the corrected image obtained in step 1 using the SGBM stereo matching method, and calculate the distance from the target to the bifocal camera according to the value in the center range of the image, and extract the corresponding preliminary registration parameters and affine transformation homography matrix according to the registration parameter index table for image registration; wherein the registration parameter index table is constructed based on the manual preliminary registration and the affine transformation homography matrix calculated by matching feature points of the preliminary registration image extracted based on LightGlue; Bi-optical image fusion module: used to convert the pixel value ratio of the bi-optical image after image registration, extract the high-frequency information and low-frequency information of the infrared image, and fuse the infrared image and the visible light image based on the high-frequency information and low-frequency information.

[0017] The present invention also provides an electronic device, comprising: Memory for storing computer programs; The processor is used to implement the steps of a method for image registration and fusion of a bifocal camera based on binocular ranging when executing a computer program.

[0018] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a method for image registration and fusion of a bifocal camera based on binocular ranging are implemented.

[0019] It can be seen from the above technical solutions that, compared with the prior art, the present invention proposes a method, system, device and storage medium for image registration and fusion of a bifocal camera based on binocular ranging. Distortion correction and stereo correction are performed through bifocal images, and the distance from the target to the bifocal camera is calculated by the SGBM stereo matching method. The registration parameter index table is constructed based on the manual preliminary registration and the affine transformation homography matrix calculated based on the LightGlue extraction of matching feature points, and the registration parameters are dynamically selected, which not only improves the real-time performance of image registration and fusion, but also reduces the complexity of registration, and avoids the computing resources and time required for the use of multi-layer deep learning models or complex feature screening algorithms in the prior art. At the same time, the infrared image and the visible light image are fused based on the extracted high-frequency information and low-frequency information of the infrared image, which also realizes the efficient fusion of visible light and infrared images, enhances the contrast and details of the final output image, and enables users to better obtain the required information. Compared with the prior art, it has the advantage of being able to make better use of multi-source information in image fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0021] Figure 1 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Embodiment 1: Embodiment 1 of the present invention discloses a dual-light camera image registration and fusion method based on binocular ranging, such as Figure 1 As shown, including: Step 1: Perform distortion correction on the bi-focal image to obtain the intrinsic and extrinsic parameters of the bi-focal camera, and perform stereo correction and mapping transformation on the bi-focal image based on the intrinsic and extrinsic parameters.

[0024] Due to the lens problems of visible light cameras and infrared cameras, lens distortion will occur during the imaging process, and the presented images will deviate from the images in the real world. Therefore, the camera needs to be calibrated to eliminate the distortion and make the image clearer and more accurate. Therefore, the infrared chessboard calibration plate is used to calibrate the distortion correction of the bi-light image. First, the distance between the bi-light camera and the calibration plate is fixed so that the chessboard is in the center of the bi-light image and the image is clear; then the chessboard is rotated to collect 20 bi-light images of the chessboard at different angles. Finally, the Opencv calibration program is used to calibrate and correct the collected bi-light images to obtain the internal and external parameters corresponding to the bi-light camera, thereby achieving the purpose of distortion correction.

[0025] Since bifocal cameras have different focal lengths, field of view angles, and resolutions, the positions and sizes of real-world targets in bifocal images will be different. Therefore, it is necessary to perform stereo matching correction on the bifocal camera so that the bifocal image can maintain consistent resolution and horizontal pixel alignment, and also provide support for subsequent ranging. Therefore, based on internal and external parameters, the stereoRectify function, initUndistortRectifyMap function, and remap function of Opencv are used to perform stereo correction and mapping transformation on the bifocal image, thereby obtaining a stereo corrected bifocal image.

[0026] Step 2: Use the SGBM stereo matching method to calculate the disparity estimation of the corrected image obtained in step 1. This method ensures the quality of the disparity map while reducing the complexity of the calculation, which is very suitable for embedded terminal devices. After calculating the disparity map, calculate the distance from the target to the bifocal camera according to the value in the center range of the map, and extract the corresponding preliminary registration parameters and affine transformation homography matrix according to the registration parameter index table for image registration; where the registration parameter index table is constructed based on the manual preliminary registration and the affine transformation homography matrix calculated by matching feature points of the preliminary registration image extracted based on LightGlue.

[0027] The distance from the target to the dual-light camera is calculated based on the value in the center range of the image, specifically: ;in, is the center point within the center range; is the width within the center range; is the height within the central range; Average the values ​​in the center range is the distance from the target to the bifocal camera.

[0028] The construction process of the registration parameter index table is as follows: At a distance of 1 meter, a dual-light camera is used to collect the image data of the infrared checkerboard calibration plate after distortion correction, and a total of 50 pairs are collected; then, through the zoom operation, the checkerboard calibration plate in the visible light image and the infrared image is manually aligned to make the checkerboard images overlap, and the zoom range parameters of the overlapping image are obtained, that is, the preliminary registration is completed. However, the preliminary registration can only ensure that most of the visible light image targets and infrared image targets within the image zoom overlap range can be registered and aligned, which will cause ghosting when the image is fused, affecting the image fusion effect. To this end, the overlapping images are secondary registered and aligned so that the infrared image targets can be accurately registered and aligned with the visible light image targets. LightGlue is a local feature matching method based on deep learning. It quickly matches the feature points extracted by SuperPoint, which can produce better correspondence and more accurate relative poses, and the matched feature points are more accurate. Therefore, for each pair of overlapping visible light and infrared images that are initially registered, LightGlue is used to extract the matching feature points and calculate the affine transformation homography matrix. Finally, the distance values, the corresponding preliminary registration parameters and the affine transformation homography matrix are made into a registration parameter index table.

[0029] Step 3: In order to ensure that the fused image maintains the color and texture information of the visible light image and the edge brightness of the infrared image, the pixel value ratio of the bi-optical image after image registration is converted to extract the high-frequency information and low-frequency information of the infrared image, and the infrared image and the visible light image are fused based on the high-frequency information and low-frequency information.

[0030] Extract high-frequency information and low-frequency information of infrared images, specifically: The pixel values ​​of the infrared image are scaled so that the pixel values ​​are within The high-frequency information and low-frequency information are extracted respectively with 0.5 as the boundary.

[0031] The infrared image and the visible light image are fused based on the high-frequency information and the low-frequency information, specifically: For high-frequency information, give weight , for low-frequency information, give weight , ; The high-frequency information and the low-frequency information are combined based on the weight to obtain a fusion coefficient, and each pixel in the visible light image is multiplied by the fusion coefficient to obtain a fused image of the infrared image and the visible light image.

[0032] The high-frequency information and low-frequency information are combined based on the weights to obtain the fusion coefficient, as follows: in, is the fusion coefficient; For high-frequency information; It is low frequency information; is the weight.

[0033] Embodiment 2: Embodiment 2 of the present invention discloses a binocular ranging-based bifocal camera image registration and fusion system using a binocular ranging-based bifocal camera image registration and fusion method, comprising: Bi-optical image preprocessing module: used to perform distortion correction on the bi-optical image, obtain the internal and external parameters of the bi-optical camera, and perform stereo correction and mapping transformation on the bi-optical image based on the internal and external parameters; Bifocal image registration module: used to calculate the disparity estimation of the corrected image obtained in step 1 using the SGBM stereo matching method, and calculate the distance from the target to the bifocal camera according to the value in the center range of the image, and extract the corresponding preliminary registration parameters and affine transformation homography matrix according to the registration parameter index table for image registration; wherein the registration parameter index table is constructed based on the manual preliminary registration and the affine transformation homography matrix calculated by matching feature points of the preliminary registration image extracted based on LightGlue; Bi-optical image fusion module: used to convert the pixel value ratio of the bi-optical image after image registration, extract the high-frequency information and low-frequency information of the infrared image, and fuse the infrared image and the visible light image based on the high-frequency information and low-frequency information.

[0034] Embodiment 3: Embodiment 3 of the present invention discloses an electronic device, including: Memory for storing computer programs; The processor is used to implement the steps of a method for image registration and fusion of a bifocal camera based on binocular ranging when executing a computer program.

[0035] Embodiment 4: Embodiment 4 of the present invention discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a method for image registration and fusion of a bifocal camera based on binocular ranging are implemented.

[0036] The embodiment of the present invention discloses a method, system, device and storage medium for image registration and fusion of a bifocal camera based on binocular ranging. Distortion correction and stereo correction are performed through bifocal images, and the distance from the target to the bifocal camera is calculated through the SGBM stereo matching method. The registration parameter index table is constructed based on the manual preliminary registration and the affine transformation homography matrix calculated based on the LightGlue extraction of matching feature points, and the registration parameters are dynamically selected, which not only improves the real-time performance of image registration and fusion, but also reduces the complexity of registration, and avoids the computing resources and time required for using multi-layer deep learning models or complex feature screening algorithms in the prior art. At the same time, the infrared image and the visible light image are fused based on the extracted high-frequency information and low-frequency information of the infrared image, which also realizes the efficient fusion of visible light and infrared images, enhances the contrast and details of the final output image, and enables users to better obtain the required information. Compared with the prior art, it has the advantage of being able to make better use of multi-source information in image fusion.

[0037] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0038] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A dual-light camera image registration and fusion method based on binocular ranging, characterized in that: include: Step 1: performing distortion correction on the bi-focal image to obtain the internal and external parameters of the bi-focal camera, and performing stereo correction and mapping transformation on the bi-focal image based on the internal and external parameters; Step 2: Use the SGBM stereo matching method to perform parallax estimation on the corrected image obtained in step 1, and calculate the distance from the target to the bifocal camera according to the value in the center range of the image, and extract the corresponding preliminary registration parameters and affine transformation homography matrix according to the registration parameter index table for image registration; wherein the registration parameter index table is constructed based on manual preliminary registration and the affine transformation homography matrix calculated by matching feature points of preliminary registration images extracted based on LightGlue; Step 3: Calculate the pixel value ratio of the registered bi-optical image, extract the high-frequency information and low-frequency information of the infrared image, and fuse the infrared image and the visible light image based on the high-frequency information and the low-frequency information.

2. The method for image registration and fusion of a bifocal camera based on binocular ranging according to claim 1, characterized in that: In step 1, the bi-optical image is calibrated and distortion corrected using an infrared checkerboard calibration plate.

3. The method for image registration and fusion of a bifocal camera based on binocular ranging according to claim 1, characterized in that: In step 1, based on the intrinsic parameters and extrinsic parameters, stereoRectify function, initUndistortRectifyMap function and remap function of Opencv are used to perform stereo correction and mapping transformation on the bi-focal image.

4. The method for image registration and fusion of a bifocal camera based on binocular ranging according to claim 1, characterized in that: In step 2, the distance from the target to the dual-light camera is calculated according to the value in the center range of the figure, specifically: ;in, is the center point within the center range; is the width within the center range; is the height within the center range; Average the values ​​in the center range is the distance from the target to the dual-light camera.

5. The method for image registration and fusion of a bifocal camera based on binocular ranging according to claim 1, characterized in that: In step 3, high-frequency information and low-frequency information of the infrared image are extracted, specifically: Scaling the pixel values ​​of the infrared image so that the pixel values ​​are within The high-frequency information and low-frequency information are extracted respectively with 0.5 as the boundary.

6. The method for image registration and fusion of a bifocal camera based on binocular ranging according to claim 1, characterized in that: In step 3, the infrared image and the visible light image are fused based on the high-frequency information and the low-frequency information, specifically: For the high frequency information, give weight , for the low-frequency information, give weight , ; The high-frequency information and the low-frequency information are combined based on the weight to obtain a fusion coefficient, and each pixel in the visible light image is multiplied by the fusion coefficient to obtain a fused image of the infrared image and the visible light image.

7. The method for image registration and fusion of a bifocal camera based on binocular ranging according to claim 6, characterized in that: The high-frequency information and the low-frequency information are combined based on the weight to obtain a fusion coefficient, as follows: in, is the fusion coefficient; For high-frequency information; It is low frequency information; is the weight.

8. A binocular ranging-based bifocal camera image registration and fusion system using the binocular ranging-based bifocal camera image registration and fusion method according to any one of claims 1 to 7, characterized in that: include: Bi-optical image preprocessing module: used to perform distortion correction on the bi-optical image, obtain the internal and external parameters of the bi-optical camera, and perform stereo correction and mapping transformation on the bi-optical image based on the internal and external parameters; Bifocal image registration module: used to perform parallax estimation calculation on the corrected image obtained in step 1 using the SGBM stereo matching method, and calculate the distance from the target to the bifocal camera according to the value in the center range of the image, and extract the corresponding preliminary registration parameters and affine transformation homography matrix according to the registration parameter index table for image registration; wherein the registration parameter index table is constructed based on manual preliminary registration and the affine transformation homography matrix calculated by matching feature points of preliminary registration images extracted based on LightGlue; Bi-optical image fusion module: used to convert the pixel value ratio of the bi-optical image after image registration, extract the high-frequency information and low-frequency information of the infrared image, and fuse the infrared image and the visible light image based on the high-frequency information and the low-frequency information.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of a method for image registration and fusion of a bifocal camera based on binocular ranging as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the method for image registration and fusion of bifocal cameras based on binocular ranging are implemented as described in any one of claims 1 to 7.

Images