A panoramic image stitching system and method for weak light enhancement and color temperature correction

Abstract

The application provides a panoramic image splicing system and method for weak light enhancement and color temperature correction. The system mainly comprises a multi-thread control module for overall controlling the execution sequence of each module; a video reading and image correction module responsible for reading four camera images and performing image correction operations; a parameter initialization module for processing images, performing key point detection and matching, and calculating the rotation matrix between images and estimating camera parameters; an image projection module for projecting each image onto a cylindrical surface according to the rotation matrix and camera parameters; an optimal seam line finding module for calculating the optimal seam line of each overlapping area; an image fusion module for fusing images according to the optimal seam line and a multi-band fusion method to obtain a panoramic image; and a panoramic image display module responsible for displaying the panoramic image. The system can realize efficient and accurate panoramic image splicing and effectively handle weak light and color temperature problems to improve image perception.

Description

Technical Field

[0001] This invention belongs to the field of image processing, and more specifically relates to a panoramic image stitching system and method for low-light enhancement and color temperature correction. Background Technology

[0002] In recent years, with the development of high technology, robotics has made rapid progress, finding widespread application not only in military and national defense, medical treatment, and industrial development, but also gradually entering daily life. Mobile robots, as an important member of intelligent robots, have the greatest advantage of being able to move freely and possess corresponding intelligent response capabilities; common examples include industrial AGVs, drones, humanoid robots, and robotic vacuum cleaners. However, in some application scenarios, the field of view of a single camera is clearly insufficient, while the field of view of multiple cameras appears cluttered. Therefore, a technology is needed to stitch together the fields of view of multiple cameras into a broader field of view.

[0003] Panoramic image stitching technology can acquire panoramic images of the surrounding environment, breaking through the limitations of a single camera's field of view and ensuring both high resolution and a large field of view are simultaneously achieved. Panoramic vision is widely used in various fields. For example, panoramic vision systems in private cars can monitor the surrounding environment to ensure driving safety; panoramic video surveillance in large supermarkets can cover all areas; and panoramic vision systems mounted on mobile robots can be used for special environments such as post-fire reconnaissance. These applications obtain comprehensive environmental information through panoramic vision, which can be used for 3D reconstruction, security monitoring, and environmental detection, among others.

[0004] In certain special environments, such as the confined spaces created by fires, electrical shafts, and elevator shafts, dim lighting and limited space often lead to poor image quality, resulting in problems such as low brightness, inconsistent color temperature, low contrast, and information loss. The varying brightness and color temperature of images captured by different cameras not only affect the subjective visual experience but also limit the effectiveness of panoramic image stitching. Therefore, how to utilize image processing technology to improve the visibility and color consistency of images under low-light conditions, making the stitched panoramic image clearer and more natural, has become an urgent problem to be solved in these special scenarios. Summary of the Invention

[0005] This invention provides a panoramic video stitching method for different color temperatures under low-light conditions. Different functional modules address issues such as image correction, image registration, image projection, and image fusion. Experiments demonstrate that this method obtains clear and natural panoramic images even under low-light conditions with inconsistent color temperatures, exhibiting strong versatility and scalability.

[0006] To achieve the above objectives, the present invention employs the following technical solution: the stitching system includes a multi-threaded control module, a video reading and image correction module, a parameter initialization module, an image projection module, an optimal stitching line finding module, and an image fusion module.

[0007] Panoramic image display module;

[0008] The multi-threaded control module is responsible for controlling the execution order of each module.

[0009] The video reading and image correction module is responsible for reading images from four cameras and performing image correction operations;

[0010] The parameter initialization module reads the processed image, performs key point detection and matching, and calculates the rotation matrix between images and estimates the camera parameters;

[0011] The image projection module projects each image onto the cylindrical surface based on the rotation matrix and camera parameters;

[0012] The optimal suture line finding module calculates the optimal suture line for each overlapping area;

[0013] The image fusion module fuses images based on the optimal stitching line and multi-band fusion method to obtain a panoramic image, which is then displayed by the panoramic image display module.

[0014] In one approach, the multi-threaded control module loads the initial parameters required by each module, loads the required functions from the open-source computer vision library OpenCV, and establishes a thread pool.

[0015] Modules that require repeated operations, such as video reading and image correction, image projection, optimal seam line finding, image fusion, and panoramic image display, are added to the thread pool.

[0016] Then, open all threads in the thread pool to enable all modules to operate in parallel;

[0017] The multi-threaded control module is also responsible for regulating and coordinating various modules, such as saving the parameters calculated by the parameter initialization module and providing them to the image projection module. When each camera has been saved by the program, the control module notifies the image projection module to start working.

[0018] In one approach, the video reading is described as follows:

[0019] First, low-light image enhancement is performed on each input image. The corresponding illumination weight matrix is ​​obtained based on the input image to guide the adaptive initialization of the illumination components. Then, under the constraint of structured illumination, the initial illumination components are optimized and smoothed through an equivalent objective function, and further nonlinear illumination adjustment is performed on them. Finally, the low-light image enhancement is achieved by combining Retinex theory.

[0020] Then, an image is randomly selected as a reference image. The average color temperature of the reference image is calculated and applied to other images to ensure that the color temperature of all images is consistent.

[0021] The image correction described uses the cameraCalibrator toolbox in MATLAB to perform camera intrinsic parameter calibration and distortion parameter calculation.

[0022] First, use OpenCV to write code to take pictures of the chessboard calibration board from different angles using buttons, taking 20 to 30 pictures;

[0023] The image is then imported into the cameraCalibrator toolbox in MATLAB. The camera distortion parameters and intrinsic parameter matrix are calculated using the cameraCalibrator toolbox. The camera intrinsic parameter matrix and distortion parameters calculated using the cameraCalibrator toolbox are then passed to the remap() function in OpenCV to obtain the image with distortion removed.

[0024] 10. A panoramic image stitching system for low-light enhancement and color temperature correction according to claim 1, characterized in that: the parameter initialization module calculates the rotation matrix between each corrected image and estimates the focal length of the camera, providing parameters for the subsequent image projection module;

[0025] First, key points are detected in each image using OpenCV's SURF detector to locate key point positions and assign feature descriptors.

[0026] The image to be detected is divided into 4 parts, and key points are only detected in the leftmost and rightmost quarter regions;

[0027] After obtaining the location and feature descriptor of the key points, the key points of each image are obtained. Then, the key points between the images are matched using the 2NN (second nearest neighbor) algorithm to obtain a set of matching point pairs for coarse registration.

[0028] The interior points of the matching point pair set are obtained by using the RANSAC algorithm. The homography matrix is ​​then obtained using the least squares method based on these interior points. The rotation matrix between images is then obtained based on the homography matrix, thereby obtaining the extrinsic parameters of the camera.

[0029] In one embodiment, the image projection module uses the following image projection model: Let P(X,Y,Z) be a point in the world coordinate system, then according to the arc length formula, the x-coordinate of the projection point p(u,v) of point P onto the cylinder is: The vertical axis is:

[0030] In one approach, the optimal suture line finding module employs an improved dynamic programming-based optimal suture line algorithm; the improvement is as follows:

[0031] The intensity difference of the energy function in the optimal stitching algorithm based on dynamic programming is changed to be obtained from the HSI color space. By extracting the object edges in the overlapping area, the stitching can avoid the object edges to a certain extent. The SSIM index of the pixels in the overlapping area is calculated and added to the energy function, thus expanding the search range when the dynamic programming finds the next node.

[0032] In one approach, the image fusion module uses a multi-band fusion method to fuse images to obtain a more natural panoramic image, and then stores the stitched panoramic image into a container.

[0033] In one approach, the panoramic image display module runs a separate thread that continuously reads the container holding the panoramic images. Whenever an image is read, it is displayed using the OpenCV imshow() function, and the image is then erased from the container to prevent memory overflow.

[0034] On the other hand, a panoramic image stitching method for low-light enhancement and color temperature correction is provided, and the panoramic image stitching method is applicable to the panoramic image stitching system.

[0035] S1. Run the video reading and image correction module to read raw images from the four cameras.

[0036] The system performs low-light enhancement and color temperature correction operations on images with low light and inconsistent color temperature. Based on the camera parameters calculated using the cameraCalibrator toolbox in MATLAB, it performs distortion correction operations on the images and saves the processed four images into a vector container for use by other modules.

[0037] S2, Run parameter initialization module, uses OpenCV's SURF detector to detect the positions of key points in adjacent images and assign feature descriptors;

[0038] The second nearest neighbor algorithm and the RANSAC algorithm are used to match key points between images. Based on these key points, the homography matrix is ​​obtained by the least squares method. Based on the homography matrix, the rotation matrix between images is obtained, and thus the extrinsic parameters of the camera are obtained.

[0039] S3. Run the image projection module. Based on the rotation matrix and cylindrical projection model between the images, select one image as the reference image and project the other images onto the same cylindrical surface that is consistent with the reference image.

[0040] S4. Run the optimal suture line finding module and image fusion module, convert each projected image to the HIS color space, calculate the energy matrix of the overlapping area of ​​each adjacent image based on the new energy function, find the optimal suture line, use the multi-band fusion method to fuse the image, and eliminate obvious splicing marks around the splicing seam.

[0041] S5. Run the panoramic image display module. Display the final panoramic image.

[0042] Beneficial effects of this invention:

[0043] This invention utilizes a multi-threaded control module to effectively achieve parallel computation among modules, thereby improving image processing speed. Simultaneously, addressing the issues of low light and color temperature inconsistencies, it ensures image clarity and consistency through low-light enhancement and color temperature correction. Furthermore, the parameter initialization module calculates the rotation matrix between images and estimates camera parameters, accurately projecting the image onto the cylindrical surface. The combination of the optimal seam finding module and the image fusion module eliminates obvious seams in the panoramic image, improving its overall aesthetic appeal. Moreover, the panoramic image display module prevents memory overflow when processing large amounts of panoramic images, enhancing the stability of panoramic image display. Therefore, this invention enables high-quality, real-time panoramic image stitching and display, possessing high practical value and promising future prospects. Attached Figure Description

[0044] Figure 1 This is a system block diagram of the present invention;

[0045] Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0046] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Typical embodiments of the invention are shown in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0047] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0048] like Figure 1 As shown, this invention discloses a panoramic image stitching system for low-light enhancement and color temperature correction. The selected camera type is a USB industrial camera with a 130-degree field of view. This camera can capture images with a resolution of 1280*720. Four identical cameras of this type are arranged in a circle at 90-degree angles. The system uses OpenCV library functions to read image data from the four cameras via a serial port.

[0049] The stitching system comprises a multi-threaded control module, a video reading and image correction module, a parameter initialization module, an image projection module, an optimal seam line finding module, an image fusion module, and a panoramic image display module. The modules are interconnected as follows: the multi-threaded control module controls the execution order of each module. The video reading and image correction module reads images from four cameras and performs image correction. The parameter initialization module reads the processed images, performs keypoint detection and matching, and calculates the rotation matrix between images and estimates camera parameters. The image projection module projects each image onto a cylindrical surface based on the rotation matrix and camera parameters. The optimal seam line finding module calculates the optimal seam line for each overlapping region. The image fusion module fuses the images using the optimal seam line and a multi-band fusion method to obtain a panoramic image, which is then displayed by the panoramic image display module.

[0050] Multi-threaded control module: To accelerate computation, this invention uses multi-threading to run the panoramic image stitching modules. The main idea is to establish a thread pool and add modules that require repeated operations, such as video reading and image correction, image projection, optimal stitching line finding, image fusion, and panoramic image display, to the thread pool. Then, all threads in the thread pool are activated, allowing all modules to operate in parallel. In addition, the control module is also responsible for coordinating the communication between the modules. For example, it saves the parameters calculated by the parameter initialization module and provides them to the image projection module. Also, when each camera has one image saved by the program, the control module notifies the image projection module to start working.

[0051] Video Reading and Image Correction Module: The system establishes four camera threads to continuously read images from the camera. First, low-light image enhancement is performed on each input image. The corresponding illumination weight matrix is ​​obtained based on the input image to guide the adaptive initialization of illumination components. Then, under the constraint of structured lighting, the initial illumination components are optimized and smoothed using an equivalent objective function, and further nonlinear illumination adjustments are performed. Finally, low-light image enhancement is achieved using Retinex theory. Next, an image is randomly selected as a reference image. The average color temperature of the reference image is calculated and applied to other images to ensure consistent color temperature across all images. Since the cameras used in this invention are all ultra-wide-angle lenses, radial and tangential distortion exist, although tangential distortion generally has a smaller impact. Images captured by the camera exhibit significant barrel distortion, requiring correction. This invention uses the MATLAB auxiliary toolbox cameraCalibrator for camera intrinsic parameter calibration and distortion parameter calculation. First, code was written in OpenCV to capture images of a checkerboard calibration board at different angles using buttons, approximately 20 to 30 images were captured. The image is then imported into the cameraCalibrator toolbox in MATLAB. The camera distortion parameters and intrinsic parameter matrix are calculated using the cameraCalibrator toolbox. The camera intrinsic parameter matrix and distortion parameters calculated using the cameraCalibrator toolbox are then passed to the remap() function in OpenCV to obtain the image with distortion removed.

[0052] The parameter initialization module primarily calculates the rotation matrix between each corrected image and estimates the camera's focal length, providing parameters for the subsequent image projection module. First, keypoints in each image are detected. This invention uses OpenCV's SURF detector to detect keypoint locations and assign feature descriptors. Since only a small portion of the images overlap, it's unnecessary to detect keypoints in the entire image. This invention divides the image into four parts, detecting keypoints only in the leftmost and rightmost quarters. After obtaining the keypoint locations and feature descriptors, the keypoints for each image are matched using the 2NN (Second Nearest Neighbor) algorithm to obtain a coarsely registered set of matching point pairs. Then, the RANSAC algorithm is used to obtain the interior points of the matching point pair set. Based on these interior points, the homography matrix is ​​calculated using the least squares method. The rotation matrix between images is then calculated from the homography matrix, thereby obtaining the camera's extrinsic parameters.

[0053] Image Projection Module: This invention arranges four USB industrial cameras at 90-degree angles. Since the images captured by each pair of cameras have significant visual differences, an image projection algorithm is needed to project each image onto the same plane. The image projection model used in this invention is as follows: Let P(X,Y,Z) be a point in the world coordinate system. Then, according to the arc length formula, the x-coordinate of the projection point p(u,v) of point P onto the cylinder is: The vertical axis is:

[0054] Optimal seam line finding module: Traditional dynamic programming-based optimal seam line algorithms often result in object disappearance and misalignment due to large parallax between adjacent shots, leading to inconsistent object positions. This invention improves upon the traditional method by retrieving intensity differences from the HSI color space in the energy function, extracting object edges in overlapping areas to allow the seam line to avoid these edges to some extent, calculating the SSIM index of pixels in the overlapping area and adding it to the energy function, and expanding the search range for the next node in dynamic programming. The improved optimal seam image fusion algorithm of this invention is more natural than traditional algorithms and significantly reduces object misalignment and disappearance.

[0055] Image fusion module: Directly stitching images together using masks will result in obvious stitching seams. Using a multi-band fusion method to fuse the images will produce a more natural panoramic image. The stitched panoramic image will then be stored in a container.

[0056] Panoramic image display module: A separate thread continuously reads the container holding panoramic images. Whenever an image is read, it is displayed using the OpenCV imshow() function, and the image is then erased from the container to prevent memory overflow.

[0057] like Figure 2 As shown, a panoramic image stitching method with low-light enhancement and color temperature correction is proposed.

[0058] S1. Run the video reading and image correction module to read raw images from the four cameras, such as... Figure 2 As shown, low-light enhancement and color temperature correction are performed on images with low light and inconsistent color temperatures. Distortion correction is performed on the images based on camera parameters calculated using MATLAB's cameraCalibrator toolbox. The processed four images are then saved into a vector container for use by other modules.

[0059] S2, Parameter Initialization Module. Using OpenCV's SURF detector, the positions of keypoints in adjacent images are detected and feature descriptors are assigned. The second nearest neighbor algorithm (PMI) + RANSAC algorithm are used to match keypoints between images. Based on these keypoints, the homography matrix is ​​obtained using the least squares method. From the homography matrix, the rotation matrix between images is calculated, thus determining the camera's extrinsic parameters.

[0060] S3. Run the image projection module. Based on the rotation matrix and cylindrical projection model between the images, select one image as the reference image, and project the other images onto the same cylindrical surface consistent with the reference image.

[0061] S4. Run the optimal seam line finding module and image fusion module. Convert each projected image to the HIS color space, calculate the energy matrix of the overlapping area of ​​each adjacent image based on the new energy function, find the optimal seam line, and use a multi-band fusion method to fuse the images, eliminating obvious splicing marks around the seam.

[0062] S5. Run the panoramic image display module. Display the final panoramic image.

[0063] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0064] It should be understood that the above detailed description of the technical solutions of the present invention with reference to preferred embodiments is illustrative and not restrictive. Those skilled in the art can modify the technical solutions described in the embodiments or make equivalent substitutions for some of the technical features based on reading this specification; however, these modifications or substitutions do not cause the essence of the corresponding technical solutions to depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A panoramic image stitching system for low-light enhancement and color temperature correction, characterized in that: The stitching system includes a multi-threaded control module, a video reading and image correction module, a parameter initialization module, an image projection module, an optimal stitch line finding module, an image fusion module, and a panoramic image display module. The multi-threaded control module is responsible for controlling the execution order of each module. The video reading and image correction module is responsible for reading images from four cameras and performing image correction operations; The parameter initialization module reads the processed image, performs key point detection and matching, and calculates the rotation matrix between images and estimates the camera parameters; The image projection module projects each image onto the cylindrical surface based on the rotation matrix and camera parameters; The optimal suture line finding module calculates the optimal suture line for each overlapping area; The image fusion module fuses images according to the optimal stitching line and multi-band fusion method to obtain a panoramic image, which is then displayed by the panoramic image display module. The multi-threaded control module loads the initial parameters of each module, loads the functions of the open-source computer vision library OpenCV, and establishes a thread pool. The modules that need to be repeatedly operated, such as video reading and image correction module, image projection module, best seam line finding module, image fusion module, and panoramic image display module, will be added to the thread pool. Then, open all threads in the thread pool to enable all modules to operate in parallel; The multi-threaded control module is also responsible for regulating and connecting various modules, saving the parameters calculated by the parameter initialization module and providing them to the image projection module. When each camera has been saved by the program, the control module notifies the image projection module to start working. The video reading process first involves performing low-light image enhancement on each input image, and then obtaining the corresponding light intensity from the input image. The weight matrix is ​​designed to guide the adaptive initialization of the illumination components. Then, under the constraint of structured illumination, the initial illumination components are optimized and smoothed through an equivalent objective function, and further nonlinear illumination adjustment is performed on them. Finally, the enhancement of low-light images is achieved by combining Retinex theory. Then, an image is randomly selected as a reference image. The average color temperature of the reference image is calculated and applied to other images to ensure that the color temperature of all images is consistent. The image correction described uses the cameraCalibrator toolbox in MATLAB to perform camera intrinsic parameter calibration and distortion parameter calculation. First, use OpenCV to write code to take pictures of the chessboard calibration board from different angles using buttons, taking 20 to 30 pictures; Then, the image is imported into the cameraCalibrator toolbox in MATLAB. The camera distortion parameters and intrinsic parameter matrix are calculated using the cameraCalibrator toolbox. The camera intrinsic parameter matrix and distortion parameters calculated by the cameraCalibrator toolbox are then passed to the remap() function in OpenCV to obtain the image with distortion removed. The parameter initialization module calculates the rotation matrix between each corrected image and estimates the camera's focal length, providing a basis for subsequent... The image projection module provides parameters; First, key points are detected in each image using OpenCV's SURF detector to locate key point positions and assign features. Feature descriptor; The image to be detected is divided into 4 parts, and key points are only detected in the leftmost and rightmost quarter regions; After obtaining the location and feature descriptor of the key points, the key points of each image are obtained. Then, the key points between the images are matched using the 2NN (second nearest neighbor) algorithm to obtain a set of matching point pairs for coarse registration. The interior points of the matching point pair set are obtained by using the RANSAC algorithm. The homography matrix is ​​obtained by using the least squares method based on these interior points. The rotation matrix between images is obtained based on the homography matrix, and thus the extrinsic parameters of the camera are obtained. The image projection module arranges four USB industrial cameras at 90 degrees. Since the images captured by each pair of cameras have significant visual differences, an image projection algorithm is used to project each image onto the same plane. The image projection model used is as follows: Let P(X, Y, Z) be a point in the world coordinate system. Then, according to the arc length formula, the x-coordinate of the projection point p(u, v) of point P onto the cylinder is: The vertical axis is: ; The optimal suture line finding module employs an improved dynamic programming-based optimal suture line algorithm; the improvement is as follows: The intensity difference of the energy function in the optimal stitching algorithm based on dynamic programming is changed to be obtained from the HSI color space. By extracting the object edges in the overlapping area, calculating the SSIM index of the pixels in the overlapping area and adding it to the energy function, the search range when dynamic programming finds the next node is expanded. The image fusion module uses a multi-band fusion method to fuse images, resulting in a more natural panoramic image, which is then stitched together. The stitched panoramic image is stored in a container; The panoramic image display module operates on a separate thread, continuously reading from the container holding panoramic images. Each time an image is read... The image is displayed using OpenCV's imshow() function, and then the image is erased from the container to prevent memory overflow.

2. A panoramic image stitching method for low-light enhancement and color temperature correction, wherein the panoramic image stitching method is applicable to the panoramic image stitching system as described in claim 1, characterized in that: S1. Run the video reading and image correction module to read the original images from the four cameras, perform low light enhancement and color temperature correction operations on images with low light and inconsistent color temperature, perform distortion correction operations on the images based on the camera parameters calculated by the cameraCalibrator toolbox in MATLAB, and save the processed four images into a vector container for use by other modules. S2, Run parameter initialization module, uses OpenCV's SURF detector to detect the positions of key points in adjacent images and assign feature descriptors; The second nearest neighbor algorithm and the RANSAC algorithm are used to match key points between images. Based on these key points, the homography matrix is ​​obtained by the least squares method. Based on the homography matrix, the rotation matrix between images is obtained, and thus the extrinsic parameters of the camera are obtained. S3. Run the image projection module. Based on the rotation matrix and cylindrical projection model between the images, select one image as the reference image and project the other images onto the same cylindrical surface that is consistent with the reference image. S4. Run the optimal suture line finding module and image fusion module, convert each projected image to the HIS color space, calculate the energy matrix of the overlapping area of ​​each adjacent image based on the new energy function, find the optimal suture line, use the multi-band fusion method to fuse the image, and eliminate obvious splicing marks around the splicing seam. S5. Run the panoramic image display module to display the final panoramic image.

Images