Chromatic aberration consistency control method based on front and rear end cooperation

Abstract

The invention belongs to the field of panoramic stitching digital image processing technologies and camera ISP technologies, and particularly relates to a chromatic aberration consistency control method based on front and rear end collaboration. The method comprises the following steps: 1, acquiring statistical information of multiple cameras; step 2, acquiring a current camera control parameter by adopting a front-end chromatic aberration consistency adjustment algorithm; 3, the current camera control parameters are issued to all the cameras, and all the cameras complete exposure imaging according to the current camera control parameters; step 4, acquiring an overlapped part image in each path of camera imaging; step 5, calculating a statistical histogram cumulative distribution function of overlapped part images in each path of camera imaging; step 6, calculating a histogram matching function according to the cumulative distribution function of the overlapped part images in each path of overlapped camera imaging, so that the histograms of the overlapped parts of the two images tend to be consistent; and step 7, according to the obtained histogram matching function, transforming the brightness channel of each pixel in the corresponding camera imaging to complete color difference consistency correction.

Description

Technical Field

[0001] This invention belongs to the fields of panoramic stitching digital image processing technology and camera ISP (Image Signal Processor) technology, specifically involving a color difference consistency control method based on front-end and back-end collaboration. Background Technology

[0002] Since the multiple cameras in a stitching system typically operate with independent automatic exposure, the exposure levels differ between the images captured by different cameras. This results in noticeable edges at the stitching points in the stitched images, making the images appear unnatural. Therefore, we need to perform exposure compensation on each image to ensure that all images have the same brightness and color; this is referred to here as color difference consistency control.

[0003] There are generally two technical approaches to solving this problem: one is to design a camera-coordinated color difference consistency control software module, which acquires AE / AWB statistical information from multiple cameras in real time, calculates the current optimal camera control parameters based on the camera statistical information using a color difference consistency adjustment algorithm, and then sends the control information to each camera to achieve multi-camera color difference consistency adjustment control. This is referred to as the camera front-end color difference consistency control algorithm.

[0004] Secondly, after acquiring the images to be stitched, relevant information about the overlapping areas of the images is statistically analyzed using algorithms such as gain compensation and histogram matching. The corresponding gain adjustment coefficients are then calculated and applied to the corresponding images to make the image intensities of the overlapping parts equal or similar, thereby achieving the overall consistency adjustment of the stitched images. This is referred to as the camera back-end color difference consistency control algorithm.

[0005] During the implementation of color difference consistency control in the splicing project, it was found that although the color difference consistency control algorithm at the camera front end can manage the ISPs of multiple cameras in a coordinated manner, there are individual differences between the cameras. For the received control parameters, there are still differences in brightness and white balance in the image, which cannot achieve good uniformity of the spliced ​​image.

[0006] The backend color difference consistency control algorithm performs well when camera scene differences are minimal. However, due to the wide dynamic range and limited overlap area of ​​the camera field of view in stitched scenes, the backend color difference consistency adjustment has limitations in scenes with significant differences in lighting conditions between different cameras (such as backlighting). For adjustments with excessively large environmental differences, inconsistencies in brightness and color still exist. This is due to inaccurate calculation of the overlap area caused by varying degrees of camera distortion and differences in field of view. Therefore, it is necessary to avoid using backend color difference consistency to handle scenes with excessively large color difference variations. Summary of the Invention

[0007] The purpose of this invention is to first use a front-end color difference consistency control algorithm to control the color difference of multiple cameras to a small deviation range, and then use a back-end color difference consistency control algorithm based on the overlapping area to calculate the color difference gain compensation value of the overlapping area, which is fed back to the corresponding front-end camera. Finally, the back-end performs real-time color difference consistency adjustment on the images from multiple cameras. This solves the inconsistency problem caused by individual camera differences that front-end color difference consistency control cannot address, and the situation where back-end color difference consistency control cannot effectively handle scenarios with large differences in lighting environments. The optimized color difference consistency effect is significantly better than that of a single-end control algorithm. Technical solution: A color difference consistency control method based on front-end and back-end collaboration, comprising the following steps: Step 1: Obtain statistical information from multiple cameras; Step 2: Use the front-end color difference consistency adjustment algorithm to obtain the current camera control parameters; Step 3: Send the current camera control parameters to each camera, and each camera completes the exposure and imaging according to the current camera control parameters; Step 4: Acquire the overlapping images from each camera; Step 5: Calculate the cumulative distribution function of the statistical histogram of the overlapping portion of the images from each camera. Step 6: Calculate the histogram matching function based on the cumulative distribution function of the overlapping parts of the images from the overlapping cameras, so that the histograms of the overlapping parts of the two images tend to be consistent. Step 7: Based on the obtained histogram matching function, transform the brightness channel of each pixel in the corresponding camera image to complete the color difference consistency correction.

[0008] Furthermore, in step one, the statistical information includes: AE histogram information, AE mean information, and AWB statistical information.

[0009] Furthermore, in step two, the current camera control parameters include: AE control information including exposure time, camera analog gain, and camera digital gain; AWB control information including white balance gain for the four channels: red channel, green Gr channel, green Gb channel, and blue channel; and color correction matrix (CCM) control information: a 3×3 matrix, where the RGB three-channel matrix is ​​multiplied by this matrix to achieve the color correction result.

[0010] Furthermore, in step four, the image In, with The overlapping part is defined as ,image In, with The overlapping part is defined as . Cumulative distribution function of histogram Defined as ; This represents the number of pixels with a grayscale value of i. It is a 256-bit array. The cumulative distribution function is defined as .

[0011] Furthermore, in step six, the cumulative distribution function is used to calculate the value from... arrive The histogram matching function T is a mapping relationship. For an 8-bit image, T is a 256-bit array for each RGB channel. for For each channel's grayscale level r, find The corresponding pixel's gray level z makes .

[0012] Furthermore, in step seven, the specific process is as follows: For the current... The brightness channel of each pixel is transformed using the mapping relationship T: The lookup table mapping transformation is completed, and the color difference consistency correction of adjacent images is completed. Similarly, it is applied to other adjacent overlapping images to complete the color difference adjustment of the entire stitched image.

[0013] Furthermore, the method also includes: Step 7: Correcting the color difference consistency of adjacent images based on the histogram matching function obtained in Step 6, thereby completing the color difference adjustment of the entire spliced ​​image.

[0014] Furthermore, in step eight, the calculated histogram matching mapping relationship T is incorporated into the CCM control information sent to each camera in step three. This completes the optimization of camera exposure and imaging.

[0015] Beneficial effects: 1. A color difference consistency control method based on front-end and back-end collaboration is proposed to solve the color difference consistency problem in image / video splicing scenarios with low overlap areas; 2. The camera-coordinated chromatic aberration consistency control method proposed in this invention optimizes the limitations of purely front-end or purely back-end independent control, utilizing the advantages of front-end and back-end control algorithms to improve the chromatic aberration consistency control effect. (See attached figures for details.) Figure 1 This is an example of the front-end and back-end collaborative color difference consistency control system architecture in this invention.

[0016] Figure 2 This is an example of separate imaging from two cameras in the image to be stitched in this invention, where there is a color difference.

[0017] Figure 3 This is an example of the original camera image stitching effect in this invention.

[0018] Figure 4 This is an example of using only the front-end camera AE / AWB to control the effect in this invention.

[0019] Figure 5 This is an example of the effect of front-end and back-end coordinated color difference consistency control based on histogram matching in this invention.

[0020] Figure 6 This is a magnified view of the splicing area in this invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0022] In this application, the terms "upper," "lower," "left," "right," "front," "rear," "top," "bottom," "inner," "outer," "middle," "vertical," "horizontal," "lateral," and "longitudinal" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are primarily for the purpose of better describing this application and its embodiments, and are not intended to limit the indicated device, element, or component to having a specific orientation, or to be constructed and operated in a specific orientation.

[0023] Furthermore, the terms "installation," "setup," "equipped with," "connection," "linking," and "socketing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral structure; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium, or an internal connection between two devices, components, or parts. Those skilled in the art can understand the specific meaning of these terms in this application based on the specific circumstances.

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0025] An implementation of a color difference consistency control method based on front-end and back-end collaboration includes the following steps: Step 1: The processor acquires multi-camera AE / AWB statistical information in real time through the splicing end, including AE histogram information, AE mean information, and AWB statistical information.

[0026] Step 2: The processor uses a front-end color difference consistency adjustment algorithm based on camera statistics to obtain the current optimal camera control parameters; the front-end color difference consistency adjustment algorithm includes three parts: lens LSC calibration, unified color temperature control, and unified exposure control.

[0027] Step 3: The processor sends control information (including AE control information containing exposure time, camera analog gain, and camera digital gain; AWB control information containing white balance gain for the four channels: red, green (Gr), green (Gb), and blue; and CCM (Color Correction Matrix) control information: a 3×3 matrix, where the RGB three-channel matrix is ​​multiplied by this matrix to achieve the color correction result) to each camera via the stitching terminal. Each camera completes the exposure and imaging according to the sent camera control parameters.

[0028] Step 4: The stitching end acquires images from the multi-camera setup in real time after front-end adjustments, including images with overlapping areas. and ,image In, with The overlapping part is defined as ,image In, with The overlapping part is defined as .

[0029] Step 5: Calculate Cumulative distribution function of histogram Defined as ( This represents the number of pixels with a grayscale value of i. (a 256-bit array) The cumulative distribution function is defined as .

[0030] Step Six: Calculate the cumulative distribution function from... arrive The histogram matching function T (which is a mapping relationship; for an 8-bit image, T is a 256-bit array for each RGB channel): that is, for For each channel's grayscale level r, find The corresponding pixel's gray level z makes .

[0031] Step 7: For the current For each color channel of each pixel, a transformation is performed using the mapping relationship T: (That is, to complete the lookup table mapping transformation), complete the color difference consistency correction of adjacent images, and similarly, apply it to other adjacent overlapping images to complete the color difference adjustment of the entire stitched image.

[0032] Step 8: Update the CCM control information with the calculated histogram matching mapping relationship T. The matrix format of the CCM control information is as follows:

[0033] target image The result is obtained by left-multiplying the source image by this matrix: (Target) (source) The histogram matching mapping relationship T is incorporated into the CCM control information sent to each camera in step three. The specific operation is as follows: Assuming the gray level r of the R channel, find the corresponding z in T, and... Replace with the original The same procedure applies to other channels. This completes the optimization of camera exposure and imaging.

[0034] Traditional image stitching color difference consistency processing either involves adjusting camera parameters at the camera end or using image processing algorithms during backend stitching based on the assumption that the color tones of the overlapping areas should be consistent. Generally, when the ambient light difference in the stitching scene is small and the overlapping area is large, a single method can achieve good color difference consistency control, without the need for overly complex color difference consistency control. The application scenarios of this invention have the following characteristics: 1. Individual differences in camera sensors and other components lead to differences in brightness and white balance even with the same exposure / white balance control parameters. 2. The camera mounting positions have low overlap areas, resulting in low robustness of gain adjustment coefficients based on the overlap area, which can easily lead to large deviations in some scenarios. 3. The application scenarios have large differences in lighting environments, in which case pure backend color difference consistency control algorithms cannot achieve good results. To address the difficulties brought about by these scenarios, this invention designs a color difference consistency control method based on frontend and backend collaboration to provide a targeted solution: The front-end color difference consistency control algorithm solves the problem of back-end processing when there are too many differences in lighting environment.

[0035] When the lighting environment of an application scenario varies too much, the imaging of multiple cameras cannot be adjusted to a good effect by a pure back-end color difference consistency control algorithm. Therefore, the intervention of a front-end color difference consistency control algorithm can control the color difference of multiple cameras to a smaller deviation range, so that the back-end algorithm can achieve good results.

[0036] The calculated histogram matching function solved the problem of fine-tuning details.

[0037] The front-end color difference consistency control algorithm has controlled the color difference of multiple cameras to a small deviation range. The back-end uses a color difference consistency control algorithm based on histogram matching of overlapping areas to calculate the histogram matching function and feed it back to the corresponding camera in the front end, further optimizing the color difference inconsistency problem caused by individual camera differences and overlapping area errors.

[0038] The backend color difference consistency control method solves the problem of color difference consistency optimization in the current spliced ​​frame.

[0039] The backend histogram matching function performs real-time color difference consistency adjustment on the images from multiple cameras. This solves the inconsistency problem caused by individual camera differences that cannot be addressed by the frontend color difference consistency control, and the problem that the histogram matching function cannot adjust the delay of the current frame when feeding back to the camera.

[0040] Example A color difference consistency control method based on front-end and back-end collaboration involves a control system comprising a camera, a stitching end (hardware device that executes the stitching algorithm and the back-end color difference consistency control algorithm), a processor end (hardware device that executes the front-end color difference consistency control algorithm), cables, and other equipment. First step, according to Figure 1 As shown in the system architecture, the camera, stitching end hardware, and processor end hardware have been successfully connected and configured. The real-time independent imaging capture from the camera is shown below. Figure 2 As shown, the effect of directly splicing them according to the spherical projection is as follows: Figure 3 As shown.

[0041] The second step involves the stitching end receiving statistical information from multiple cameras, including AE histogram information, AE mean information, and AWB statistical information.

[0042] Step 2: After the statistical information is received, it is transmitted via PCIe to the front-end color difference consistency control algorithm module residing on the processor.

[0043] Step 3: After the front-end color difference consistency control algorithm module on the processor side completes its calculations, it transmits the obtained control information (including AE control information, AWB control information, and CCM control information) to the stitching end via PCIe. The stitching end sends the control information in its buffer to each camera via differential signals. When the camera receives the control information, it performs frame synchronization and adjusts the exposure and white balance according to the control information.

[0044] Step 4: Each camera completes exposure and imaging according to the issued camera control parameters. The images obtained from the same cameras as in Step 1 are then stitched together to create the desired effect, such as... Figure 4 As shown.

[0045] Step 5: Following steps four, five, and six of the present invention, the back-end color difference consistency control algorithm residing at the stitching end performs real-time mapping calculation using the calculated histogram matching function T after acquiring the current frame. Then, following step seven, the calculated histogram matching mapping relationship T is converted into a matrix format of CCM control information and sent to each camera.

[0046] Step 6: At the stitching end, complete the stitching of the image by coordinating and controlling the color difference consistency between the front and back ends based on histogram matching, such as... Figure 5 As shown in the enlarged image of the splicing details. Figure 6 As shown. With Figure 3 A comparison of images without color difference consistency adjustment and images with only front-end camera color difference consistency control reveals a significant improvement, resulting in a more realistic and natural effect. The implementation method illustrates the operational steps and verification results of this invention.

[0047] The above description is merely a specific embodiment of the present invention, providing a detailed description of the invention. Parts not covered herein are conventional techniques. However, the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A color difference consistency control method based on front-end and back-end collaboration, characterized in that: The steps are as follows: Step 1: Obtain statistical information from multiple cameras; Step 2: Use the front-end color difference consistency adjustment algorithm to obtain the current camera control parameters; Step 3: Send the current camera control parameters to each camera, and each camera completes the exposure and imaging according to the current camera control parameters; Step 4: Acquire the overlapping images from each camera; Step 5: Calculate the cumulative distribution function of the statistical histogram of the overlapping portion of the images from each camera. Step 6: Calculate the histogram matching function based on the cumulative distribution function of the overlapping images from the overlapping cameras to make the histograms of the overlapping parts of the two images more consistent. Step 7: Based on the obtained histogram matching function, transform the brightness channel of each pixel in the corresponding camera image to complete the color difference consistency correction.

2. The method according to claim 1, characterized in that: In step one, the statistical information includes: AE histogram information, AE mean information, and AWB statistical information.

3. The method according to claim 2, characterized in that: In step two, the current camera control parameters include: AE control information including exposure time, camera analog gain, and camera digital gain; AWB control information including white balance gain for the four channels: red channel, green Gr channel, green Gb channel, and blue channel; and color correction matrix (CCM) control information: a 3×3 matrix, where the RGB three-channel matrix is ​​multiplied by this matrix to achieve the color correction result.

4. The method according to claim 3, characterized in that: In step four, the image In, with The overlapping part is defined as ,image In, with The overlapping part is defined as . Cumulative distribution function of histogram Defined as ; This represents the number of pixels with a grayscale value of i. It is a 256-bit array. The cumulative distribution function is defined as .

5. The method according to claim 4, characterized in that: In step six, the cumulative distribution function is used to calculate the value from... arrive The histogram matching function T is a mapping relationship. For an 8-bit image, T is a 256-bit array for each RGB channel. for For each channel's grayscale level r, find The corresponding pixel's gray level z makes .

6. The method according to claim 5, characterized in that: Step seven involves the following specific steps: For the current... The brightness channel of each pixel is transformed using the mapping relationship T: The lookup table mapping transformation is completed, and the color difference consistency correction of adjacent images is completed. Similarly, it is applied to other adjacent overlapping images to complete the color difference adjustment of the entire stitched image.

7. The method according to claim 6, characterized in that: The method further includes: Step 7: Correcting the color difference consistency of adjacent images based on the histogram matching function obtained in Step 6, thereby completing the color difference adjustment of the entire spliced ​​image.

8. The method according to claim 7, characterized in that: In step eight, the calculated histogram matching mapping relationship T is incorporated into the CCM control information sent to each camera in step three. This completes the optimization of camera exposure and imaging.

Images