White balance matching method of multi-camera system

Abstract

The invention provides a white balance matching method for a multi-camera system. The white balance matching method comprises the following steps: S1, determining a primary camera and a secondary camera; s2, calibrating a Planck curve of each camera; s3, the white balance gain of the secondary camera is calculated; S301, the white balance gain of the primary camera is obtained; s302, obtaining a white balance drop point of the main camera; s303, calculating the distance between the main white balance drop point and each calibration point on the Planck curve; s304, different weights are given to all points on the Planck curve according to the distance; s305, weighting each point on the curve to obtain a white balance drop point of secondary shooting; and S306, calculating a secondary shooting gain. Through the method, the difference of white balance effects of a multi-camera system is reduced, and the applicable scene content is wider; moreover, the method can be used in pure-color scenes with large image content change and non-rich colors among multiple cameras.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a white balance matching method for a multi-camera system. Background Art

[0002] In the prior art, in a multi-camera integrated system, due to the differences in the field of view angles of each camera, the photosensitivity of each camera also varies. Even if the same white balance algorithm is used, the images captured by multiple cameras will have different white balance effects, affecting the viewing experience.

[0003] The white balance of existing multi-camera acquisition systems is either not matched, that is, the white balance effect of each camera is completely independent, or it is simply matched based on the grayscale world algorithm.

[0004] However, without matching, the white balance results between multiple cameras vary significantly. Using a grayscale world algorithm for matching doesn't account for overlapping images between multiple cameras. Using a global image for grayscale world matching results in large errors and is easily affected by changes in scene content.

[0005] In addition, terms commonly used in the prior art include:

[0006] Automatic white balance AWB: Auto White Balance in English, its purpose is to "restore neutral-colored objects to neutral colors regardless of the light source", where neutral-colored objects are objects that reflect the spectrum uniformly.

[0007] Color temperature: A quantitative measure of color expressed in Kelvin temperature (K). Kelvin defines this by the color properties of light emitted by a blackbody, an ideal light source, at different temperatures. When a blackbody is heated, it initially emits red, then gradually brightens, becoming yellow, then white, and finally blue. Red is a warm color with a low color temperature. Blue is a cool color with a high color temperature.

[0008] Planck curve: On the CIE-1931 xyY chromaticity diagram, a curve describes the color change of a black body when heated to different temperatures. In white balance algorithms, the Planck curve is often used to describe the camera module's response to neutral-colored objects under light sources of different color temperatures.

[0009] Color card: A standard card printed with colored blocks and gray blocks (color cards vary from manufacturer to manufacturer) used for image color calibration.

[0010] Grayscale world algorithm: Assuming that the average reflectance of light from each object in the image is generally constant, the average values ​​of the R, G, and B components of the digital image tend to be the same. Therefore, for an unwhite-balanced digital image, the global R, G, and B values ​​of the image are calculated. Typically, the R and B channels are compensated to achieve the same average values ​​for the three channels, thus achieving white balance.

[0011] Field of view: The field of view of the camera module to capture images. The larger the field of view, the more scene content can be captured.

[0012] White balance gain: A constant is multiplied by the R, G, and B channels of a digital image (generally, only the R and B channels are gain processed, and the following text also applies to the R and B channels) to achieve a white balance effect. This constant changes with the scene; even for the same scene, the white balance gains of digital images captured by different camera modules may vary.

[0013] R, G, B channels: the red, green, and blue component channels of a digital image. The RGB channels below have the same meaning. Summary of the Invention

[0014] In order to solve the above problems, the purpose of this application is to reduce the differences in white balance effects of multi-camera systems through this method, and to apply it to a wider range of scene contents; and it can be used in pure color scenes with large changes in image content between multiple cameras and not rich colors.

[0015] Specifically, the present invention provides a white balance matching method for a multi-camera system, the method comprising the following steps:

[0016] S1. Determine the primary and secondary cameras:

[0017] By reading the FOV parameter (Field of View) in each camera's technical manual, a larger value indicates a wider field of view. Alternatively, place each camera in the system in the same position to capture a scene. Determine the camera with the largest field of view as the primary camera, and name the remaining cameras Secondary 1, Secondary 2, and so on.

[0018] S2. Calibrate the Planck curve of each camera:

[0019] Calibrate the main camera and secondary camera 1, secondary camera 2, etc. in a standard light box. The response results to the gray blocks on the color card under different light sources are as follows. The number of calibrated light sources can be determined according to the actual usage. Assume that ct1 to ct8 represent eight light sources. The response results are expressed as the mean value of the R channel in the gray block divided by the mean value of the G channel, and the mean value of the B channel divided by the mean value of the G channel. These are recorded as the calibration point coordinates. The coordinates of the main camera calibration point are marked as (X z1 , Y z1), (X z2 , Y z2 ),......(X zi , Y zi ), (X zi ,Y zi ) represents the coordinates of the main camera under the i-th light source, X zi >0,Y zi >0; coordinates of the second-photographed calibration point are (X c11 ,Y c11 ), (X c12 ,Y c12 )......(X c1i ,Y c1i ), the coordinates of the secondary j calibration point are marked as (X cj1 ,Y cj1 ), (X cj2 ,Y cj2 )......(X cji ,Y cji ), (X cji ,Y cji ) represents the coordinates of the jth shot under the i-th light source, X cji >0,Y cji >0;

[0020] S3. Calculate the secondary shot white balance gain:

[0021] S301: Obtaining the white balance gain of the main camera. In this application, the white balance gain of the main camera is calculated by other algorithms in the prior art and can be used directly.

[0022] S302: Obtaining the white balance point of the main camera;

[0023] S303: Calculating the distance between the main camera white balance point and each calibration point on the Planck curve;

[0024] S304: assigning different weights to each point on the Planck curve according to the distance;

[0025] S305: Obtain the white balance point of the secondary shot after weighting each point on the curve;

[0026] S306: Calculate the secondary shooting gain.

[0027] The step S3 further comprises:

[0028] S301: First, the main camera white balance gain is obtained according to the preset automatic white balance algorithm of the main camera. The gain values ​​of the R and B channels are R zgain , B zgain This method assumes that the white balance gain of the primary camera has been calculated and used directly, and only requires the secondary camera to be closer to the primary camera;

[0029] S302: Gain value R for the main camera R and B channels zgain , B zgain Taking the reciprocal, we get

[0030] Note it as the main camera white balance point;

[0031] S303: Calculate the main camera white balance point and the calibration points (X z1 , Y z1 ), (X z2 , Y z2 ),...(X zi , Y zi ) distance;

[0032] S304: Assign different weights according to the distance, denoted as w1, w2, ...w i ;

[0033] S305: Use the weights w1, w2, ...w of each landing point of the main camera i is the secondary camera 1 calibration point (X c11 ,Y c11 ), (X c12 ,Y c12 )......(X c1i ,Y c1i ) is weighted, and the weighted average of the secondary camera 1 calibration points is obtained to obtain the secondary camera 1 white balance point R c1l , B c1l , the calculation formula is:

[0034] R c1l =(w1×X c11 +w2×X c12 +...+w i ×X c1i ) / (w1+w2+...+w i )

[0035] B c1l =(w1×Y c11 +w2×Y c12 +...+w i ×Y c1i ) / (w1+w2+...+w i )

[0036] Similarly, the white balance point of the jth shot is R cjl , B cjl , the calculation formula is:

[0037] R cjl =(w1×X cj1+w2×X cj2 +...+w i ×X cji ) / (w1+w2+...+w i )

[0038] B cjl =(w1×Y cj1 +w2×Y cj2 +...+w i ×Y cji ) / (w1+w2+...+w i )

[0039] S306: Finally calculate the white balance gain of secondary camera 1 - R c1gain1 , B c1gain1 , that is, the white balance point R of the second shot c1l , B c1l Take the reciprocal:

[0040]

[0041] Similarly, the white balance gain R of the jth shot cjgain1 , B cjgain1 , the calculation formula is:

[0042]

[0043] In step S303, the distance calculation method includes Euclidean distance and block distance. If Euclidean distance is used, assuming that the calibration point (X z1 ,Y z1 ) and the main camera white balance point (R zl ,B zl ), the Euclidean distance calculation formula between the two is: The formula for calculating block distance is d b1 =|X z1 -R zl |+|Y z1 -B zl |.

[0044] In step S304, the weight distribution method includes the weight in the form of Gaussian function and the weight of the inverse of the distance; assuming that the distance between the white balance point and each calibration point is d1, d2, ... d i , if Gaussian function is used to assign weights, then Among them, A and s are parameters that are adjusted according to the usage, and e is a natural constant. If the inverse of the distance is used to assign weights, then

[0045] Therefore, the advantages of this application are:

[0046] 1. Match the white balance of a multi-camera system to reduce the difference in image quality between lenses;

[0047] 2. Use the Planck curve calibration method to perform white balance matching, which is applicable to a wider range of scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0049] Figure 1 It is a schematic diagram of the response results to the gray block on the color card under different light sources.

[0050] Figure 2 is a flow chart of this method.

[0051] Figure 3 3 is a flow chart of calculating the secondary shot white balance gain in step S3 of the method. DETAILED DESCRIPTION

[0052] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail with reference to the accompanying drawings.

[0053] This application provides a white balance matching method for a multi-camera system. Each camera in the multi-camera system is labeled as primary camera, secondary camera 1, secondary camera 2, etc. The white balance effect of each secondary camera approaches that of the primary camera. The following steps are for matching the white balance of secondary camera 1. The steps for the other secondary cameras are the same:

[0054] like Figure 2 As shown, the method is a method for achieving gain matching between primary and secondary cameras based on a calibrated Planck curve, comprising the following steps:

[0055] S1. Determine the primary and secondary cameras:

[0056] By reading the FOV parameter (Field of View) in each camera's technical manual, a larger value indicates a larger field of view. Alternatively, place each camera in the system in the same position to capture a scene. This will determine the camera with the largest field of view as the primary camera, and the remaining cameras as secondary camera 1, secondary camera 2, and so on.

[0057] S2. Calibrate the Planck curve of each camera:

[0058] Calibrate the main camera and secondary camera 1, secondary camera 2, etc. in a standard light box, and the response results to the gray blocks on the color card under different light sources are as follows: Figure 1As shown in the figure (the number of calibrated light source types can be determined according to actual usage), ct1 to ct8 in the figure represent eight light sources, such as A light, H light, D50 light, D65 light, etc. The response result is represented by the mean value of the R channel divided by the mean value of the G channel, and the mean value of the B channel divided by the mean value of the G channel in the gray block; , recorded as the calibration point coordinates, the main camera calibration point coordinates are marked as (X z1 , Y z1 ), (X z2 , Y z2 ),......(X zi , Y zi ), (X zi ,Y zi ) represents the coordinates of the main camera under the i-th light source, X zi >0,Y zi >0; coordinates of the second-photographed calibration point are (X c11 ,Y c11 ), (X c12 ,Y c12 )......(X c1i ,Y c1i ), the coordinates of the secondary j calibration point are marked as (X cj1 ,Y cj1 ), (X cj2 ,Y cj2 )......(X cji ,Y cji ), (X cji ,Y cji ) represents the coordinates of the jth shot under the i-th light source, X cji >0,Y cji >0;

[0059] Under the same light source, the primary and secondary cameras may have different responses, which can cause differences in overall image color. This difference in image color necessitates white balance matching. Alternatively, the primary and secondary cameras may have different image content, affecting the user experience. Even if the responses are identical, applying the original white balance algorithm to the primary and secondary camera images separately may produce different results. This is because white balance algorithms typically use image content to estimate gain, resulting in different overall color tones for the primary and secondary cameras.

[0060] S3. Calculate the secondary white balance gain: the implementation method is as follows Figure 3 As shown, it can be divided into the following steps:

[0061] S301: First, the main camera white balance gain is obtained according to the preset main camera automatic white balance algorithm, wherein the gain values ​​of the R and B channels are R and B, respectively. zgain , B zgain Taking the most commonly used gray world algorithm as an example, where Rsum , G sum , B sum Respectively represent the sum of the pixel values ​​of the R, G, and B channels of the image captured by the primary camera. There are many types of primary camera white balance algorithms, not limited to the grayscale world algorithm mentioned above. The implementation of the primary camera automatic white balance algorithm is not within the scope of this patent because this method assumes that the white balance gain of the primary camera has been calculated and simply moves the secondary camera closer to the primary camera. In this application, the primary camera white balance gain is calculated by other algorithms in the prior art, so it is used directly.

[0062] S302: Gain value R for the main camera R and B channels zgain , B zgain Taking the reciprocal, we get

[0063] Note it as the main camera white balance point;

[0064] S303: Calculate the main camera white balance point and the calibration points (X z1 , Y z1 ), (X z2 , Y z2 ),...(X zi , Y zi ), there are many ways to calculate distance, such as Euclidean distance, block distance, etc. Taking Euclidean distance as an example, assuming that the calibration point (X z1 ,Y z1 ) and the main camera white balance point (R zl ,B zl ), the Euclidean distance calculation formula between the two is: The formula for calculating block distance is d b1 =|X z1 -R zl |+|Y z1 -B zl |;

[0065] S304: Assign different weights according to the distance, denoted as w1, w2, ...w i There are many ways to assign weights, such as weights in the form of Gaussian functions, weights in the inverse of distances, etc. Assuming that the distances between the white balance point and each calibration point are d1, d2, ...d i , if Gaussian function is used to assign weights, then Among them, A and s are parameters that are adjusted according to the usage, and e is a natural constant. If the inverse of the distance is used to assign weights, then

[0066] S305: Use the weights w1, w2, ...w of each landing point of the main camera i is the secondary camera 1 calibration point (Xc11 ,Y c11 ), (X c12 ,Y c12 )......(X c1i ,Y c1i ) is weighted, and the weighted average of the secondary camera 1 calibration points is obtained to obtain the secondary camera 1 white balance point R cl , B cl , the calculation formula is:

[0067] R c1l =(w1×X c11 +w2×X c12 +...+w i ×X c1i ) / (w1+w2+...+w i )

[0068] B c1l =(w1×Y c11 +w2×Y c12 +...+w i ×Y c1i ) / (w1+w2+...+w i );

[0069] Similarly, the white balance point of the jth shot is R cjl , B cjl , the calculation formula is:

[0070] R cjl =(w1×X cj1 +w2×X cj2 +...+w i ×X cji ) / (w1+w2+...+w i )

[0071] B cjl =(w1×Y cj1 +w2×Y cj2 +...+w i ×Y cji ) / (w1+w2+...+w i );

[0072] S306: Finally calculate the white balance gain of secondary camera 1 - R c1gain1 , B c1gain1 : White balance point R for the second shot c1l , B c1l Take the reciprocal:

[0073]

[0074] Similarly, the white balance gain R of the jth shot cjgain1 , Bcjgain1 , the calculation formula is:

[0075]

[0076] In this method, matching is achieved by calculating the white balance gain of the secondary camera using the primary camera's white balance gain. As described in the background, even if the primary and secondary cameras use the same algorithms, the calculated results can differ significantly in actual visual perception. Therefore, matching is achieved by using the secondary camera's white balance gain as a reference, linking the two rather than calculating them separately.

[0077] To sum up, the process of implementing this method is mainly:

[0078] 1. Calibrate the gray block response results of each camera under different light sources;

[0079] 2. Calculate the white balance gain value of the secondary shot based on the calibration results.

[0080] Since the primary camera has the largest field of view and rich scene content, the other secondary cameras use the primary camera as a reference for matching. Since the primary camera's response to various light sources has been pre-calibrated, the current light source information is inferred based on the primary camera's white balance gain value. This light source information is then passed to the secondary cameras for use. Similarly, the secondary cameras' calibrated Planck curve is used to obtain the final white balance gain.

[0081] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A white balance matching method for a multi-camera system, characterized in that: The method comprises the following steps: S1. Determine the primary and secondary cameras: By reading the FOV parameter (Field of View) in each camera's technical manual, a larger value indicates a wider field of view. Alternatively, place each camera in the system in the same position to capture a scene. Determine the camera with the largest field of view as the primary camera, and name the remaining cameras Secondary 1, Secondary 2, and so on. S2. Calibrate the Planck curve of each camera: Calibrate the main camera and secondary camera 1, secondary camera 2, etc. in a standard light box. The response results to the gray blocks on the color card under different light sources are as follows. The number of calibrated light sources can be determined according to the actual usage. Assume that ct1 to ct8 represent eight light sources. The response results are expressed as the mean value of the R channel in the gray block divided by the mean value of the G channel, and the mean value of the B channel divided by the mean value of the G channel. These are recorded as the calibration point coordinates. The coordinates of the main camera calibration point are marked as (X z1 , Y z1 ), (X z2 , Y z2 ),......(X z1 , Y z1 ), (X zi ,Y zi ) represents the coordinates of the main camera under the i-th light source, X zi >0,Y zi >0; coordinates of the second-photographed calibration point are (X c11 ,Y c11 ), (X c12 ,Y c12 )......(X c1i ,Y c1i ), the coordinates of the secondary j calibration point are marked as (X cj1 ,Y cj1 ), (X cj2 ,Y cj2 )......(X cji ,Y cji ), (X cji ,Y cji ) represents the coordinates of the jth shot under the i-th light source, X cji >0,Y cji >0; S3. Calculate the secondary shot white balance gain: S301: Obtaining the white balance gain of the main camera; S302: Obtaining the white balance point of the main camera; S303: Calculating the distance between the main camera white balance point and each calibration point on the Planck curve; S304: assigning different weights to each point on the Planck curve according to the distance; S305: Obtain the white balance point of the secondary shot after weighting each point on the curve; S306: Calculate the secondary shooting gain.

2. The white balance matching method for a multi-camera system according to claim 1, wherein: The step S3 further comprises: S301: First, the main camera white balance gain is obtained according to the preset automatic white balance algorithm of the main camera. The gain values ​​of the R and B channels are R zgain , B zgain ; S302: Gain value B for the main camera R and B channels zgain , B zgain Taking the reciprocal, we get Note it as the main camera white balance point; S303: Calculate the main camera white balance point and the calibration points (X z1 , Y z1 ), (X z2 , Y z2 ),...(X zi , Y zi ) distance; S304: Assign different weights according to the distance, denoted as w1, w2, ...w i ; S305: Use the weights w1, w2, ...w of each landing point of the main camera i is the secondary camera 1 calibration point (X c11 ,Y c11 ), (X c12 ,Y c12 )......(X c1i ,Y c1i ) is weighted, and the weighted average of the secondary camera 1 calibration points is obtained to obtain the secondary camera 1 white balance point R c1l , B c1l , the calculation formula is: R c1l =(w1×X c11 +w2×X c12 +...+w i ×X c1i ) / (w1+w2+...+w i ) B c1l =(w1×Y c11 +w2×Y c12 +...+w i ×Y c1i ) / (w1+w2+...+w i ) Similarly, the white balance point of the jth shot is R cjl , B cjl , the calculation formula is: R cjl =(w1×X cj1 +w2×X cj2 +...+w i ×X cji ) / (w1+w2+...+w i ) B cjl =(w1×Y cj1 +w2×Y cj2 +...+w i ×Y cji ) / (w1+w2+...+w i ) S306: Finally calculate the white balance gain of secondary camera 1 - R c1gain1 , B c1gain1 , that is, the white balance point R of the second shot c1l , B c1l Take the reciprocal: Similarly, the white balance gain R of the jth shot cjgain1 , B cjgain1 , the calculation formula is:

3. The white balance matching method for a multi-camera system according to claim 1, wherein: In step S303, the distance calculation method includes Euclidean distance and block distance. If Euclidean distance is used, assuming that the calibration point (X z1 ,Y z1 ) and the main camera white balance point (R zl ,B zl ), the Euclidean distance calculation formula between the two is: The formula for calculating block distance is d b1 =|X z1 -R zl |+|Y z1 -B zl |.

4. The white balance matching method for a multi-camera system according to claim 1, wherein: In step S304, the weight distribution method includes the weight in the form of Gaussian function and the weight of the inverse of the distance; assuming that the distance between the white balance point and each calibration point is d1, d2, ... d i , if Gaussian function is used to assign weights, then Among them, A and s are parameters that are adjusted according to the usage, and e is a natural constant. If the inverse of the distance is used to assign weights, then

Images