Camera calibration method and device, electronic equipment and storage medium

Abstract

The invention provides a camera calibration method and device, electronic equipment and a storage medium, and relates to the technical field of camera calibration, and the method comprises the steps: carrying out the white balance and brightness alignment processing of an original image, so as to obtain a processed first image; constructing a target function based on the color correction matrix, wherein the target function at least contains a noise regularization item; and iterating the color correction matrix until the iterated color correction matrix enables a target function value to meet a preset condition, and taking the iterated color correction matrix as a target color correction matrix, the target function value being associated with the first image. Therefore, by constructing the target function based on the color correction matrix and continuously iterating the target function, the optimal color correction matrix can be obtained, and due to the fact that the regularization item representing the noise amplification degree is added to the target function, the influence on the image noise can be better controlled.

Description

Technical Field

[0001] The present disclosure relates to the technical field of camera calibration, and particularly to a camera calibration method, apparatus, electronic device, and storage medium. Background Art

[0002] Due to the differences between the response characteristics of the human eye and those of a camera, a characterization process needs to be introduced in the camera pipeline. Camera characterization refers to correcting individual differences of a camera to eliminate the inherent biases in image capture and processing. Each camera has some specific hardware and software differences, which can cause problems such as color distortion, brightness non-linearity, and noise during image acquisition. The purpose of camera characterization is to calibrate and adjust the camera so that it can capture and process images more accurately, and restore the details and colors of the real scene as much as possible.

[0003] A color correction matrix is required in the camera characterization process to achieve color correction. The accuracy of the color correction matrix is the key to whether the camera can obtain accurate colors that conform to human perception.

[0004] Therefore, how to determine a reliable and accurate color correction matrix to achieve accurate calibration of the camera is an urgent problem to be solved at present. Summary of the Invention

[0005] The present application provides a camera calibration method, apparatus, electronic device, and storage medium, aiming to solve at least one of the technical problems in the related art to some extent.

[0006] The first aspect of the embodiments of the present application provides a camera calibration method, including:

[0007] Performing white balance and brightness alignment processing on the original image to obtain a first processed image;

[0008] Constructing an objective function based on the color correction matrix, where the objective function contains at least a noise regularization term;

[0009] Iterating the color correction matrix until the objective function value satisfies a preset condition after iteration, and then taking the color correction matrix after iteration as the target color correction matrix, where the objective function value is associated with the first image.

[0010] The second aspect of the embodiments of the present application provides a camera calibration apparatus, including:

[0011] A first processing module, configured to perform white balance and brightness alignment processing on the original image to obtain a first processed image;

[0012] A first construction module for constructing an objective function based on a color correction matrix, where the objective function contains at least a noise regularization term;

[0013] An iteration module for iterating the color correction matrix until the objective function value satisfies a preset condition after iteration, and then taking the color correction matrix after iteration as the target color correction matrix, where the objective function value is associated with the first image.

[0014] An embodiment of the third aspect of the present application provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the camera calibration method of the embodiments of the present application.

[0015] An embodiment of the fourth aspect of the present application provides a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to cause the computer to execute the camera calibration method disclosed in the embodiments of the present application.

[0016] In the embodiments of the present disclosure, first, the original image is subjected to white balance and brightness alignment processing to obtain a processed first image, then an objective function is constructed based on the color correction matrix, the objective function contains at least a noise regularization term, and then the color correction matrix is iterated until the objective function value satisfies a preset condition after iteration, and then the color correction matrix after iteration is taken as the target color correction matrix, where the objective function value is associated with the first image. Thus, by constructing an objective function based on the color correction matrix and continuously iterating the objective function, an optimal color correction matrix can be obtained. Since the objective function adds a regularization term representing the degree of noise amplification, the influence on image noise can be better controlled and the color accuracy can be balanced.

[0017] Additional aspects and advantages of the present disclosure will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and / or additional aspects and advantages of the present disclosure will become apparent and be readily understood from the following description of the embodiments in conjunction with the drawings, where:

[0019] Figure 1 is a flowchart of a camera calibration method according to a first embodiment of the present disclosure;

[0020] Figure 2 is a flowchart of a camera calibration method according to a second embodiment of the present disclosure;

[0021] Figure 3 is a schematic flowchart of a camera calibration method provided according to the third embodiment of the present disclosure;

[0022] Figure 4 is a schematic diagram of a camera calibration device according to an embodiment of the present disclosure;

[0023] Figure 5 is a block diagram of an electronic device for implementing the camera calibration method according to an embodiment of the present disclosure. Detailed implementation manners

[0024] Embodiments of the present disclosure will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary only for explaining the present disclosure and should not be construed as a limitation of the present disclosure. On the contrary, the embodiments of the present disclosure include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0025] It should be noted that the existing camera calibration process usually needs to simulate the relevant processing process of the camera pipeline to obtain the RGB (primary colors) values before the color correction module. The camera pipeline calibration process mainly involves: 1. Measuring the standard values of the color card under the target light source; 2. Acquiring the standard color card image under the calibration light source; 3. Successively applying black level subtraction, lens shading correction, and white balance to the original image directly output by the camera; 4. Picking up the average RGB values of each color block of the standard color card; 5. Using the gray color blocks to achieve alignment with the standard RGB brightness; 6. Calculating the average color difference between the values of the color blocks other than the gray color blocks after applying the color correction matrix and the standard RGB in the Lab color space, and then iterating the objective function to obtain the color correction matrix. The problem with this calibration process is that the calibration process does not consider the influence of non-standard gamma functions, and the objective function does not consider the influence of noise, so it cannot reliably balance color accuracy and noise level. Therefore, the present application proposes a camera calibration method, device, electronic device, and storage medium, aiming to at least solve one of the technical problems in the above solutions to a certain extent.

[0026] Among them, it should be noted that the execution subject of the camera calibration method in this embodiment can be a camera calibration device, which can be implemented in a software and / or hardware manner, and this device can be configured in any electronic device with an image acquisition device, such as a mobile phone, a tablet, a camera, which is not limited herein.

[0027] In the embodiments of the present disclosure, the "camera calibration device" will be used as the execution subject to execute the camera calibration method, which will be simply referred to as the "device" hereinafter for description, and this is not limited herein.

[0028] Figure 1 is a schematic flowchart of a camera calibration method provided according to the first embodiment of the present disclosure. As Figure 1 shown, the method includes:

[0029] S101: Perform white balance and brightness alignment processing on the original image to obtain a first processed image.

[0030] Among them, the original image is the initial image of the target color card to be processed, which can be the initial image data of the target color card collected by the camera sensor and is also called an unprocessed image (RAW image).

[0031] Among them, the target color card can be a pre-selected standard color chart. Among them, the standard color chart is a tool for color measurement and calibration, which usually consists of a series of color patches with known colors. The color values of these color patches are accurately measured and calibrated and can be used as reference standards for comparing and correcting the color performance of image acquisition devices.

[0032] As an example, the standard ColorChecker color chart can be selected as the target color card.

[0033] Among them, the first image can be the image obtained after preprocessing the original image.

[0034] Optionally, after the image acquisition device acquires the original image, preprocessing can be performed on the original image, such as black level subtraction, lens shading correction, white balance and brightness alignment processing, etc., which are not limited here.

[0035] It should be noted that camera characterization is generally divided into two independent parts, namely color correction and white balance. Among them, white balance mainly refers to adjusting the white point in the image to real white to achieve color constancy, while color correction is to eliminate the inherent color deviation of the camera sensor.

[0036] Among them, white balance is a method for correcting the color temperature deviation in the image. When the light source of the scene photographed by the camera changes, it will cause the color temperature of the image to change, making white no longer look like real white. The role of white balance is to eliminate this color temperature deviation by adjusting the gains of the three basic color channels (red, green, blue) in the image, so that white looks real and pure in the image.

[0037] It should be noted that images captured under different scenarios and lighting conditions may have different brightness levels. Brightness alignment can make the brightness of these images consistent, thereby improving the overall consistency and visual perception of the images. In some cases, details in the images may be difficult to observe due to too low or too high brightness. Through brightness alignment processing, the brightness distribution of the images can be balanced, making the details more clearly visible.

[0038] S102: Construct an objective function based on the color correction matrix, where the objective function contains at least a noise regularization term.

[0039] It should be noted that the color correction matrix is used to correct the color of the first image, so that the corrected first image can accurately conform to human eye perception.

[0040] Among them, the objective function can be a function containing the target to be optimized. In the embodiments of the present disclosure, the color correction matrix can be used as the target to be optimized, that is, taking the color correction matrix as a variable, continuously adjusting the parameters of the color correction matrix, changing the value of the objective function, so as to achieve the purpose of optimizing the target.

[0041] In the embodiments of the present disclosure, the design of the objective function takes into account the influence of noise, so a noise regularization term is added. Among them, the noise regularization term can be associated with the color correction matrix. This noise regularization term can be only related to the color correction matrix.

[0042] Optionally, the objective function can also contain a color difference error term. In camera characterization calibration, the color difference error term can be used to measure the color shift of the camera on different color channels.

[0043] The construction process of the noise regularization term is described below. Among them, the image noise can be represented by a covariance matrix. Based on the noise propagation theory, the covariance matrix ∑ corresponding to the image after applying the color correction matrix to the first image y can be calculated using the following formula: ∑ y ≈A∑ x A T .

[0044] Among them, ∑ x is the covariance matrix of the first image. Taking the three-channel RGB as an example, ∑ x the diagonal elements of represent variances, and the remaining elements represent covariances.

[0045] Among them, A is the color correction matrix, which can be expressed as:

[0046] It should be noted that, in order to determine the influence of the color correction matrix on noise, the embodiments of the present disclosure have made a non-generality simplification. Since the covariance of an image is usually significantly smaller than the variance, Σ can be ignored. x Among them, the non-diagonal elements are ignored, and when considering the influence of noise, only the change of the diagonal of the covariance matrix Σ y is concerned. Generally, the noise levels of the three channels of the original image are basically the same, that is, σ 11 ≈σ 22 ≈σ 33 .

[0047]

[0048] When ignoring the non-diagonal elements of the above Σ x and Σ y , and σ 11 ≈σ 22 ≈σ 33 , the average value Ω of the amplification ratio of the noise variances of the three channels of the image can be considered as the noise regularization term:

[0049]

[0050] Furthermore, the chromatic aberration error term and the noise regularization term can be combined to construct an objective function.

[0051] For example, the objective function can be where λ is the coefficient of the noise regularization term, A is the color correction matrix, N is the number of color patches in the color card, w i is the weight corresponding to the i-th color patch, and ΔE i is the chromatic aberration corresponding to the i-th color patch.

[0052] S103: Iterate the color correction matrix until the objective function value satisfies a preset condition for the iterated color correction matrix, and then use the iterated color correction matrix as the target color correction matrix.

[0053] Among them, the objective function value is the function value corresponding to the objective function and any color correction matrix.

[0054] Among them, the objective function value is associated with the first image, and the first image can affect the chromatic aberration error term value in the objective function value.

[0055] Among them, the preset condition can be the condition that makes the objective function optimal. As an implementation method, the embodiments of the present disclosure can consider the objective function to be the optimal objective function when the objective function takes the minimum value, and use the color correction matrix that can make the objective function take the minimum value as the target color correction matrix.

[0056] Specifically, after each iteration of the color correction matrix, the first image can be processed based on the iterated color correction matrix and converted to the LAB color space, so as to determine the chromatic aberration error term value corresponding to the first image after color correction in the LAB color space. Combining the noise regularization term value and the chromatic aberration error term value, the objective function value can be determined.

[0057] Specifically, after determining the target color correction matrix, the parameters of the camera can be configured, that is, the data of the target color correction matrix is loaded into the image processing parameters of the camera.

[0058] Optionally, if any color correction matrix can make the objective function obtain the minimum value, then this any color correction matrix can be used as the target color correction matrix to realize the characterization calibration of the camera.

[0059] Among them, after determining the target color correction matrix, the first image after preprocessing is color-corrected using the target color correction matrix.

[0060] In the embodiment of the present disclosure, first, the original image is subjected to white balance and brightness alignment processing to obtain the processed first image, then an objective function is constructed based on the color correction matrix, the objective function at least contains a noise regularization term, and then the color correction matrix is iterated until the objective function value satisfies a preset condition after iteration, then the iterated color correction matrix is used as the target color correction matrix, where the objective function value is associated with the first image. Thus, by constructing an objective function based on the color correction matrix and continuously iterating the objective function, an optimal color correction matrix can be obtained. Since the objective function adds a regularization term representing the degree of noise amplification, the influence on image noise can be better controlled and the color accuracy can be balanced.

[0061] Figure 2 It is a schematic flowchart of a camera calibration method according to the second embodiment of the present disclosure. As Figure 2 shown, the method includes:

[0062] S201: Determine the original image corresponding to the target color card and each reference color block in the target color card.

[0063] Specifically, the target color card can be photographed by the camera, so that the original image of the target color card can be collected.

[0064] Among them, the target color card can be a pre-selected standard color chart. Among them, the standard color chart is a tool for color measurement and calibration, and it is usually composed of a series of color patches with known colors. The color values of these color patches are accurately measured and calibrated, and can be used as a reference standard for comparing and correcting the color performance of image acquisition devices.

[0065] As an example, the standard ColorChecker color chart can be selected as the target color card.

[0066] Among them, the reference color patches can be specific color patches used in the preprocessing process of the camera's response values. Among them, during the white balance process, the reference color patches are used as reference points. The camera analyzes the color information of the reference color patches and adjusts the color temperature and hue of the image according to their measured values, so as to achieve a more accurate color performance.

[0067] For example, in the standard ColorChecker color chart, the 19th to 24th color patches are gray scale color patches. The 20th to 23rd color patches can be selected as the reference color patches for preprocessing, or the 20th to 22nd color patches can also be selected as the reference color patches, which is not limited here. In the standard ColorChecker color chart, the 19th to 24th color patches are respectively: pure black, dark gray, medium gray, light gray, light gray and pure white.

[0068] As an implementation method, the 19th and 24th color patches can be removed, that is, pure black and pure white are removed, and the 20th to 23rd color patches are retained as the reference color patches.

[0069] In the embodiments of the present disclosure, before determining each reference color patch in the target color card, the original image is first subjected to black level removal processing and lens shadow correction to obtain the original image in Bayer format.

[0070] S202: Determine the white balance conversion matrix based on the color channel values of each reference color patch.

[0071] Optionally, the gain parameters can be calculated first based on a preset gain calculation model and the color channel values corresponding to each reference color patch in each color channel. Then, based on the gain parameters and the color channel values corresponding to each color channel, the diagonal parameters constituting the white balance conversion matrix can be calculated. Then, the white balance conversion matrix can be formed based on each diagonal parameter.

[0072] Among them, the white balance conversion matrix can be in the form of a diagonal matrix, including multiple diagonal elements, that is, diagonal parameters.

[0073] Among them, the preset gain calculation model can be a pre-constructed model for calculating gain parameters.

[0074] Specifically, the gain parameter can be calculated based on the color channel values of the three color channels (red, green, and blue) corresponding to each reference color patch.

[0075] Among them, the preset gain calculation formula can be: max_gain = max([r / g, b / g, 1.0]).

[0076] Among them, max_gain is the gain parameter.

[0077] As an implementation method, when calculating r / g and b / g, it is first necessary to determine the color channel values corresponding to the three color channels of each reference color patch. For example, if the reference color patches include s1, s2, and s3, then it is necessary to determine the three color channel values r1, g1, and b1 corresponding to s1 for the red, green, and blue color channels respectively. Similarly, the three color channel values r2, g2, and b2 corresponding to s2, and the three color channel values r3, g3, and b3 corresponding to s3 also need to be obtained.

[0078] Furthermore, the r1 / g1 and b1 / g1 corresponding to the s1 reference color patch, the r2 / g2 and b2 / g2 corresponding to the s2 reference color patch, and the r3 / g3 and b3 / g3 corresponding to the s3 reference color patch can be calculated. After that, the average of r1 / g1, r2 / g2, and r3 / g3 can be taken to obtain r / g corresponding to the reference color patches s1, s2, and s3, and the average of b1 / g1, b2 / g2, and b3 / g3 can be taken to obtain b / g corresponding to the reference color patches s1, s2, and s3.

[0079] It should be noted that the above example is only for illustrative purposes and does not limit the present disclosure.

[0080] Furthermore, the diagonal parameters k0, k1, and k2 in the white balance conversion matrix can be calculated through the following calculation formula.

[0081]

[0082] k1 = max_gain

[0083]

[0084] Furthermore, after obtaining the specific values of the diagonal parameters k0, k1, and k2, the white balance conversion matrix can be constructed as follows:

[0085]

[0086] S203: Based on the white balance conversion matrix, process the original image to obtain a second image.

[0087] Among them, the second image is the image obtained after the white balance conversion matrix acts on the original image.

[0088] Specifically, the white balance conversion matrix can be applied to the original image through the following formula:

[0089]

[0090] Among them, R, G, and B are the red channel value, green channel value, and blue channel value corresponding to each pixel point in the original image respectively. Among them, R′, G′, and B′ are the red channel value, green channel value, and blue channel value corresponding to each pixel point in the second image respectively.

[0091] S204: Perform brightness alignment processing on the second image to obtain the first image.

[0092] Specifically, according to the first color channel value corresponding to any color channel of each reference color block in the second image, and the color block standard value corresponding to each reference color block and any color channel, then according to the first color channel value and color block standard value corresponding to each reference color block, the first brightness correction coefficient corresponding to each reference color block can be calculated.

[0093] Furthermore, the average value of the first brightness correction coefficients corresponding to each reference color block is used as the second brightness correction coefficient, and then based on the second brightness correction coefficient, the second image can be subjected to brightness alignment processing to obtain the first image.

[0094] In the embodiments of the present disclosure, when performing brightness alignment processing, the selected color channel can be the green color channel. The following will be described with the green color channel, and this is not limited herein.

[0095] Among them, the color block standard value can be the color channel average value corresponding to each pixel point in any color block and the green color channel. The color block standard values corresponding to each reference color block are usually different. The color block standard value is a determined value obtained through instrument measurement and conversion.

[0096] When determining the first color channel value corresponding to any reference color block, first, the average value of the green color channel values corresponding to each pixel point of any reference color block in the second image and the green color channel can be determined, and then this average value of the green color channel values can be used as the first color channel value corresponding to the any reference color block.

[0097] For example, for the reference color block s1 with N pixel points, then the respective color channel values G1, G2, G3...G corresponding to the N pixel points and the green channel in the second image can be used. NThe average of is taken as the first color channel value G1 corresponding to the reference color block s1. If the reference color block also includes s2, s3, and s4, the first color channel values G2, G3, and G4 corresponding to s2, s3, and s4 can be determined similarly.

[0098] The reference color blocks s1, s2, s3, and s4 have corresponding color block standard values G1′, G2′, G3′, and G4′ on the green color channel.

[0099] As an example, the first brightness correction coefficient ratio1 corresponding to the reference color block s1 may be obtained by the following formula:

[0100]

[0101] Similarly, the first brightness correction coefficients ratio2, ratio3, ratio4 corresponding to the reference color blocks s2, s3, s4 can be determined. Then, the average value ratio' of ratio1 and ratio2, ratio3, ratio4 can be used as the second brightness correction coefficient.

[0102] Furthermore, the brightness alignment of the second image can be performed using the following formula:

[0103]

[0104] Among them, R′, G′ and B′ are the red channel value, green channel value and blue channel value corresponding to each pixel in the second image, respectively, and R″, G″, B″ are the red channel value, green channel value and blue channel value corresponding to each pixel in the first image, respectively.

[0105] It should be noted that, before the iterative process begins, the color correction matrix may be initialized.

[0106] S205: Constructing a color difference error term based on the weight value assigned to each color block in the target color card.

[0107] As a possible implementation method, in the embodiment of the present disclosure, when constructing the color difference error term, it can be constructed based on the average color difference of each color block of the target color card in the Lab color space. The specific calculation formula is as follows:

[0108]

[0109] Where N is the number of all color blocks on the target color card, and i is the serial number of any color block in the target color card. i is the weight pre-assigned to the color block i. It should be noted that different weights can be set for the color blocks to emphasize more important color blocks.

[0110] Among them, when calculating the color difference ΔE corresponding to the color block i i , the color difference formula can be used for calculation. There are many existing color difference formulas. For example, the CIE DE2000 color difference formula can be selected, which is not limited here.

[0111] S206: Construct a noise regularization term based on the color correction matrix containing unknown parameters.

[0112] The construction process of the noise regularization term is described below. Among them, the image noise can be represented by the covariance matrix. Based on the noise propagation theory, the covariance matrix ∑ corresponding to the image after applying the color correction matrix to the first image y can be calculated using the following formula: ∑ y ≈A∑ x A T .

[0113] Among them, ∑ x is the covariance matrix of the first image. Taking the three-channel RGB as an example, ∑ x the diagonal elements represent variances, and the remaining elements represent covariances.

[0114] Among them, A is the color correction matrix, which can be expressed as: b0, b1, b2, b3, b4, b5, b6, b7, b8 in the color correction matrix are the unknown parameters it contains. It should be noted that in the subsequent steps, the unknown parameters in the color correction matrix can be continuously assigned and iterated until the color correction matrix that makes the objective function meet the preset conditions is finally obtained.

[0115] It should be noted that in order to determine the influence of the color correction matrix on the noise, the embodiments of the present disclosure have made a non-generality simplification. Since the covariance of the general image is significantly smaller than the variance, the non-diagonal elements in ∑ x can be ignored. At the same time, when considering the influence of noise, only the change of the diagonal of the covariance matrix ∑ y is concerned. Generally, the noise levels of the three channels of the original image are basically the same, that is, σ 11 ≈σ 22 ≈σ 33 .

[0116]

[0117] After ignoring the non-diagonal elements of the above ∑ x and ∑ y , and σ 11 ≈σ 22 ≈σ 33In this case, the average value Ω of the amplification ratio of the noise variances of the three channels of the image can be considered as the noise regularization term:

[0118]

[0119] S207: Based on the noise regularization term and the color difference error term, construct an objective function.

[0120] Among them, the objective function is a function that takes the minimum value.

[0121] Specifically, the objective function can be constructed by combining the color difference error term and the noise regularization term.

[0122] For example, the objective function can be where λ is the coefficient of the noise regularization term, A is the color correction matrix, N is the number of color patches in the color card, w i is the weight corresponding to the i-th color patch, and ΔE i is the color difference corresponding to the i-th color patch.

[0123] S208: Iterate the parameters in the color correction matrix to obtain different color correction matrices.

[0124] Specifically, the parameters in the color correction matrix can be continuously iteratively updated, so that different color correction matrices can be determined.

[0125] S209: Based on the first image, determine the color difference error term value and the noise regularization term value corresponding to each color correction matrix.

[0126] It should be noted that since the noise regularization term value is directly related to the color correction matrix, the noise regularization term value can be calculated based on the following formula:

[0127]

[0128] Specifically, when obtaining the color difference error term value, the first image can be converted to the LAB color space according to each color correction matrix, and then the average color difference can be calculated through the formula as the color difference error term value, which is not limited here.

[0129] S210: According to the color difference error term value and the noise regularization term value corresponding to each color correction matrix, determine the objective function value corresponding to each color correction matrix.

[0130] Specifically, through the following formula, based on the color difference error term value and the noise regularization term value Ω corresponding to each color correction matrix, determine the objective function value corresponding to each color correction matrix:

[0131]

[0132] Among them, λ is the regularization term coefficient.

[0133] S211: If the objective function value corresponding to any color correction matrix is the minimum among all objective function values, then determine any color correction matrix such that the objective function meets the preset conditions, and take any color correction matrix as the target color correction matrix.

[0134] It should be noted that if the objective function value corresponding to any color correction matrix is the minimum among all objective function values, it means that this any color correction matrix can make both the color difference error term value and the noise regularization term value relatively small.

[0135] Among them, the noise regularization term can effectively control the complexity of the correction matrix and avoid overfitting. The color difference error term can reflect the difference degree between the color measured by the camera and the reference color. Therefore, the smaller the sum of these two indicators, the better the matching effect of the target color correction matrix to the reference color, and the more accurately it can restore the reference color.

[0136] Specifically, the smaller the sum of the noise regularization term and the color difference error term, it means that the target color correction matrix can minimize the complexity of the correction matrix while minimizing the gap between the color measured by the camera and the reference color to the greatest extent, thereby better adapting to the characteristics of the camera and achieving a more accurate and reliable color correction effect.

[0137] In the embodiments of the present disclosure, first, determine the original image corresponding to the target color card and each reference color block in the target color card. Then, based on the color channel values of each reference color block, determine the white balance conversion matrix. Then, based on the white balance conversion matrix, process the original image to obtain a second image. Then, perform brightness alignment processing on the second image to obtain a first image. Based on the weight value assigned to each color block in the target color card, construct a color difference error term. Based on the color correction matrix containing unknown parameters, construct a noise regularization term. Based on the noise regularization term and the color difference error term, construct an objective function. Among them, the objective function is a minimum value function. Iterate the parameters in the color correction matrix to obtain different color correction matrices. Based on the first image, determine the color difference error term value and the noise regularization term value corresponding to each color correction matrix. According to the color difference error term value and the noise regularization term value corresponding to each color correction matrix, determine the objective function value corresponding to each color correction matrix. If the objective function value corresponding to any color correction matrix is the minimum among all objective function values, then determine any color correction matrix such that the objective function meets the preset conditions. Thus, an objective function based on the color difference error term and the noise regularization term can be constructed, which can better control the influence on image noise.

[0138] Figure 3It is a schematic flowchart of a camera calibration method provided according to the third embodiment of the present disclosure. As Figure 3 shown, the method includes:

[0139] S301: Determine the original image corresponding to the target color card and each reference color block in the target color card.

[0140] S302: Determine the white balance conversion matrix based on the color channel values of each reference color block.

[0141] S303: Process the original image based on the white balance conversion matrix to obtain a second image.

[0142] It should be noted that the specific implementation manners of steps S301 - S303 can refer to the above embodiments and will not be elaborated here.

[0143] S304: Apply the non - standard gamma curve and the standard inverse gamma curve to the second image successively.

[0144] It should be noted that in the camera pipeline, generally after passing through the color correction module, the gamma function will be applied to convert linear data to non - linear data, and then the display end will apply the inverse transformation to convert the non - linear data back to linear data. If the two operations are inverse to each other, the color will not change.

[0145] Among them, the gamma curve is used to convert linear RGB to non - linear RGB, and the inverse gamma curve is used to convert non - linear RGB back to linear RGB.

[0146] In some cases, the gamma function on the camera pipeline does not adopt the standard form but has certain adjustments for brightness and contrast. Therefore, in this actual process, it is necessary to consider in advance during the color matrix calibration process. Apply the non - standard gamma curve to the camera response, and then apply the standard inverse gamma curve to simulate the above process. At the same time, the brightness alignment also needs to be adjusted to adapt to this change.

[0147] S305: According to the first color channel value corresponding to any color channel of each reference color block in the second image, and the color block standard value corresponding to each reference color block and any color channel.

[0148] S306: Calculate the first brightness correction coefficient corresponding to each reference color block according to the first color channel value and the color block standard value corresponding to each reference color block.

[0149] S307: Take the average value of the first brightness correction coefficients corresponding to each reference color block as the second brightness correction coefficient.

[0150] S308: Perform brightness alignment processing on the second image based on the second brightness correction coefficient to obtain a first image.

[0151] S309: Construct an objective function based on the color correction matrix, where the objective function contains at least a noise regularization term.

[0152] S310: Iterate the color correction matrix until the objective function value satisfies a preset condition after iteration, and then use the color correction matrix after iteration as the target color correction matrix, where the objective function value is associated with the first image.

[0153] It should be noted that the specific implementation manners of steps S305 - S310 can refer to the above embodiments and will not be elaborated here.

[0154] In the embodiments of the present disclosure, first, determine the original image corresponding to the target color card and each reference color block in the target color card. Then, based on the color channel values of each reference color block, determine the white balance conversion matrix. Then, based on the white balance conversion matrix, process the original image to obtain a second image. Then, successively apply the non - standard gamma curve and the standard inverse gamma curve to the second image. Then, according to the first color channel value corresponding to any color channel of each reference color block in the second image and the standard value of the color block corresponding to each reference color block and the any color channel, then calculate the first brightness correction coefficient corresponding to each reference color block according to the first color channel value and the color block standard value corresponding to each reference color block. Then, use the average value of the first brightness correction coefficients corresponding to each reference color block as the second brightness correction coefficient. Finally, based on the second brightness correction coefficient, perform brightness alignment processing on the second image to obtain the first image. Then, construct an objective function based on the color correction matrix, where the objective function contains at least a noise regularization term. Finally, iterate the color correction matrix until the objective function value satisfies a preset condition after iteration, and then use the color correction matrix after iteration as the target color correction matrix. Thus, considering the influence of the non - standard gamma curve in the calibration process, it is more in line with the actual camera pipeline processing process, the calibration result is more accurate, and certain adjustments are made in the brightness alignment during the pre - processing process to adapt to this change.

[0155] Figure 4 is a schematic diagram of a camera calibration device according to another embodiment of the present disclosure. As Figure 4 shown, the camera calibration device 400 includes:

[0156] A first processing module 410, configured to perform white balance and brightness alignment processing on the original image to obtain a processed first image;

[0157] A first construction module 420, configured to construct an objective function based on the color correction matrix, where the objective function contains at least a noise regularization term;

[0158] An iterative module 430 is configured to iterate the color correction matrix until the objective function value meets a preset condition after iteration. Then, the iterated color correction matrix is used as the target color correction matrix, where the objective function value is associated with the first image.

[0159] Optionally, the first processing module includes:

[0160] A first determination unit configured to determine the original image corresponding to the target color card and each reference color block in the target color card;

[0161] A second determination unit configured to determine a white balance conversion matrix based on the color channel values of each reference color block;

[0162] A first processing unit configured to process the original image based on the white balance conversion matrix to obtain a second image;

[0163] A second processing unit configured to perform brightness alignment processing on the second image to obtain the first image.

[0164] Optionally, the first determination unit is further configured to:

[0165] Perform black level removal processing and lens shading correction on the original image.

[0166] Optionally, the second determination unit is specifically configured to:

[0167] Calculate gain parameters based on a preset gain calculation model and the color channel values of each reference color block corresponding to each color channel;

[0168] Calculate each diagonal parameter constituting the white balance conversion matrix based on the gain parameters and the color channel values corresponding to each color channel;

[0169] Form the white balance conversion matrix based on each diagonal parameter.

[0170] Optionally, the second processing unit is specifically configured to:

[0171] According to the first color channel value corresponding to any color channel of each reference color block in the second image and the standard value of the color block corresponding to each reference color block and the any color channel;

[0172] Calculate the first brightness correction coefficient corresponding to each reference color block according to the first color channel value and the color block standard value corresponding to each reference color block;

[0173] Take the mean of the first brightness correction coefficients corresponding to each of the reference color patches as the second brightness correction coefficient;

[0174] Based on the second brightness correction coefficient, perform brightness alignment processing on the second image to obtain the first image.

[0175] Optionally, the second processing unit is further configured to:

[0176] Apply the non-standard gamma curve and the standard inverse gamma curve to the second image successively.

[0177] Optionally, the first construction module is specifically configured to:

[0178] Based on the weight value assigned to each color patch in the target color card, construct a color difference error term;

[0179] Based on the color correction matrix including unknown parameters, construct the noise regularization term;

[0180] Based on the noise regularization term and the color difference error term, construct the target function, where the target function is a function that takes the minimum value.

[0181] Optionally, the iteration module is specifically configured to:

[0182] Iterate the parameters in the color correction matrix to obtain different color correction matrices;

[0183] Based on the first image, determine the color difference error term value and the noise regularization term value corresponding to each of the color correction matrices;

[0184] According to the color difference error term value and the noise regularization term value corresponding to each of the color correction matrices, determine the target function value corresponding to each of the color correction matrices;

[0185] If the target function value corresponding to any one color correction matrix is the minimum value among all the target function values, then determine that the any one color correction matrix makes the target function satisfy the preset condition.

[0186] In the embodiments of the present disclosure, first, the original image is subjected to white balance and brightness alignment processing to obtain a first processed image. Then, an objective function is constructed based on a color correction matrix, and the objective function contains at least a noise regularization term. Then, the color correction matrix is iterated until the objective function value satisfies a preset condition after iteration, and the iterated color correction matrix is used as the target color correction matrix, where the objective function value is associated with the first image. Thus, by constructing an objective function based on the color correction matrix and continuously iterating the objective function, an optimal color correction matrix can be obtained. Since the objective function adds a regularization term representing the degree of noise amplification, the influence on image noise can be better controlled and the color accuracy can be balanced.

[0187] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0188] Figure 5 A block diagram of an exemplary computer device suitable for implementing the embodiments of the present application is shown. Figure 5 The computer device 12 shown is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present application.

[0189] As Figure 5 shown, the computer device 12 is presented in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0190] The bus 18 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the multiple bus structures. For example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnection (PCI) bus.

[0191] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0192] Memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (Random Access Memory; hereinafter referred to as: RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 can be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, commonly referred to as a "hard disk drive").

[0193] Although Figure 5 not shown in, a disk drive for reading and writing removable non-volatile disks (such as "floppy disks") and an optical disk drive for reading and writing removable non-volatile optical disks (such as: Compact Disc Read Only Memory; hereinafter referred to as: CD-ROM), Digital Video Disc Read Only Memory; hereinafter referred to as: DVD-ROM) or other optical media) may be provided. In these cases, each drive can be connected to bus 18 through one or more data media interfaces. Memory 28 may include at least one program product having a set (such as at least one) of program modules configured to perform the functions of the embodiments of the present application.

[0194] A program / utility 40 having a set (at least one) of program modules 42 can be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment. Program modules 42 generally perform the functions and / or methods described in the embodiments of the present application.

[0195] The computer device 12 can also communicate with one or more external devices 14 (such as keyboards, pointing devices, monitors 24, etc.), and can also communicate with one or more devices that enable users to interact with the computer device 12, and / or communicate with any device that enables the computer device 12 to communicate with one or more other computing devices (such as network cards, modems, etc.). Such communication can be carried out through the input / output (I / O) interface 22. Moreover, the computer device 12 can also communicate with one or more networks (such as a Local Area Network (LAN), a Wide Area Network (WAN), and / or a public network, such as the Internet) through the network adapter 20. As shown in the figure, the network adapter 20 communicates with other modules of the computer device 12 through the bus 18. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in combination with the computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0196] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the camera calibration method mentioned in the foregoing embodiments.

[0197] Those skilled in the art will readily think of other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include common general knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and embodiments are only regarded as exemplary, and the true scope and spirit of the present application are pointed out by the following claims.

[0198] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

[0199] It should be noted that in the description of the present application, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0200] Any process or method description, whether in a flowchart or otherwise described herein, can be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a specific logical function or process. The scope of the preferred embodiments of the present application includes additional implementations, where the functions may be performed in a substantially simultaneous manner or in the reverse order according to the functions involved, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0201] It should be understood that various parts of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application specific integrated circuits with appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0202] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried out in implementing the above-described embodiment methods can be completed by instructing relevant hardware through a program. The said program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0203] In addition, in each embodiment of the present application, the functional units can be integrated into one processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. When the above integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage media mentioned above can be read-only memories, magnetic disks, optical discs, etc.

[0204] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0205] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A method for camera calibration, characterized in that, Including: Performing white balance and brightness alignment processing on the original image to obtain a processed first image; Constructing an objective function based on a color correction matrix, where the objective function at least contains a noise regularization term; Iterating the color correction matrix until the objective function value satisfies a preset condition after iteration, and then taking the color correction matrix after iteration as the target color correction matrix, where the objective function value is associated with the first image.

2. The method according to claim 1, characterized in that, The performing white balance and brightness alignment processing on the original image to obtain a processed first image includes: Determining the original image corresponding to the target color card and each reference color block in the target color card; Determining a white balance conversion matrix based on the color channel values of each reference color block; Processing the original image based on the white balance conversion matrix to obtain a second image; Performing brightness alignment processing on the second image to obtain the first image.

3. The method according to claim 2, wherein Before determining the original image corresponding to the target color card and each reference color block in the target color card, it further includes: Performing black level removal processing and lens shadow correction on the original image.

4. The method according to claim 3, wherein The determining a white balance conversion matrix based on the color channel values of each reference color block includes: Calculating gain parameters based on a preset gain calculation model and the color channel values of each reference color block corresponding to each color channel; Calculating each diagonal parameter constituting the white balance conversion matrix based on the gain parameters and the color channel values corresponding to each color channel; Forming the white balance conversion matrix based on each diagonal parameter.

5. The method according to claim 2, wherein The performing brightness alignment processing on the second image to obtain the first image includes: According to the first color channel value corresponding to any color channel of each reference color block in the second image and the color block standard value corresponding to each reference color block and the any color channel; Calculating a first brightness correction coefficient corresponding to each reference color block according to the first color channel value and the color block standard value corresponding to each reference color block; Taking the mean value of the first brightness correction coefficients corresponding to each reference color block as a second brightness correction coefficient; Performing brightness alignment processing on the second image based on the second brightness correction coefficient to obtain the first image.

6. The method according to claim 5, characterized in that, Before according to the first color channel value corresponding to any color channel of each reference color block in the second image and the color block standard value corresponding to each reference color block and the any color channel, it further includes: Successively applying a non-standard gamma curve and a standard inverse gamma curve to the second image.

7. The method according to claim 1, characterized in that, The constructing an objective function based on a color correction matrix includes: Constructing a color difference error term based on the weight value assigned to each color block in the target color card; Constructing the noise regularization term based on the color correction matrix containing unknown parameters; Constructing the objective function based on the noise regularization term and the color difference error term, where the objective function is a minimum value function.

8. The method according to claim 7, wherein The iterating the color correction matrix until the objective function value satisfies a preset condition includes: Iterate the parameters in the color correction matrix to obtain different color correction matrices; Based on the first image, determine the chromatic aberration error term value and the noise regularization term value corresponding to each color correction matrix; According to the chromatic aberration error term value and the noise regularization term value corresponding to each color correction matrix, determine the objective function value corresponding to each color correction matrix; If the objective function value corresponding to any color correction matrix is the minimum among all the objective function values, then determine that the any color correction matrix makes the objective function satisfy a preset condition.

9. A camera calibration device, characterized in that, It includes: A first processing module, configured to perform white balance and brightness alignment processing on the original image to obtain a processed first image; A first construction module, configured to construct an objective function based on the color correction matrix, and the objective function at least contains a noise regularization term; An iteration module, configured to iterate the color correction matrix until the objective function value satisfies a preset condition after iteration, and then use the color correction matrix after iteration as the target color correction matrix, where the objective function value is associated with the first image.

10. An electronic device, including: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1-8.

11. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.

Images