Color restoration method suitable for multiple exposure mechanism

Abstract

The present invention provides a color restoration method suitable for a multiple-exposure mechanism, comprising: obtaining Bayer images at different exposure levels; obtaining Bayer images at different exposure levels after applying white balance gain; obtaining a first matrix at different exposure levels; obtaining tristimulus value images under equal-energy light sources at different exposure levels; obtaining fused tristimulus value images under equal-energy light sources; obtaining a second matrix; obtaining an image after applying a color correction matrix; and obtaining an image after applying a gamma curve. The present invention can optimize the problem of color cast in the final image caused by inconsistent white balance gains corresponding to different regions in a scene under mixed light sources. Furthermore, the present invention simultaneously considers the gamma curve and target saturation during the calculation process, and the calculated color correction matrix matches the gamma curve and target saturation, effectively resolving the problem of abnormal color restoration under high dynamic range imaging conditions.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a color restoration method applicable to a multiple exposure mechanism. Background Art

[0002] In high dynamic range scenarios, in order to collect as much information as possible, a multiple exposure mechanism is used to capture multiple frames of images and fuse them together to obtain an image that covers the entire dynamic range. In this scenario, two or more light sources with different color temperatures may appear.

[0003] In this case, due to the limitations of the white balance gain and color correction matrix application method, the traditional method may cause color cast in the color restoration of local areas. Summary of the Invention

[0004] The purpose of the present invention is to provide a color restoration method suitable for a multiple exposure mechanism, which takes into account the gamma curve and target saturation. The calculated color correction matrix can match the gamma curve and target saturation, and can effectively solve the problem of abnormal color restoration under high dynamic range imaging conditions.

[0005] To solve the above technical problems, the present invention provides a color restoration method applicable to a multiple exposure mechanism, comprising the following steps:

[0006] Perform multiple exposures based on a predetermined exposure mechanism to obtain Bayer images at different exposure levels;

[0007] determining white balance gains of the Bayer images at different exposure levels to obtain Bayer images after applying the white balance gains at different exposure levels;

[0008] Based on the Bayer images after applying the white balance gain at different exposure levels and the corresponding ambient light source color temperature, a first matrix at different exposure levels is obtained, where the first matrix is ​​a matrix of tristimulus values ​​under an equal energy light source converted from the Bayer image values ​​after applying the white balance gain at the ambient light source color temperature;

[0009] Obtaining tristimulus value images under equal energy light sources at different exposure levels based on the Bayer images after applying the white balance gain at different exposure levels and the first matrix;

[0010] fusing the tristimulus value images under the equal energy light source at different exposure levels to obtain a fused tristimulus value image under the equal energy light source;

[0011] Determine a gamma curve and target saturation suitable for the current scene to obtain a second matrix, where the second matrix is ​​a matrix for converting tristimulus values ​​to RGB values ​​under an equal energy light source;

[0012] Obtaining an image after applying a color correction matrix based on the fused tristimulus value image under the equal energy light source and the second matrix; and

[0013] An image after applying the gamma curve is obtained based on the image after applying the color correction matrix and the gamma curve.

[0014] Optionally, the method for calculating the first matrix based on the Bayer images after applying the white balance gain at different exposure levels and the corresponding color temperature of the ambient light source includes:

[0015] Obtaining tristimulus values ​​of the image under the color temperature of the ambient light source, and simultaneously obtaining a Bayer image value after applying a white balance gain under the color temperature of the ambient light source;

[0016] Calculate and convert the tristimulus values ​​under the color temperature of the ambient light source into the Bell image value matrix after applying the white balance gain;

[0017] Calculate the conversion matrix of the Bayer image value after applying the white balance gain under the color temperature of the ambient light source to the tristimulus value matrix under the color temperature of the ambient light source, and simultaneously obtain the conversion matrix of the tristimulus value under the color temperature of the ambient light source to the tristimulus value under the D65 light source; and

[0018] The first matrix is ​​calculated.

[0019] Optionally, before obtaining the first matrix, the method further includes calibrating the Bayer image value after applying white balance gain under a commonly used standard light source to convert it into a tristimulus value matrix under an equal energy light source.

[0020] Optionally, a method for calculating the first matrix includes:

[0021] A first initial matrix is ​​calculated based on a matrix of converting the Bayer image values ​​after applying the white balance gain at the ambient light color temperature to the tristimulus values ​​and a matrix of converting the tristimulus values ​​at the ambient light color temperature to the tristimulus values ​​under a D65 light source, where the first initial matrix is ​​a matrix of converting the Bayer image values ​​after applying the white balance gain at the initial ambient light color temperature to the tristimulus values ​​under a D65 light source;

[0022] Calculate the actual tristimulus values ​​under the D65 light source, and calculate the actual Lab value under the D65 light source;

[0023] Comparing the actual Lab value under the D65 light source with the Lab value under the standard D65 light source to obtain the actual saturation of each color block in the image, thereby obtaining the actual average saturation of the image;

[0024] Calculating a difference between the actual average saturation and the target saturation, and determining whether it is necessary to adjust the tristimulus values ​​of the image under the color temperature of the ambient light source according to whether the difference is less than a set threshold;

[0025] If the difference is less than the set threshold, the first initial matrix is ​​a conversion of the Bayer image values ​​under the actual ambient light source color temperature after applying the white balance gain to the tristimulus value matrix under the D65 light source, and the calculation ends. Then, the tristimulus value matrix under the D65 light source is converted to the tristimulus value matrix under the equal energy light source to obtain the first matrix;

[0026] If the difference is not less than the set threshold, the tristimulus values ​​of the image under the color temperature of the ambient light source are adjusted, and the above steps after obtaining the tristimulus values ​​of the image under the color temperature of the ambient light source are repeated.

[0027] Optionally, the target saturation is calculated as follows:

[0028] The calculation formula of the actual saturation is:

[0029] Among them, a s with b s is the standard Lab value, a t with b t is the target Lab value, a a with b a is the actual Lab value.

[0030] Optionally, the standard Lab value a s with b s Determined according to the calibration parameters, the target Lab value a t with b t Obtained according to the target saturation.

[0031] Optionally, the gamma curve applicable to the current scene is obtained based on the histogram information of the fused tristimulus value image under the equal-energy light source; and the target saturation is obtained based on the brightness of the current scene and the exposure time of the image sensor.

[0032] Optionally, the method of obtaining the second matrix includes:

[0033] Obtaining a matrix of tristimulus values ​​converted to RGB values ​​under a standard D65 light source, and simultaneously obtaining the tristimulus values ​​under the actual D65 light source;

[0034] Calculate the LRGB value of each color block and calculate the sRGB value;

[0035] Calculate the tristimulus values ​​of the actual display screen under the D65 light source, and calculate the Lab value of the actual display screen under the D65 light source;

[0036] Calculating a difference between the Lab value of the actual display screen under the D65 light source and the target Lab value, and determining whether the Lab value of the actual display screen under the D65 light source needs to be adjusted based on whether the difference is less than a set threshold;

[0037] If the difference is less than the set threshold, the standard tristimulus value-to-RGB value matrix under the D65 light source is converted to the actual tristimulus value-to-RGB value matrix under the D65 light source, and the calculation ends; then, the tristimulus value matrix under the equal-energy light source is converted to the tristimulus value matrix under the D65 light source to obtain the second matrix;

[0038] If the difference is not less than the set threshold, adjusting the Lab value of the actual display screen under the D65 light source to obtain an adjusted Lab value under the D65 light source;

[0039] Calculate the tristimulus values ​​under the adjusted D65 light source;

[0040] The adjusted tristimulus value-to-RGB value matrix under the D65 light source is calculated and the steps after obtaining the tristimulus value-to-RGB value matrix under the standard D65 light source are repeated.

[0041] Optionally, after calculating the LRGB value of each color block and before calculating the sRGB value, the method further includes: multiplying the LRGB value of each color block by a saturation adjustment matrix to obtain the LRGB value after saturation adjustment.

[0042] Optionally, the LRGB value after saturation adjustment is applied to the gamma curve to obtain an sRGB value.

[0043] Optionally, the sRGB value is reversed using a standard gamma and a standard RGB value to tristimulus value conversion matrix under a D65 light source to obtain the tristimulus values ​​of the actual display screen under a D65 light source.

[0044] In the color restoration method for a multiple-exposure mechanism provided by the present invention, first, Bayer images at different exposure levels are obtained; then, the white balance gains of the Bayer images at different exposure levels are determined to obtain Bayer images after applying the white balance gains at different exposure levels; then, based on the Bayer images after applying the white balance gains at different exposure levels and the corresponding ambient light source color temperature, a first matrix at different exposure levels is obtained; based on a gamma curve and target saturation applicable to the current scene, a second matrix is ​​obtained; and finally, based on the Bayer images after applying the white balance gains at different exposure levels, the first matrix, the second matrix, and the gamma curve, an image after applying the gamma curve is obtained. During implementation, the present invention can optimize the problem of color cast in the final image caused by inconsistent white balance gains corresponding to different regions in a scene under mixed light sources. Furthermore, during the calculation process, the present invention simultaneously considers the gamma curve and the target saturation, and the calculated color correction matrix matches the gamma curve and the target saturation, effectively resolving the problem of abnormal color restoration under high dynamic range imaging conditions.

[0045] Furthermore, the difference between the Lab value of the actual display screen under the D65 light source and the target Lab value is calculated, and whether the Lab value of the actual display screen under the D65 light source needs to be adjusted is determined based on whether the difference is less than a set threshold; if the difference is less than the set threshold, the standard tristimulus value-to-RGB value matrix under the D65 light source is converted to the actual tristimulus value-to-RGB value matrix under the D65 light source, and the calculation is terminated to ultimately obtain a second matrix; if the difference is not less than the set threshold, the Lab value of the actual display screen under the D65 light source is adjusted to obtain the adjusted Lab value under the D65 light source, and then the adjusted tristimulus value-to-RGB value matrix under the D65 light source is obtained by calculation, and then the second matrix is ​​calculated, that is, the Lab value of the target color block is adaptively adjusted, thereby further improving the calculation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Those skilled in the art should understand that the drawings are provided for a better understanding of the present invention, but do not constitute any limitation on the scope of the present invention.

[0047] Figure 1 4 is a flow chart of a color restoration method applicable to a multiple exposure mechanism provided by an embodiment of the present invention.

[0048] Figure 2 This is a flowchart of a color restoration method applicable to a multiple exposure mechanism provided by an embodiment of the present invention.

[0049] Figure 3 This is a flowchart of calibration of a first matrix provided by an embodiment of the present invention.

[0050] Figure 4 This is a calculation flow chart for obtaining the tristimulus values ​​of a calibration image color block under the color temperature of an ambient light source, provided by an embodiment of the present invention.

[0051] Figure 5 This is a calculation flow chart of the second matrix provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0052] In high dynamic range scenarios, in order to capture information in a larger dynamic range as much as possible, a multiple exposure mechanism is used to capture multiple frames of images and fuse them to obtain an image that covers the entire dynamic range.

[0053] In this scenario, there may be two or more light sources with different color temperatures. To solve the problem of mixed light sources, traditional white balance and color correction mechanisms mainly have the following two methods:

[0054] The first method is to select one of the mixed light sources, or select the balanced white balance gain and the corresponding color correction matrix as the common white balance and common color correction matrix. This method may cause local color cast problems due to inappropriate local white balance gain.

[0055] The second method is to apply white balance gains to the Bayer images of different exposures, fuse them, and then apply a common color correction matrix to the fused image. This method may cause unnatural transition areas and color casts due to inconsistent dimensionality of the source data.

[0056] At the same time, in order to display (high dynamic range) HDR scenes in (low dynamic range) LDR display devices, a non-standard gamma curve will be applied to the data after the color correction matrix is ​​applied, and this operation will further highlight the color cast problem in local areas.

[0057] Based on the above problems, an embodiment of the present invention provides a color restoration method suitable for a multiple exposure mechanism. The method first obtains Bayer images at different exposure levels. Then, the white balance gain of the Bayer images at different exposure levels is determined to obtain Bayer images after applying the white balance gain at different exposure levels. Next, a first matrix at different exposure levels is obtained based on the Bayer images after applying the white balance gain at different exposure levels and the corresponding ambient light source color temperature. A second matrix is ​​obtained based on a gamma curve and a target saturation applicable to the current scene. Finally, an image after applying the gamma curve is obtained based on the Bayer images after applying the white balance gain at different exposure levels, the first matrix, the second matrix, and the gamma curve. During implementation, the present invention can optimize the problem of color cast in the final image caused by inconsistent white balance gains corresponding to different regions in a scene under mixed light sources. In addition, the present invention simultaneously considers the gamma curve and the target saturation during the calculation process. The calculated color correction matrix matches the gamma curve and the target saturation, effectively solving the problem of abnormal color restoration under high dynamic range imaging conditions.

[0058] To make the objects, advantages, and features of the present invention more clearly apparent, the present invention is further described below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the drawings are all in a very simplified form and are not drawn to scale. They are only used to conveniently and clearly assist in illustrating the purposes of the embodiments of the present invention. In addition, the structures shown in the drawings are often part of the actual structure. In particular, different drawings may need to illustrate different focuses and sometimes use different scales.

[0059] As used in the present invention, the singular forms "a", "an", and "the" include plural objects, the term "or" is generally used to include the meaning of "and / or", the term "several" is generally used to include the meaning of "at least one", and the term "at least two" is generally used to include the meaning of "two or more". In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features specified as "first", "second", and "third" may explicitly or implicitly include one or at least two of the features, unless the content clearly indicates otherwise.

[0060] Figure 1 This is a flow chart of a color restoration method applicable to a multiple exposure mechanism provided by an embodiment of the present invention. Figure 1 As shown, an embodiment of the present invention provides a color restoration method applicable to a multiple exposure mechanism, comprising the following steps:

[0061] S1: Expose the image multiple times based on the established exposure mechanism to obtain Bayer images at different exposure levels;

[0062] S2: determining the white balance gain of the Bayer image at different exposure levels, and obtaining the Bayer image after applying the white balance gain at different exposure levels;

[0063] S3: obtaining a first matrix at different exposure levels based on the Bayer images after applying the white balance gain and the corresponding ambient light color temperature at different exposure levels, wherein the first matrix is ​​a matrix of tristimulus values ​​under an equal energy light source converted from the Bayer image values ​​after applying the white balance gain at the ambient light color temperature;

[0064] S4: obtaining tristimulus value images under equal energy light sources at different exposure levels based on the Bayer images after applying the white balance gain at different exposure levels and the first matrix;

[0065] S5: fusing the tristimulus value images under the equal energy light source at different exposure levels to obtain a fused tristimulus value image under the equal energy light source;

[0066] S6: Determine a gamma curve and target saturation suitable for the current scene, and obtain a second matrix, where the second matrix is ​​a matrix for converting tristimulus values ​​to RGB values ​​under an equal energy light source;

[0067] S7: obtaining an image after applying a color correction matrix based on the fused tristimulus value image under the equal energy light source and the second matrix; and

[0068] S8: Obtaining an image after applying a gamma curve based on the image after applying the color correction matrix and the gamma curve.

[0069] In step S1, the image is exposed multiple times based on a predetermined exposure mechanism to obtain Bayer images at different exposure levels. Figure 2 As shown, the image output by the image sensor can be exposed three times, namely short exposure, normal exposure and long exposure, to obtain a Bayer image (RAW_S), a Bayer image (RAW_N) and a Bayer image (RAW_L) respectively. Of course, four or more exposures with different exposure amounts can also be performed. The present invention does not limit the number of exposures and the exposure amount.

[0070] In step S2, the white balance gain of the Bayer image under different exposure amounts is determined to obtain the Bayer image after applying the white balance gain under different exposure amounts.

[0071] First, determine the white balance gain of the Bayer image at different exposures, apply the white balance gain to the corresponding Bayer image at different exposures, and obtain the Bayer image after applying the white balance gain at different exposures. For example, please refer to Figure 2As shown, white balance statistics are performed on the Bayer images at different exposures, and white balance gains are calculated to obtain white balance gain S under short exposure, white balance gain N under normal exposure, and white balance gain L under long exposure. Next, the white balance gains are applied to the corresponding Bayer images at different exposures to obtain Bayer images with applied white balance gains at different exposures, namely, the Bayer image with applied white balance gains (WBRAW_S), the Bayer image with applied white balance gains (WBRAW_N), and the Bayer image with applied white balance gains (WBRAW_L).

[0072] In step S3, based on the Bayer image after applying the white balance gain under different exposure levels and the corresponding ambient light source color temperature, the first matrix (WBCam2EXYZ) under different exposure levels is obtained. The first matrix is ​​the three-color stimulation value matrix under equal energy light source converted from the Bayer image value after applying the white balance gain under the ambient light source color temperature.

[0073] Figure 3 is a calibration flow chart of the first matrix provided by one embodiment of the present invention, Figure 4 This is a calculation flow chart for obtaining the tristimulus values ​​of each color block of the calibration image under the color temperature of the ambient light source provided by an embodiment of the present invention. Figure 3 and Figure 4 As shown, the method for calculating the first matrix (WBCam2EXYZ) includes:

[0074] Sub-step S31: obtaining tristimulus values ​​(DestXYZ) of the image at the color temperature of the ambient light source, and simultaneously obtaining a Bayer image value (WBCam) after applying a white balance gain at the color temperature of the ambient light source.

[0075] Please refer to Figure 4 As shown, the method for obtaining the tristimulus values ​​(DestXYZ) of each color block of the calibration image under the color temperature of the ambient light source includes: first executing sub-step S311: obtaining the Lab value (D50Lab) of the image under the D50 color temperature; then executing sub-step S312: calculating and obtaining the tristimulus values ​​(D50XYZ) of the image under the D50 color temperature; then executing sub-step S313: calculating and obtaining the tristimulus values ​​(DestXYZ) of the image under the color temperature of the ambient light source.

[0076] At the same time, a Bayer image value (WBCam) after applying the white balance gain at the ambient light source color temperature is obtained based on the Bayer image after applying the white balance gain. Furthermore, before obtaining the tristimulus values ​​of the image at the ambient light source color temperature, or before calculating the first matrix, a calibration can be performed to convert the Bayer image value after applying the white balance gain under a standard light source into a tristimulus value matrix under an equal energy light source. This calibration can be performed using methods known to those skilled in the art and will not be further described in the present invention.

[0077] Sub-step S32: Calculate and obtain a Bayer image value matrix (DestXYZ2WBCam) obtained by converting the tristimulus values ​​at the ambient light color temperature to the Bayer image value matrix (DestXYZ2WBCam) after applying the white balance gain. Exemplarily, the tristimulus values ​​(DestXYZ) of the image at the ambient light color temperature are fitted with the Bayer image value (WBCam) after applying the white balance gain at the ambient light color temperature to obtain the Bayer image value matrix (DestXYZ2WBCam) obtained by converting the tristimulus values ​​at the ambient light color temperature to the Bayer image value matrix (DestXYZ2WBCam) after applying the white balance gain.

[0078] Sub-step S33: Calculate the conversion matrix of the three-stimulus values ​​(WBCam2DestXYZ) from the Bayer image values ​​after applying the white balance gain at the ambient light color temperature, and simultaneously obtain the conversion matrix of the three-stimulus values ​​(DestXYZ2D65XYZ) from the ambient light color temperature to the three-stimulus values ​​(DestXYZ2WBCam) under the D65 light source. The conversion matrix of the three-stimulus values ​​(DestXYZ2WBCam) after applying the white balance gain to the Bayer image values ​​at the ambient light color temperature can be calculated using a calculation method known to those skilled in the art. This will not be described in detail in this embodiment of the present invention.

[0079] Then, in the next sub-step, a first matrix can be calculated. This matrix can be obtained by multiplying the Bayer image values ​​after applying white balance gain at the ambient light color temperature to the tristimulus value matrix (WBCam2DestXYZ) by the tristimulus value matrix under the D65 light source (DestXYZ2D65XYZ). This matrix can then be multiplied by the tristimulus value matrix under the D65 light source (D65XYZ2EXYZ) to obtain the first matrix (WBCam2EXYZ). In this embodiment, the actual saturation needs to be compared with the target saturation to determine whether the tristimulus values ​​(DestXYZ) of the image at the ambient light color temperature need to be adjusted. Specifically, the following sub-steps are included.

[0080] Sub-step S34: Based on the conversion matrix of the three-color stimulus values ​​(WBCam2DestXYZ) obtained by applying the white balance gain to the Bayer image values ​​under the color temperature of the ambient light source and the conversion matrix of the three-color stimulus values ​​under the color temperature of the ambient light source and the D65 light source (DestXYZ2D65XYZ), a first initial matrix is ​​calculated. The first initial matrix is ​​the conversion matrix of the three-color stimulus values ​​(initial WBCam2D65XYZ) obtained by applying the white balance gain to the Bayer image values ​​under the initial color temperature of the ambient light source and the D65 light source. It is subsequently necessary to determine whether to adjust the first initial matrix.

[0081] Sub-step S35: Calculate and obtain the actual tristimulus values ​​(D65XYZ) under the D65 light source. Exemplarily, the tristimulus value matrix (initial WBCam2D65XYZ) under the D65 light source is converted based on the initial Bayer image value after applying white balance gain under the color temperature of the ambient light source (i.e., the first initial matrix), and the Bayer image value (WBCam) after applying white balance gain under the color temperature of the ambient light source obtained from the Bayer image to obtain the actual tristimulus values ​​(D65XYZ) under the D65 light source. It should be noted that the actual tristimulus values ​​under the D65 light source here refer to the tristimulus values ​​under the D65 light source of the actual scene, and what is obtained in this step are the tristimulus values ​​of each color block of the image under the D65 light source of the actual scene.

[0082] Then, the actual Lab value (D65Lab) under the D65 light source is calculated, and the Lab value of each color block under the D65 light source in the actual scene is obtained according to the tristimulus values ​​of each color block in the image under the D65 light source in the actual scene.

[0083] Sub-step S36: Compare the actual Lab value (D65Lab) under the D65 light source with the Lab value under the standard D65 light source to obtain the actual saturation of each color block in the image. a , thereby obtaining the actual average saturation of the image.

[0084] In the case of consistent hue, assuming that the standard Lab value of a color block is L s 、a s 、b s , the target Lab value is L t 、a t 、b t , the actual Lab value is L a 、a a 、b a ,So,

[0085] The target saturation is calculated as:

[0086] The actual saturation is calculated as:

[0087] The standard Lab value is obtained by calibration parameters; the target Lab value is obtained according to the target saturation; and the actual Lab value is obtained by the above calculation.

[0088] After the actual saturation of each color block in the image is obtained according to the above formula, the actual average saturation of the image is obtained by averaging.

[0089] Sub-step S37: Calculate the difference between the actual average saturation and the target saturation, and determine whether to adjust the tristimulus values ​​DestXYZ of the image at the ambient light color temperature based on whether the difference is less than a set threshold. For example, the set threshold may be 0.01%, but is not limited thereto.

[0090] If the difference is less than the set threshold, that is, if yes, then execute sub-step S38; if the difference is not less than the set threshold, that is, if no, then execute sub-step S39.

[0091] Sub-step S38: The first initial matrix is ​​the conversion of the Bayer image values ​​under the actual ambient light color temperature after applying the white balance gain to the tristimulus value matrix under the D65 light source (WBCam2D65XYZ), and the calculation ends. The tristimulus value matrix under the D65 light source is then converted to the tristimulus value matrix under the equal-energy light source (D65XYZ2EXYZ), thereby obtaining the first matrix (WBCam2EXYZ). Specifically, WBCam2D65XYZ is multiplied by D65XYZ2EXYZ to obtain WBCam2EXYZ.

[0092] Sub-step S39: Adjust the tristimulus values ​​(DestXYZ) of the image at the ambient light color temperature, and repeat the above steps after obtaining the tristimulus values ​​of the image at the ambient light color temperature. In this step, the tristimulus values ​​(DestXYZ) of the image at the ambient light color temperature are adjusted to obtain adjusted tristimulus values ​​at the ambient light color temperature (adjusted DestXYZ). The adjusted tristimulus values ​​at the ambient light color temperature are then used to replace the tristimulus values ​​(DestXYZ) of the image at the ambient light color temperature in step S21. The subsequent steps are then repeated until the first matrix (WBCam2EXYZ) is determined.

[0093] The above is the calculation method of the first matrix under a certain standard light source. The same method can be used to obtain the first matrix under different standard light sources. For specific application examples, please refer to Figure 2As shown, the first matrix S under short exposure, the first matrix N under normal exposure, and the first matrix L under long exposure can be obtained.

[0094] In step S4, based on the Bayer images after applying the white balance gain at different exposure levels and the first matrix, tristimulus value images under equal energy light sources at different exposure levels are obtained.

[0095] Please continue to refer to Figure 1 and 2 As shown, in step S2, Bayer images after applying white balance gains at different exposure levels are obtained, and in step S3, a first matrix at different exposure levels is obtained. The Bayer images after applying white balance gains at different exposure levels are combined with the first matrix to obtain tristimulus value images under equal energy light sources at different exposure levels. Specifically, a tristimulus value image (EXYZ_S) under equal energy light sources at short exposure, a tristimulus value image (EXYZ_N) under equal energy light sources at normal exposure, and a tristimulus value image (EXYZ_L) under equal energy light sources at long exposure are obtained.

[0096] In step S5, the tristimulus images under the equal-energy light source at different exposure levels are fused to obtain a fused tristimulus image under the equal-energy light source. Specifically, the tristimulus images under the equal-energy light source at short exposure, normal exposure, and long exposure are fused to obtain a fused tristimulus image under the equal-energy light source (Combined EXYZ), i.e., a high dynamic range image.

[0097] In step S6, a gamma curve and a target saturation suitable for the current scene are determined to obtain a second matrix, which is a matrix for converting tristimulus values ​​to RGB values ​​under an equal energy light source.

[0098] Specifically, first, a gamma curve and target saturation suitable for the current scene are determined. The gamma curve suitable for the current scene is obtained based on the histogram information of the fused tristimulus value image under the equal energy light source; the target saturation is obtained based on the brightness of the current scene and information such as the exposure time and gain of the image sensor.

[0099] Next, based on the gamma curve and the target saturation, a second matrix (EXYZ2LRGB) is calculated, where the second matrix is ​​a matrix for converting tristimulus values ​​to RGB values ​​under an equal energy light source.

[0100] Figure 5 This is a calculation flow chart of the second matrix provided by an embodiment of the present invention. Figure 5 As shown, the method for calculating the second matrix (EXYZ2LRGB) includes:

[0101] Sub-step S61: Obtain a standard D65 light source tristimulus value to RGB value matrix (standard D65XYZ2LRGB), and simultaneously obtain the tristimulus values ​​(D65XYZ) of each color block of the calibration image obtained in sub-step S35 under the D65 light source. Exemplarily, the standard D65 light source tristimulus value to RGB value matrix is ​​obtained based on the international RGB color space standard.

[0102] Sub-step S62: Based on the tristimulus value to RGB value matrix under the standard D65 light source (standard D65XYZ2LRGB) and the tristimulus values ​​of each color block in the calibration image under the D65 light source (D65XYZ), calculate the LRGB value of each color block.

[0103] Then, the LRGB value of each color block is multiplied by the saturation adjustment matrix (the initial value of the matrix is ​​a matrix with a saturation of 100) to obtain the LRGB value after saturation adjustment.

[0104] Sub-step S63: Calculate and obtain the sRGB value. Apply the LRGB value after saturation adjustment to the gamma curve to obtain the sRGB value.

[0105] Sub-step S64: Calculate the tristimulus values ​​of the actual display screen under the D65 light source (D65XYZ of the actual display screen), and calculate the Lab value of the actual display screen under the D65 light source (D65Lab of the actual display screen). Use standard gamma and a standard RGB value to D65 light source tristimulus value conversion matrix to reversely calculate the sRGB values ​​to obtain the tristimulus values ​​of the actual display screen under the D65 light source. The standard RGB value to D65 light source tristimulus value conversion matrix can be obtained based on the standard D65 light source tristimulus value to RGB value conversion matrix in sub-step S61. Next, the Lab value of the actual display screen under the D65 light source is calculated based on the tristimulus values ​​of the actual display screen under the D65 light source. The specific calculation process is not repeated here.

[0106] Sub-step S65: Calculate the difference between the actual display's Lab value under the D65 illuminant (D65Lab of the actual display) and the target Lab value, and determine whether to adjust the actual display's Lab value under the D65 illuminant (D65Lab of the actual display) based on whether the difference is less than a set threshold. For example, the set threshold may be 0.01%, but is not limited thereto.

[0107] If the difference is less than the set threshold, that is, if yes, then execute sub-step S66; if the difference is not less than the predetermined setting, that is, if no, then execute sub-step S67.

[0108] Sub-step S66: The standard tristimulus value-to-RGB value matrix under the D65 light source (standard D65XYZ2LRGB) is converted to the actual tristimulus value-to-RGB value matrix under the D65 light source (D65XYZ2LRGB), and the calculation ends. Then, based on the tristimulus value matrix under the equal-energy light source, the tristimulus value matrix under the D65 light source (EXYZ2D65XYZ) is converted to obtain the second matrix (EXYZ2LRGB). Specifically, D65XYZ2LRGB is multiplied by EXYZ2D65XYZ to obtain EXYZ2LRGB.

[0109] Sub-step S67: adjusting the Lab value of the actual display screen under the D65 light source (D65Lab of the actual display screen) to obtain an adjusted Lab value under the D65 light source (adjusted D65Lab).

[0110] After executing step S67 , sub-step S68 is then executed: calculating and obtaining the adjusted tristimulus values ​​under the D65 light source (adjusted D65XYZ).

[0111] Then, step S69 is performed: an adjusted tristimulus value-to-RGB value matrix under the D65 light source (adjusted D65XYZ2LRGB) is calculated, and the steps after obtaining the tristimulus value-to-RGB value matrix under the standard D65 light source (standard D65XYZ2LRGB) are repeated. That is, the adjusted tristimulus value-to-RGB value matrix under the D65 light source is obtained, and the adjusted tristimulus value-to-RGB value matrix under the D65 light source (adjusted D65XYZ2LRGB) is used to replace the standard tristimulus value-to-RGB value matrix under the D65 light source (standard D65XYZ2LRGB) in sub-step S61, and the subsequent steps are repeated until the second matrix is ​​determined.

[0112] In step S7 , an image (LRGB) after applying a color correction matrix is ​​obtained based on the fused tristimulus value image (CombinedEXYZ) under the equal energy light source and the second matrix (EXYZ2LRGB).

[0113] In step S8 , an image (sRGB) after applying a gamma curve is obtained based on the image (LRGB) after applying the color correction matrix and the gamma curve.

[0114] In the color restoration method for a multiple-exposure mechanism provided by the present invention, first, Bayer images at different exposure levels are obtained; then, the white balance gains of the Bayer images at different exposure levels are determined to obtain Bayer images after applying the white balance gains at different exposure levels; then, based on the Bayer images after applying the white balance gains at different exposure levels and the corresponding ambient light source color temperature, a first matrix at different exposure levels is obtained; based on a gamma curve and target saturation applicable to the current scene, a second matrix is ​​obtained; and finally, based on the Bayer images after applying the white balance gains at different exposure levels, the first matrix, the second matrix, and the gamma curve, an image after applying the gamma curve is obtained. During implementation, the present invention can optimize the problem of color cast in the final image caused by inconsistent white balance gains corresponding to different regions in a scene under mixed light sources. Furthermore, during the calculation process, the present invention simultaneously considers the gamma curve and the target saturation, and the calculated color correction matrix matches the gamma curve and the target saturation, effectively resolving the problem of abnormal color restoration under high dynamic range imaging conditions.

[0115] Furthermore, the Lab value of the actual display screen under the D65 light source (D65XYZ of the actual display screen) is compared with the target Lab value to determine whether the Lab value of the actual display screen under the D65 light source needs to be adjusted; if the difference is less than the set threshold, the matrix of converting the tristimulus values ​​under the standard D65 light source to RGB values ​​is the second matrix, and the calculation is terminated; if the difference is not less than the set threshold, the Lab value of the actual display screen under the D65 light source is adjusted to obtain the adjusted Lab value under the D65 light source, and then the adjusted tristimulus value-to-RGB value matrix under the D65 light source is obtained by calculation, and then the second matrix is ​​calculated, that is, the Lab value of the target color block is adaptively adjusted, thereby further improving the calculation accuracy.

[0116] The above description is only a description of the preferred embodiment of the present invention, and does not limit the scope of the rights of the present invention. Any person skilled in the art can make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A color restoration method suitable for a multiple exposure mechanism, characterized in that: The color restoration method comprises the following steps: Expose the image multiple times based on a predetermined exposure mechanism to obtain Bayer images at different exposure levels; determining white balance gains of the Bayer images at different exposure levels to obtain Bayer images after applying the white balance gains at different exposure levels; Based on the Bayer images after applying the white balance gain at different exposure levels and the corresponding ambient light source color temperature, a first matrix at different exposure levels is obtained, where the first matrix is ​​a matrix of tristimulus values ​​under an equal energy light source converted from the Bayer image values ​​after applying the white balance gain at the ambient light source color temperature; Obtaining tristimulus value images under equal energy light sources at different exposure levels based on the Bayer images after applying the white balance gain at different exposure levels and the first matrix; fusing the tristimulus value images under the equal energy light source at different exposure levels to obtain a fused tristimulus value image under the equal energy light source; Determine a gamma curve and target saturation suitable for the current scene to obtain a second matrix, where the second matrix is ​​a matrix for converting tristimulus values ​​to RGB values ​​under an equal energy light source; Obtaining an image after applying a color correction matrix based on the fused tristimulus value image under the equal energy light source and the second matrix; and An image after applying the gamma curve is obtained based on the image after applying the color correction matrix and the gamma curve.

2. The color restoration method applicable to a multiple exposure mechanism according to claim 1, wherein: The method of obtaining a first matrix based on the Bayer images after applying the white balance gain and the corresponding ambient light source color temperature under different exposures includes: Obtaining tristimulus values ​​of the image under the color temperature of the ambient light source, and simultaneously obtaining a Bayer image value after applying a white balance gain under the color temperature of the ambient light source; Calculate and convert the tristimulus values ​​under the color temperature of the ambient light source into the Bell image value matrix after applying the white balance gain; Calculate the conversion matrix of the Bayer image value after applying the white balance gain under the color temperature of the ambient light source to the tristimulus value matrix under the color temperature of the ambient light source, and simultaneously obtain the conversion matrix of the tristimulus value under the color temperature of the ambient light source to the tristimulus value under the D65 light source; and The first matrix is ​​calculated.

3. The color restoration method applicable to a multiple exposure mechanism according to claim 2, wherein: Before obtaining the first matrix, the method further includes calibrating the Bayer image value after applying white balance gain under a commonly used standard light source to convert it into a tristimulus value matrix under an equal energy light source.

4. The color restoration method applicable to a multiple exposure mechanism according to claim 2, wherein: Methods for calculating the first matrix include: A first initial matrix is ​​calculated based on a matrix of converting the Bayer image values ​​after applying the white balance gain at the ambient light color temperature to the tristimulus values ​​and a matrix of converting the tristimulus values ​​at the ambient light color temperature to the tristimulus values ​​under a D65 light source, where the first initial matrix is ​​a matrix of converting the Bayer image values ​​after applying the white balance gain at the initial ambient light color temperature to the tristimulus values ​​under a D65 light source; Calculate the actual tristimulus values ​​under the D65 light source, and calculate the actual Lab value under the D65 light source; Comparing the actual Lab value under the D65 light source with the Lab value under the standard D65 light source to obtain the actual saturation of each color block in the image, thereby obtaining the actual average saturation of the image; Calculating a difference between the actual average saturation and the target saturation, and determining whether it is necessary to adjust the tristimulus values ​​of the image under the color temperature of the ambient light source according to whether the difference is less than a set threshold; If the difference is less than the set threshold, the first initial matrix is ​​a conversion of the Bayer image values ​​under the actual ambient light source color temperature after applying the white balance gain to the tristimulus value matrix under the D65 light source, and the calculation ends. Then, the tristimulus value matrix under the D65 light source is converted to the tristimulus value matrix under the equal energy light source to obtain the first matrix; If the difference is not less than the set threshold, the tristimulus values ​​of the image under the color temperature of the ambient light source are adjusted, and the above steps after obtaining the tristimulus values ​​of the image under the color temperature of the ambient light source are repeated.

5. The color restoration method applicable to a multiple exposure mechanism according to claim 4, wherein: The target saturation is calculated as follows: The calculation formula of the actual saturation is: Among them, a s with b s is the standard Lab value, a t with b t is the target Lab value, a a with b a is the actual Lab value.

6. The color restoration method applicable to a multiple exposure mechanism according to claim 5, wherein: The standard Lab value a s with b s Determined according to the calibration parameters, the target Lab value a t with b t Obtained according to the target saturation.

7. The color restoration method applicable to a multiple exposure mechanism according to claim 4, wherein: The gamma curve suitable for the current scene is obtained based on the histogram information of the fused tristimulus value image under the equal energy light source; and the target saturation is obtained based on the brightness of the current scene and the exposure time of the image sensor.

8. The color restoration method applicable to a multiple exposure mechanism according to claim 7, wherein: Methods for obtaining the second matrix include: Obtaining a matrix of tristimulus values ​​converted to RGB values ​​under a standard D65 light source, and simultaneously obtaining the tristimulus values ​​under the actual D65 light source; Calculate the LRGB value of each color block and calculate the sRGB value; Calculate the tristimulus values ​​of the actual display screen under the D65 light source, and calculate the Lab value of the actual display screen under the D65 light source; Calculating a difference between the Lab value of the actual display screen under the D65 light source and the target Lab value, and determining whether the Lab value of the actual display screen under the D65 light source needs to be adjusted based on whether the difference is less than a set threshold; If the difference is less than the set threshold, the standard tristimulus value-to-RGB value matrix under the D65 light source is converted to the actual tristimulus value-to-RGB value matrix under the D65 light source, and the calculation ends; then, the tristimulus value matrix under the equal-energy light source is converted to the tristimulus value matrix under the D65 light source to obtain the second matrix; If the difference is not less than the set threshold, adjusting the Lab value of the actual display screen under the D65 light source to obtain an adjusted Lab value under the D65 light source; Calculate the tristimulus values ​​under the adjusted D65 light source; The adjusted tristimulus value-to-RGB value matrix under the D65 light source is calculated and the steps after obtaining the tristimulus value-to-RGB value matrix under the standard D65 light source are repeated.

9. The color restoration method applicable to a multiple exposure mechanism according to claim 8, wherein: After calculating the LRGB value of each color block and before calculating the sRGB value, the method further includes: multiplying the LRGB value of each color block by a saturation adjustment matrix to obtain an LRGB value after saturation adjustment.

10. The color restoration method applicable to a multiple exposure mechanism according to claim 9, wherein: The LRGB value after saturation adjustment is applied to the gamma curve to obtain an sRGB value.

11. The color restoration method applicable to a multiple exposure mechanism according to claim 8, wherein: The sRGB value is reversed using a standard gamma and standard RGB value conversion matrix to convert the tristimulus values ​​under the D65 light source to obtain the tristimulus values ​​of the actual display screen under the D65 light source.

Images