White balance processing method, device, electronic device and storage medium

Abstract

The present disclosure provides a white balance processing method, comprising: in response to a result indicating that a sensor's operating mode is a high dynamic range acquisition mode, adjusting the sensor's digital gain using the channel gain corresponding to a historical image; performing white balance processing on multiple frames of exposed images using the sensor to obtain an original image, wherein transition regions in the original image exhibit a target color; and performing full-image color correction on the original image based on the channel gain of the original image and the color temperature of the target environment in which the sensor is located to obtain a target image that matches the color temperature. The present disclosure also provides a white balance processing device, electronic device, and storage medium.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a white balance processing method, device, electronic device, and storage medium. Background Art

[0002] In the field of in-vehicle image processing, white balance color cast causes color cast on the edges of bright objects, affecting the machine vision algorithm's edge detection and, in turn, the recognition of objects, especially traffic signs, traffic lights, and lane markings. For vehicles equipped with autonomous driving equipment, dashcam images are usually directly processed by the front camera's ISP (Image Signal Processor). Users will also have a poor visual experience when viewing dashcam images with white balance color cast.

[0003] Based on this, AWB (Automatic White Balance) technology is usually used to perform AWB processing on the original image collected by the sensor, and then obtain the R_gain red channel gain (Red_gain, red channel gain) and B_gain (Blue_gain, blue channel gain) in the target environment corresponding to the original image, and use R_gain and B_gain to adjust the gain parameters in the ISP, so that the ISP can finally correct the color of the original image so that the output target image conforms to the actual color temperature of the target environment, avoiding color cast at the edges of objects in the image.

[0004] HDR (High Dynamic Range) sensors are capable of presenting images with a wider brightness range and sharper details. Compared to linear sensors, they are more suitable for scenarios such as automotive applications that require higher detail capture and a higher brightness range. However, because the original image obtained by the HDR sensor is the result of stitching together multiple frames of exposure images at different exposure levels, the highlighted objects it presents will have color fringing. The aforementioned AWB technology only has color calibration capabilities and cannot meet the requirements of identifying and removing color fringes. Therefore, it cannot accurately recognize objects and cannot provide users with a better visual experience. Summary of the Invention

[0005] In order to solve at least one of the above technical problems, the present disclosure provides a white balance processing method, device, electronic device and storage medium.

[0006] One aspect of the present disclosure provides a white balance processing method, which may include: in response to an identification result that the operating mode of the sensor is a high dynamic range acquisition mode, adjusting the digital gain of the sensor using the channel gain corresponding to the historical image; using the sensor to perform white balance processing on multiple frames of exposure images to obtain an original image, wherein the transition area in the original image presents a target color; and based on the channel gain of the original image and the color temperature of the target environment in which the sensor is located, performing full-image color correction on the original image to obtain a target image that matches the color temperature.

[0007] In some embodiments, using the sensor to perform white balance processing on multiple frames of exposure images to obtain the original image may include: the sensor performing white balance processing on each frame of the exposure image respectively to obtain at least a highlight correction image, a dark area correction image and a general correction image, wherein the transition areas in the highlight correction image, the dark area correction image and the general correction image all present the target color; performing feature cropping on the highlight correction image, the dark area correction image and the general correction image respectively to obtain highlight correction segments corresponding to high brightness areas, dark area correction segments corresponding to low brightness areas and general correction segments corresponding to general brightness areas; and splicing the highlight correction segments, the dark area correction segments and the general correction segments in sequence to obtain the original image.

[0008] In some embodiments, in response to the recognition result that the operating mode of the sensor is a high dynamic range acquisition mode, before adjusting the digital gain of the sensor using the channel gain corresponding to the historical image, it may include: determining the channel gain of each color channel based on the calibration parameters corresponding to the historical image, wherein the color channels include at least a red channel and a blue channel.

[0009] In some embodiments, the channel gain is expressed as: cali_x_gain = 1 / cali_x_g = g_avg / x_avg; wherein, cali_x_gain represents the channel gain of the target channel, including at least the channel gain cali_r_gain of the red channel and the channel gain cali_b_gain of the blue channel; cali_x_g represents the calibration parameters of the target channel, including at least the calibration parameters cali_r_g of the red channel and the calibration parameters cali_b_g of the blue channel; g_avg represents the average number of green pixels in the historical image; x_avg represents the average number of target color pixels in the historical image, including at least the average number of red pixels r_avg and the average number of blue pixels b_avg.

[0010] In some embodiments, before determining the channel gain of each color channel based on the calibration parameters corresponding to the historical image, it also includes: determining the calibration parameters of the sensor under various standard light sources, wherein the calibration parameters include at least a blue channel calibration parameter and a red channel calibration parameter, and the reciprocal of the calibration parameters is used to characterize the channel gain of each color channel.

[0011] In some embodiments, determining the calibration parameters of the sensor under various standard light sources includes: capturing an original image of a target gray card under the standard light source, counting initial calibration parameters of the original image of the target gray card, and using the reciprocal of the initial calibration parameters as an initial channel gain; using the initial channel gain as an initial digital gain of the sensor; using the sensor to capture an original image of a grayscale block, and analyzing a deviation value of the original image of the grayscale block, wherein the deviation value is used to characterize the color accuracy of the original image captured by the sensor, and the deviation value is inversely proportional to the color accuracy; in response to a judgment result that the deviation value is greater than a maximum limit value, adjusting the initial digital gain using the deviation value to obtain a process digital gain; and until the deviation value is less than or equal to the maximum limit value, using the reciprocal of the process digital gain as the calibration parameter of the sensor under the standard light source, and storing it in a device memory.

[0012] In some embodiments, before adjusting the digital gain of the sensor using the channel gain corresponding to the historical image in response to the recognition result that the operating mode of the sensor is the high dynamic range acquisition mode, it includes: determining the operating mode of the sensor, wherein the operating mode at least includes the high dynamic range acquisition mode and the linear acquisition mode.

[0013] In some embodiments, after determining the operating mode of the sensor, the method further includes: in response to an identification result that the operating mode of the sensor is the linear acquisition mode, obtaining the channel gain of the original image and the color temperature of the target environment in which the sensor is located; adjusting the digital gain of an image signal processor using the channel gain of the original image; and performing full-image color correction on the original image using the image signal processor based on the color temperature to obtain a target image that matches the color temperature.

[0014] In some embodiments, before using the sensor to perform white balance processing on multiple frames of exposure images to obtain the original image, the process includes: using the sensor to capture multiple frames of exposure images, wherein the exposure images include at least a highlight exposure image for presenting details of a high-brightness area, a dark area exposure image for presenting details of a low-brightness area, and a general exposure image for presenting a general brightness area.

[0015] In some embodiments, before performing full-image color correction on the original image based on the color temperature of the target environment in which the sensor is located to obtain a target image that matches the color temperature, the method includes: compensating the white balance state information to mask the weight of the channel gain of the historical image in the white balance state information to obtain target state information for characterizing the white balance state of the target environment; and estimating the color temperature of the target environment based on the target state information.

[0016] Another aspect of the present disclosure provides a white balance processing device, which may include: a sensor parameter adjustment module, a raw image generation module, and a full-image color correction module. The sensor parameter adjustment module is configured to adjust the digital gain of the sensor using the channel gain corresponding to the historical image in response to a recognition result that the sensor's operating mode is a high dynamic range acquisition mode; the raw image generation module is configured to use the sensor to perform white balance processing on multiple frames of exposure images to obtain a raw image, wherein the transition region in the raw image exhibits a target color; and the full-image color correction module is configured to perform full-image color correction on the raw image based on the channel gain of the raw image and the color temperature of the target environment in which the sensor is located, to obtain a target image that matches the color temperature.

[0017] Yet another aspect of the present disclosure provides an electronic device, which may include: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor performs the white balance processing method described in any one of the above embodiments.

[0018] Another aspect of the present disclosure provides a readable storage medium, wherein the readable storage medium stores execution instructions, and when the execution instructions are executed by a processor, they are used to implement the white balance processing method described in any of the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0020] Figure 1 A schematic diagram of the white balance processing method architecture of the related art;

[0021] Figure 2 A flow chart for generating white balance gain of an original image of the related art;

[0022] Figure 3 is a flow chart of a white balance processing method according to an exemplary embodiment of the present disclosure;

[0023] Figure 4 Schematic diagram of the white balance processing method according to an exemplary embodiment of the present disclosure;

[0024] Figure 5 A flow chart for generating calibration parameters according to an exemplary embodiment of the present disclosure;

[0025] Figure 6 A flowchart for generating white balance gain of an original image according to an exemplary embodiment of the present disclosure;

[0026] Figure 7 A flowchart of compensation of white balance state information according to an exemplary embodiment of the present disclosure;

[0027] Figure 8 A flowchart of state information analysis according to an exemplary embodiment of the present disclosure; and

[0028] Figure 9 Schematic diagram of a white balance processing device according to an exemplary embodiment of the present disclosure.

[0029] Description of Reference Numerals

[0030] 1000 White Balance Processing Device

[0031] 1002 sensor parameter adjustment module

[0032] 1004 Original Image Generation Module

[0033] 1006 Full Image Color Correction Module

[0034] 1100 bus

[0035] 1200 processor

[0036] 1300 Memory

[0037] 1400 Other circuits DETAILED DESCRIPTION

[0038] The present disclosure will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to explain the relevant content and are not intended to limit the present disclosure. It should also be noted that, for ease of description, only the portions relevant to the present disclosure are shown in the accompanying drawings.

[0039] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0040] Unless otherwise stated, the exemplary embodiments / examples shown are to be understood as providing exemplary features of various details of some ways in which the technical concepts of the present disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of the various embodiments / examples may be further combined, separated, interchanged, and / or rearranged without departing from the technical concepts of the present disclosure.

[0041] The use of cross hatching and / or shading in the accompanying drawings is generally used to make the boundaries between adjacent components clear. As such, unless otherwise indicated, the presence or absence of cross hatching or shading does not convey or indicate any preference or requirement for the specific materials, material properties, dimensions, proportions, commonalities between the components shown, and / or any other characteristics, attributes, properties, etc. of the components. In addition, in the accompanying drawings, the sizes and relative sizes of the components may be exaggerated for clarity and / or descriptive purposes. When the exemplary embodiments can be implemented differently, the specific process sequence can be performed in a different order than described. For example, two successively described processes can be performed substantially simultaneously or in an order opposite to the order described. In addition, the same figure numbers represent the same components.

[0042] When a component is referred to as being “on,” “over,” “connected to,” or “coupled to” another component, the component may be directly on, directly connected to, or directly coupled to the other component, or intervening components may be present. However, when a component is referred to as being “directly on,” “directly connected to,” or “directly coupled to” another component, there are no intervening components present. For this purpose, the term “connected” may refer to a physical connection, an electrical connection, etc., with or without intervening components.

[0043] The terms used herein are for the purpose of describing specific embodiments and are not intended to be restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, the features, integral bodies, steps, operations, parts, assemblies and / or their groups stated are indicated, but the presence or addition of one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups is not excluded. It should also be noted that, as used herein, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, and as such, they are used to explain the inherent deviations of the measured values, calculated values ​​and / or values ​​provided that will be recognized by those of ordinary skill in the art.

[0044] Figure 1 A schematic diagram of the white balance processing method architecture of the related art; Figure 2The white balance gain generation flow chart of the original image of the related art. Before introducing the white balance processing method of the present invention, first combine Figures 1 to 2 The white balance processing method of related technologies and its defects are briefly described.

[0045] The white balance processing method of the related art is mainly proposed to solve the color cast problem of the original image captured by the linear sensor. Its main process is as follows: the sensor captures a 12-bit to 16-bit original image; the original image is preprocessed by the ISP (Image Signal Processor) to obtain a preprocessed original image; the preprocessed original image is simultaneously sent to the white balance gain processing module and AWB (Automatic White Balance) module of the ISP. The AWB module performs AWB statistics on the preprocessed original image to obtain AWB status information of the preprocessed original image, and uses its AWB core algorithm to analyze the AWB status information of the original image to obtain a white balance gain corresponding to the original image; the ISP uses the white balance gain to perform white balance processing on the preprocessed original image to obtain a target image.

[0046] The linear sensor captures only one frame of image under a specific lighting environment, and this is used as the original image. Since the original image output by the linear sensor is not a fitted image, at least the transition areas it presents do not have color edges. Therefore, the target image can be obtained by performing full-image color correction using the ISP.

[0047] White balance processing is one of the processing processes for the original image. Before white balance processing, the original image will undergo a series of preprocessing including vignetting correction, bad pixel correction and exposure control. Since this disclosure mainly explains the white balance processing process, the relevant implementation methods of the preprocessing will not be elaborated in detail.

[0048] The AWB module has the ability to count AWB status information and also has the ability to determine CCT (Correlated Color Temperature, relative color temperature) and white balance gain using the core algorithm. Figure 2The process of the AWB core algorithm of the AWB module of the related art is briefly explained. The AWB module first performs state information analysis based on the AWB state information of the preprocessed original image, and then estimates the CCT based on the results of the state information analysis to determine the CCT of the lighting environment corresponding to the original image. Furthermore, based on the results of the state information analysis, the CCT of the lighting environment corresponding to the original image, and the calibration parameters of the sensor under different standard light sources, the white balance gain of the original image is calculated to determine the channel gain (i.e., white balance gain) corresponding to the original image, where the channel gain can be the red channel gain R_gain and the blue channel gain B_gain. Furthermore, R_gain and B_gain are applied to the R_gain register and B_gain register within the ISP, and the CCT is synchronized to the ISP to implement ISP white balance gain adjustment, ultimately obtaining the adjusted white balance gain of the ISP.

[0049] Obviously, the white balance processing methods demonstrated in related art are only applicable to the case where a linear sensor captures a single frame of image as the original image, which lacks color fringing. They are unable to address the color fringing issues of the original image generated by an HDR (High Dynamic Range) sensor, which is composed of multiple frames fitted together. Furthermore, the sensor should have a unique CCT for each moment of illumination. The CCT serves as the input condition for different functional modules on the ISP. These modules can use different functional parameters under the influence of different CCTs. Functional parameters can include, for example, a color transformation matrix. If the CCT is inaccurate, the color accuracy of the image output by the ISP will also deteriorate. Functional parameters can also include, for example, lens vignetting correction parameters. If the CCT is inaccurate, the image output by the ISP will exhibit color cast, vignetting, or center color cast. Because the sensor's digital gain is not adjusted in the related art, the AWB module does not consider masking the weight of the historical channel gain when executing the AWB core algorithm. However, with respect to the architecture proposed in this disclosure that adjusts the sensor's digital gain, this processing mode is affected by the weight of the historical channel gain, and the obtained CCT cannot represent the light source characteristics of the current lighting environment. Therefore, the current AWB core algorithm is not suitable for the overall architecture of the white balance processing method proposed in this disclosure.

[0050] The present disclosure proposes a white balance processing method to solve the problem that related technologies cannot process color edges in the original images output by HDR sensors; at the same time, in order to adapt to the overall architecture of the present disclosure, some data processing processes in the related technologies are also optimized.

[0051] Figure 3 is a flow chart of a white balance processing method according to an exemplary embodiment of the present disclosure; Figure 4Schematic diagram of the white balance processing method according to an exemplary embodiment of the present disclosure; Figure 5 A flow chart for generating calibration parameters according to an exemplary embodiment of the present disclosure; Figure 6 A flowchart for generating white balance gain of an original image according to an exemplary embodiment of the present disclosure; Figure 7 A flowchart of compensation of white balance state information according to an exemplary embodiment of the present disclosure; Figure 8 This is a flowchart of the state information analysis of an exemplary embodiment of the present disclosure. Figures 3 to 8 The white balance processing method S100 proposed in the present disclosure is described.

[0052] Step S102 : In response to the recognition result that the working mode of the sensor is the high dynamic range acquisition mode, the digital gain of the sensor is adjusted using the channel gain corresponding to the historical image.

[0053] The sensor is the device used to capture images. Depending on its characteristics, it can have different operating modes, such as high dynamic range acquisition mode and linear acquisition mode. Different operating modes will use different white balance processing methods. This means that this method is also applicable to white balancing raw images captured by linear sensors.

[0054] High dynamic range acquisition mode means that the sensor is an HDR sensor, which has the ability to display details of both highlights and shadows in an image. HDR sensors are more suitable for the recognition of traffic signs, traffic lights, lane lines, etc. in the automotive field, as well as the recording and acquisition of driving recorders. Therefore, using HDR sensors as image acquisition devices in the automotive field will become an inevitable trend.

[0055] Before outputting a raw image, an HDR sensor captures multiple exposure frames. The raw image is the result of fitting these multiple exposure frames. If each exposure frame exhibits color cast, color fringing will appear in the transition area between adjacent frames after fitting. Therefore, when the sensor detects that it is operating in high dynamic range acquisition mode, adjusting the sensor's digital gain can eliminate color cast issues in each exposure frame before fitting, thereby preventing color fringing in the transition area of ​​the fitted result.

[0056] Based on the actual lighting conditions, the exposure image includes at least a highlight exposure image corresponding to high-brightness areas, a dark exposure image corresponding to low-brightness areas, and a normal exposure image corresponding to medium-brightness areas. The HDR sensor controls the exposure intensity by adjusting the exposure duration to render details and content in objects of varying brightness. Specifically, a short exposure duration can render details and content in high-brightness areas, a long exposure duration can render details and content in low-brightness areas, and a moderate exposure duration can render details and content in medium-brightness areas.

[0057] The historical image is the previous output result of the original image output by the sensor, and the historical image and the original image are adjacent in output timing. The channel gain corresponding to the historical image is the white balance gain obtained based on the historical image, including the red channel gain and the blue channel gain. Based on the above, in related technologies, the current original image is usually used to adjust the white balance gain of the ISP, while this method uses the channel gain obtained last time to adjust the digital gain of the sensor, so that the balance of the RGB three channels is achieved before the original image is fitted. Therefore, the original image after fitting will not have color cast or color in the transition area (especially the edge of the highlight object), which fundamentally avoids the color edge problem of the original image.

[0058] The digital gain of a sensor refers to the parameter used in the sensor to perform white balance processing on an image, and is used to characterize the light source characteristics of the lighting environment.

[0059] Step S104 : performing white balance processing on the multiple frames of exposure images using a sensor to obtain original images.

[0060] Among them, the sensor performs white balance processing on multiple frames of exposure images so that the transition area in the original image presents the target color.

[0061] Specifically, step S104 may be implemented as follows: the sensor performs white balance processing on each frame of exposure image respectively to obtain at least a highlight correction image, a dark area correction image, and a general correction image, wherein the transition areas in the highlight correction image, the dark area correction image, and the general correction image all present the target color; feature cropping is performed on the highlight correction image, the dark area correction image, and the general correction image respectively to obtain highlight correction segments corresponding to high-brightness areas, dark area correction segments corresponding to low-brightness areas, and general correction segments corresponding to general brightness areas; the highlight correction segments, the dark area correction segments, and the general correction segments are spliced ​​in sequence to obtain the original image.

[0062] In other words, each acquired exposure image is white-balanced before fitting to overcome color casts. The corresponding portion of each corrected image that matches the exposure level is then cropped. Finally, these cropped segments are stitched together into a complete image based on the actual scene position. This ensures that the transition between adjacent segments is free of color fringing, and details and content are clearly presented at all brightness levels.

[0063] Among them, the target color representation is the color that matches the actual lighting environment and is controlled by the digital gain of the sensor; then when the white part in each exposure image can present a true white state, then when the splicing edges of the fragments corresponding to each exposure image are fitted, no color edges in the transition area will appear.

[0064] Step S106 : performing full-image color correction on the original image based on the channel gain of the original image and the color temperature of the target environment where the sensor is located, so as to obtain a target image that matches the color temperature.

[0065] The channel gain (ie, white balance gain) of the original image is essentially the correction factor of the corresponding color channel, including the red channel gain and the blue channel gain, which is used to adjust the average value of the number of pixels of various colors in the original image.

[0066] Color temperature (CCT) refers to the actual light source color of the target environment. When the target environment is fixed, the corresponding CCT should be unique and fixed.

[0067] Due to equipment limitations, the captured original image may exhibit color cast, meaning the CCT of the captured image deviates from the actual CCT seen by the human eye. To address this, color correction is performed on the original image using the channel gain derived from the original image and the CCT of the target environment (i.e., the actual lighting environment). This restores the actual light source color of the target environment and produces a target image that matches the CCT of the target environment.

[0068] In some embodiments, before step S102 , the method may further include: determining a channel gain of each color channel based on calibration parameters corresponding to the historical image, where the color channels include at least a red channel and a blue channel.

[0069] The calibration parameter corresponding to the historical image is the number of pixels for which the deviation between the original image and the illumination environment is less than or equal to the maximum limit, and the reciprocal of this parameter is the channel gain corresponding to the historical image. Since the adjustment of the sensor's digital gain is based on the channel gain of the historical image, the channel gain corresponding to the historical image needs to be determined before step S102. Different light sources correspond to different calibration parameters, and before this step, there are some preparatory steps to determine the calibration parameters of each standard light source (which will be described later).

[0070] The channel gain can be expressed as:

[0071] cali_x_gain=1 / cali_x_g=g_avg / x_avg;

[0072] Among them, cali_x_gain represents the channel gain of the target channel, including at least the channel gain cali_r_gain of the red channel and the channel gain cali_b_gain of the blue channel; cali_x_g represents the calibration parameters of the target channel, including at least the calibration parameters cali_r_g of the red channel and cali_b_g of the blue channel; g_avg represents the average number of green pixels in the historical image; x_avg represents the average number of target color pixels in the historical image, including at least the average number of red pixels r_avg and the average number of blue pixels b_avg.

[0073] In some embodiments, before determining the channel gain of each color channel based on the calibration parameters corresponding to the historical image, it also includes: determining the calibration parameters of the sensor under various standard light sources, wherein the calibration parameters include at least a blue channel calibration parameter and a red channel calibration parameter, and the reciprocal of the calibration parameter is used to characterize the channel gain of each color channel.

[0074] Standard light sources are artificial light sources that simulate various ambient lighting conditions, allowing off-site environments like production plants and production rooms to achieve lighting effects that are essentially the same as those found in specific environments. Standard light sources can be H, A, TL84, D50, D65, D75, U30, and others. The aforementioned examples all refer to the color temperatures of standard light sources.

[0075] Different working modes of the sensor use different calibration parameter determination processes. Figure 5 The calibration parameter determination process in various working modes is explained.

[0076] When the sensor operates in linear acquisition mode, the calibration parameter determination process is as follows: Determine whether the sensor operates in HDR acquisition mode. If not, set the sensor's pre-HDR gain to 0. This process is equivalent to issuing a linear acquisition mode calibration parameter instruction. Furthermore, the sensor is used to capture an 18% gray card under each of the aforementioned standard light sources, with the card positioned in the center of the image and occupying 50% of the image area. The sensor exposure time is adjusted to maintain normal image exposure, and then a raw image of any scene under standard light is captured. The number of red and blue pixels in the center of the raw image, occupying 10% of the field of view, is counted to obtain the calibration parameters cali_r_g for the red channel and cali_b_g for the blue channel.

[0077]

[0078]

[0079]

[0080] Among them, cali_r_gain = r_avg / g_avg; cali_b_gain = b_avg / g_avg.

[0081] Where i is the serial number of the region of interest in the image captured by the sensor, ri represents the number of red pixels in the i-th region of interest, N is the total number of pixels in the region of interest (N satisfies the requirement of being zero when multiplied by 4, that is, N mod 4 = 0), bi is the number of blue pixels in the i-th region of interest; g_avg is the average number of green pixels, gri represents the total number of red and green pixels in the i-th region of interest, and gbi represents the total number of blue and green pixels in the i-th region of interest.

[0082] Each standard light source corresponds to a unique calibration parameter, and the calibration parameters cali_r_g and cali_b_g are eventually written into the sensor device memory.

[0083] When the sensor's operating mode is high dynamic range acquisition mode, the process for determining the calibration parameters is as follows: Determine whether the sensor's operating mode is HDR acquisition mode. If so, set the sensor's pre-HDR gain to 1 (all three RGB channels are set to 1). This process is equivalent to issuing a calibration parameter instruction for HDR acquisition mode. Furthermore, set various standard light sources, and capture an original image of an 18% gray card under each standard light source. Then, determine the red channel calibration parameter cali_r_g and the blue channel calibration parameter cali_b_g according to the calibration parameter acquisition process for linear acquisition mode. This will not be repeated here.

[0084] The difference is that after obtaining cali_r_g and cali_b_g, their reciprocals are taken: cali_r_gain = 1 / cali_r_g, cali_b_gain = 1 / cali_b_g, to obtain the channel gains of the corresponding channels and set them to the corresponding digital gains in the HDR sensor, pre-HDR gains. When the HDR sensor synthesizes the raw image, it uses the sensor's built-in synthesis algorithm to synthesize multiple frames of exposure images inside the sensor. The synthesized raw image will have nonlinear components. Therefore, the gain obtained by the reciprocal is written to the digital gain of the HDR sensor. The white balance state of the raw image may deviate from the corresponding standard light source. Based on this, it is necessary to verify the white balance results of the digital gain and fine-tune the digital gain that does not meet expectations to ensure the accuracy of the calibration parameters.

[0085] The verification process involves capturing an original image of an X-Rite standard 24-color chart under a selected standard light source, with the chart centered within the original image and occupying 75% of the original image's field of view. The deltaC deviation values ​​of the grayscale blocks in the original image (i.e., blocks 20 to 23 of the 24-color chart) are analyzed. When multiple grayscale blocks are selected (or a single grayscale block is optional), deltaC can be the average or weighted average of the deviation values ​​for each grayscale block. Furthermore, a maximum deltaC limit is set. A lower deltaC indicates higher color accuracy. Therefore, when deltaC is less than or equal to the maximum limit, the white balance result is reliable. Otherwise, the sensor's digital gain needs to be adjusted. This involves repeatedly capturing the 24-color chart so that the deviation values ​​of the grayscale blocks in the original image are less than or equal to the maximum limit. DeltaC typically ranges from 0 to 5. Finally, when deltaC is less than or equal to the maximum limit, the inverse of the optimized digital gain is used as the white balance calibration parameter for the HDR sensor under the standard light source. Subsequently, switch to other standard light sources and repeat the above process to obtain the calibration parameters corresponding to each standard light source and store them in the device memory.

[0086] Specifically, an original image of a target gray card is captured under a standard light source, initial calibration parameters of the original image of the target gray card are counted, and the inverse of the initial calibration parameters is used as an initial channel gain; the initial channel gain is used as an initial digital gain of the sensor; the original image of the grayscale block is collected by the sensor, and the deviation value of the original image of the grayscale block is analyzed, wherein the deviation value is used to characterize the color accuracy of the original image collected by the sensor, and the deviation value is inversely proportional to the color accuracy; in response to a judgment result that the deviation value is greater than a maximum limit value, the initial digital gain is adjusted using the deviation value to obtain a process digital gain; and until the deviation value is less than or equal to the maximum limit value, the inverse of the process digital gain is used as the calibration parameter of the sensor under the standard light source and stored in the device memory.

[0087] In some embodiments, before step S102 , the method includes: determining an operating mode of the sensor, wherein the operating mode includes at least a high dynamic range acquisition mode and a linear acquisition mode.

[0088] Since different working modes of the sensor will match different white balance processing methods, this method provides a working mode identification step. It can be seen that this method also supports the white balance processing method of the linear acquisition mode of the related technology.

[0089] In some embodiments, in response to the recognition result that the operating mode of the sensor is a linear acquisition mode, the channel gain of the original image and the color temperature of the target environment in which the sensor is located are obtained; the channel gain of the original image is used to adjust the digital gain of the image signal processor; based on the color temperature, the image signal processor is used to perform full-image color correction on the original image to obtain a target image that matches the color temperature.

[0090] In some embodiments, before step S104, the method further includes: using a sensor to capture multiple frames of exposure images, wherein the exposure images at least include a highlight exposure image for presenting details of a high-brightness area, a dark area exposure image for presenting details of a low-brightness area, and a general exposure image for presenting a general brightness area.

[0091] This step is generated based on the characteristics of the HDR sensor and will not be described in detail here.

[0092] In some embodiments, after step S104, the process further includes: compressing the original image by the sensor into a 12-bit to 16-bit original image compression package; before processing by the ISP, the compressed original image package needs to be decompressed to obtain the original image using a 20-bit or 24-bit to 16-bit lookup table mapping; further, preprocessing is performed by the ISP, and the preprocessing may include vignetting correction, bad pixel correction, and exposure control, etc., which will not be described in detail here. After obtaining the preprocessed original image, the AWB module is used to perform AWB statistics on the original image to obtain its AWB status information; at the same time, the preprocessed original image is also sent to the ISP white balance gain unit for subsequent white balance processing. The above processes are similar to the white balance processing methods of the related art, and reference can be made to the above text, which will not be described in detail here.

[0093] The difference between this method and related technologies is that the AWB module executes the process of the AWB core algorithm. Specifically, Figure 6As shown, a new step of compensating the white balance state information is added, that is, before step S106, the following steps are further provided: compensating the white balance state information to shield the weight of the channel gain of the historical image in the white balance state information, obtaining target state information for characterizing the white balance state of the target environment; and estimating the color temperature of the target environment based on the target state information.

[0094] Specifically, based on the principle of the AWB module executing the AWB core algorithm, the processing can be viewed as a linear process. Assuming I is the image processed by the AWB module, I_raw is the original image, and G is the abbreviation for the gain output by the AWB module, then I = I_raw * G. This formula shows that the input to the ISP is I, which is equivalent to the product of the original image I_raw, which reflects the current lighting environment, and the digital gain (pre-HDR gain) calculated from the previous frame (i.e., G). Therefore, the purpose of white balance compensation is to compensate for the pre-HDR gain of the previous frame, thereby eliminating the interference of the pre-HDR gain on the color temperature of the current image. The compensated image then reflects the current lighting environment. Compensation can be applied directly to the white balance statistics. This method requires no additional hardware support, only modifications to the underlying software, and is supported by common ISP architectures.

[0095] More specifically, after obtaining the compensated white balance state information, the state information is analyzed to obtain weighted white balance state information, and the weighted white balance state information is used to estimate the CCT; then, the white balance gain is calculated based on the CCT estimation value and the weighted white balance state information, and the result of the white balance gain calculation (the channel gain of the original image) is synchronized to the white balance state information compensation unit and written to the HDR sensor at the same time; finally, the ISP white balance gain is adjusted for the ISP using the CCT estimation value and the result of the white balance gain calculation, so that the adjusted white balance gain is written to the ISP register.

[0096] Specifically, the white balance state information compensation is implemented by the white balance state information compensation unit, which is mainly used to compensate for the current input AWB state information. Since this method controls the digital gain pre-HDRgain of the sensor, the output original image cannot reflect the white balance state of the current lighting environment. The target state information obtained after the white balance state information compensation can reflect the current white balance state. For the specific implementation process, please refer to Figure 7 , define the AWB_gain structure. When the AWB module is executed for the first time, both pre_gain and cur_gain are 1.000. Define the AWB_sates (i.e. white balance status information) structure. The calculation of the target status information after compensation is:

[0097] compst_awb_stats.r_g=raw_awb_stats.r_g / pre_gain.r_gain;

[0098] compst_awb_stats.b_g=raw_awb_stats.b_g / pre_gain.b_gain;

[0099] sensor_gain_en is the enable bit of the digital gain before HDR synthesis (sensor pre-HDR gain): if it is 1, sensor pre-HDR gain is enabled, pre_gain = pre_gain * cur_gain, where the pre_gain to the right of the "=" is the previous channel gain, and its product with the current channel gain cur_gain is used as the new pre_gain (i.e., the pre_gain to the left of the "="). The new pre_gain is used as compensation for the next AWB state information ( / pre_gain); if it is not 1, sensor pre-HDR gain is not enabled, and pre_gain and cur_gain maintain the default values. That is, when sensor_gain_en = 0, this control method is applicable to linear sensors.

[0100] Specifically, during the initialization process, the internal program code of the white balance status information can be expressed as:

[0101]

[0102] After reading the gain pre_gain, the white balance state information is compensated, that is, each parameter is processed by " / pre_gain" to obtain the target state information. Its internal program code can be expressed as:

[0103]

[0104] Check whether sensor_gain_en=1 (the sensor enable bit) is 1. If not, do not adjust pre_gain and cur_gain, leaving them at their default values. If it is, read cur_gain from the sensor. Furthermore, multiply pre_gain by cur_gain to assign the next frame's pre_gain value to the new pre_gain used for the next compensation, where pre_gain = pre_gain * cur_gain.

[0105] The internal program code of the new pre_gain can be expressed as:

[0106] pre_gain.r_gain=pre_gain.r_gain*cur_gain.r_gain;

[0107] pre_gain.b_gain=pre_gain.b_gain*cur_gain.b_gain;

[0108] Specifically, the process of state analysis of the target state information after compensation can be referred to Figure 8 The weighted white balance state information generated can represent the state point information of the current lighting environment and serve as the input for subsequent white balance gain and CCT estimation. When performing state analysis, the output result is a series of state point information output after luminance weighting, gray area weighting, and CCT weighting, as well as the weight values ​​taken for each CCT interval.

[0109] More specifically, the steps for obtaining weighted white balance status information for each CCT interval are as follows: obtaining compensated white balance status information, removing overexposed and dark pixels from the sample; removing non-gray pixels from the sample based on gray region boundary parameters; weighting gray region pixels based on gray region weight parameters; calculating white balance status information for each CCT interval based on white balance calibration parameters under various light sources; generating white balance status information for each CCT interval; and performing illumination weighting on the white balance status information based on the illumination weight parameters, ultimately obtaining weighted white balance status information for each CCT interval. This implementation directly utilizes existing technology, and the detailed process is not further described.

[0110] The white balance processing method proposed in the present disclosure is applicable to HDR sensors and linear sensors, and solves the color edge problem of images generated by HDR sensors. In addition, based on the architecture proposed for HDR sensors, a white balance compensation step is added to ensure that the final output target image matches the color temperature of the target environment.

[0111] Figure 9 Schematic diagram of a white balance processing device according to an exemplary embodiment of the present disclosure.

[0112] like Figure 9As shown, the present disclosure proposes a white balance processing device 1000, which may include: a sensor parameter adjustment module 1002, a raw image generation module 1004, and a full-image color correction module 1006. The sensor parameter adjustment module 1002 is configured to adjust the digital gain of the sensor using the channel gain corresponding to the historical image in response to the recognition result that the operating mode of the sensor is the high dynamic range acquisition mode; the raw image generation module 1004 is configured to use the sensor to perform white balance processing on multiple exposure frames to obtain a raw image, wherein the transition area in the raw image presents a target color; and the full-image color correction module 1006 is configured to perform full-image color correction on the raw image based on the channel gain of the raw image and the color temperature of the target environment in which the sensor is located, to obtain a target image that matches the color temperature.

[0113] The various modules of the white balance processing device 1000 are proposed to execute various steps of the white balance processing method. The implementation principles and execution steps thereof can be referred to above and will not be described in detail here.

[0114] The device may include corresponding modules for executing each or several steps in the above flowchart. Therefore, each step or several steps in the above flowchart may be executed by a corresponding module, and the device may include one or more of these modules. The module may be one or more hardware modules specifically configured to execute the corresponding steps, or implemented by a processor configured to execute the corresponding steps, or stored in a computer-readable medium for execution by a processor, or implemented by some combination thereof.

[0115] The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0116] Bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component Architecture (EISA) bus. Buses can be classified as address buses, data buses, control buses, and the like. For ease of illustration, this figure shows only one connecting line, but this does not imply that there is only one bus or only one type of bus.

[0117] The white balance processing device proposed in the present disclosure is applicable to HDR sensors and linear sensors, and solves the color edge problem of images generated by HDR sensors. In addition, based on the architecture proposed for HDR sensors, a white balance compensation step is added to ensure that the final output target image matches the color temperature of the target environment.

[0118] Any process or method description in the flowchart or otherwise described herein can be understood to represent a module, fragment or portion of code including one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong. The processor performs the various methods and processes described above. For example, the method embodiments of the present disclosure can be implemented as a software program that is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).

[0119] The logic and / or steps represented in the flowchart or otherwise described herein may be embodied in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device).

[0120] For the purposes of this specification, a "readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use with or in conjunction with an instruction execution system, device or apparatus. More specific examples (a non-exhaustive list) of readable storage media include the following: an electrical connection having one or more wires (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), a fiber optic device, and a portable read-only memory (CDROM). In addition, the readable storage medium can even be paper or other suitable medium on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a memory.

[0121] It should be understood that various parts of the present disclosure can be implemented using hardware, software, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement the hardware: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0122] Those skilled in the art will understand that all or part of the steps of the above-mentioned implementation method can be accomplished by instructing related hardware through a program, and the program can be stored in a readable storage medium. When the program is executed, it includes one or a combination of the steps of the method implementation method.

[0123] Furthermore, the functional units in the various embodiments of the present disclosure may be integrated into a single processing module, each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules. If the integrated modules are implemented as software functional modules and sold or used as independent products, they may also be stored in a readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0124] The present disclosure also provides an electronic device, including: a memory, the memory storing execution instructions; and a processor or other hardware module, the processor or other hardware module executing the execution instructions stored in the memory, so that the processor or other hardware module performs a white balance gain compensation method.

[0125] The present disclosure further provides a readable storage medium, in which execution instructions are stored. When the execution instructions are executed by a processor, the method for compensating a white balance gain is implemented.

[0126] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present disclosure. In this specification, the schematic representations of the above terms are not necessarily the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described may be combined in a suitable manner in any one or more embodiments / methods or examples. In addition, those skilled in the art may combine and combine the different embodiments / methods or examples described in this specification and the features of the different embodiments / methods or examples, unless they are mutually inconsistent.

[0127] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0128] Those skilled in the art will appreciate that the above embodiments are merely intended to clearly illustrate the present disclosure and are not intended to limit the scope of the present disclosure. Other changes or modifications may be made based on the above disclosure, and such changes or modifications are still within the scope of the present disclosure.

Claims

1. A white balance processing method, characterized in that: include: In response to an identification result that the operating mode of the sensor is a high dynamic range acquisition mode, adjusting the digital gain of the sensor using a channel gain corresponding to the historical image; Utilizing the sensor to perform white balance processing on multiple frames of exposure images to obtain original images, the method includes: performing white balance processing on each frame of the exposure image by the sensor to obtain at least a highlight correction image, a dark area correction image, and a general correction image, wherein transition areas in the highlight correction image, the dark area correction image, and the general correction image all present a target color; performing feature cropping on the highlight correction image, the dark area correction image, and the general correction image to obtain highlight correction segments corresponding to high brightness areas, dark area correction segments corresponding to low brightness areas, and general correction segments corresponding to general brightness areas; sequentially splicing the highlight correction segments, the dark area correction segments, and the general correction segments to obtain the original image; wherein the transition areas in the original image present the target color; and Performing full-image color correction on the original image based on the channel gain of the original image and the color temperature of the target environment in which the sensor is located, so as to obtain a target image that matches the color temperature; The historical image is a previous output result of the original image output by the sensor, and the historical image and the original image are adjacent in output time sequence; Before performing full-image color correction on the original image based on the color temperature of the target environment in which the sensor is located to obtain a target image matching the color temperature, the method includes: The white balance state information is compensated to shield the weight of the channel gain of the historical image in the white balance state information to obtain target state information for characterizing the white balance state of the target environment; and the color temperature of the target environment is estimated based on the target state information.

2. The white balance processing method according to claim 1, wherein: Before adjusting the digital gain of the sensor using the channel gain corresponding to the historical image in response to the recognition result that the working mode of the sensor is the high dynamic range acquisition mode, the method includes: The channel gain of each color channel is determined based on the calibration parameters corresponding to the historical image, wherein the color channels include at least a red channel and a blue channel.

3. The white balance processing method according to claim 2, wherein: The channel gain is expressed as: cali_x_gain=1 / cali_x_g=g_avg / x_avg; Among them, cali_x_gain represents the channel gain of the target channel, including at least the channel gain cali_r_gain of the red channel and the channel gain cali_b_gain of the blue channel; cali_x_g represents the calibration parameters of the target channel, including at least the calibration parameters cali_r_g of the red channel and the calibration parameters cali_b_g of the blue channel; g_avg represents the average number of green pixels in the historical image; x_avg represents the average number of target color pixels in the historical image, including at least the average number of red pixels r_avg and the average number of blue pixels b_avg.

4. The white balance processing method according to claim 3, wherein: Before determining the channel gain of each color channel based on the calibration parameters corresponding to the historical image, the method further includes: Calibration parameters of the sensor under various standard light sources are determined, wherein the calibration parameters include at least a blue channel calibration parameter and a red channel calibration parameter, and the reciprocals of the calibration parameters are used to characterize the channel gain of each color channel.

5. The white balance processing method according to claim 4, wherein: Determining the calibration parameters of the sensor under various standard light sources includes: Taking an original image of a target gray card under the standard light source, calculating initial calibration parameters of the original image of the target gray card, and using the reciprocal of the initial calibration parameters as the initial channel gain; Using the initial channel gain as the initial digital gain of the sensor; Using the sensor to collect an original image of a grayscale block, and analyzing a deviation value of the original image of the grayscale block, wherein the deviation value is used to represent the color accuracy of the original image collected by the sensor, and the deviation value is inversely proportional to the color accuracy; In response to a determination that the deviation value is greater than a maximum limit value, adjusting the initial digital gain using the deviation value to obtain a process digital gain; and Until the deviation value is less than or equal to the maximum limit, the reciprocal of the process digital gain is used as the calibration parameter of the sensor under the standard light source and stored in the device memory.

6. The white balance processing method according to claim 1, wherein: Before adjusting the digital gain of the sensor using the channel gain corresponding to the historical image in response to the recognition result that the working mode of the sensor is the high dynamic range acquisition mode, the method includes: An operating mode of the sensor is determined, wherein the operating mode includes at least the high dynamic range acquisition mode and the linear acquisition mode.

7. The white balance processing method according to claim 6, wherein: After determining the operating mode of the sensor, the method further includes: In response to a recognition result that the operating mode of the sensor is the linear acquisition mode, obtaining a channel gain of the original image and a color temperature of a target environment in which the sensor is located; Adjusting the digital gain of an image signal processor using the channel gain of the original image; Based on the color temperature, the image signal processor is used to perform full-image color correction on the original image to obtain a target image that matches the color temperature.

8. The white balance processing method according to claim 1, wherein: Before performing white balance processing on the multiple frames of exposure images using the sensor to obtain original images, the method includes: The sensor is used to collect multiple frames of exposure images, wherein the exposure images include at least a highlight exposure image for presenting details of a high-brightness area, a dark area exposure image for presenting details of a low-brightness area, and a general exposure image for presenting a general-brightness area.

9. A white balance processing device, characterized in that: include: a sensor parameter adjustment module, configured to adjust the digital gain of the sensor using the channel gain corresponding to the historical image in response to a recognition result that the operating mode of the sensor is the high dynamic range acquisition mode; an original image generation module, configured to perform white balance processing on multiple frames of exposure images using the sensor to obtain original images, comprising: performing white balance processing on each frame of the exposure image by the sensor to obtain at least a highlight correction image, a dark area correction image, and a general correction image, wherein transition areas in the highlight correction image, the dark area correction image, and the general correction image all present a target color; performing feature cropping on the highlight correction image, the dark area correction image, and the general correction image to obtain highlight correction segments corresponding to high brightness areas, dark area correction segments corresponding to low brightness areas, and general correction segments corresponding to general brightness areas; and sequentially splicing the highlight correction segments, the dark area correction segments, and the general correction segments to obtain an original image; wherein the transition areas in the original image present the target color; and a full-image color correction module, configured to perform full-image color correction on the original image based on the channel gain of the original image and the color temperature of the target environment in which the sensor is located, so as to obtain a target image that matches the color temperature; The historical image is a previous output result of the original image output by the sensor, and the historical image and the original image are adjacent in output time sequence; Before performing full-image color correction on the original image based on the color temperature of the target environment in which the sensor is located to obtain a target image matching the color temperature, the method includes: The white balance state information is compensated to shield the weight of the channel gain of the historical image in the white balance state information to obtain target state information for characterizing the white balance state of the target environment; and the color temperature of the target environment is estimated based on the target state information.

10. An electronic device, characterized in that: include: a memory storing execution instructions; as well as A processor, wherein the processor executes the execution instruction stored in the memory, so that the processor performs the white balance processing method according to any one of claims 1 to 8.

11. A readable storage medium, characterized in that: The readable storage medium stores an execution instruction, and when the execution instruction is executed by the processor, it is used to implement the white balance processing method according to any one of claims 1 to 8.

Images