Automatic white balance method, adjusting system and white balance terminal

Abstract

The invention provides an automatic white balance method, an adjustment system and a white balance terminal, and the method comprises the steps: obtaining a to-be-corrected image through an image collection module, and estimating the scene type of the to-be-corrected image through a scene information extraction module; according to the scene category, applying pre-calibrated white point information corresponding to the scene category; the white point information comprises white point division areas of different color temperatures and respective white point weights of the different color temperatures. Calculating three primary color weighted accumulated pixel values of red, green and blue of all pixel points falling into a white point division area included in the scene category; calculating white balance gain coefficients of three primary colors of red, green and blue; different scene categories have different color temperature distribution preferences, white point division areas, weights and color preferences are adjusted in real time according to scene classification, the influence of interference color blocks is reduced, and more accurate white balance parameters are obtained. When scene classification does not change for a long time, the stability is improved by limiting the gain adjustment amount and the convergence range of white balance; deployment is easy, and universality is high.

Description

Technical Field

[0001] The invention belongs to the technical field of white balance adjustment, and in particular relates to an automatic white balance method and adjustment system, and a white balance terminal. Background Art

[0002] White balance: restores the color of the white area in the scene to the white perceived by the human eye under different color temperature environments. Objects will have a series of color casts under different lighting environments or color temperatures, and the camera's white balance corrects the color cast of the object and restores it to the object's original color.

[0003] Limitations of traditional white balance algorithms: 1. Interference points and white points may have similar landing points; counting too many interference points will cause inaccurate weighted cumulative pixel values ​​of the three primary colors of red, green and blue, resulting in deviations in the calculation of the white balance gain coefficients of the three primary colors, which in turn causes image color cast. 2. In a mixed color temperature motion scene, the proportion of white points under different color temperatures covered by the image statistical area changes, and the statistical value of the white point area fluctuates accordingly, causing the automatic white balance (AWB) to be unstable.

[0004] A limitation of a public deep learning-based white balance algorithm: inputting a color-biased image will produce a color-free image. This type of task dataset is easy to build, but the output image resolution is high, the model has a large number of parameters and computation, cannot meet real-time requirements, and is difficult to deploy.

[0005] Another limitation of the white balance algorithm based on deep learning is that it takes a color-biased image as input, estimates the white balance gain parameter, and then applies it to the color-biased image to obtain a color-free image. This type of task model is relatively small, but the response characteristics of different modules are different, making it difficult to find a suitable mapping relationship. Data set collection and training work needs to be repeated for different modules, and the versatility is not strong. Summary of the invention

[0006] The object of the present invention is to provide an automatic white balance method and adjustment system, and a white balance terminal. Since different categories have different color temperature distribution preferences, the present invention can adjust the white point division area, weight and color preference in real time according to the scene classification information, minimize the influence of interfering color blocks, and obtain more accurate white balance parameters. When the scene classification does not change for a long time, the stability can be improved by limiting the gain adjustment amount and convergence range of the white balance. The scene classification detection task technology of the present invention is mature, the model parameters are few, the reasoning speed is fast, and it is easy to deploy. The present invention does not need to estimate accurate white balance parameters in the deep learning stage, nor does it need to repeat training of different modules, and has strong versatility.

[0007] The present invention provides an automatic white balance method, comprising:

[0008] S1. The image acquisition module acquires the image to be corrected, and estimates the scene category of the image to be corrected through the scene information extraction module; according to the scene category, the pre-calibrated white point information corresponding to the scene category is applied; the white point information includes white point division areas of different color temperatures and white point weights of different color temperatures;

[0009] S2, determining whether the pixel points of the area of ​​interest in the image to be corrected fall into the white point division area included in the scene category; calculating the weighted cumulative pixel values ​​of the three primary colors of red, green and blue of all the pixel points falling into the white point division area included in the scene category, which are rv, gv and bv respectively;

[0010] S3, calculate the white balance gain coefficients of the three primary colors of red, green and blue, which are IRgain, IGgain and IBgain respectively; so that rv*IRgain=gv*IGgain=bv*IBgain holds;

[0011] S4. When it is detected according to the inter-frame scene information that the scene category has not changed for a long time, the gain adjustment amount and the convergence range of the white balance are limited.

[0012] Furthermore, the calibration used in step S1 specifically includes:

[0013] S01, using the image sensor in the image acquisition module to shoot the gray card images at different color temperatures in the light box; calibrating the white balance reference points of the different color temperatures, and fitting the color temperature curve according to the white balance reference points of the different color temperatures, and drawing the white point areas corresponding to the different color temperatures based on the color temperature curve to obtain an initial calibrated white point area distribution map;

[0014] S02. Photographing gray card images under different scene categories, drawing white point division areas corresponding to the different scene categories and containing the different color temperatures, and assigning the respective white point weights to the different color temperatures.

[0015] Further, in step S01, the pixel points in the middle area of ​​the gray card under each color temperature are taken as valid pixel points; for all valid pixel points under one of the color temperatures, the unweighted cumulative pixel values ​​of the three primary colors of red, green and blue are calculated, the ratio of the unweighted cumulative pixel value of the red primary color to the unweighted cumulative pixel value of the green primary color is taken as the horizontal coordinate value of the white balance reference point of the one color temperature, and the ratio of the unweighted cumulative pixel value of the blue primary color to the unweighted cumulative pixel value of the green primary color is taken as the vertical coordinate value of the white balance reference point of the one color temperature, to obtain the white balance reference point of the one color temperature; similarly, the white balance reference point of each color temperature is calculated; a color temperature curve is fitted according to the white balance reference points of different color temperatures, and a certain envelope graph is drawn with the point on the color temperature curve as the center point, and the set covered by all the envelope graphs is the initial calibration white point area distribution map;

[0016] In step S02, the pixel points in the middle area of ​​the gray card under each of the scene categories are taken as valid pixel points; for all the valid pixel points in the middle area of ​​the gray card under each of the scene categories, the B / G and R / G values ​​of each of the valid pixel points are calculated, and the white point position of each of the valid pixel points in the coordinate system is correspondingly drawn; according to the white point position, the white point area during the calibration in step S01, i.e., the size of the envelope graph and the white point weight, i.e., the weight value falling within the range of the envelope graph, are adjusted as the white point calibration result of the scene.

[0017] Furthermore, the scene categories include: indoor, semi-indoor and outdoor; the outdoor is subdivided into sunny, cloudy, sunset and dusk according to different color temperatures; the indoor scene and the semi-indoor scene both include: six color temperatures: H, A, CWF, D50, D65 and D75;

[0018] In the indoor scene, the white point weights corresponding to the six color temperatures are the same;

[0019] The white point weight corresponding to the D65 color temperature of the semi-indoor scene is larger than the white point weight corresponding to the D65 color temperature in the initial calibration color temperature curve.

[0020] Furthermore, the four outdoor scenes of sunny day, cloudy day, sunset and dusk all include three color temperatures: D50, D65 and D75;

[0021] The white point weight corresponding to the D50 color temperature of the sunny scene is larger than the white point weight corresponding to the D50 color temperature in the initial calibration color temperature curve; the white point weight corresponding to the D65 color temperature in the sunny scene is larger than the white point weight corresponding to the D65 color temperature in the initial calibration color temperature curve;

[0022] The white point weights corresponding to the D50 color temperature, the D65 color temperature and the D75 color temperature in the cloudy scene are larger than the white point weights corresponding to the same color temperatures in the initial calibration color temperature curve diagram;

[0023] The white point weight corresponding to the D50 color temperature in the sunset scene is larger than the white point weight corresponding to the D50 color temperature in the initial calibration color temperature curve;

[0024] The white point weights corresponding to the D65 color temperature and the D75 color temperature in the dusk scene are larger than the white point weights corresponding to the same color temperatures in the initial calibration color temperature curve.

[0025] Furthermore, after step S3 and before step S4, the method further includes:

[0026] According to different scene categories, the calculated white balance gain coefficients of the three primary colors of red, green and blue are adjusted with preference; wherein the gain adjustment coefficients of the three primary colors of red, green and blue are Rshift, Gshift and Bshift respectively; the adjusted white balance gain coefficients of the three primary colors are Rgain, Ggain and Bgain respectively; Rgain = IRgain*Rshift; Ggain = IGgain*Gshift; Bgain = IBgain*Bshift.

[0027] Further, in step S4, the parameters of the convergence range include red, green and blue primary color gain coefficient thresholds corresponding to different scenes, which are SceneThreRgain, SceneThreGgain and SceneThreBgain respectively;

[0028] The restrictions when calculating the white balance gain coefficient of the nth frame are as follows:

[0029] Rgain n =Rgain n-1 +step, and Rgain n <SceneThreRgain;

[0030] Ggain n =Ggain n-1 +step, and Ggain n <SceneThreGgain;

[0031] Bgain n =Bgain n-1 +step, and Bgain n <SceneThreBgain。

[0032] The adjusted white balance gain coefficients of the three primary colors in the nth frame are Rgain and n , Ggain n and Bgain n The adjusted white balance gain coefficients of the three primary colors in the n-1th frame are Rgain n-1 , Ggain n-1 and Bgain n-1 ; step is the gain adjustment amount.

[0033] The present invention also provides an automatic white balance adjustment system, comprising:

[0034] An image acquisition module, used for acquiring an image to be corrected;

[0035] A scene information extraction module, used to estimate the scene category of the image to be corrected;

[0036] The white balance calculation and correction module is used to calculate the white balance gain coefficients of the three primary colors of red, green and blue, which are IRgain, IGgain and IBgain respectively; so that rv*IRgain=gv*IGgain=bv*IBgain holds; the weighted cumulative pixel values ​​of the three primary colors of red, green and blue are rv, gv and bv respectively.

[0037] Furthermore, the image acquisition module includes an image sensor and a preprocessing module, and the preprocessing module performs image signal processing on the image acquired by the image sensor;

[0038] The scene information extraction module includes: a neural network processor; the neural network processor deploys a trained deep learning network to estimate the scene category corresponding to the image to be corrected from the image to be corrected; and outputs the scene category to the white balance calculation and correction module to guide the white balance calculation and correction processing.

[0039] The present invention also provides a white balance terminal, comprising:

[0040] A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the above-mentioned automatic white balance method by executing the computer instructions.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention provides an automatic white balance method and adjustment system, and a white balance terminal, comprising: S1, an image acquisition module acquires an image to be corrected, and estimates the scene category of the image to be corrected through a scene information extraction module; according to the scene category, applying pre-calibrated white point information corresponding to the scene category; the white point information includes white point division areas of different color temperatures and respective white point weights of different color temperatures; S2, judging whether the pixel points of the area of ​​interest in the image to be corrected fall into the white point division area included in the scene category; calculating the weighted cumulative pixel values ​​of the three primary colors of red, green and blue of all the pixel points falling into the white point division area included in the scene category, which are rv, gv and bv respectively; S3, calculating the white balance gain coefficients of the three primary colors of red, green and blue, which are IRgain, IGgain and IBgain respectively; so that rv*IRgain=gv*IGgain=bv*IBgain holds;

[0043] S4. When it is detected according to the inter-frame scene information that the scene category has not changed for a long time, the gain adjustment amount and the convergence range of the white balance are limited.

[0044] Since different categories have different color temperature distribution preferences, the present invention can adjust the white point division area, weight and color preference in real time according to the scene classification information, minimize the influence of interfering color blocks, and obtain more accurate white balance parameters. When the scene classification does not change for a long time, the stability can be improved by limiting the gain adjustment amount and convergence range of the white balance. The scene classification detection task technology of the present invention is mature, with few model parameters, fast inference speed, and easy deployment. The present invention does not need to estimate accurate white balance parameters in the deep learning stage, nor does it need to repeat training of different modules, and has strong versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 The figure is a flow chart of an automatic white balance method according to an embodiment of the present invention.

[0046] Figure 2 This is an initial calibration color temperature curve fitted in step S01 in the automatic white balance method of an embodiment of the present invention.

[0047] Figure 3 Schematic diagram of white point information corresponding to different scene categories in the automatic white balance method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent from the following description. It should be noted that the accompanying drawings are in a very simplified form and use an inaccurate scale, which is only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0049] In the drawings, the sizes of layers, regions, elements and their relative sizes may be exaggerated for clarity. Like reference numerals denote like elements throughout.

[0050] It should be understood that when an element or layer is referred to as "on ...", "adjacent to ...", "connected to" or "coupled to" other elements or layers, it can be directly on, adjacent to, connected to or coupled to other elements or layers, or there can be intervening elements or layers. On the contrary, when an element is referred to as "directly on ...", "directly adjacent to ...", "directly connected to" or "directly coupled to" other elements or layers, there is no intervening element or layer. It should be understood that although the terms first, second, third, etc. can be used to describe various elements, components, regions, layers and / or parts, these elements, components, regions, layers and / or parts should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer or part from another element, component, region, layer or part. Therefore, without departing from the teachings of the present application, the first element, component, region, layer or part discussed below can be represented as the second element, component, region, layer or part. And when the second element, component, region, layer or part is discussed, it does not indicate that the present application necessarily has the first element, component, region, layer or part.

[0051] Spatially relative terms such as "under", "beneath", "below", "under", "above", "above", etc., may be used here for convenience of description to describe the relationship between an element or feature shown in the figure and other elements or features. It should be understood that in addition to the orientation shown in the figure, the spatial relationship terms are intended to also include different orientations of the device in use and operation. For example, if the device in the accompanying drawings is turned over, then the elements or features described as "under other elements" or "under it" or "under it" will be oriented as "on" other elements or features. Therefore, the exemplary terms "under" and "under" may include both upper and lower orientations. The device can be oriented otherwise (rotated 90 degrees or other orientations) and the spatial descriptors used herein are interpreted accordingly.

[0052] The purpose of the terms used herein is only to describe specific embodiments and is not intended to be limiting of the present application. When used herein, the singular forms "one", "an" and "said / the" are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "consisting of" and / or "comprising", when used in this specification, determine the presence of the features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or groups. When used herein, the term "and / or" includes any and all combinations of the relevant listed items.

[0053] The embodiment of the present invention provides an automatic white balance method, such as Figure 1 As shown, including:

[0054] S1. The image acquisition module acquires the image to be corrected, and estimates the scene category of the image to be corrected through the scene information extraction module; according to the scene category, the white point information corresponding to the pre-calibrated scene category is applied; the white point information includes the white point division areas of different color temperatures and the white point weights of different color temperatures;

[0055] S2, determining whether the pixel points of the area of ​​interest in the image to be corrected fall into the white point division area included in the scene category; calculating the weighted cumulative pixel values ​​of the three primary colors of red, green and blue of all the pixel points falling into the white point division area included in the scene category, which are rv, gv and bv respectively;

[0056] S3, calculate the white balance gain coefficients of the three primary colors of red, green and blue, which are IRgain, IGgain and IBgain respectively; so that rv*IRgain=gv*IGgain=bv*IBgain holds;

[0057] S4. When it is detected that the scene category has not changed for a long time according to the scene information between frames, the gain adjustment amount and the convergence range of the white balance are limited.

[0058] The following describes in detail the steps of the automatic white balance method according to the embodiment of the present invention.

[0059] The image acquisition module includes an image sensor, which records (calibrates) the image sensor's response to white under various light sources (color temperatures). By comparing the currently calculated white point with the lookup table and going to the nearest calibration white point, the correct white balance can be calculated.

[0060] The pre-calibration used in step S1 specifically includes:

[0061] S01, such as Figure 2 As shown, the image sensor in the image acquisition module is used to capture gray card images at different color temperatures (such as H, A, CWF, D50, D65 and D75) in the light box; the white balance reference points at different color temperatures are calibrated ( Figure 2 The color temperature curve is fitted according to the white balance reference points of different color temperatures ( Figure 2 The white point areas corresponding to different color temperatures are drawn based on the color temperature curve to obtain the initial calibration white point area distribution map.

[0062] Take the middle area of ​​the gray card (it is believed that the illumination in this area is more uniform and the calibration is more accurate). Assume that the middle area of ​​the gray card has m horizontal rows and n vertical columns with a total of m*n valid pixels. The middle area of ​​the gray card accounts for about 50% of the total area of ​​the gray card. In the human eye, the pixels of the three primary colors of red, green and blue are in equal proportion, and the pixels of the three primary colors of gray are also in equal proportion. Therefore, all valid pixels (gray) in the middle area of ​​the gray card at each color temperature are considered to be white points.

[0063] Specifically, at a color temperature, for all valid pixel points, the unweighted cumulative pixel values ​​of the three primary colors of red, green and blue are calculated, and the ratio of the unweighted cumulative pixel value of the red primary color (R) to the unweighted cumulative pixel value of the green primary color (G) is used as the horizontal coordinate value of the white balance reference point of the color temperature, and the ratio of the unweighted cumulative pixel value of the blue primary color (B) to the unweighted cumulative pixel value of the green primary color (G) is used as the vertical coordinate value of the white balance reference point of the color temperature, so as to obtain the white balance reference point at the color temperature. Similarly, the white balance reference point at each color temperature is calculated. A color temperature curve is fitted according to the white balance reference points of different color temperatures, and a certain envelope graph is drawn with the point on the color temperature curve as the center point. The set covered by all envelope graphs is the initial calibration white point area distribution map. The envelope refers to the boundary surrounded by an object or a set. Exemplarily, the envelope graph can be a closed area surrounded by an ellipse, a circle, a square, a rectangle and other curves. The white point area corresponding to a certain color temperature is actually a set within the neighborhood range of the white balance reference point of the color temperature on the color temperature curve. The size of this neighborhood has no exact regulations and is related to the response of the sensor itself. It is an empirical value during calibration. The set covered by all envelope graphs is considered to be the possible value of white objects in the real scene at all color temperatures. Colors that do not fall within this range are not considered white. The depth of the figure represents the weight, also known as the white point weight.

[0064] If the gray card image captured by the image sensor is raw data, it needs to be interpolated into RGB image format data through any interpolation method. Different color temperatures are general terms, and more color temperatures can be calibrated as needed. The more types, the more accurate the white balance gain will be. H light source: halogen tungsten household light source, color temperature is 2300K; A light source: kitchen window spotlight, color reference lamp, color temperature is 2856K; CWF light source: cold white store light source, color temperature is 4150K; D50 light source: international standard artificial daylight, color temperature is 5000K; D65 light source: international standard artificial daylight, color temperature is 6500K; D75 standard light source is northern daylight, color temperature is 7500K.

[0065] S02. Photographing gray card images under different scene categories, drawing white point division areas corresponding to different scene categories and containing different color temperatures, and assigning respective white point weights to different color temperatures.

[0066] The pixels in the middle area of ​​the gray card under each scene category are taken as valid pixels; for all valid pixels in the middle area of ​​the gray card under each scene category, the B / G and R / G values ​​of each valid pixel are calculated, and the white point position of each valid pixel in the coordinate system is correspondingly drawn. According to these white point positions, the white point area (the size of the envelope figure) and the white point weight (the weight value falling within the range of the envelope figure) during the calibration in step S01 are adjusted as the white point calibration result of the scene.

[0067] Finally, for each scene category, there is a unique weighted color temperature curve that has been fine-tuned for the scene based on S01 calibration. After the scene weight is fine-tuned, the size and position of the envelope graph corresponding to the same color temperature in S02 and S01 have slight changes, that is, the weight of the white point area corresponding to the same color temperature is fine-tuned according to the scene. Assuming that for a sunny scene, based on the XY axis color temperature curve calibrated by S01, the position and size of these envelope graphs, also known as the distribution of the white point area, are fine-tuned according to the actual landing points of B / G and R / G on the XY axis reflected by the gray card shot of the sunny scene.

[0068] like Figure 2 and Figure 3 As shown in the figure, the scene categories include: indoor, semi-indoor and outdoor; outdoor is subdivided into sunny, cloudy, sunset and dusk according to different color temperatures. Both indoor and semi-indoor scenes contain six color temperatures: H, A, CWF, D50, D65 and D75. In indoor scenes, the white point weights corresponding to the six color temperatures are the same. The white point weight corresponding to the D65 color temperature of the semi-indoor scene is larger than the white point weight corresponding to the D65 color temperature in the initial calibration color temperature curve.

[0069] The four outdoor scenes of sunny day, cloudy day, sunset and dusk all contain three color temperatures: D50, D65 and D75. The white point weight corresponding to the D50 color temperature of the sunny scene is larger than the white point weight corresponding to the D50 color temperature in the initial calibration color temperature curve; the white point weight corresponding to the D65 color temperature in the sunny scene is larger than the white point weight corresponding to the D65 color temperature in the initial calibration color temperature curve. The white point weights corresponding to the D50 color temperature, D65 color temperature and D75 color temperature in the cloudy scene are larger than the white point weights corresponding to the same color temperature in the initial calibration color temperature curve. The white point weight corresponding to the D50 color temperature in the sunset scene is larger than the white point weight corresponding to the D50 color temperature in the initial calibration color temperature curve. The white point weight corresponding to the D65 color temperature and D75 color temperature in the dusk scene is larger than the white point weights corresponding to the same color temperature in the initial calibration color temperature curve.

[0070] Calculating white balance specifically includes:

[0071] S1. The image acquisition module acquires the image to be corrected, and estimates the scene category of the image to be corrected through the scene information extraction module; specifically, the image acquisition module includes an image sensor, which captures the image to be corrected and sends it to the neural network processor (NPU) in the scene information extraction module. The neural network processor deploys a trained deep learning network, which can estimate the scene category corresponding to the image to be corrected from the input image to be corrected. The deep learning network estimates the scene category corresponding to the image to be corrected from the image with color cast or small color cast in the image to be corrected. According to the scene category, the white point information corresponding to the scene category in the pre-calibration step S02 is applied; the white point information includes white point division areas of different color temperatures and white point weights of different color temperatures.

[0072] S2. Determine whether the pixel points of the area of ​​interest in the image to be corrected fall into the white point division area included in the scene category; calculate the weighted cumulative pixel values ​​of the three primary colors of red, green and blue of all the pixel points falling into the white point division area included in the scene category, which are rv, gv and bv respectively.

[0073] The image to be corrected is the original image without white balance. If the image to be corrected is not in RGB format, it needs to be converted into RGB format. For all pixels of the image to be corrected, the red channel pixel value R, green channel pixel value G and blue channel pixel value B of any pixel point P are obtained, and R / G and B / G are calculated, so that the position of the pixel point P in the coordinate system under the scene category can be found.

[0074] If the pixel point P is in the envelope graph (i.e., the white point division area) drawn by the color temperature curve in S02, the pixel point P is considered to be a possible white point, then its R, G and B values ​​are multiplied by the weight coefficients calibrated in the envelope graph, and the weighted cumulative pixel values ​​of the three primary colors are accumulated to obtain rv, gv and bv respectively.

[0075] Exemplarily, the image to be corrected has a total of 2*2 pixels and a total of 4 pixels. The number of pixels that the image to be corrected can contain is not limited, and there can actually be tens of thousands of pixels; this example only takes a small number of pixels as an example for easy understanding. If the scene is sunny, then calculate whether the pixel point falls in the color temperature curve envelope graph, that is, the gray area near the curve. Assume that pixel 0 and pixel 3 fall in, and pixel 0 falls near D65, with a weight value of 2; pixel 3 falls near D50, with a weight value of 1.1, and calculate the weighted cumulative pixel value (weighted accumulated value) of the three primary colors, R0 is the red channel pixel value of pixel 0; R3 is the red channel pixel value of pixel 3; G0 is the green channel pixel value of pixel 0; G3 is the green channel pixel value of pixel 3; B0 is the blue channel pixel value of pixel 0; B3 is the blue channel pixel value of pixel 3.

[0076] rv=R0*2+R3*1.1; gv=G0*2+G3*1.1; bv=B0*2+B3*1.1.

[0077] S3. Calculate the white balance gain coefficients of the three primary colors of red, green and blue, which are IRgain, IGgain and IBgain respectively, so that rv*IRgain=gv*IGgain=bv*IBgain holds.

[0078] Optionally, according to different scene categories, the calculated white balance gain coefficients of the three primary colors of red, green and blue are adjusted with preference; the gain adjustment coefficients of the three primary colors of red, green and blue are Rshift, Gshift and Bshift respectively; the adjusted white balance gain coefficients of the three primary colors are Rgain, Ggain and Bgain respectively; Rgain = IRgain*Rshift; Ggain = IGgain*Gshift; Bgain = IBgain*Bshift.

[0079] The white balance calculation has actually been completed in S3, but the calculated gain coefficient can be fine-tuned for different scene categories to change the color temperature preference of different scenes. After completing the S3 white balance operation, you can actually test different scene images. For the actual test results, adjust the gain value in the form of gain adjustment coefficients according to customer needs. If the actual test results are satisfactory and no adjustment is required, then the gain adjustment coefficients of the three primary colors can be set to 1.

[0080] In step S4, when it is detected that the scene category has not changed for a long time according to the scene information between frames, the gain adjustment amount and the convergence range of the white balance are limited. The parameters of the convergence range include the gain coefficient thresholds of the red, green and blue primary colors corresponding to different scenes, which are SceneThreRgain, SceneThreGgain and SceneThreBgain respectively;

[0081] The restrictions when calculating the white balance gain coefficient of the nth frame are as follows:

[0082] Rgain n =Rgain n-1 +step, and Rgain n <SceneThreRgain;

[0083] Ggain n =Ggain n-1 +step, and Ggain n <SceneThreGgain;

[0084] Bgain n =Bgain n-1+step, and Bgain n <SceneThreBgain。

[0085] The adjusted white balance gain coefficients of the three primary colors in the nth frame are Rgain and n , Ggain n and Bgain n ; The adjusted white balance gain coefficients of the three primary colors in the n-1th frame are Rgain n-1 , Ggain n-1 and Bgain n-1 ; step is the gain adjustment amount.

[0086] Determine whether the scene has changed; if the scene has changed, change the position and size of the white point statistical area; adjust the weight of the corresponding color temperature reference point, apply scene preference; automatic white balance calculation results. If the scene has not changed, the scene is stable and the automatic white balance calculation results are calculated.

[0087] Since different categories have different color temperature distribution preferences, it is possible to adjust the white point division area, weight and color preference in real time according to the scene classification information to minimize the impact of interfering color blocks and obtain more accurate white balance parameters. When the scene classification has not changed for a long time, the stability can be improved by limiting the gain adjustment amount and convergence range of the white balance. The scene classification detection task technology of the present invention is mature, with few model parameters, fast inference speed and easy deployment. The present invention does not require accurate white balance parameters to be estimated in the deep learning stage, nor does it require repeated training of different modules, and has strong versatility.

[0088] The present invention also provides an automatic white balance adjustment system, comprising:

[0089] An image acquisition module, used for acquiring an image to be corrected;

[0090] A scene information extraction module, used to estimate the scene category of the image to be corrected;

[0091] The white balance calculation and correction module is used to calculate the white balance gain coefficients of the three primary colors of red, green and blue, which are IRgain, IGgain and IBgain respectively; so that rv*IRgain=gv*IGgain=bv*IBgain holds; the weighted cumulative pixel values ​​of the three primary colors of red, green and blue are rv, gv and bv respectively.

[0092] The image acquisition module includes an image sensor and a preprocessing module, and the preprocessing module performs image signal processing on the image acquired by the image sensor;

[0093] The scene information extraction module includes: a neural network processor; the image to be corrected is captured by the image sensor and sent to the neural network processor (NPU) in the scene information extraction module. The neural network processor deploys a trained deep learning network. The deep learning network estimates the scene category corresponding to the image to be corrected through images with color cast or small color cast in the image to be corrected, and outputs the scene category to the white balance calculation and correction module to guide the white balance calculation and correction processing.

[0094] The present invention also provides a white balance terminal, comprising:

[0095] The memory and the processor are connected to each other for communication. The memory stores computer instructions. The processor executes the computer instructions to perform the automatic white balance method.

[0096] The computer program can be divided into one or more modules, one or more modules are stored in the memory and executed by the processor to complete the various modules of the present invention. One or more modules can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0097] The processor referred to in this embodiment may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0098] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0099] In summary, the present invention provides an automatic white balance method and adjustment system, and a white balance terminal, including: S1, an image acquisition module acquires an image to be corrected, and estimates the scene category of the image to be corrected through a scene information extraction module; according to the scene category, the pre-calibrated white point information corresponding to the scene category is applied; the white point information includes white point division areas of different color temperatures and white point weights of different color temperatures; S2, judging whether the pixel points of the area of ​​interest in the image to be corrected fall into the white point division area included in the scene category; calculating the weighted cumulative pixel values ​​of the three primary colors of red, green and blue of all the pixel points falling into the white point division area included in the scene category, which are rv, gv and bv respectively; S3, calculating the white balance gain coefficients of the three primary colors of red, green and blue, which are IRgain, IGgain and IBgain respectively; so that rv*IRgain=gv*IGgain=bv*IBgain holds; S4, according to the inter-frame scene information, when it is monitored that the scene category has not changed for a long time, limiting the gain adjustment amount and convergence range of the white balance.

[0100] Since different categories have different color temperature distribution preferences, the present invention can adjust the white point division area, weight and color preference in real time according to the scene classification information, minimize the influence of interfering color blocks, and obtain more accurate white balance parameters. When the scene classification does not change for a long time, the stability can be improved by limiting the gain adjustment amount and convergence range of the white balance. The scene classification detection task technology of the present invention is mature, with few model parameters, fast inference speed, and easy deployment. The present invention does not need to estimate accurate white balance parameters in the deep learning stage, nor does it need to repeat training of different modules, and has strong versatility.

[0101] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the method disclosed in the embodiment, since it corresponds to the device disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0102] The above description is only a description of the preferred embodiment of the present invention, and is not any limitation on the scope of rights of the present invention. Any technical personnel in this field can make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. An automatic white balance method, characterized in that: include: S1, the image acquisition module acquires the image to be corrected, and the scene information extraction module estimates the scene category of the image to be corrected; According to the scene category, applying pre-calibrated white point information corresponding to the scene category; the white point information includes white point division areas of different color temperatures and respective white point weights of different color temperatures; S2, determining whether the pixel points of the area of ​​interest in the image to be corrected fall into the white point division area included in the scene category; Calculate the weighted cumulative pixel values ​​of the three primary colors of red, green and blue of all the pixel points falling into the white point divided area included in the scene category, which are rv, gv and bv respectively; S3, calculate the white balance gain coefficients of the three primary colors of red, green and blue, which are IRgain, IGgain and IBgain respectively; so that rv*IRgain=gv*IGgain=bv*IBgain holds; S4. When it is detected according to the inter-frame scene information that the scene category has not changed for a long time, the gain adjustment amount and the convergence range of the white balance are limited.

2. The automatic white balance method according to claim 1, wherein: The calibration used in step S1 specifically includes: S01, using the image sensor in the image acquisition module to shoot the gray card images at different color temperatures in the light box; calibrating the white balance reference points of the different color temperatures, and fitting the color temperature curve according to the white balance reference points of the different color temperatures, and drawing the white point areas corresponding to the different color temperatures based on the color temperature curve to obtain an initial calibrated white point area distribution map; S02. Photographing gray card images under different scene categories, drawing white point division areas corresponding to the different scene categories and containing the different color temperatures, and assigning the respective white point weights to the different color temperatures.

3. The automatic white balance method according to claim 2, wherein: In step S01, the pixel points in the middle area of ​​the gray card under each color temperature are taken as valid pixel points; for all valid pixel points under one color temperature, the unweighted cumulative pixel values ​​of the three primary colors of red, green and blue are calculated, the ratio of the unweighted cumulative pixel value of the red primary color to the unweighted cumulative pixel value of the green primary color is taken as the abscissa value of the white balance reference point of the one color temperature, and the ratio of the unweighted cumulative pixel value of the blue primary color to the unweighted cumulative pixel value of the green primary color is taken as the ordinate value of the white balance reference point of the one color temperature, to obtain the white balance reference point of the one color temperature; Similarly, the white balance reference point of each color temperature is calculated; a color temperature curve is fitted according to the white balance reference points of different color temperatures, and a certain envelope graph is drawn with a point on the color temperature curve as the center point, and the set covered by all the envelope graphs is the initial calibration white point area distribution map; In step S02, the pixel points in the middle area of ​​the gray card under each of the scene categories are taken as valid pixel points; for all the valid pixel points in the middle area of ​​the gray card under each of the scene categories, the B / G and R / G values ​​of each of the valid pixel points are calculated, and the white point position of each of the valid pixel points in the coordinate system is correspondingly drawn; according to the white point position, the white point area during the calibration in step S01, i.e., the size of the envelope graph and the white point weight, i.e., the weight value falling within the range of the envelope graph, are adjusted as the white point calibration result of the scene.

4. The automatic white balance method according to claim 2, wherein: The scene categories include: indoor, semi-indoor and outdoor; the outdoor is subdivided into sunny, cloudy, sunset and dusk according to different color temperatures; the indoor scene and the semi-indoor scene both include: six color temperatures: H, A, CWF, D50, D65 and D75; In the indoor scene, the white point weights corresponding to the six color temperatures are the same; The white point weight corresponding to the D65 color temperature of the semi-indoor scene is larger than the white point weight corresponding to the D65 color temperature in the initial calibration color temperature curve.

5. The automatic white balance method according to claim 4, wherein: The four outdoor scenes of sunny day, cloudy day, sunset and dusk all include three color temperatures: D50, D65 and D75; The white point weight corresponding to the D50 color temperature of the sunny scene is larger than the white point weight corresponding to the D50 color temperature in the initial calibration color temperature curve; the white point weight corresponding to the D65 color temperature in the sunny scene is larger than the white point weight corresponding to the D65 color temperature in the initial calibration color temperature curve; The white point weights corresponding to the D50 color temperature, the D65 color temperature and the D75 color temperature in the cloudy scene are larger than the white point weights corresponding to the same color temperatures in the initial calibration color temperature curve diagram; The white point weight corresponding to the D50 color temperature in the sunset scene is larger than the white point weight corresponding to the D50 color temperature in the initial calibration color temperature curve; The white point weights corresponding to the D65 color temperature and the D75 color temperature in the dusk scene are larger than the white point weights corresponding to the same color temperatures in the initial calibration color temperature curve.

6. The automatic white balance method according to claim 1, wherein: After step S3 and before step S4, the method further includes: According to different scene categories, the calculated white balance gain coefficients of the three primary colors of red, green and blue are adjusted with preference; wherein the gain adjustment coefficients of the three primary colors of red, green and blue are Rshift, Gshift and Bshift respectively; the adjusted white balance gain coefficients of the three primary colors are Rgain, Ggain and Bgain respectively; Rgain = IRgain*Rshift; Ggain = IGgain*Gshift; Bgain = IBgain*Bshift.

7. The automatic white balance method according to claim 6, wherein: In step S4, the parameters of the convergence range include red, green and blue primary color gain coefficient thresholds corresponding to different scenes, which are SceneThreRgain, SceneThreGgain and SceneThreBgain respectively; The restrictions when calculating the white balance gain coefficient of the nth frame are as follows: Rgain n =Rgain n-1 +step, and Rgain n <SceneThreRgain; Ggain n = Ggain n-1 + step, and Ggain n < SceneThreGgain; Bgain n = Bgain n-1 + step, and Bgain n <SceneThreBgain。 The adjusted white balance gain coefficients of the three primary colors in the nth frame are Rgain and n , Ggain n and Bgain n The adjusted white balance gain coefficients of the three primary colors in the n-1th frame are Rgain n-1 , Ggain n-1 and Bgain n-1 ; step is the gain adjustment amount.

8. An automatic white balance adjustment system, characterized in that: include: An image acquisition module, used for acquiring an image to be corrected; A scene information extraction module, used to estimate the scene category of the image to be corrected; The white balance calculation and correction module is used to calculate the white balance gain coefficients of the three primary colors of red, green and blue, which are IRgain, IGgain and IBgain respectively; so that rv*IRgain=gv*IGgain=bv*IBgain holds; the weighted cumulative pixel values ​​of the three primary colors of red, green and blue are rv, gv and bv respectively.

9. The automatic white balance adjustment system as claimed in claim 8, characterized in that: The image acquisition module includes an image sensor and a preprocessing module, and the preprocessing module performs image signal processing on the image acquired by the image sensor; The scene information extraction module includes: a neural network processor; The neural network processor deploys a trained deep learning network to estimate the scene category corresponding to the image to be corrected from the image to be corrected; and outputs the scene category to the white balance calculation and correction module to guide the white balance calculation and correction processing.

10. A white balance terminal, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the automatic white balance method according to any one of claims 1 to 7 by executing the computer instructions.

Images