Adaptive white balance method based on color temperature estimation

Abstract

The invention belongs to the technical field of image processing, and particularly relates to an adaptive white balance method based on color temperature estimation. The method comprises the following steps: step 1, carrying out color temperature correction gain modeling on a determined camera, namely calibrating white balance correction gains of a gray scene in different color temperature irradiation environments, only calculating a gain GainRed of a red Red component and a gain GainBlue of a blue Blue component by taking a green Green component of Bayer format data as a reference, and during calculation, distinguishing a Green component corresponding to the red Red component as GreenR and a Green component corresponding to the blue Blue component as GreenB; 2, calculating the color temperature correction gain of the real-time image blocks; and 3, respectively correcting the infrared component and the blue component of the real-time Bayer image by using the red gain GainR and the blue gain GainB to complete automatic white balance correction. According to the self-adaptive white balance method, the white balance precision is improved, and the adaptability to different illumination environments is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an adaptive white balance method based on color temperature estimation. Background Art

[0002] White balance corrects the appearance of white objects in an image so that they appear white under varying lighting conditions. White balance algorithms adjust the gain of each channel in the image to eliminate color distortion caused by the color temperature of the light source. The automatic white balance feature uses the image's grayscale distribution, tonal distribution, or scene detection algorithms to estimate the light source's color temperature and apply the appropriate white balance correction.

[0003] Typical white balance methods include the grayscale world method and the perfect reflection method. With the development of automatic white balance correction technology, a variety of improved methods have emerged. For example, (1) the fusion method of grayscale world and perfect reflection. The grayscale world algorithm assumes that the overall color of the image is gray on average, and achieves white balance by adjusting the RGB channel gain. The perfect reflection method is based on the assumption that "the brightest point is white", but is easily disturbed by the highlight area. Current research often combines the two, such as screening the effective white point area through dynamic thresholding, and combining the grayscale world statistical information to achieve parameter adaptive adjustment. (2) Dynamic thresholding and block processing method. The dynamic thresholding algorithm divides the image into blocks (such as 12 sub-areas), combines the color space (such as YCrCb) to analyze local features, and screens pixels with white features for gain calculation. This method improves the robustness of complex scenes through regional processing. (3) Intelligent correction based on color temperature estimation. By analyzing the relationship between the color temperature of the light source and the RGB channels, a color temperature-gain mapping model is established. For example, in the YCrCb space, areas with a Cb / Cr ratio close to 0 and high brightness are detected as white points, and the gain is adjusted in combination with the color temperature curve. (4) End-to-end network driven by deep learning. Based on a convolutional neural network (CNN), the mapping relationship from the original image to the white balance correction is directly learned. It can handle complex lighting and non-uniform color temperature scenes. The output of traditional algorithms (such as the grayscale world) is used as the input of the neural network, and the correction parameters are dynamically adjusted in combination with the attention mechanism. This type of method has been applied in mobile phone cameras and drone image processing.

[0004] The above technology still has the following defects: (1) When estimating and correcting the color temperature of 8-bit color images after tone mapping, color deviation still exists under extreme lighting conditions (such as mixed light sources and low illumination), and the accuracy needs to be improved; (2) There is a lack of reference data for color temperature calibration, and the adaptability in unknown environments is poor; (3) The computing resources consumed are large, which is not conducive to parallel acceleration optimization and it is difficult to ensure the real-time processing of large-resolution video images. Summary of the Invention

[0005] The present invention provides an adaptive white balance method based on color temperature estimation. The adaptive white balance method of the present invention improves white balance accuracy and enhances adaptability to different lighting environments.

[0006] The method of the present invention is an improvement on the gray world method, and comprises steps of color temperature correction gain modeling, calculating color temperature correction gain of real-time image blocks, screening color temperature correction gain of block images, generating image color temperature correction gain, and correcting real-time images.

[0007] The adaptive white balance method based on color temperature estimation of the present invention comprises the following specific steps: Step 1: Model the color temperature correction gain for the camera. This means calibrating the white balance correction gain for gray scenes under different color temperature illumination environments. Using the green component of the Bayer format data as the reference, only the red component gain (GainRed) and the blue component gain (GainBlue) are calculated. During the calculation, the Green component corresponding to the Red component is distinguished as GreenR, and the Green component corresponding to the Blue component is distinguished as GreenB. The detailed process is as follows: Step 1.1: Use a color temperature box to collect the RAW format Bayer data of grayscale scenes under different color temperature illumination environments

[10] ; Step 1.2: Separate each frame of Data to obtain the color components DRed, DGreenR, DGreenB, and DBlue; Step 1.3: Use the grayscale world method to calculate the correction gain GainBlue and GainRed of each frame image. Among them, num is the number of pixels of each type component, which is also 1 / 4 of the number of pixels of the whole image; Step 1.4: Calculate the gain reciprocals BlueGain

[10] and RedGain

[10] based on the sample gains GainBlue

[10] and GainRed

[10] . First, perform quadratic curve fitting / piecewise linear fitting on GainBlue

[10] to obtain the fitting parameters. Then, use the fitting parameters to calculate the gain value GainBlueFit

[10000] in the illuminance color temperature range [1,10000]. Secondly, calculate the reciprocal of GainBlueFit

[10000] to obtain BlueGainFit

[10000] . Finally, extract the blue gain reciprocal sample BlueGain[gnum] according to the commonly used color temperature. In the same way, obtain the infrared gain reciprocal sample RedGain[rnum]. Where gnum=rnum.

[0008] Step 2: Calculate the color temperature correction gain of the real-time image block Step 2.1: Divide the Bayer image into several image blocks, where the number of horizontal blocks boxNumH and the number of vertical blocks boxNumV are both even numbers, so that the number of pixels of different colors is equal; Step 2.2: Calculate the correction gains GainBlueBox

[256] and GainRedBox

[256] of each image block using the gray world method; Step 2.3: Convert the gains GainBlueBox

[256] and GainRedBox

[256] to their inverses to obtain the inverse gains BlueGainBox

[256] and RedGainBox

[256] ; Step 2.4: The reciprocal gain BlueGainBox

[256] and RedGainBox

[256] form coordinate points and map them to the full-image reciprocal gain BlueGain[gnum] and RedGain[gnum] curve coordinate system calibrated in step 1; Step 2.5: Calculate the distance between the reciprocal gain coordinate point of each block and the reciprocal gain curve coordinate system of the entire image; if the distance is less than the threshold distTh (which can be 0.095), retain the reciprocal gain value of the coordinate point and record its distance value; otherwise, discard it; Step 2.6: Take the weighted average of all retained reciprocal gains to obtain the reciprocal gains RGain and BGain of the current real-time image frame. The weight calculation method is to first calculate the reciprocal of the distance value and then normalize it to obtain the weight value of each reciprocal gain; Step 2.7: Calculate the final red gain GainR = 1 / RGain and the blue gain GainB = 1 / BGain; Step 3: Use the red gain GainR and blue gain GainB to correct the infrared component and blue component of the real-time Bayer image respectively to complete the automatic white balance correction.

[0009] The Bayer image after white balance correction is subjected to other subsequent ISP processing to obtain the final image.

[0010] In the above solution, step 1.4 extracts the inverse of the blue gain and the inverse of the infrared gain to create samples based on the commonly used color temperature. The extraction method refers to the following different color temperature scenarios: (1) 1800K~1900K: Candlelight.

[0011] (2) 2000K~3000K: dawn or dusk.

[0012] (3) 2700K~3200K: Tungsten filament bulb.

[0013] (4) 3200K~3400K: studio lighting, quartz lamp.

[0014] (5) 3500K: warm white fluorescent light.

[0015] (6) 4500K: White cool fluorescent light, the moon at moonrise.

[0016] (7) 4000K~5000K: early morning or afternoon sunlight.

[0017] (8) 4200K~5000K: Flash bulb.

[0018] (9) 5500K: Daylight, electronic flash.

[0019] (10) 6500K: Daylight fluorescent lamp.

[0020] (11) 6500K~7500K: cloudy.

[0021] (12) 8000K~9000K: Blue sky with white clouds.

[0022] (13) 10000K: sunny and sunny (14) 12000K~25000K: Clear blue sky.

[0023] In step 1.1, the color temperature range can be divided into 10 levels from 1000K to 10000K.

[0024] The method of the present invention calibrates and estimates the automatic white balance method based on a RAW format Bayer image, thereby obtaining the best illumination color temperature estimation result for the image. Based on the actual selected optical lens and image sensor, the present invention collects RAW format Bayer image data of grayscale images of different color temperatures for calibration, and performs fitting and interpolation to generate a color temperature correction gain curve model, which serves as a reference for color temperature estimation of real-time video images. The weighted summation of block gains that conform to the reference model is screened, thereby improving the accuracy of automatic white balance. The present invention implements a hardware parallel acceleration algorithm for automatic white balance by estimating the blocks of real-time video images, and implements a white point detection module of the dynamic threshold algorithm in an FPGA, achieving real-time processing speed.

[0025] The beneficial effects of the method of the present invention are: (1) Using Bayer images to estimate color temperature improves the accuracy of illumination estimation; (2) Using the Bayer image to calibrate the color temperature gain correction curve model, the weighted sum of the block gains is screened to improve the white balance accuracy; (3) Block estimation is used to reduce the interference of color image blocks, thereby improving the adaptability of the white balance method of the present invention to different lighting environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Flow chart of the method of the present invention.

[0027] Figure 2 To calibrate the color temperature correction gain inverse curve, the blue line is the fitting curve, and the infrared line is the sampling point link curve to form a coordinate system.

[0028] Figure 3 This is a block effect diagram.

[0029] Figure 4 The gain inverse of the image block with red characters meets the distance screening condition, while the gain inverse of the image block with yellow characters does not meet the distance screening condition.

[0030] Figure 5 The position of the reciprocal coordinate point of the block distance in the coordinate system of the reciprocal curve of color temperature correction gain. The red circle meets the distance screening condition, and the black circle does not meet the distance screening condition.

[0031] Figure 6 The inverse of the weighted average gain after screening, and the position of the blue circle in the coordinate system of the inverse gain curve.

[0032] Figure 7 Image without white balance correction.

[0033] Figure 8 An image subjected to white balance correction according to the present invention. DETAILED DESCRIPTION

[0034] The adaptive white balance method based on color temperature estimation of the present invention includes the following implementation steps.

[0035] Step 1: Modeling the color temperature correction gain. For camera-type devices, once the optical lens and image sensor (such as CMOS, SSD, etc.) are determined, the color perception characteristics are determined. To achieve automatic white balance correction for real-time video images, the color temperature correction gain is first modeled. This calibrates the white balance correction gain for gray scenes under different color temperature illumination environments. Here, the green component of the Bayer format data is used as the reference, and only the gain (GainRed) of the red component and the gain (GainBlue) of the blue component are calculated. During the calculation, the Green component corresponding to the Red component is distinguished as GreenR, and the Green component corresponding to the Blue component is distinguished as GreenB.

[0036] Detailed steps are as in 1.1-1.4.

[0037] Step 1.1: Use a color temperature box to collect the RAW format Bayer data of grayscale scenes under different color temperature illumination environments

[10] . For example, the color temperature range is from 1000K to 10000K and is collected in 10 levels.

[0038] Step 1.2: Separate each frame of Data to obtain the color components DRed, DGreenR, DGreenB, and DBlue; Step 1.3: Calculate the correction gains GainBlue and GainRed for each frame using the grayscale world method; Among them, num is the number of pixels of each type component, which is also 1 / 4 of the number of pixels of the whole image; Step 1.4: Calculate the gain inverses BlueGain

[10] and RedGain

[10] based on the sample gains GainBlue

[10] and GainRed

[10] , as follows: Figure 2 As shown in the figure, a color temperature curve is established to match the color temperature. The distance between the color temperature of each frame of the video and the color temperature curve is calculated. Sampling can establish segmented segments, reducing the amount of calculation and facilitating the implementation of embedded computing programs.

[0039] First, perform quadratic curve fitting / piecewise linear fitting on GainBlue

[10] to obtain the fitting parameters; then use the fitting parameters to calculate the gain value GainBlueFit

[10000] in the illuminance color temperature range [1,10000]; then calculate the reciprocal of GainBlueFit

[10000] to obtain BlueGainFit

[10000] , and finally, extract the blue gain reciprocal sample BlueGain[gnum] according to the common color temperature. The infrared gain reciprocal sample RedGain[rnum] is obtained in the same way; where gnum=rnum, the value is generally 7 to 15 sample points, and here it is 10 sample points. The extraction method can refer to the following different color temperature scenarios: (1) 1800K~1900K: Candlelight.

[0040] (2) 2000K~3000K: dawn or dusk.

[0041] (3) 2700K~3200K: Tungsten filament bulb.

[0042] (4) 3200K~3400K: studio lighting, quartz lamp.

[0043] (5) 3500K: warm white fluorescent light.

[0044] (6) 4500K: White cool fluorescent light, the moon at moonrise.

[0045] (7) 4000K~5000K: early morning or afternoon sunlight.

[0046] (8) 4200K~5000K: Flash bulb.

[0047] (9) 5500K: Daylight, electronic flash.

[0048] (10) 6500K: Daylight fluorescent lamp.

[0049] (11) 6500K~7500K: cloudy.

[0050] (12) 8000K~9000K: Blue sky with white clouds.

[0051] (13) 10000K: sunny and sunny (14) 12000K~25000K: Clear blue sky.

[0052] Step 2: Calculate the color temperature correction gain of the real-time image blocks. For example, calculate the color temperature correction gain of a 1920*1080 resolution image by dividing the blocks.

[0053] Step 2.1: Divide the Bayer image into several image blocks, where the horizontal block number boxNumH and the vertical block number boxNumV are both even numbers, so that the number of pixels of different colors is equal. For a 1920*1080 resolution image, the number of blocks boxNumH=16 and boxNumV=16 is as follows: Figure 3 As shown in the figure, the height of the image block is boxHei = floor(1080 / boxNumH), and the width of the image block is boxWid = ceil(1920 / boxNumV).

[0054] Step 2.2: Use the gray world method to calculate the correction gain GainBlueBox

[256] and GainRedBox

[256] of each image block, as Figure 4 shown.

[0055] Step 2.3: Convert the gains GainBlueBox

[256] and GainRedBox

[256] into inverses to obtain the inverse gains BlueGainBox

[256] and RedGainBox

[256] .

[0056] Step 2.4: Use the reciprocal gain BlueGainBox

[256] and RedGainBox

[256] to form coordinate points and map them to the full-graph reciprocal gain BlueGain[gnum] and RedGain[gnum] curve coordinate system calibrated in step 1, as shown in the following example: Figure 5 shown.

[0057] Step 2.5: Calculate the distance from the reciprocal gain coordinate point of each block to the reciprocal gain curve coordinate system of the entire image; if the distance is less than the threshold distTh (which can be 0.095), retain the reciprocal gain value of the coordinate point and record its distance value; otherwise, discard it, such as Figure 6 shown.

[0058] Step 2.6: Take the weighted average of all retained reciprocal gains to obtain the reciprocal gains RGain and BGain of the current real-time image frame. The weight calculation method is to first calculate the reciprocal of the distance value and then normalize it to obtain the weight value of each reciprocal gain.

[0059] Step 2.7: Calculate the final red gain GainR = 1 / RGain and the blue gain GainB = 1 / BGain; Step 3: Use the red gain GainR and blue gain GainB to correct the infrared component and blue component of the real-time Bayer image respectively to complete the automatic white balance correction.

[0060] The Bayer image after white balance correction is subjected to other subsequent ISP processing to obtain the final image. Figure 7 This is an image without white balance correction. Figure 8 This is an image that has been subjected to white balance correction according to the method of the present invention.

Claims

1. An adaptive white balance method based on color temperature estimation, characterized in that: The specific steps are as follows: Step 1: Model the color temperature correction gain for the camera. This means calibrating the white balance correction gain for gray scenes under different color temperature illumination environments. Using the green component of the Bayer format data as the reference, only the red component gain (GainRed) and the blue component gain (GainBlue) are calculated. During the calculation, the Green component corresponding to the Red component is distinguished as GreenR, and the Green component corresponding to the Blue component is distinguished as GreenB. The detailed process is as follows: Step 1.1: Use a color temperature box to collect the RAW format Bayer data of grayscale scenes under different color temperature illumination environments [10]; Step 1.2: Separate each frame of Data to obtain the color components DRed, DGreenR, DGreenB, and DBlue; Step 1.3: Use the grayscale world method to calculate the correction gain GainBlue and GainRed of each frame image. ； Among them, num is the number of pixels of each type component, which is also 1 / 4 of the number of pixels of the whole image; Step 1.4: Calculate the gain reciprocals BlueGain[10] and RedGain[10] based on the sample gains GainBlue[10] and GainRed[10]. First, perform quadratic curve fitting / piecewise linear fitting on GainBlue[10] to obtain the fitting parameters. Then, use the fitting parameters to calculate the gain value GainBlueFit[10000] in the illuminance color temperature range [1,10000]. Secondly, calculate the reciprocal of GainBlueFit[10000] to obtain BlueGainFit[10000]. Finally, extract the blue gain reciprocal sample BlueGain[gnum] according to the commonly used color temperature. In the same way, obtain the infrared gain reciprocal sample RedGain[rnum]. Where gnum=rnum. Step 2: Calculate the color temperature correction gain of the real-time image block Step 2.1: Divide the Bayer image into several image blocks, where the number of horizontal blocks boxNumH and the number of vertical blocks boxNumV are both even numbers, so that the number of pixels of different colors is equal; Step 2.2: Calculate the correction gains GainBlueBox[256] and GainRedBox[256] of each image block using the gray world method; Step 2.3: Convert the gains GainBlueBox[256] and GainRedBox[256] to their inverses to obtain the inverse gains BlueGainBox[256] and RedGainBox[256]; Step 2.4: The reciprocal gain BlueGainBox[256] and RedGainBox[256] form coordinate points and map them to the full-image reciprocal gain BlueGain[gnum] and RedGain[gnum] curve coordinate system calibrated in step 1; Step 2.5: Calculate the distance between the reciprocal gain coordinate point of each block and the reciprocal gain curve coordinate system of the entire image; if the distance is less than the threshold distTh, retain the reciprocal gain value of the coordinate point and record its distance value; otherwise, discard it; Step 2.6: Take the weighted average of all retained reciprocal gains to obtain the reciprocal gains RGain and BGain of the current real-time image frame. The weight calculation method is to first calculate the reciprocal of the distance value and then normalize it to obtain the weight value of each reciprocal gain; Step 2.7: Calculate the final red gain GainR = 1 / RGain and the blue gain GainB = 1 / BGain; Step 3: Use the red gain GainR and blue gain GainB to correct the infrared component and blue component of the real-time Bayer image respectively to complete the automatic white balance correction.

2. The adaptive white balance method according to claim 1, wherein: Step 1.4 extracts the blue gain inverse and red gain inverse samples based on the commonly used color temperature. The extraction method refers to the following different color temperature scenarios: (1) 1800K~1900K: candlelight; (2) 2000K~3000K: dawn or dusk; (3) 2700K~3200K: tungsten filament bulb; (4) 3200K~3400K: studio lighting, quartz lamp; (5) 3500K: warm white fluorescent lamp; (6) 4500K: White cool fluorescent light, moon at moonrise; (7) 4000K~5000K: early morning or afternoon sunlight; (8) 4200K~5000K: flash bulb; (9) 5500K: Daylight, electronic flash; (10) 6500K: Daylight fluorescent lamp; (11) 6500K~7500K: cloudy; (12) 8000K~9000K: blue sky with white clouds; (13) 10000K: sunny and sunny; (14) 12000K~25000K: Clear blue sky.

3. The adaptive white balance method according to claim 1, wherein: In step 1.1, the color temperature range is from 1000K to 10000K and is divided into 10 levels.

Images