White balance processing method, device, storage medium, and program product

Abstract

This application provides a white balance processing method, apparatus, storage medium, and program product, relating to the field of image data processing technology. The method includes: during video recording, identifying the scene type corresponding to one of the recorded image data; adjusting the weights of the target region and non-target regions in the image data according to the scene type, so that the pixel statistics of the target region have an effect in white balance calculation; calculating the white balance gain parameter of the target region based on the adjusted weights; and performing white balance processing on the image data according to the white balance gain parameter to generate a target image; wherein, during video recording, the calculation and update frequency of the white balance gain parameter is related to the scene complexity of each recorded image data, and the scene complexity is used to indicate the movement speed of the subject in each image data. The method of this application can reduce color deviation in image data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image data processing, and in particular to a white balance processing method and device, a storage medium and a program product. BACKGROUND

[0002] Auto White Balance (AWB) is used to ensure that the neutral color in the image data can be restored truly under different light source conditions, thereby avoiding color deviation problems caused by differences in light source color temperature.

[0003] The related technology estimates the light source color temperature value by analyzing the pixel statistical information of the image data, and determines the white balance gain parameter according to the estimated light source color temperature value to realize white balance processing. SUMMARY

[0004] The present application provides a white balance processing method, device, storage medium and program product to reduce color deviation of image data.

[0005] In a first aspect, the present application provides a white balance processing method, comprising: in video recording, identifying a scene type corresponding to one of the recorded image data; adjusting the weight of a target region in the image data and the weight of a non-target region in the image data according to the scene type, so that the pixel statistical information of the target region has an effect in white balance calculation; calculating a white balance gain parameter of the target region based on the adjusted weight; and performing white balance processing on the image data according to the white balance gain parameter to generate a target image; wherein in the video recording, the calculation update frequency of the white balance gain parameter is related to the scene complexity of each of the recorded image data, and the scene complexity is used to indicate the moving speed of the subject in each of the image data.

[0006] In a possible embodiment, the scene type includes a portrait scene, a blue sky scene, a shadow scene, a warm light scene, a fluorescent light scene or a mixed light source scene.

[0007] In a possible embodiment, the white balance processing method further comprises: in the fluorescent light scene, compensating for the influence of excessive green light by adjusting a color deviation value Duv in a related light source color temperature CCT space, wherein Duv is a general value or an individual value related to user preference; and performing white balance processing on the image data according to the white balance gain parameter to generate a target image includes performing white balance processing on the image data according to the white balance gain parameter and performing color adjustment on the image data according to the color deviation value Duv to generate a target image.

[0008] In a possible embodiment, the adjusting the color deviation value Duv in the relevant light source color temperature CCT space includes: offsetting a Duv coordinate corresponding to the image data in a negative Duv direction by a preset value, and the preset value belongs to 5%-15%.

[0009] In a possible embodiment, the weight of the target region is greater than or equal to a first weight value, the weight of the non-target region is less than or equal to a second weight value, and the first weight value is greater than the second weight value; in a fluorescent lamp scene, the weight of the target region is adjusted to be greater than or equal to the first weight value, and the weight of the region where the green object is located is adjusted to be less than or equal to the second weight value.

[0010] In a possible embodiment, in video recording, if it is determined through the recorded image data that the video is of a subject that is static, the first frequency is used to calculate and update the white balance gain parameter; if it is determined through the recorded image data that the video is of a subject that is dynamic, the second frequency is used to calculate and update the white balance gain parameter, and the second frequency is less than the first frequency.

[0011] In a possible embodiment, the white balance gain parameter of the target region is calculated based on the adjusted weight, including: in a mixed light source scene, a plurality of light source components in the mixed light source scene are separated through a multi-light source separation module, and an independent white balance gain parameter of the target region is independently calculated for each light source; and the white balance gain parameter of the target region is obtained based on the independent white balance gain parameters and a weighted fusion algorithm.

[0012] In a second aspect, the present application provides a white balance processing device, including: a scene type identification module, a weight adjustment module, a white balance gain parameter calculation module, and a target image generation module. The scene type identification module is configured to identify a scene type corresponding to one of the recorded image data in video recording. The weight adjustment module is configured to adjust the weight of the target region in the image data and the weight of the non-target region in the image data according to the scene type, so that the pixel statistical information of the target region has an effect in white balance calculation. The white balance gain parameter calculation module is configured to calculate the white balance gain parameter of the target region based on the adjusted weight. The target image generation module is configured to perform white balance processing on the image data according to the white balance gain parameter, to generate a target image. In video recording, the calculation update frequency of the white balance gain parameter is related to the scene complexity of each of the recorded image data, and the scene complexity is used to indicate the moving speed of the subject in each of the image data.

[0013] In a third aspect, the present application provides an electronic device, including: a processor, and a memory connected with the processor in communication; the memory stores computer execution instructions; and the processor executes the computer execution instructions stored in the memory, to realize the method of the first aspect.

[0014] In a fourth aspect, the present application provides a computer readable storage medium, and the computer readable storage medium stores computer execution instructions. When the computer execution instructions are executed by a processor, the computer execution instructions are used to implement the method in the first aspect.

[0015] In a fifth aspect, the present application provides a computer program product, and the computer program product includes a computer program. When the computer program is executed by a processor, the computer program implements the method in the first aspect.

[0016] According to the embodiments of the present application, different color features are exhibited by different scene types in practice, and the scene types are identified and distinguished by identifying the scene types corresponding to the image data, so that subsequent fine or differentiated white balance processing is facilitated, and the white balance processing effect is improved.

[0017] In addition, the embodiments of the present application can perform differentiated processing on the target region and the non-target region in the image data by adjusting the weight of the target region and the weight of the non-target region in the image data according to the scene type, so as to reduce the interference of the non-target region. The white balance gain parameter of the target region is calculated based on the adjusted weight, so that the white balance gain parameter of the target region is the focus of subsequent white balance processing, the interference of the non-target region is reduced, and the white balance processing effect is improved.

[0018] Furthermore, the computing update frequency of the white balance gain parameter is adaptively determined according to the subject movement speed in the video recording scene, so as to reduce the stability influence of the subject movement on the picture color in the video recording scene.

[0019] To sum up, the white balance processing method of the embodiments of the present application can reduce the color deviation of the image data and improve the color accuracy of the image data. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0021] Figure 1 FIG. 1 is a schematic diagram of the white balance processing method, device, storage medium and program product of the embodiments of the present application applied in a mobile terminal photography scene;

[0022] Figure 2 FIG. 2 is a flowchart of the white balance processing method of the embodiments of the present application;

[0023] Figure 3 FIG. 3 is a schematic diagram of the white balance processing device of the embodiments of the present application;

[0024] Figure 4 FIG. 4 is a structural schematic diagram of the electronic device of the embodiments of the present application.

[0025] The present application has been shown and described with reference to the preferred embodiments. Equivalent mechanisms and methods incorporating one or more of the above-mentioned aspects can also be used. The application is not limited to the examples described herein. The scope of the application should be determined with reference to the appended claims. DETAILED DESCRIPTION

[0026] The exemplary embodiments will be described in detail with reference to the drawings. Unless specified otherwise, the same or similar components in the drawings are identified with the same or similar reference numerals. The embodiments described in the following exemplary embodiments are not meant to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the appended claims.

[0027] The technical terms related to the embodiments of the present application will be described below.

[0028] Color temperature of a light source: a physical quantity used to describe the color characteristics of a light source, with the unit of Kelvin (K). Color temperature of a light source is defined by comparing the color of light emitted by an ideal black body at different temperatures. Color temperature of a light source describes whether the light source is “warm” or “cool”. Roughly speaking, the color of light sources such as candlelight and tungsten lamps is orange-yellow, which is called “warm light”, corresponding to low color temperature of light source, with the color temperature of light source being 2500K-3500K. The color of light sources such as midday sunlight and fluorescent lamps is relatively neutral, corresponding to high color temperature of light source, with the color temperature of light source being 4500K-6500K. The color of light sources in scenes such as overcast days, shadows, and blue skies is blue, which is called “cool light”, corresponding to high color temperature of light source, with the color temperature of light source being 7000K-9000K+.

[0029] Standard light source: a standardized lighting condition used for unified color evaluation and comparison. Standard light sources are defined by the International Commission on Illumination (CIE) to ensure consistency and comparability in evaluating the color of a light source when observing the same object at different times, places, and devices.

[0030] Taking some standard light sources as examples, the color temperature of an A light source (incandescent / tungsten lamp) is 2856K, with red and yellow light, and lack of blue light. Taking a cool white fluorescent lamp (CWF) and a tubular lamp with a display index of 84 (TL84) in a fluorescent lamp light source as examples, the color temperature of CWF is 4150K-4200K, and the color temperature of TL84 is 4000K.

[0031] White balance: based on the three primary color mixing rules of light and the color constancy of the human eye, through active intervention in the signal intensity of each channel in the RGB color space (Red-Green-Blue Color Space), offset the color deviation caused by the color temperature of the light source (Correlated Color Temperature, CCT), and restore the neutral color (white or gray) in the image data. Through white balance processing, the RGB values of the neutral color are close to the same.

[0032] Figure 1 The white balance processing method, device, storage medium and program product of the embodiments of the present application are applied to a mobile terminal photography scene.

[0033] As shown in Figure 1 , in a mobile terminal photography scene, user 1 records a video through the camera application 3 of the mobile terminal 2 such as a smartphone or a tablet computer, and the camera application 3 calls the image sensor of the mobile terminal to collect image data to generate image data 4. The white balance processing method, device, storage medium and program product of the embodiments of the present application can obtain target image 5 by performing white balance processing on image data 4.

[0034] In addition to the mobile terminal photography scene, the white balance processing method, device, storage medium and program product involved in the embodiments of the present application can also be applied to industrial detection and machine vision, security monitoring systems, unmanned aerial vehicles and intelligent driving scenes.

[0035] In the industrial detection and machine vision scene, in the industrial production line or automatic equipment, the color of the object is accurately identified through the camera, such as food sorting, electronic component detection and the like. In this scene, the image data captured by the camera is adjusted by white balance to reduce the color deviation of the image data and accurately identify the color of the object.

[0036] In the security monitoring system scene, the monitoring camera needs to maintain the color consistency of the image data under complex lighting conditions (such as day and night alternation, indoor / outdoor mixed environment) to ensure the reliability of the monitoring data. In this scene, the image data collected by the monitoring camera is adjusted by white balance to reduce the color deviation of the image data and clearly display the subject in the image data.

[0037] In the unmanned aerial vehicle and intelligent driving scene, for example, in the shadow scene, backlight scene, multi-light source scene and the like, real-time white balance adjustment is crucial for target recognition and environment perception.

[0038] In the above scenes, Auto White Balance (AWB) technology is used for white balance adjustment, which needs to take into account color accuracy, user visual preference, complex scene and other challenges.

[0039] In the related art, the color temperature of a light source is estimated by analyzing global pixel statistical information of image data, and white balance processing is performed according to the estimated color temperature of the light source to determine a white balance gain parameter. This approach has the problem that the accuracy of calculating the white balance gain parameter is disturbed by some regions, so the white balance processing still has a large color deviation.

[0040] In addition, in the related art, when performing white balance processing in a video recording scenario, a fixed frequency is used to calculate the white balance gain parameter, and in the case of high complexity of the scene of the image data, the color deviation is large.

[0041] The white balance processing method, device, equipment, storage medium and program product provided by the present application aim to solve the above technical problems of the prior art.

[0042] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present application will be described below with reference to the drawings.

[0043] Figure 2 The flowchart of the white balance processing method of the embodiments of the present application. The white balance processing method of the embodiments of the present application can be executed by an electronic device, specifically by an image data processing module of the electronic device. The image data processing module, for example, includes a processor of a smartphone or a camera. As shown in Figure 2 The white balance processing method includes steps S101 to S104.

[0044] S101, in video recording, for one of the recorded image data, identifying the scene type corresponding to the image data.

[0045] For example, the scene type corresponding to the image data can be identified by a deep learning model. Specifically, it can include: preprocessing the image data, extracting scene features in the preprocessed image data; determining the scene type based on the scene features.

[0046] The preprocessing, for example, includes image data size adjustment, normalization, and mapping color space.

[0047] The image data size adjustment is used to process the image data into a unified model input size, which can be 224 (number of pixel points) x 224 (number of pixel points) or 384 x 384.

[0048] The normalization is used to map the pixel value of each pixel point of the image data from [0, 255] to [0, 1] or [-1, 1], and subtract the mean value and divide by the standard deviation.

[0049] Exemplarily, the image data can be in a raw format, which refers to an unprocessed data format captured by an image sensor. The raw format can be mapped to an RGB color space through a Demosaicing process or a Color Interpolation process. The image data in the raw format is unprocessed, and thus is more authentic and more accurate in scene type recognition. The image data in the raw format is more optimal for subsequent white balance processing.

[0050] The deep learning model may, for example, be a convolutional neural network (CNN), a residual network (ResNet), or the like. The specific type of the deep learning model is not limited herein. The deep learning model can be trained using training samples. The training samples are image data samples and have scene type labels. The deep learning model trained using the training samples can be used to identify the scene type corresponding to the image data.

[0051] In one specific implementation, the scene types include a portrait scene, a blue sky scene, a shadow scene, a warm light scene, a fluorescent light scene, or a mixed light source scene.

[0052] The portrait scene can be understood as a scene with a person as the main subject, which is usually focused on a face or an upper body and emphasizes expressions, skin color, and background blurring. For the portrait scene, especially the scene focused on a face, the skin color restoration requirement is high, but it is prone to be yellow or too white.

[0053] The blue sky scene can be understood as a scene with a blue sky as the main subject, which is common in outdoor landscape photography. The blue sky scene has a large proportion of blue pictures, is prone to overexposure (the sky turns white) or underexposure (the ground is too dark), and the like.

[0054] The shadow scene can be understood as a scene in the shade, such as a scene under a tree, a scene in the backlight of a building, a scene in an overcast environment, and the like. The shadow scene has weak light, a dark overall, a high color temperature of the light source, and a blue picture, and the like.

[0055] The warm light scene can be understood as a scene under a light source with a low color temperature, such as a scene under an incandescent lamp, a scene under the setting sun, a scene under candlelight, and the like. The warm light scene has a low color temperature of the light source, a yellow or orange or red overall picture, and a dark picture, and the like.

[0056] The fluorescent light scene can be understood as a scene under fluorescent light (daylight) illumination. The fluorescent light scene has a non-uniform color temperature of the light source and a green or cyan picture, and the like.

[0057] The mixed light source scene can be understood as a light source with two or more different color temperatures, directions or types of light sources. The mixed light source scene has the characteristics of complex light and inconsistent colors in different regions.

[0058] The above scenarios are only examples and not limitations of the scenario types.

[0059] S102, adjust the weight of the target region in the image data and the weight of the non-target region in the image data according to the scene type, so that the pixel statistical information of the target region has an effect in the white balance calculation.

[0060] The target region can be understood as a region that is adapted to the scene type and mainly reflects neutral colors. The non-target region is other regions in the image data except the target region.

[0061] Taking the above scenario type as an example, since the human eye is more sensitive to skin color, the face region is the key perspective focus, and the target region in the image data of the portrait scene includes the face region.

[0062] In the blue sky scene, the user wants to keep the blue color of the sky. If the sky region is used as a white reference for subsequent white balance processing, the blue color of the sky in the image data will become gray. Therefore, the target region in the image data of the blue sky scene includes the non-sky region.

[0063] In the shadow scene, the neutral color object in the shadow can represent the color temperature of the current light source, so the target region in the image data of the shadow scene includes the region where the neutral object in the shadow is located.

[0064] In the warm light scene, the neutral color object can reflect the yellow color that needs to be neutralized, so the target region in the image data of the warm light scene includes the region where the neutral object is located.

[0065] The target region in the image data of the fluorescent lamp scene includes the region where the neutral object is located.

[0066] The mixed light source scene has the characteristics of complex light and inconsistent colors in different regions, and the background region can be ignored, and the main body of the image data is focused on, so the target region in the image data of the mixed light source scene includes the region where the main body is located.

[0067] It should be noted that generally, according to visual habits and image data shooting habits, the main body is usually located in the center region of the image data, and the target region mentioned in the above embodiments is also generally located in the center region of the image data.

[0068] It can be understood that the weight of the target region in the image data can be adjusted to be higher, and the weight of the non-target region in the image data can be adjusted to be lower. Thus, the region with neutral color can be used as a reference for white balance processing, so that the subsequent white balance processing is more accurate.

[0069] The pixel statistical information of the target region can be understood as a mathematical statistics on the RGB values of all pixel points or specific region pixel points of the target region, so as to extract information reflecting overall or local features. Since the light source color temperature represents the color of the light source, the color of the light source affects the RGB proportion of the light reflected by the object, and the RGB values of the pixel points corresponding to the target region, the pixel statistical information of the target region can be used to deduce the light source color temperature, and then the white balance gain parameter of the target region is determined according to the mapping relationship between the light source color temperature and the white balance gain parameter.

[0070] The pixel statistical information may, for example, include average R / G, B / G ratio, color histogram, average brightness, brightness histogram, saturation distribution, etc.

[0071] S103: calculating the white balance gain parameter of the target region based on the adjusted weight.

[0072] In a possible embodiment, the weight of the target region may, for example, be adjusted to 1, and the weight of the non-target region may, for example, be adjusted to 0, i.e. the non-target region is ignored.

[0073] In another possible embodiment, the weight of the target region may, for example, be adjusted to be greater than or equal to a first weight value, the weight of the non-target region may, for example, be adjusted to be less than or equal to a second weight value, and the first weight value is greater than the second weight value. That is, the white balance gain parameter of the target region is focused on. The value range of the first weight value and the second weight value is [0, 1] and the sum of the first weight value and the second weight value is 1.

[0074] The step S103 of calculating the white balance gain parameter of the target region based on the adjusted weight may include: calculating the white balance gain parameter of the target region based on the adjusted weight of the target region and the pixel statistical information of the target region.

[0075] In addition, in another embodiment of the present application, the white balance gain parameter of the image data can also be calculated based on the adjusted weight of the target region, the adjusted weight of the non-target region, and the pixel statistical information of the image data. The white balance gain parameter of the image data can be used for the white balance processing of the image data in the following.

[0076] S104, performing white balance processing on the image data according to the white balance gain parameter to generate a target image.

[0077] For example, the electronic device can perform white balance processing on the image data by applying the gain indicated by the white balance gain parameter to the RGB values of the image data, to generate a target image.

[0078] In the video recording, the calculation update frequency of the white balance gain parameter is related to the scene complexity of each image data recorded, and the scene complexity is used to indicate the moving speed of the subject in each image data.

[0079] The moving speed of the subject in the image data can be determined according to a ratio of a displacement change amount of the subject in each image data to a quantity of the image data.

[0080] For example, the calculation update frequency of the white balance gain parameter can be positively correlated with a scene complexity of each recorded image data, and the scene complexity is positively correlated with the moving speed of the subject in each image data.

[0081] According to the embodiment of the present application, the color features of different scene types are different, and the scene types are identified and distinguished by identifying the scene type corresponding to the image data, so as to facilitate subsequent fine or differentiated white balance processing, and improve the white balance processing effect.

[0082] In addition, the embodiment of the present application adjusts the weight of the target region in the image data and the weight of the non-target region in the image data according to the scene type, so as to perform differentiated processing on the target region and the non-target region in the image data, and reduce the interference of the non-target region. The white balance gain parameter of the target region is calculated based on the adjusted weight, so that the white balance gain parameter of the target region is the focus of subsequent white balance processing, the interference of the non-target region is reduced, and the white balance processing effect is improved.

[0083] Furthermore, the calculation update frequency of the white balance gain parameter is adaptively determined according to the moving speed of the subject in the video recording scene, so as to reduce the stability influence of the subject movement on the picture color in the video recording scene.

[0084] In summary, the white balance processing method of the embodiment of the present application can reduce the color deviation of the image data and improve the color accuracy of the image data.

[0085] In a possible embodiment, in the video recording, if it is determined through the recorded image data that the video recording is a video with a static subject, the white balance gain parameter is calculated and updated at a first frequency; if it is determined through the recorded image data that the video recording is a video with a dynamic subject, the white balance gain parameter is calculated and updated at a second frequency, and the second frequency is less than the first frequency.

[0086] When the moving speed of the subject is less than a speed threshold, the subject can be determined to be static. When the moving speed of the subject is greater than or equal to the speed threshold, the subject can be determined to be dynamic.

[0087] The speed threshold, the first frequency and the second frequency can be pre-set according to experimental results.

[0088] In the embodiment of the present application, in the case of a static subject, the color of the image data is stable, and the white balance gain parameter is updated by high-frequency calculation, which can quickly adapt to changes in illumination and has higher color accuracy. In the case of a dynamic subject, the color of the image data is unstable, and the white balance gain parameter is updated by low-frequency calculation, which can prevent color jumping and the like.

[0089] In a possible embodiment, the reference data required for performing the white balance processing method of the present application can be obtained through testing in a laboratory stage, and the reference data that passes the testing is written into the electronic device.

[0090] The reference data, for example, includes at least one of an AWB (Auto White Balance) calibration table, a light source color temperature probability table, a brightness threshold, a saturation threshold, a valid area, and an invalid area.

[0091] The AWB calibration table can be understood as a standard white balance gain parameter required for different light source color temperatures.

[0092] The determination method of the AWB calibration table will be described below: the AWB calibration table is obtained by collecting image data of a standard light source, calculating the white balance gain parameter corresponding to the image data of the standard light source, and constructing a mapping relationship between the light source color temperature and the white balance gain parameter. The process requires a standard light box to provide a standard light source, a standard color card, and the like.

[0093] Specifically, collecting the image data of the standard light source includes fixing the electronic device (which has an image sensor) in front of the standard light box and aligning the standard color card to ensure uniform illumination. Under each standard light source, the image data is obtained by photographing the image data through the electronic device with the image sensor. Calculating the white balance gain parameter corresponding to the image data of the standard light source includes selecting a neutral color block on the standard color card, reading the average RGB value of the neutral color block, taking the G channel as a reference, calculating the R gain and B gain that restore the neutral color block to a proportion close to 1:1:1 of the RGB value as the white balance gain parameter. For each standard light source, the data points of the light source color temperature and the white balance gain of the standard light source can be obtained through the above steps. The AWB calibration table (AWB Calibration Table) can be obtained by fitting the data points of a plurality of standard light sources.

[0094] The light source color temperature probability table can be understood as defining how to map the pixel statistical information of the image data to the light source color temperature value with the highest confidence. By adjusting the light source color temperature probability table, the sensitivity of a specific light source color temperature range can be changed.

[0095] The brightness threshold and the saturation threshold are preset. In the embodiments of the present application, the pixels with too high or too low brightness can be excluded by the brightness threshold. The pixels with too high or too low saturation can be excluded by the saturation threshold. These pixels can be noise and can affect the effect of the white balance processing.

[0096] The effective area can be understood as an area that must participate in the white balance processing.

[0097] The ineffective area can be understood as an area that does not participate in the white balance processing. The effective area and the ineffective area can be preset according to experimental results.

[0098] The above reference data can be obtained by relevant personnel in the laboratory stage. The reference data can also support updating. For example, the reference data can be updated in the form of installation package upgrade. The installation package is an installation package of an application program with a white balance processing function. The electronic device upgraded by the installation package can use the updated reference data for white balance processing.

[0099] For each scene type, the weight of the target area in the image data and the weight of the non-target area in the image data can be adjusted, and the white balance gain parameter of the target area is calculated based on the adjusted weights to perform white balance processing.

[0100] In the following, the application of some other embodiments of the present application to specific scene types will be described in detail in combination with the characteristics of various scene types.

[0101] In a possible embodiment, the image data of the warm light scene can also be detected to determine the area of the warm light area in the warm light scene. The light source color temperature is corrected according to the area of the warm light area, and the white balance gain parameter is adjusted. The light source color temperature here is determined by the pixel statistical information of the image data of the warm light scene.

[0102] Specifically, in the case that the warm light area is large, the light source color temperature determined by the pixel statistical information of the image data of the warm light scene is lower than the actual light source color temperature. This leads to poor effect of the white balance processing according to the inaccurate (lower than the actual light source color temperature) light source color temperature.

[0103] The light source color temperature determined by the pixel statistical information of the image data of the warm light scene is specifically determined according to a light source color temperature probability table.

[0104] In the embodiments of the present application, in the case that the area of the warm light area is greater than an area threshold, the light source color temperature determined by the pixel statistical information of the image data of the warm light scene can be increased to compensate for the influence of the lower light source color temperature value. Furthermore, the white balance gain parameter calculated according to the increased light source color temperature is also more accurate. Therefore, the effect of the white balance processing can be improved.

[0105] The area threshold value can be preset according to experimental results.

[0106] In a possible embodiment, in the case that the area of the warm light region is greater than the area threshold value in the warm light scene, the brightness threshold value can also be adjusted to exclude dark pixel points. In the warm light scene, the dark pixel points have a higher probability of being noise, and the excluded pixel points do not participate in the calculation of the pixel statistical information and thus do not participate in the calculation of the white balance gain parameter. By excluding the dark pixel points, the effect of the white balance processing can be improved.

[0107] Similarly, in a possible embodiment, in the case that the area of the warm light region is greater than the area threshold value in the warm light scene, the saturation threshold value can also be adjusted to exclude oversaturated pixel points. In the warm light scene, the oversaturated pixel points have a higher probability of being noise, and the effect of the white balance processing can be improved by adjusting the saturation threshold value.

[0108] In a possible embodiment, the image data of the shadow scene can also be detected to determine whether a target whose color information meets a preset condition exists in the image data of the shadow scene. If so, the light source color temperature is corrected and the white balance gain parameter is adjusted.

[0109] The preset condition includes that the blue chroma is greater than a threshold value. That is, the target whose color information meets the preset condition is a blue target. The preset condition can also include that the area of the blue target is greater than a threshold value.

[0110] Specifically, if there is a large piece of blue or a blue target with a deeper color in the shadow scene, the light source color temperature determined by the pixel statistical information of the image data of the shadow scene will be higher than the actual light source color temperature. This leads to poor effects of the white balance processing according to the inaccurate (higher than the actual light source color temperature) light source color temperature.

[0111] The light source color temperature determined by the pixel statistical information of the image data of the shadow scene is specifically determined according to a light source color temperature probability table.

[0112] In the embodiment of the present application, in the case that a target whose color information meets a preset condition exists in the image data of the shadow scene, the light source color temperature determined by the pixel statistical information of the image data of the warm light scene can be reduced to compensate for the influence of a higher light source color temperature. Furthermore, the white balance gain parameter calculated according to the reduced light source color temperature is also more accurate. In this way, the effect of the white balance processing can be improved.

[0113] It should be noted that the increase and decrease ranges of the light source color temperature can be determined according to a preset mapping table of experimental results.

[0114] In the case of reducing the color temperature of the light source as described above, it is necessary to note that there is a case where the image data of the shadow scene includes a blue sky region. In the process of reducing the color temperature of the light source for white balance processing, the color of the blue sky region can be affected. In a possible embodiment, the image data of the shadow scene can be detected to determine whether the image data of the shadow scene includes a blue sky region. If so, the saturation or color consistency parameter of the adjusted image data is adjusted. By adjusting the saturation or color consistency parameter, specific colors such as blue can be independently controlled under different light source color temperatures to avoid distortion of specific colors during white balance processing.

[0115] For a fluorescent lamp scene, the spectrum of the fluorescent lamp is discontinuous, with a sharp peak in the green band, resulting in a strong green signal perceived by the image sensor, and thus a greenish image data.

[0116] In a possible embodiment, in a fluorescent lamp scene, the weight of the target region is adjusted to be greater than or equal to the first weight value, and the weight of the region where the green object is located is adjusted to be less than or equal to the second weight value.

[0117] In the embodiments of the present application, the region where the green object is located can be further distinguished on the basis of distinguishing the target region and the non-target region. By adjusting the weight of the target region to be greater than or equal to the first weight value, and adjusting the weight of the region where the green object is located to be less than or equal to the second weight value, the focus of the white balance processing is further placed on the target region, and the interference of the region where the green object is located on the white balance processing under the fluorescent lamp scene is reduced. Thus, the color deviation of the white balance processing under the fluorescent lamp scene can be reduced, and the color accuracy of the white balance processing under the fluorescent lamp scene can be improved.

[0118] In another possible embodiment, in addition to the white balance processing of the image data of the fluorescent lamp scene, color deviation compensation in the "green-magenta" direction can be additionally added to handle the greenish problem. The color deviation compensation in the "green-magenta" direction is realized by a color deviation value (Delta u, v, Duv).

[0119] Specifically, in a possible embodiment, in a fluorescent lamp scene, the white balance processing method can further include: compensating for the influence of excessive green light by adjusting a color deviation value Duv in a related light source color temperature CCT space, wherein Duv is a general value or an individual value related to user preference; and the white balance processing of the image data according to the white balance gain parameter in step S104 to generate a target image includes: white balance processing of the image data according to the white balance gain parameter and color adjustment of the image data according to the color deviation value Duv to generate a target image.

[0120] In a possible embodiment, the adjusting the color deviation value Duv in the relevant light source color temperature CCT space comprises: offsetting the Duv coordinate corresponding to the image data by a preset value in the negative Duv direction, and the preset value belongs to 5%-15%. Specifically, the Duv coordinate corresponding to the image data of the fluorescent lamp scene can be offset by a preset value in the negative Duv direction.

[0121] Duv is positive, that is, the light source is green; Duv is negative, that is, the light source is magenta; and Duv is 0, that is, the light source has no deviation in the "green-magenta" direction. In the embodiment of the application, by offsetting the Duv coordinate corresponding to the image data of the fluorescent lamp scene by a preset value in the negative Duv direction, color deviation compensation in the "green-magenta" direction can be added to the image data to process the problem of greenish picture of the image data.

[0122] In a possible embodiment, the white balance gain parameter of the target region is calculated based on the adjusted weight, comprising: in a mixed light source scene, separating a plurality of light source components in the mixed light source scene, and independently calculating an independent white balance gain parameter of the target region for each light source; and obtaining the white balance gain parameter of the target region based on the independent white balance gain parameters and a weighted fusion algorithm.

[0123] For example, the image data is subjected to semantic segmentation by a deep learning model, and the plurality of light source components in the mixed light source scene and the target region corresponding to each light source can be identified. The deep learning model for semantic segmentation can be the same as or different from the deep learning model for identifying the scene type. In the case of the same, the training samples of the two models are different, resulting in different functions of the two models. In the case of the different, the deep learning model for semantic segmentation may, for example, be an encoder-decoder, a fully convolutional network, etc.

[0124] The independent white balance gain parameter of the target region is independently calculated for each light source, and the calculation manner is the same as in the above embodiment, that is, the pixel statistical information of the target region is calculated, the light source color temperature value is calculated based on the pixel statistical information of the target region, and the corresponding white balance gain parameter is mapped according to the light source color temperature value, which will not be described in detail here.

[0125] The weight of the independent white balance gain parameter corresponding to each light source can be set in advance according to experimental results.

[0126] According to the embodiments of the application, by separating the mixed light source and independently processing, the color inconsistency problem caused by the single and global white balance gain parameter can be improved, and the common overall color deviation problem in the mixed light source scene can also be inhibited. That is, the embodiments of the application can reduce the color deviation of the image data and improve the color accuracy of the image data.

[0127] It should be noted that, in the presence of the effective area and the invalid area, the target area, the non-target area, the invalid area and the effective area in the embodiment of the present application constitute a hierarchical differential processing mechanism of image data. Among them, the effective area must participate in the white balance processing, the target area participates in the white balance processing with a higher weight, the non-target area participates in the white balance processing with a lower weight, and the invalid area does not participate in the white balance processing. The core of the white balance calculation can be forced to be anchored in the area most critical to color judgment through the effective area and the target area. The influence of the area meaningless or easy to mislead the color judgment can be reduced through the invalid area and the non-target area. The necessity difference of various areas participating in the white balance processing can facilitate to dynamically adapt to various scene types and stably and efficiently reduce the color deviation of image data.

[0128] Among them, the target area and the effective area can be completely coincident, partially coincident or non-coincident. The target area and the non-target area are non-coincident, and the non-target area and the invalid area can be completely coincident, partially coincident or non-coincident.

[0129] Figure 3 A structural schematic diagram of a white balance processing device provided by the embodiment of the present application. The white balance processing device can be applied to an electronic device. As shown in the figure, Figure 3 The white balance processing device provided by the embodiment of the present application includes a scene type identification module 310, a weight adjustment module 320, a white balance gain parameter calculation module 330 and a target image generation module 340.

[0130] The scene type identification module 310 is used to identify the scene type corresponding to the image data in the video recording.

[0131] The weight adjustment module 320 is used to adjust the weight of the target area in the image data and the weight of the non-target area in the image data according to the scene type, so that the pixel statistical information of the target area has an effect in the white balance calculation.

[0132] The white balance gain parameter calculation module 330 is used to calculate the white balance gain parameter of the target area based on the adjusted weight.

[0133] The target image generation module 340 is used to perform white balance processing on the image data according to the white balance gain parameter, so as to generate a target image; wherein in the video recording, the calculation update frequency of the white balance gain parameter is related to the scene complexity of each image data recorded, and the scene complexity is used to indicate the moving speed of the subject in each image data.

[0134] In a possible embodiment, the scene type includes a portrait scene, a blue sky scene, a shadow scene, a warm light scene, a fluorescent lamp scene or a mixed light source scene.

[0135] In a possible embodiment, the white balance processing apparatus further includes a color deviation value adjustment module configured to compensate the influence of excessive green light by adjusting a color deviation value Duv in a correlated color temperature CCT space of a light source in a fluorescent lamp scene, wherein the Duv is a general value or an individual value related to user preference; and the target image generation module includes a target image generation sub-module configured to perform white balance processing on the image data according to the white balance gain parameter and perform color adjustment on the image data according to the color deviation value Duv, to generate the target image.

[0136] In a possible embodiment, the color deviation value adjustment module includes an offset sub-module configured to offset the Duv coordinate corresponding to the image data to a negative Duv direction by a preset value, and the preset value belongs to 5%-15%.

[0137] In a possible embodiment, the weight of the target region is greater than or equal to a first weight value, the weight of the non-target region is less than or equal to a second weight value, and the first weight value is greater than the second weight value; in the fluorescent lamp scene, the weight of the target region is adjusted to be greater than or equal to the first weight value, and the weight of the region where the green object is located is adjusted to be less than or equal to the second weight value.

[0138] In a possible embodiment, in video recording, if it is determined through the recorded image data that the video being recorded is a video with a static subject, the white balance gain parameter is updated at a first frequency; and if it is determined through the recorded image data that the video being recorded is a video with a dynamic subject, the white balance gain parameter is updated at a second frequency, and the second frequency is less than the first frequency.

[0139] In a possible embodiment, the white balance gain parameter calculation module includes a separation sub-module and a weighting sub-module, the separation sub-module is configured to separate a plurality of light source components in a mixed light source scene through a multi-light source separation module, and calculate an independent white balance gain parameter of the target region for each light source independently; and the weighting sub-module is configured to obtain the white balance gain parameter of the target region based on each independent white balance gain parameter and a weighted fusion algorithm.

[0140] Figure 4 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the electronic device provided by the embodiment of the present application includes a processor 401 and a memory 402. Figure 4 As shown in FIG. 1, the electronic device further includes an image sensor 403. Optionally, the device further includes a communication component 404. The processor 401, the memory 402, the image sensor 403 and the communication component 404 are connected through a bus 405. Figure 4

[0141] The electronic device may, for example, be a smart phone, a camera, a security monitoring device, etc. ​

[0142] In the implementation process, the image sensor 403 can collect image data for video recording.

[0143] The memory 402 stores codes, and the processor 401 executes the codes stored in the memory 402 to perform the method of the above method embodiments.

[0144] The specific implementation process of the processor 401 can refer to the above method embodiments, which have similar implementation principles and technical effects, and details are not described here.

[0145] In the above Figure 4 In the embodiment shown, it should be understood that the processor 401 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly embodied as hardware processor execution or combined with hardware and software modules in the processor for execution.

[0146] The memory 402 can include a high-speed RAM memory, and can also include a non-volatile storage NVM, such as at least one disk memory.

[0147] The image sensor 403 can be understood as an electronic element that converts optical signals (photons) into electrical signals (voltage / current).

[0148] The bus 405 can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus 405 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 405 in the drawings of the present application does not limit to only one bus or one type of bus.

[0149] The embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores computer execution instructions. When the computer execution instructions are executed by a processor, the computer execution instructions are used to implement the method of the above method embodiments.

[0150] The computer readable storage medium described above can be realized by any type of volatile or nonvolatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0151] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium, and can write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0152] The embodiment of the present application provides a computer program product, comprising a computer program, the computer program is executed by the processor to realize the method provided by any of the above embodiments of the present application.

[0153] It should be noted that, for the above-mentioned method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action order described, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0154] Further need to be explained is that although each step in the flowchart is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise stated in this paper, the execution of these steps has no strict order limit, and these steps can be executed in other order. Moreover, at least part of the steps in the flowchart can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed with at least part of other steps or other steps or stages of sub-steps or stages, or alternately.

[0155] It should be understood that the above-described apparatus embodiments are merely illustrative, and the apparatus of the present application can also be implemented in other manners. For example, the division of units / modules in the above-described embodiments is merely a logical function division, and the actual implementation can be in another manner. For example, a plurality of units / modules or components can be combined, or can be integrated into another system, or some features can be omitted or not executed.

[0156] In addition, each functional unit / module in each embodiment of the present application can be integrated into one unit / module, or each unit / module can be physically present alone, or two or more units / modules can be integrated together. The integrated unit / module can be implemented in the form of hardware or in the form of a software program module.

[0157] The integrated unit / module, if implemented in the form of hardware, can be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory (RRAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), an enhanced dynamic random access memory (EDRAM), a high-bandwidth memory (HBM), a hybrid memory cube (HMC), etc.

[0158] If the integrated units / modules are implemented in the form of software program modules and sold or used as independent products, they can be stored in a computer readable memory. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a memory and includes a number of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the embodiments of the method of the present application. The aforementioned memory includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0159] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0160] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The application is intended to cover any variations, uses or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice in the art to which the application pertains or can relate. The specification and examples are to be regarded as exemplary only, and the true scope and spirit of the application are indicated by the following claims.

[0161] It should be understood that the application is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is limited only by the appended claims.

Claims

1. A white balance processing method characterized by comprising: The method comprises: In video recording, for one of the recorded image data, a scene type corresponding to the image data is identified; According to the scene type, the weight of a target region in the image data and the weight of a non-target region in the image data are adjusted, so that the pixel statistical information of the target region has an effect in white balance calculation, and the target region is a neutral color region adapted to the scene type; A white balance gain parameter of the target region is calculated based on the adjusted weight; The image data is subjected to white balance processing according to the white balance gain parameter to generate a target image; In the video recording, the calculation update frequency of the white balance gain parameter is related to the scene complexity of each image data recorded, and the scene complexity is used to indicate the moving speed of the subject in the image data. In the video recording, if it is judged through the recorded image data that the recorded video is a video with a static subject, a first frequency is used to calculate and update the white balance gain parameter; if it is judged through the recorded image data that the recorded video is a video with a dynamic subject, a second frequency is used to calculate and update the white balance gain parameter, and the second frequency is less than the first frequency.

2. The method of claim 1, wherein, The scene type includes a portrait scene, a blue sky scene, a shadow scene, a warm light scene, a fluorescent lamp scene or a mixed light source scene.

3. The method of claim 2, wherein, The method further comprises: In the fluorescent lamp scene, the influence of excessive green light is compensated by adjusting a color deviation value Duv in a related light source color temperature CCT space, wherein the Duv is a general value or an individual value related to user preference; The white balance processing of the image data according to the white balance gain parameter to generate a target image comprises: The white balance processing of the image data according to the white balance gain parameter and the color adjustment of the image data according to the color deviation value Duv are performed to generate the target image.

4. The method of claim 3, wherein, The adjustment of the color deviation value Duv in the related light source color temperature CCT space comprises: The Duv coordinate corresponding to the image data is offset by a preset value in the negative Duv direction, and the preset value belongs to 5%-15%.

5. The white balance processing method according to any one of claims 2 to 4, characterized by, The weight of the target region is greater than or equal to a first weight value, and the weight of the non-target region is less than or equal to a second weight value, and the first weight value is greater than the second weight value; In the fluorescent lamp scene, the weight of the target region is adjusted to be greater than or equal to the first weight value, and the weight of the region where the green object is located is adjusted to be less than or equal to the second weight value.

6. The white balance processing method according to any one of claims 2 to 4, characterized by, The calculation of the white balance gain parameter of the target region based on the adjusted weight comprises: In the mixed light source scene, a plurality of light source components in the mixed light source scene are separated, and an independent white balance gain parameter of the target region is independently calculated for each light source; The white balance gain parameter of the target region is obtained based on each independent white balance gain parameter and a weighted fusion algorithm.

7. An electronic device, comprising: The method comprises: A processor and a memory connected in communication with the processor; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory to realize the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method of any one of claims 1 to 6.

9. A computer program product, characterised in that, A computer program is included, which, when executed by a processor, implements the method of any one of claims 1 to 6.

Images