White balance processing method and device, electronic equipment and storage medium
Through the multi-color gamut white point estimation method, multiple networks in the white point estimation model are used for image processing, which solves the problem of insufficient white balance processing accuracy in the prior art, and achieves higher accuracy and robust white balance adjustment.
Patent Information
- Application Number
- CN202410233572.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-29
- Publication Date
- 2025-07-25
AI Technical Summary
In the prior art, the accuracy of automatic white balance processing is poor.
By obtaining the image of the target color space, the first and second white point estimation networks in the white point estimation model are used to estimate white points in multi-color gamut, and a variety of feature vectors and white point estimation results are output, and a variety of white point estimation results are combined for white balance processing.
Improve the accuracy and robustness of white balance processing, and achieve more accurate white balance adjustment.
Smart Images
Figure CN120378757A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of image processing, and in particular, to a white balance processing method, apparatus, electronic device, and storage medium. Background Art
[0002] Accurate color reproduction is an important part of image processing in an imaging system. Since a camera imaging system does not have color constancy similar to the human visual system, it is necessary to perform automatic white balance correction on the images captured by the camera to restore the true colors of the image objects.
[0003] In the related art, the accuracy of automatic white balance processing is poor. Summary of the Invention
[0004] The present application aims to solve at least one of the technical problems in the related art to some extent.
[0005] To this end, the present application proposes a white balance processing method, apparatus, electronic device, and storage medium, so as to obtain multiple white point estimation results based on the input of multiple color gamuts, and then perform white balance processing based on the multiple white point estimation results, thereby improving the accuracy of white balance processing.
[0006] An embodiment of one aspect of the present application proposes a white balance processing method, including:
[0007] Obtaining a first image in a target color space obtained by processing an original image;
[0008] Inputting the first image into a first white point estimation network in a white point estimation model to perform white point estimation, and outputting a first feature vector and a first white point estimation result in a first color gamut, as well as a second feature vector and a second white point estimation result in a second color gamut;
[0009] Inputting the first feature vector and the second feature vector into a second white point estimation network in the white point estimation model to perform white point estimation, and obtaining a third white point estimation result;
[0010] Performing white balance processing on the original image according to the first white point estimation result, the second white point estimation result, and the third white point estimation result.
[0011] An embodiment of another aspect of the present application proposes a white balance processing apparatus, including:
[0012] An obtaining module, configured to obtain a first image in a target color space obtained by processing an original image;
[0013] A first estimation module, configured to input the first image into a first white point estimation network in a white point estimation model to perform white point estimation, and output a first feature vector of a first color gamut, a first white point estimation result, a second feature vector of a second color gamut, and a second white point estimation result;
[0014] A second estimation module, configured to input the first feature vector and the second feature vector into a second white point estimation network in the white point estimation model to perform white point estimation, and obtain a third white point estimation result;
[0015] A processing module, configured to perform white balance processing on the original image according to the first white point estimation result, the second white point estimation result, and the third white point estimation result.
[0016] Another embodiment of this application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method described in the foregoing aspect is implemented.
[0017] Another embodiment of this application provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in the foregoing aspect is implemented.
[0018] Another embodiment of this application provides a computer program product, on which a computer program is stored. When the program is executed by a processor, the method described in the foregoing aspect is implemented.
[0019] The white balance processing method, device, electronic device, and storage medium provided by this application obtain a first image of a target color space obtained by processing an original image, input the first image into a first white point estimation network in a white point estimation model to perform white point estimation, output a first feature vector of a first color gamut, a first white point estimation result, a second feature vector of a second color gamut, and a second white point estimation result, input the first feature vector and the second feature vector into a second white point estimation network in the white point estimation model to perform white point estimation, obtain a third white point estimation result, and perform white balance processing on the original image according to the first white point estimation result, the second white point estimation result, and the third white point estimation result, so as to achieve obtaining multiple white point estimation results based on inputs of multiple color gamuts, and further perform white balance processing based on multiple white point estimation results, thereby improving the accuracy of white balance processing.
[0020] Additional aspects and advantages of this application will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of this application. Description of the Drawings
[0021] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of embodiments in conjunction with the accompanying drawings, where:
[0022] Figure 1 It is a schematic flowchart of a white balance processing method provided by an embodiment of the present application;
[0023] Figure 2 It is a schematic flowchart of another white balance processing method provided by an embodiment of the present application;
[0024] Figure 3 It is one of the schematic diagrams of a white point estimation result provided by an embodiment of the present application;
[0025] Figure 4 It is another schematic diagram of a white point estimation result provided by an embodiment of the present application;
[0026] Figure 5 It is a schematic structural diagram of a white point estimation model provided by an embodiment of the present application;
[0027] Figure 6 It is a schematic flowchart of another white balance processing method provided by an embodiment of the present application;
[0028] Figure 7 It is a schematic structural diagram of another white balance processing device provided by an embodiment of the present application;
[0029] Figure 8 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners
[0030] The embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, but should not be construed as limiting the present application.
[0031] The white balance processing method, device, electronic device, and storage medium of the embodiments of the present application will be described below with reference to the accompanying drawings.
[0032] Figure 1 It is a schematic flowchart of a white balance processing method provided by an embodiment of the present application.
[0033] The execution subject of the white balance processing method in the embodiment of the present application is a white balance processing device, which can be set in an electronic device. The electronic device can be a smart phone, a smart watch, a smart wearable device, a palm computer, etc., which are not limited in this embodiment.
[0034] Such as Figure 1As shown, the method may include the following steps:
[0035] Step 101: Obtain a first image in a target color space obtained by processing an original image.
[0036] The original image is an original image collected by an image sensor, usually Bayer data RGrGbB. Bayer data is a specific color filter array (CFA) format used when the image sensor captures an image.
[0037] The target color space corresponds to the sensor. That is, there are differences in the spectral responses of different sensors, so different sensors correspond to different color spaces. As an implementation, different sensors can be unified into the same color space, that is, the target color space, so as to maintain high generalization and accuracy even if the spectral response differences between different types of sensors are large. The target color space is the color space of the reference image sensor. If the sensor that collects the original image is used as the reference sensor, then the color space corresponding to the sensor that collects the original image is the target color space. If another sensor is used as the reference sensor, then the color space selected as the reference sensor is the target color space.
[0038] The first image is obtained by processing the original image. As an implementation, the original image can be preprocessed to obtain the first image, such as a three-channel linear RGB image. The preprocessing includes: performing lens shading correction and black level subtraction on the original image to reduce the influence of luminance uniformity (luma shading), color uniformity (color shading), and dark current on the image quality of the input image of the model, and then converting the original image into an RGB image. Among them, the R and B channels remain unchanged, and the average of Gr and Gb is taken to obtain the G channel, and local average downsampling is performed on the obtained RGB image to obtain a small-size RGB image. Furthermore, the small-size RGB image is converted into the target color space corresponding to the reference sensor to obtain the first image, so as to perform processing in the same color space.
[0039] Step 102: Input the first image into the first white point estimation network in the white point estimation model to perform white point estimation, and output the first feature vector and the first white point estimation result of the first color gamut, as well as the second feature vector and the second white point estimation result of the second color gamut.
[0040] The white point estimation model is pre-trained and includes a first white point estimation network and a second white point estimation network.
[0041] In the embodiments of the present application, the first image is input into the first white point estimation network for white point estimation, and the features of two color gamuts and the white point estimation results are obtained, which are respectively called the first feature of the first color gamut and the first white point estimation result, and the second feature vector and the second white point estimation result of the second color gamut. Among them, the first feature is a chromaticity feature, which indicates the difference between different color channels. For example, it is a chromaticity logarithmic histogram feature. Among them, the second feature is a semantic feature related to white point estimation, which can indicate the area of focus in the image, such as an object in the image, such as a face. The white point estimation result indicates the color deviation of the white point. The white point is the point of the white pixel. The white point estimation result is used to determine the white balance parameter, where the white balance parameter includes the white balance gain value. Among them, the second color gamut is the three-channel RGB color gamut, where RGB are the three color channels, which are the red RED color channel, the blue Blue color channel, and the green Green color channel respectively. The first color gamut is a color gamut determined based on the three-channel RGB color gamut, or a color gamut determined based on the chromaticity logarithmic feature. As an implementation, it is a new color channel determined based on the differences between the channel values of the R color channel, the G color channel, and the B color channel among the three color channels. Each new color channel in the first color gamut indicates the difference between one color channel among the three color channels and the channel values of the other two color channels respectively. Among them, the white point in the image refers to a certain area of an image being pure white, that is, there is no any fine level, and such an area is identified.
[0042] Step 103: Input the first feature vector and the second feature vector into the second white point estimation network in the white point estimation model for white point estimation to obtain the third white point estimation result.
[0043] In the embodiments of the present application, the second white point estimation network performs white point estimation based on the first feature vector of the first color gamut and the second feature vector of the second color gamut, realizes white point estimation of the features of the two color gamuts based on the chromaticity difference feature of the first color gamut and the semantic feature of the second color gamut, and by simultaneously focusing on the features of the two color gamuts, realizes the complementarity of the features of the two color gamuts, obtains a more accurate third white point estimation result, and then determines the corresponding third white point estimation result according to the third white point estimation result.
[0044] Step 104: Perform white balance processing on the original image according to the first white point estimation result, the second white point estimation result, and the third white point estimation result.
[0045] In an implementation of the embodiment of the present application, according to the set weight value, the first white point estimation result, the second white point estimation result, and the third white point estimation result can be weighted and added to obtain the target white point estimation result. According to the target white point estimation result, the corresponding target white balance parameter is determined, and the original image is subjected to white balance processing according to the target white balance parameter, that is, the gain values of the red channel, the blue channel, and the green channel included in the target white balance parameter are used to calculate the red (Red, R), green (Green, G), and blue (Blue, B) values of each pixel after adjustment, so as to correct the color of the original image. By performing white balance adjustment through multiple output white point estimation results, the characteristics of multiple color gamuts are considered, and good algorithm generalization is achieved according to the semantic information of the second color gamut. At the same time, the first color gamut is also used to obtain more accurate white point estimation accuracy. Through the complementary advantages of the second color gamut and the first color gamut, the network learns more robust features and outputs multiple white point estimation results, improving the accuracy and robustness of white balance adjustment.
[0046] In the white balance processing method of the embodiment of the present application, a first image in the target color space obtained by processing the original image is acquired, and the first image is input into the first white point estimation network in the white point estimation model for white point estimation, and a first feature vector and a first white point estimation result in the first color gamut, as well as a second feature vector and a second white point estimation result in the second color gamut are output. The first feature vector and the second feature vector are input into the second white point estimation network in the white point estimation model for white point estimation to obtain a third white point estimation result. According to the first white point estimation result, the second white point estimation result, and the third white point estimation result, the original image is subjected to white balance processing to achieve obtaining multiple white point estimation results based on the input of multiple color gamuts, and then performing white balance processing based on multiple white point estimation results, improving the accuracy of white balance processing.
[0047] Based on the above embodiment, Figure 2 is a schematic flowchart of another white balance processing method provided by the embodiment of the present application, as Figure 2 shown, the method includes the following steps:
[0048] Step 201, determine the color space conversion matrix.
[0049] Among them, the color space conversion matrix includes the conversion relationship between the first color space corresponding to the target image sensor for acquiring the original image and the target color space.
[0050] Since there are differences in spectral response sensitivity between different image sensors, it will cause the overlap degree of the white point distribution space in the white map of different image sensors to deteriorate, as Figure 3As shown in the white map in , the white dots of different colors are relatively scattered, which will make the generalization of the white dot estimation model for processing data from different sensors poor. Therefore, in order to improve the generalization of the model, this application adopts the mapping of the secondary color space, that is, all are mapped to the same target color space, such as Figure 4 As shown in the white map in , after unifying the color space, the consistency of the standard white dot GT white dot in the white map space distribution has been greatly improved, realizing that even if the spectral response differences between different types of sensors are large, a high generalization can still be maintained.
[0051] In one implementation manner of the embodiments of this application, the first spectral response sensitivity of the first image sensor used as a reference and the second spectral response sensitivity of the target image sensor are obtained, and a color space conversion matrix is determined according to the first spectral response sensitivity and the second spectral response sensitivity. Among them, the color space corresponding to the first image sensor used as a reference is the target color space, and the first image sensor is randomly determined from the image data of multiple image sensors used as training samples during the model training of the white dot estimation model, so as to realize the unification of the data of multiple sensors in the same color space before entering the white dot estimation model, and avoid the differences existing due to different spectral response sensitivities of different sensors.
[0052] Among them, the method for determining the color space conversion matrix according to the first spectral response sensitivity and the second spectral response sensitivity is as follows:
[0053] As one implementation manner, according to the first spectral response sensitivity, the pixel values of the 24-color card of the first image sensor used as a reference are determined:
[0054]
[0055] Among them, d refers to the identification values of 24 color blocks in the 24-color card. For example, d = 1 indicates the first color block, d ∈ {1, 2,..., 24}, k ∈ {R, G, B}.
[0056] As one implementation manner, the pixel values of the 24-color card of the target image sensor are calculated by the following formula:
[0057]
[0058] Among them, d refers to the identification values of 24 color blocks in the 24-color card, d ∈ {1, 2,..., 24}, k ∈ {R, G, B}.
[0059] Map the first image to the target color space, and according to the mapping relationship, determine the color space conversion matrix as:
[0060]
[0061] wherein, R λ，k represents the spectral reflectance of a 24-color card under E light, where E is the standard light, and the spectral radiant power within the visible light is a constant value, also known as the equal-energy spectrum or equal-energy white light; represents the spectral response sensitivity of the target image sensor; represents the spectral response sensitivity of the first image sensor as a reference; L λ represents the spectral power density of the light source.
[0062] Step 202, according to the color space conversion matrix, convert the original image from the first color space to the target color space to obtain the first image.
[0063] In one implementation manner of the embodiment of the present application, as Figure 5 shown, preprocess the original image to obtain the preprocessed original image RGB, and then, perform color space mapping on the preprocessed original image RGB according to the color space conversion matrix to obtain the first image RGB in the target color space ctm = RGB·CTM;
[0064] wherein, the preprocessing method can refer to the explanation in the foregoing embodiment, with the same principle, which will not be elaborated here; RGB ctm is the first image in the target color space obtained by mapping the preprocessed original image from the first color space to the target color space.
[0065] Step 203, perform feature extraction on the first image to obtain the chromaticity feature of the first image.
[0066] In the embodiment of the present application, as Figure 3 shown, input the first image into the chromaticity feature extraction module for chromaticity feature extraction to obtain the chromaticity feature of the first image. Among them, the chromaticity feature of the first image is a feature in the first color gamut, which can be a chromaticity logarithmic histogram feature. In the implementation of the present application, the chromaticity logarithmic histogram feature is determined based on the differences between the channel values of the three channels of R, G, and B, where the channel values include at least one of the luminance value and the color value. Among them, the chromaticity histogram shows the color information of each pixel point in the image in the form of a histogram, so as to reflect the distribution and proportion of colors in the image. Among them, the three channels of R, G, and B in this step and subsequent steps are the red RED color channel, the blue Blue color channel, and the green Green color channel.
[0067] In the embodiment of the present application, taking the channel value as the luminance value for illustration, first, for each pixel point in the first image, determine the average luminance value of the three color channels of this pixel point wherein, I Ris the brightness value of the R channel of the pixel, I B is the brightness value of the B channel of the pixel; I G is the brightness value of the G channel of the pixel.
[0068] For the pixel, determine the difference between the brightness value of the R channel and the brightness value of the G channel and the difference between the brightness value of the R channel and the brightness value of the B channel
[0069] Determine the difference between the brightness value of the G channel and the brightness value of the R channel and determine the difference between the brightness value of the G channel and the brightness value of the B channel
[0070] Determine the difference between the brightness value of the B channel and the brightness value of the R channel and determine the difference between the brightness value of the B channel and the brightness value of the G channel
[0071] Furthermore, according to the average brightness values of the three color channels of the pixel and the brightness differences between the three color channels, determine the chromaticity logarithmic histogram feature corresponding to the pixel
[0072] where c refers to the identifiers of the three color channels in the first color gamut, c ∈ {1, 2, 3}, where ε is a small positive constant used to ensure the effectiveness of logarithmic calculation; σ is also an empirical constant; o and p represent the range of logarithmic space coordinates.
[0073] Normalize the chromaticity logarithmic histogram feature of the pixel to obtain the normalized chromaticity logarithmic histogram feature FT of the pixel op = norm(H I ), Similarly, the chromaticity logarithmic histogram features of all pixels in the first image can be determined, thereby obtaining the chromaticity feature of the first image.
[0074] Step 204, input the chromaticity feature into the first color gamut sub-network in the first white point estimation network for white point estimation, and output the first feature vector and the first white point estimation result.
[0075] As Figure 5 shown, where the first feature vector is an intermediate hidden feature extracted by the first color gamut sub-network based on the chromaticity feature. As an implementation, the first color gamut sub-network uses grouped convolution and point convolution to reduce the model computing power, and uses a feature pyramid structure to construct an excellent feature extractor, so as to improve that the first feature vector extracted carries feature information related to white point estimation.
[0076] Among them, the first color gamut sub-network determines the first white point estimation result corresponding to the first image according to the chromaticity characteristics, and determines the corresponding first white balance parameter according to the first white point estimation result. The first white balance parameter includes a first white balance gain value, and the first white balance gain value is used to perform white balance adjustment on the original image corresponding to the first image, that is, adjust the color temperature of the original image for white balance correction.
[0077] In the embodiment of the present application, the white point estimation network is trained according to the image data collected by multiple sensors. Specifically, a reference first image sensor is determined from multiple sensors, and each sensor to be recognized is used as a target sensor to establish a correspondence between the first color space of the target sensor and the target color space of the first image sensor. For each target sensor, the original image of the target sensor is mapped to the target color space to achieve processing in the same target color space. Among them, each original image is labeled with the true white point estimation result. Furthermore, based on the differences between the first white point estimation result, the second white point estimation result, and the third white point estimation result output by the model and the labeled true white point estimation result, the model parameters are adjusted until the difference is minimized, or the number of times of model training reaches the set number of times, then the white point estimation model training is completed.
[0078] Step 205: Input the first image into the second color gamut sub-network in the first white point estimation network for white point estimation, and output a second feature vector and a second white point estimation result.
[0079] As an implementation manner, the second color gamut sub-network uses grouped convolution and point convolution to reduce the model computing power, and uses a feature pyramid structure to construct a feature extractor, and uses an attention mechanism for the multi-level feature maps output by the feature pyramid to focus on the white areas related to white point estimation in the first image, so as to improve the accuracy of the input second feature vector.
[0080] Among them, the second color gamut sub-network determines the second white point estimation result corresponding to the first image according to the semantic features in the image carried by the first image, and determines the corresponding second white balance parameter according to the second white point estimation result. The second white point estimation result includes a second white balance gain value, and the second white balance gain value is used to perform white balance adjustment on the original image corresponding to the first image, that is, adjust the color temperature of the original image for white balance correction.
[0081] Step 206: Input the first feature vector and the second feature vector into the second white point estimation network in the white point estimation model for white point estimation to obtain a third white point estimation result.
[0082] In the embodiment of the present application, the white point estimation model further includes a second white point estimation network. The input of the second white point estimation network includes a first feature vector and a second feature vector. That is to say, the second white point estimation network simultaneously focuses on the features of two color gamuts, infers and outputs a white point estimation result based on the features of the two color gamuts, and determines a third white point estimation result according to the white point estimation result. The third white point estimation result includes a third white balance gain value.
[0083] As an example, the second white point estimation network is the deep learning network FusionNet.
[0084] Step 207: Perform white balance processing on the original image according to the first white point estimation result, the second white point estimation result, and the third white point estimation result.
[0085] In an implementation manner of the embodiment of the present application, the inverse matrix CTM of the color space conversion matrix is determined -1 , and according to the inverse matrix, the first white point estimation result, the second white point estimation result, and the third white point estimation result are respectively converted to the first color space to obtain a first target white point estimation result, a second target white point estimation result, and a third target white point estimation result. The specific calculation method is as follows:
[0086] WP multi =WP i ·CTM -1 ={rg i ,1,bg i};
[0087] where i ∈ {1, 2, 3}, i being 1 represents the first white point estimation result WP1, i being 2 represents the second white point estimation result WP2; i being 3 represents the third white point estimation result WP3. Among them, {rg i ,1,bg i} represents each target white point estimation result. Among them, when the i value is 1, it is the first target white point estimation result, when the i value is 2, it is the second target white point estimation result, and when the i value is 3, it is the third target white point estimation result. Among them, rg i indicates the color deviation value between red and green, and bg i indicates the color deviation value between blue and green. Among them, rg i and bg i are used to determine the deviation degree of the white area in the image, that is, can be used to determine the white balance gain value, that is represents the gain value for adjusting the red component; represents the gain value for adjusting the blue component, and 1 represents the gain value for the green component.
[0088] It should be noted that since the human eye is most sensitive to light in the green light wavelength range (480nm - 600nm) in the spectrum, and the number of green pixel points collected in the Bayer array is the largest, current cameras usually fix the gain value of the green component and then adjust the gain values of the red and blue components respectively to achieve the adjustment of the red and blue components.
[0089] Perform white balance processing on the original image according to the first target white point estimation result, the second target white point estimation result, and the third target white point estimation result, that is, add the first target white point estimation result, the second target white point estimation result, and the third target white point estimation result, determine the white balance parameter according to the addition result, and perform white balance adjustment on the original image according to the white balance parameter. Specifically, it can be implemented through the following formula:
[0090]
[0091] Among them, S r 、S Gr 、S Gb and S B respectively represent the Bayer data obtained by the current sensor, and R, Gr, Gb, and B are the output results of the final white balance calibration.
[0092] Among them, steps 206 and 207 can refer to the explanations in the foregoing embodiments. The principles are the same and will not be elaborated here.
[0093] The white balance processing method of the embodiment of the present application designates the first image sensor as the reference image sensor, and converts the data of the target image sensor to be processed currently into the color space corresponding to the first image sensor, which can greatly reduce the influence of the large difference in spectral effects between image sensors on the accuracy of white point estimation. While using the semantic information of the second color gamut to obtain better algorithm generalization, the first color gamut is also used to obtain more accurate white point estimation accuracy. Through the complementary advantages of RGB and the color gamut, the network feature extractor learns more robust features, achieving higher accuracy and generalization performance.
[0094] Based on the above embodiments, the embodiment of the present application provides another white balance processing method. Figure 6 It is a schematic flowchart of another white balance processing method provided by the embodiment of the present application. As Figure 6 shown, this method includes the following steps:
[0095] Step 601, obtain the first image in the target color space obtained by processing the original image.
[0096] Step 602: Input the first image into a first white point estimation network in a white point estimation model to perform white point estimation, and output a first eigenvector and a first white point estimation result of a first color gamut, and a second eigenvector and a second white point estimation result of a second color gamut.
[0097] Step 603: Input the first eigenvector and the second eigenvector into a second white point estimation network in a white point estimation model to perform white point estimation, and obtain a third white point estimation result.
[0098] Among them, steps 601 to 603 can refer to the relevant explanations in the aforementioned embodiments, and the principles are the same, so they will not be repeated here.
[0099] Step 604: Obtain a white balance adjustment strategy.
[0100] In one implementation of an embodiment of the present application, the white balance adjustment strategy can be determined based on the computing power of the electronic device, including an accuracy priority strategy and a computing power priority strategy. Electronic devices with strong computing power can adopt an accuracy priority strategy, and electronic devices with limited computing power can adopt a computing power priority strategy.
[0101] Step 605 , determining that the white balance adjustment strategy is an accuracy priority strategy, and performing white balance processing on the original image according to the first white point estimation result, the second white point estimation result, and the third white point estimation result.
[0102] In one implementation of the embodiment of the present application, if the white balance adjustment strategy is a precision priority strategy, the inverse matrix of the color space conversion matrix is determined, and according to the inverse matrix, the first white point estimation result, the second white point estimation result, and the third white point estimation result are respectively converted to the first color space to obtain the first target white point estimation result, the second target white point estimation result, and the third target white point estimation result. According to the first target white point estimation result, the second target white point estimation result, and the third target white point estimation result, the original image is white balanced, which can meet the requirements of electronic devices with strong computing power for white balance accuracy. Among them, the relevant explanations in the aforementioned embodiments are also applicable to this embodiment, and the principles are the same, which will not be repeated here.
[0103] Step 606 , in response to the white balance adjustment strategy being a computing power priority strategy, white balance processing is performed on the original image according to the first white point estimation result.
[0104] In the embodiments of the present application, if the white balance adjustment strategy is a computing power priority strategy, a first color gamut subnet with lower computing power requirements can be used for white point estimation to output a first white point estimation result. The first white balance parameter is determined according to the first white point estimation result, and the original image is subjected to white balance processing according to the first white balance parameter, which can run on electronic devices with more limited computing power and meet the requirements of electronic devices with different computing powers. Among them, the relevant explanations in the foregoing embodiments are also applicable to this embodiment and will not be repeated here.
[0105]
[0106] Among them, S r 、S Gr 、S Gb and S B respectively represent the Bayer data obtained by the current sensor, and R, Gr, Gb, and B are the output results of the final white balance calibration.
[0107] It should be noted that multiple white point estimation results can be output in the present application. By combining any two white point estimation results, the obtained white point estimation results can all be used for adjusting the white balance parameters to improve the reliability of parameter adjustment.
[0108] In the embodiments of the present application, the first white point estimation result also needs to be mapped to the first color space according to the inverse matrix of the color space conversion matrix, and the original image is subjected to white balance processing in the first color space.
[0109] In the white balance processing method of the embodiments of the present application, based on obtaining the white balance adjustment strategy, a suitable white point estimation branch can be selected according to the platform computing power and accuracy requirements, and the corresponding white balance adjustment parameters are determined to achieve greater compatibility with electronic devices with different computing powers.
[0110] Based on the above embodiments, the evaluation accuracy data of the white point estimation model can well reflect the benefits of the present application in improving the accuracy of automatic white balance AIAWB and reducing the error rate.
[0111] The present application can constrain the network to learn more robust features by using a dual color gamut and multi-group output, which is beneficial to improving the accuracy of the subnet and reducing error cases.
[0112] To implement the above embodiments, the embodiments of the present application also propose a white balance processing device.
[0113] Figure 7 It is a schematic structural diagram of a white balance processing device provided by the embodiments of the present application.
[0114] As Figure 7 shown, the device may include:
[0115] An acquisition module 71, configured to acquire a first image in a target color space obtained by processing an original image;
[0116] A first estimation module 72, configured to input the first image into a first white point estimation network in a white point estimation model to perform white point estimation, and output a first feature vector of a first color gamut, a first white point estimation result, a second feature vector of a second color gamut, and a second white point estimation result;
[0117] A second estimation module 73, configured to input the first feature vector and the second feature vector into a second white point estimation network in the white point estimation model to perform white point estimation, and obtain a third white point estimation result;
[0118] A processing module 74, configured to perform white balance processing on the original image according to the first white point estimation result, the second white point estimation result, and the third white point estimation result.
[0119] Further, in an implementation manner of the embodiment of the present application, the first estimation module 72 is specifically configured to:
[0120] Extract features of the first image to obtain a chromaticity feature of the first image;
[0121] Input the chromaticity feature into a first color gamut sub-network in the first white point estimation network to perform white point estimation, and output the first feature vector and the first white point estimation result;
[0122] Input the first image into a second color gamut sub-network in the first white point estimation network to perform white point estimation, and output the second feature and the second white point estimation result.
[0123] In an implementation manner of the embodiment of the present application, the apparatus further includes a first determination module, configured to:
[0124] Determine a color space conversion matrix; wherein, the color space conversion matrix includes a conversion relationship between a first color space corresponding to a target image sensor that acquires the original image and the target color space;
[0125] According to the color space conversion matrix, convert the original image from the first color space to the target color space.
[0126] In an implementation manner of the embodiment of the present application, the first determination module is further configured to:
[0127] Acquire a first spectral response sensitivity of a first image sensor used as a reference and a second spectral response sensitivity of the target image sensor;
[0128] Determine the color space conversion matrix according to the first spectral response sensitivity and the second spectral response sensitivity.
[0129] In an implementation manner of the embodiment of the present application, the apparatus further includes a second determination module, configured to:
[0130] Obtain a white balance adjustment strategy;
[0131] Determine that the white balance adjustment strategy is an accuracy - first strategy.
[0132] In an implementation manner of the embodiment of the present application, the processing module 74 is further configured to:
[0133] Determine the inverse matrix of the color space conversion matrix;
[0134] According to the inverse matrix, convert the first white - point estimation result, the second white - point estimation result, and the third white - point estimation result to the first color space respectively, to obtain a first target white - point estimation result, a second target white - point estimation result, and a third target white - point estimation result;
[0135] Perform white balance processing on the original image according to the first target white - point estimation result, the second target white - point estimation result, and the third target white - point estimation result.
[0136] In an implementation manner of the embodiment of the present application, the processing module 74 is further configured to:
[0137] Obtain a white balance adjustment strategy;
[0138] In response to the white balance adjustment strategy being a computing - power - first strategy, perform white balance processing on the original image according to the first white - point estimation result.
[0139] It should be noted that the foregoing explanation of the method embodiment is also applicable to the apparatus of this embodiment, and will not be elaborated here.
[0140] In the white balance processing apparatus of the embodiment of the present application, obtain a first image in a target color space obtained by processing an original image, input the first image into a first white - point estimation network in a white - point estimation model for white - point estimation, output a first feature vector in a first color gamut and a first white - point estimation result, as well as a second feature vector in a second color gamut and a second white - point estimation result, input the first feature vector and the second feature vector into a second white - point estimation network in the white - point estimation model for white - point estimation, obtain a third white - point estimation result, and perform white balance processing on the original image according to the first white - point estimation result, the second white - point estimation result, and the third white - point estimation result, so as to realize obtaining multiple white - point estimation results based on the input of multiple color gamuts, and further perform white balance processing based on multiple white - point estimation results, thereby improving the accuracy of white balance processing.
[0141] To implement the above embodiments, the present application further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method described in the foregoing method embodiments is implemented.
[0142] To implement the above embodiments, the present application further provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the method described in the foregoing method embodiments is implemented.
[0143] To implement the above embodiments, the present application further provides a computer program product, on which a computer program is stored. When the computer program is executed by a processor, the method described in the foregoing method embodiments is implemented.
[0144] Figure 8 FIG. is a block diagram of an electronic device provided by an embodiment of the present application. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0145] Refer to Figure 8 , the electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.
[0146] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.
[0147] The memory 804 is configured to store various types of data to support the operation of the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, and the like. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.
[0148] The power component 806 provides power to various components of the electronic device 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.
[0149] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can not only sense the boundaries of the touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.
[0150] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) that is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.
[0151] The I / O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which can be a keyboard, a click wheel, buttons, etc. These buttons can include, but are not limited to: a home button, a volume button, a power-on button, and a lock button.
[0152] The sensor assembly 814 includes one or more sensors for providing an assessment of the status of various aspects of the electronic device 800. For example, the sensor assembly 814 can detect the on / off state of the electronic device 800, the relative positioning of components, such as the display and keypad of the electronic device 800. The sensor assembly 814 can also detect a change in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and a change in the temperature of the electronic device 800. The sensor assembly 814 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0153] The communication component 816 is configured to facilitate communication between the electronic device 800 and other devices in a wired or wireless manner. The electronic device 800 can access a wireless network based on communication standards, such as WiFi, 4G, or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0154] In an exemplary embodiment, the electronic device 800 can be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above method.
[0155] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of the electronic device 800 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.
[0156] In the description of this specification, the descriptions with reference to terms such as "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.
[0157] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of this application, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically defined.
[0158] Any process or method description shown in a flowchart or described in other ways herein may be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a customized logical function or process, and the scope of the preferred embodiments of this application includes additional implementations, where the functions may be executed not in the order shown or discussed, including in a substantially simultaneous manner according to the functions involved or in a reverse order, which should be understood by those skilled in the art to which the embodiments of this application pertain.
[0159] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in combination with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion having one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other suitable processing as necessary, and then storing it in a computer memory.
[0160] It should be understood that various parts of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.
[0161] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the method of the above embodiments can be completed by a program instructing relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.
[0162] In addition, each functional unit in various embodiments of the present application may be integrated into a processing module, may exist physically alone for each unit, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0163] The above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disc, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.
Claims
1. A white balance processing method, characterized in that, Including: Obtaining a first image in a target color space obtained by processing an original image; Inputting the first image into a first white point estimation network in a white point estimation model for white point estimation, and outputting a first feature vector and a first white point estimation result of a first color gamut, as well as a second feature vector and a second white point estimation result of a second color gamut; Inputting the first feature vector and the second feature vector into a second white point estimation network in the white point estimation model for white point estimation to obtain a third white point estimation result; Performing white balance processing on the original image according to the first white point estimation result, the second white point estimation result, and the third white point estimation result.
2. The method according to claim 1, characterized in that, The step of inputting the first image into a first white point estimation network in a white point estimation model for white point estimation, and outputting a first feature vector and a first white point estimation result of a first color gamut, as well as a second feature vector and a second white point estimation result of a second color gamut, includes: Performing feature extraction on the first image to obtain a chromaticity feature of the first image; Inputting the chromaticity feature into a first color gamut sub-network in the first white point estimation network for white point estimation, and outputting the first feature vector and the first white point estimation result; Inputting the first image into a second color gamut sub-network in the first white point estimation network for white point estimation, and outputting the second feature and the second white point estimation result.
3. The method according to claim 1, characterized in that The method further includes: Determining a color space conversion matrix; wherein, the color space conversion matrix includes a conversion relationship between a first color space corresponding to a target image sensor that acquires the original image and the target color space; Converting the original image from the first color space to the target color space according to the color space conversion matrix.
4. The method according to claim 3, wherein The step of determining the color space conversion matrix includes: Obtaining a first spectral response sensitivity of a first image sensor used as a reference and a second spectral response sensitivity of the target image sensor; Determining the color space conversion matrix according to the first spectral response sensitivity and the second spectral response sensitivity.
5. The method according to claim 2, wherein Before performing white balance processing on the original image according to the first white point estimation result, the second white point estimation result, and the third white point estimation result, it further includes: Obtaining a white balance adjustment strategy; Determining that the white balance adjustment strategy is an accuracy-first strategy.
6. The method according to claim 5, characterized in that The step of performing white balance processing on the original image according to the first white point estimation result, the second white point estimation result, and the third white point estimation result includes: Determining an inverse matrix of the color space conversion matrix; According to the inverse matrix, respectively converting the first white point estimation result, the second white point estimation result, and the third white point estimation result to the first color space to obtain a first target white point estimation result, a second target white point estimation result, and a third target white point estimation result; Performing white balance processing on the original image according to the first target white point estimation result, the second target white point estimation result, and the third target white point estimation result.
7. The method according to claim 2, characterized in that The method further includes: Obtaining a white balance adjustment strategy; In response to the white balance adjustment strategy being the computing power priority strategy, perform white balance processing on the original image according to the first white point estimation result.
8. A white balance processing device, characterized in that, It includes: An acquisition module, configured to acquire a first image in a target color space obtained by processing an original image; A first estimation module, configured to input the first image into a first white point estimation network in a white point estimation model to perform white point estimation, and output a first feature vector and a first white point estimation result in a first color gamut, as well as a second feature vector and a second white point estimation result in a second color gamut; A second estimation module, configured to input the first feature vector and the second feature vector into a second white point estimation network in the white point estimation model to perform white point estimation, and obtain a third white point estimation result; A processing module, configured to perform white balance processing on the original image according to the first white point estimation result, the second white point estimation result, and the third white point estimation result.
9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method described in any one of claims 1-7 is implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the method described in any one of claims 1-7 is implemented.