A white balance processing method, system and electronic device

Abstract

Embodiments of the present application provide a white balance processing method, system and electronic equipment, the method comprising: inputting a to-be-corrected image into a pre-trained white balance inference model to obtain a color category probability map corresponding to the to-be-corrected image; based on the color category probability map, using a corresponding relationship between a standard color value corresponding to each predefined color category and a current frame sensor response of the to-be-corrected image to solve a white balance gain matrix; and applying the white balance gain matrix to the to-be-corrected image to complete white balance correction for the to-be-corrected image. The embodiments break through the limitation of traditional algorithms relying only on gray areas, can fully utilize rich color information in the scene for constraint, and improve the white balance correction accuracy and robustness in complex or gray area missing scenes.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a white balance processing method, system, and electronic device. Background Technology

[0002] White balance is a color correction technique in digital imaging. Its purpose is to eliminate or correct color cast problems in images caused by differences in the color temperature of light sources in the shooting scene, so that white objects appear white in the final image, thereby restoring the true colors of the scene.

[0003] Different light sources have different color temperature characteristics. For example, high color temperature sunlight appears bluish, while low color temperature tungsten lamp light appears yellowish. Because camera image sensors lack the color constancy adaptability of the human eye, raw sensor images captured under different light sources will exhibit an overall color cast corresponding to the color temperature of the light source. To compensate for this color temperature difference, white balance technology adjusts the gain ratio of the red, green, and blue channels of the image to counteract the color deviation caused by the light source's color temperature, restoring white objects to white, and using this as a benchmark to achieve color correction across the entire image.

[0004] As imaging applications become increasingly complex, shooting environments often involve mixed illumination from multiple light sources with different color temperatures, such as a mixture of warm indoor lighting and natural sunlight from outside. However, currently, there is no solution that can fully utilize known color information for precise white balance correction and perform continuous, zone-by-zone correction in multi-color-temperature mixed scenarios. Summary of the Invention

[0005] The purpose of this application is to provide a white balance processing method, system, and electronic device to achieve accurate white balance correction of images and restore the true colors of scenes in complex lighting environments.

[0006] A first aspect of this application provides a white balance processing method, the method comprising: The image to be corrected is input into a pre-trained white balance inference model to obtain a color category probability map corresponding to the image to be corrected; wherein, the white balance inference model is trained by supervised learning based on scene pairing data and is used to infer the predefined color category to which each pixel belongs from the original sensor image, the scene pairing data includes the original sensor image and a white balance corrected reference image in the same scene, and the predefined color category includes a color reference class; Based on the color category probability map, the white balance gain matrix is ​​solved by utilizing the correspondence between the standard color value corresponding to each predefined color category and the current frame sensor response of the image to be corrected. The white balance gain matrix is ​​applied to the image to be corrected to complete the white balance correction of the image.

[0007] In the above implementation process, a pixel-level color category probability map is generated through a pre-trained white balance inference model. The white balance gain matrix is ​​then solved by utilizing the relationship between the standard color values ​​of various known color categories in the scene and the current sensor response value. This overcomes the limitation of traditional algorithms that rely solely on gray areas, fully utilizing the rich color information in the scene for constraints, thus improving the accuracy and robustness of white balance correction in complex scenes or scenes lacking gray areas. Even if there are no neutral gray areas in the scene, the accurate white balance gain can still be solved by using the mapping relationship between the standard color values ​​of known colored objects (such as red signs and green plants) and the current response.

[0008] Furthermore, the white balance inference model is trained through the following steps: Obtain the scene pairing data; The reference image is segmented into color reference regions to generate a label mask; the label mask divides scene pixels into the multiple predefined color categories. Using the original sensor image or the simulated original sensor image as input, and the labeled mask as a supervision signal, a convolutional neural network is trained to obtain the white balance inference model.

[0009] In the above implementation process, the pixel-level color category label mask obtained from the segmentation of the reference image is used as a supervision signal to train the convolutional neural network. The model learns the mapping relationship from the original sensor response to the intrinsic color semantic composition of the scene, so that it can "recall" the prior knowledge of the color composition of similar scenes during the inference stage.

[0010] Further, the step of segmenting the reference image into color reference regions and generating a labeled mask includes: The red, green and blue three-channel color values ​​of each pixel in the reference image are calculated by difference with the red, green and blue three-channel color values ​​of the standard color space corresponding to each color category in the predefined color card. The color category whose difference value meets the preset requirements is selected as the initial classification result of the corresponding pixel. The initial classification results of each pixel in the reference image are clustered to obtain the annotation mask.

[0011] In the above implementation, combining pixel-wise nearest neighbor matching based on standard color values ​​with clustering post-processing can efficiently generate high-quality pixel-level color category label masks for reference images. This solves the problem of high cost and time consumption required to manually create dense pixel-level ground truth labels for supervised learning tasks, improving the efficiency and quality of training data preparation.

[0012] Further, the step of training a convolutional neural network using the original sensor image or the simulated original sensor image as input and the labeled mask as a supervision signal to obtain the white balance inference model includes: The original sensor image or the simulated original sensor image is input into the convolutional neural network, and forward propagation is used to obtain a predicted probability map of each pixel belonging to each predefined color category; Based on the predicted probability map and the labeled mask, the loss value is calculated using an asymmetric loss function; the asymmetric loss function assigns unequal loss weights to different predefined color categories, so that the convolutional neural network imposes different penalties for classification errors of different color categories during training. Based on the loss value, backpropagation is performed on the convolutional neural network to update the parameters of the convolutional neural network, thereby obtaining the white balance inference model.

[0013] In the above implementation process, color category supervised training is performed on the convolutional neural network, and an asymmetric loss function is introduced to apply differentiated penalties to the misclassification of neutral gray class and color reference class during the training process. In particular, the classification accuracy of neutral gray class pixels is enhanced to avoid deviations in the subsequent white balance gain solution due to neutral gray class being incorrectly classified into other color categories. At the same time, data augmentation is performed by combining simulated original sensor images to expand the illumination diversity of the training data, so that the trained white balance inference model has good generalization ability for images to be corrected with different color temperatures and different light source types.

[0014] Further, the original sensor images of the simulation are obtained through the following steps: The reference image is converted from a nonlinear color domain to a linear light domain to obtain a linear reference image; Unequal gain adjustments are applied to each color channel of the linear reference image to simulate the original channel response state of the image sensor under different light source conditions without white balance correction, thereby obtaining a linear image with channel imbalance. Based on the linear image, the original sensor image of the simulation is obtained.

[0015] In the above implementation process, a realistic original sensor image is synthesized from a color-corrected reference image, eliminating the need to repeatedly capture paired data under different lighting conditions, thus reducing the cost of training data acquisition. Simultaneously, the paired data generated in this way achieves naturally noise-free registration on the supervisory signal, i.e., the labeled mask, which helps improve the accuracy and generalization ability of model training.

[0016] Furthermore, the method also includes: When the image to be corrected includes multiple illumination areas of light sources with different color temperatures, the image to be corrected is divided into multiple sub-regions based on the color category probability map. The local white balance gain is calculated for each color reference category within each sub-region. Spatially smooth interpolation is performed on the local white balance gain of each sub-region to generate a pixel-by-pixel gain field; The gain field is applied to the image to be corrected to complete continuous white balance correction for the image to be corrected.

[0017] In the above implementation process, the image is divided into sub-regions corresponding to different color temperature light sources using a color category probability map, and the local gain of each region is solved independently. Then, a pixel-wise gain field is formed through spatial smooth interpolation, which effectively solves the problem of global white balance failure in mixed light source scenes. It can perform continuous, smooth, and accurate differentiated color correction on objects affected by different color temperatures in the same image, eliminate color banding between regions, and achieve high-quality white balance in multi-color temperature scenes.

[0018] Further, the step of dividing the image to be corrected into regions based on the color category probability map includes: The maximum probability category of each pixel in the color category probability map is selected to obtain the preliminary segmentation result; Pixels with confidence levels below a preset threshold in the preliminary segmentation results are filtered out to obtain a deterministic segmentation result; Based on the deterministic segmentation results, the image to be corrected is divided into regions.

[0019] In the above implementation process, by selecting the highest probability class of each pixel in the color category probability map and filtering out low-confidence pixels, a reliable deterministic segmentation result is generated, and the image to be corrected is divided into regions accordingly. This effectively avoids interference from low-confidence pixels on region boundaries, improving the accuracy and robustness of region division in multi-light source scenes. At the same time, region division based on the deterministic segmentation result makes the subsequent local white balance gain calculation of each sub-region more accurate, thereby achieving smooth and natural continuous white balance correction under multi-color temperature mixed illumination and improving the color reproduction quality in complex lighting scenes.

[0020] Furthermore, the predefined color categories also include neutral gray; the step of solving the white balance gain matrix based on the color category probability map, using the correspondence between the standard color value corresponding to each predefined color category and the sensor response of the current frame of the image to be corrected, includes: Pixels belonging to the neutral gray class in the color category probability map are selected as candidate neutral points; The white balance gain matrix is ​​solved based on the sensor response of the candidate neutral point and the standard color value corresponding to the neutral gray class, as well as the sensor response and corresponding standard color value of each color reference class.

[0021] In the above implementation process, the sensor response of the candidate neutral point and its standard color value are used together with the correspondence between each color reference class to solve the white balance gain matrix. The neutral gray area that may exist in the scene is fully utilized as an additional constraint. It is optimized in conjunction with the color reference class, which effectively suppresses the gain calculation deviation caused by insufficient number or uneven distribution of color reference classes. At the same time, the introduction of neutral gray class enhances the adaptability to non-color scenes (such as large area gray cards, cement walls, etc.) and improves the generalization performance of white balance algorithm in various scenes.

[0022] A second aspect of this application provides a white balance processing system, the system comprising: The inference module is used to input the image to be corrected into a pre-trained white balance inference model to obtain a color category probability map corresponding to the image to be corrected; wherein, the white balance inference model is trained by supervised learning based on scene pairing data, and is used to infer the predefined color category to which each pixel belongs from the original sensor image, the scene pairing data includes the original sensor image and a white balance corrected reference image in the same scene, and the predefined color category includes a color reference class; The calculation module is used to solve the white balance gain matrix based on the color category probability map and by utilizing the correspondence between the standard color value corresponding to each predefined color category and the current frame sensor response of the image to be corrected. The correction module is used to apply the white balance gain matrix to the image to be corrected, thereby completing the white balance correction for the image to be corrected.

[0023] A third aspect of this application provides an electronic device, the electronic device comprising: processor; Memory used to store processor-executable instructions; Wherein, when the processor invokes the executable instructions, it implements any of the methods described in the first aspect. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1A schematic flowchart of a white balance processing method provided in an embodiment of this application; Figure 2 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0027] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0028] Traditional automatic white balance methods, such as the gray-world method and the perfect reflection method, are generally based on specific statistical assumptions (e.g., assuming the average color of the entire scene is gray). They work reasonably well in scenes with a single light source or clear light source information. However, in some application scenarios (such as large areas of monochrome backgrounds or nighttime fill light shooting), the scene does not contain a reliable neutral gray reference area, and traditional algorithms will fail due to the lack of anchor points.

[0029] In recent years, deep learning-based white balance methods have gradually emerged. These methods train neural networks on large-scale data to learn the mapping relationship from input images to correction parameters in an end-to-end manner. However, such schemes face the problem of limited model generalization ability. For novel or rare lighting scenes not covered in the training data, their correction effect is uncontrollable, and performing complete forward inference for each scene results in high computational costs.

[0030] Even more challenging is the fact that real-world shooting scenarios often involve a mixture of multiple light sources with different color temperatures, such as warm indoor lighting combined with natural sunlight from outside. Most automatic white balance methods, whether traditional statistically based or deep learning-based, can only generate a single set of global correction parameters and apply them to the entire image. They cannot differentiate between areas affected by different light sources, resulting in inaccurate colors in some areas even after correction.

[0031] To address any of the aforementioned problems, this application proposes a data-driven automatic white balance scheme based on scene memory. By training a lightweight neural network on a reference image with known correct white balance, the neural network explicitly "remembers" the affiliation of each color region in the scene. When performing white balance calculations on newly captured frames of the same scene, the output of this neural network replaces the traditional algorithm's reliance on gray references, thereby fundamentally solving problems such as gray misjudgment and lack of gray references.

[0032] The solution includes the following steps: Step S1: Obtain scene pairing data: Two acquisitions were performed on the same scene: a reference RGB image I was obtained after preliminary white balance correction by manual or traditional algorithms. ref (e.g., JPEG / PNG after ISP processing); Acquire the original RAW sensor image I under the same lighting conditions (the same scene refers to the same physical scene captured in two separate acquisitions—i.e., the same scene content, the same subject, the same shooting position, etc. Additionally, in scenes where the subject is moving, the same physical scene should also include approximately the same shooting time to ensure consistency in the content captured in both acquisitions). raw .

[0033] Step S2: Construct training labels: For reference image I ref Perform color reference region segmentation to generate a pixel-level labeled mask M, dividing the scene pixels into N predefined color categories (including N). One color reference class (N-1 color categories can correspond to the top three rows of a common 24-color chart, totaling 3*6 types and 18 categories) and one neutral gray (the last row of the 24 color patches). With this classification result, the AWB algorithm knows how to restore the corresponding color patch to its corresponding color. In other words, it can make reasonable use of non-gray information. Traditional algorithms actually only find gray patches and then restore them to gray. Segmentation can be performed by semi-automatic tools (such as a semi-global matching segmentation algorithm based on the CIELAB color space, which calculates the pixel-by-pixel difference between each pixel (RGB value) of the manually corrected JPG image and the corresponding CIELAB RGB value on the color chart, takes the closest class as the classification result, and clusters the relevant classification results. Of course, if there are errors in the classification results, they can be corrected by human interaction.

[0034] It should be noted that the color distance calculation between a pixel and the standard color chart can be performed either in the RGB three-channel color space or in the CIELAB color space using only the A and B chromaticity channels (ignoring the L luminance channel). Since luminance information contributes little to color category discrimination and luminance varies significantly under different lighting conditions, in practical applications, the AB components of the LAB space (A component represents the red-green axis, and B component represents the yellow-blue axis) can be used to calculate the Euclidean distance to obtain more stable chromaticity classification results. The three-channel difference calculation in the RGB space is also applicable to this solution; both methods can be flexibly chosen based on scenario requirements or computational resources.

[0035] Step S3: Neural Network Training (Scene Memory Writing): RAW sensor image I raw Or its synthetic simulation version (simulation version refers to I) ref The raw image is obtained by performing an inverse transform (which may include a series of inverse ISP operations). This raw image is used as network input, and the segmentation mask M obtained in step S2 is used as the supervision signal to train or fine-tune a lightweight convolutional neural network F. θ This will cause it to output a probability map of each pixel belonging to each color category: F θ (I raw →P∈RN×H×WF θ (I raw → P∈R N×H×W ; The training loss employs an asymmetric loss function designed for color segmentation tasks. This function strictly penalizes false positive errors that misclassify colored pixels as other color categories, while tolerating false negative errors that classify colored pixels as background (neutral gray), thus ensuring that the color categories predicted by the network have high reliability.

[0036] Step S4: Scene color prior inference (scene memory retrieval): For newly acquired RAW images of the same scene I raw ′, using the trained network F θ Infer the color category probability map P′ for each pixel; perform the argmax operation on P′ to obtain a deterministic segmentation result, and filter pixels with confidence scores below the threshold.

[0037] Step S5: AWB Gain Calculation: Based on the segmentation results of step S4, the white balance gain matrix G=[gR,gG,gB] is solved by using the correspondence between the known standard color values ​​of each color category (such as the standard LAB values ​​of the Macbeth Color Checker color chart) and the current frame sensor response, through the least squares method or linear constraint optimization, so that each reference color block is closest to its standard color value after gain correction.

[0038] Step S6: White Balance Correction The calculated gain matrix G is applied to the entire RAW image frame to complete the white balance correction.

[0039] Furthermore, the solution also includes multi-color temperature zone white balance: For scenarios with multiple light sources of different color temperatures (such as a mixture of indoor daylight and artificial lighting), a regional white balance extension scheme is further provided: Based on the spatial distribution segmentation map obtained in step S4, the scene is divided into regions; step S5 is performed separately for the color reference category in each region to calculate the local white balance gain of each region; spatial smoothing interpolation is performed on the gain of each region to generate a pixel-by-pixel gain field, thereby realizing continuous white balance correction in multi-color temperature scenes.

[0040] This scheme outputs semantically meaningful color categories via the network, independent of the existence of neutral gray areas in the scene, fundamentally avoiding the limitations of the gray world assumption. Through a data-driven scene memory mechanism, even if the scene is entirely composed of colored areas, AWB calculation can be completed using the standard values ​​of known color categories. All color prior knowledge is explicitly encoded through supervised learning, eliminating the need to manually adjust complex heuristic parameters for different scenes. The output is a mask of the color reference area, which can be seamlessly integrated into the gain calculation module in existing ISP pipelines without replacing the entire AWB module. The regional scheme can simultaneously handle multiple light sources with different color temperatures in the scene, which is usually ineffective for related methods.

[0041] Based on this, the embodiments of this application provide a white balance processing method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating a white balance processing method provided in an embodiment of this application.

[0042] In this embodiment, the method includes: Step S10: Input the image to be corrected into the pre-trained white balance inference model to obtain the color category probability map corresponding to the image to be corrected; wherein, the white balance inference model is trained by supervised learning based on scene pairing data, and is used to infer the predefined color category to which each pixel belongs from the original sensor image, the scene pairing data includes the original sensor image and the white balance corrected reference image in the same scene, and the multiple predefined color categories include multiple color reference classes; Step S20: Based on the color category probability map, solve the white balance gain matrix by utilizing the correspondence between the standard color value corresponding to each predefined color category and the current frame sensor response of the image to be corrected; Step S30: Apply the white balance gain matrix to the image to be corrected to complete the white balance correction of the image to be corrected.

[0043] It should be noted that the image to be corrected is the raw sensor image (i.e., RAW image) that requires white balance correction. This image reflects the linear response data directly acquired by the sensor under specific lighting conditions without any subsequent processing.

[0044] The white balance inference model can be a white balance algorithm model based on physical assumptions, such as the gray world model or the perfect reflection model, or it can be a data-driven neural network learning model. This embodiment does not limit this, as long as the model can generate a color category probability map that represents the probability of each pixel belonging to the neutral gray class and each color reference class based on the input image information through feature extraction and classification inference.

[0045] The color category probability map is a distribution map used to characterize the probability (i.e., confidence level or probability value) that each pixel in the image to be corrected belongs to multiple predefined color categories.

[0046] Multiple predefined color categories are several color classifications that are pre-divided for analyzing the color composition of an image. These multiple predefined color categories include, but are not limited to: warm color categories that appear yellowish or orangeish under low color temperature light sources, cool color categories that appear bluish under high color temperature light sources, color-cast categories that appear greenish under specific artificial light sources, and neutral color categories that serve as a white balance correction benchmark.

[0047] A color reference class refers to a set of non-neutral color pixels or regions with a clear color tendency identified among the multiple predefined color categories. These pixels are used as reference information for analyzing the global color cast of an image or for assisting in inferring the color of a light source, such as the category corresponding to an orange-red object affected by a light source in a low color temperature scene.

[0048] Optionally, the plurality of predefined color categories include a plurality of color reference classes and a neutral gray class.

[0049] Neutral gray refers to a set of pixels or regions identified as having spectral neutral characteristics among the multiple predefined color categories. These pixels have equal red, green, and blue channel values ​​under ideal standard white light. The corresponding white or gray object surfaces in the image serve as reference white points for determining white balance correction gain. By performing color cast analysis and compensation calculations on the neutral gray, the entire image can achieve accurate color reproduction.

[0050] In the specific implementation, an end-to-end white balance inference model, which is a convolutional neural network, is pre-trained. The input of this model is the image to be corrected, and its output is not a traditional white balance gain value, but a color category probability map corresponding to the spatial dimension of the input image.

[0051] The color category probability map is used to characterize the probability that each pixel in the image to be corrected belongs to multiple predefined color categories. These predefined color categories are divided into two main categories: multiple color reference classes and one neutral gray class. The color reference classes are used to characterize various typical colored objects in the scene with known standard color values ​​(e.g., 18 colored blocks defined based on a standard 24-color chart), and the neutral gray class is used to characterize neutral gray objects or regions in the scene (e.g., 6 achromatic gray blocks in a 24-color chart).

[0052] Through this design, the white balance inference model not only learns the ability to find grayscale in the original sensor image, but also, using semantic segmentation, learns the ability to identify known colors in the original sensor image. This lays the foundation for high-precision white balance correction using non-grayscale color information.

[0053] It should be understood that the correspondence between the standard color values ​​of each predefined color category and the sensor response of the current frame refers to the deviation mapping relationship between the standard red, green, and blue channel color values ​​of each predefined color category (including neutral gray and various color reference categories) under ideal spectral response (i.e., the sensor response values ​​that the object of this category should have under standard white light) and the actual sensor response values ​​(i.e., the original RGB values ​​collected after being affected by unknown scene light sources) of each pixel in the current image to be corrected, which are classified into that category based on the color category probability map. This correspondence reflects how the color temperature characteristics of the current scene light source systematically shift the color representation of each category of object.

[0054] The white balance gain matrix is ​​a transformation matrix used for global color correction of the entire image to be corrected. It can be represented as a 3×3 diagonal matrix (or a 3-dimensional gain vector), where the diagonal elements correspond to the red channel gain coefficient GR, the green channel gain coefficient GG, and the blue channel gain coefficient GB, respectively. This gain matrix acts on the original RGB response value of each pixel in the image. By multiplying it by the gain coefficient corresponding to each channel, the sensor response value affected by unknown light sources is corrected to a color value approximating that under standard light sources, thereby numerically achieving the white balance goal of restoring white or gray objects in the scene to white.

[0055] Solving the white balance gain matrix refers to the process of collecting the actual sensor response values ​​of pixels belonging to a specific predefined color category (especially neutral gray) in the current frame based on the color category probability map, combining the known standard color values ​​of each predefined color category under a standard light source, constructing a set of mapping equations between the standard color values ​​and the actual response values, and calculating the optimal red, green, and blue channel gain coefficients that minimize the overall deviation between the corrected color values ​​of each reference pixel and the standard color values ​​through the least squares method, weighted optimization, or other numerical solution methods, thereby obtaining the white balance gain matrix applied to the entire image.

[0056] The essence of white balance correction is to find a set of gain matrices G=[gR,gG,gB] that corrects the current sensor response value of each pixel to near its expected standard color value. This application solves for this gain matrix by constructing an optimization problem, for example, using the least squares method or linear constraint optimization method, to minimize the overall difference between the color value presented by all classified pixels (especially those classified as color reference class and neutral gray class) and their corresponding standard color value after gain matrix correction. The gain matrix solved in this way is derived based on the comprehensive constraints of multiple known color information in the scene, which is far more accurate than statistical methods that rely solely on grayscale assumptions.

[0057] Understandably, by applying the global gain matrix obtained from the solution to every pixel of the entire frame of the image to be corrected, the final image after white balance correction can be obtained.

[0058] In this embodiment, a pixel-level color category probability map is generated using a pre-trained white balance inference model. The white balance gain matrix is ​​then calculated by leveraging the relationship between the standard color values ​​of various known color categories in the scene and the current sensor response value. This approach overcomes the limitation of traditional algorithms that rely solely on gray areas, fully utilizing the rich color information in the scene for constraint, thus improving the accuracy and robustness of white balance correction in complex scenes or scenes lacking gray areas. Even if neutral gray areas are completely absent in the scene, the accurate white balance gain can still be calculated by mapping the standard color values ​​of known colored objects (such as red signs or green plants) to the current response.

[0059] Based on any of the above embodiments, the white balance inference model is trained through the following steps: Obtain the scene pairing data; The reference image is segmented into color reference regions to generate a label mask; the label mask divides scene pixels into the multiple predefined color categories. Using the original sensor image or the simulated original sensor image as input, and the labeled mask as a supervision signal, a convolutional neural network is trained to obtain the white balance inference model. It should be noted that the white balance inference model is trained using a scene memory writing method.

[0060] Scene pairing data refers to a set of image pairs acquired using imaging equipment within the same scene. Each image pair includes a white-balanced reference image and an unbalanced original sensor image. The reference image represents the scene's true color information under standard illumination (e.g., a D65 light source) and can serve as a true reference for color correction. The original sensor image represents the scene's original sensor response information under illumination conditions to be processed (e.g., indoor light sources with unknown color temperatures, mixed light sources, etc.) and is affected by color cast. The reference image and the original sensor image in this scene pairing data have a pixel-level spatial correspondence.

[0061] Color reference region segmentation refers to performing region-level pixel classification processing on the reference image in the scene pairing data according to multiple predefined color categories (such as neutral gray, low color temperature warm, high color temperature cool, specific light source color cast, etc.). This segmentation process aims to extract pixel regions belonging to neutral gray and each color reference category from the reference image, discretizing the continuous image scene into several regions with clear color category affiliations, thereby providing pixel-level supervision information that matches the color category probability map format for subsequent model training.

[0062] A label mask is a pixel-level classification label map with the same spatial resolution as the reference image, generated after the color reference region segmentation process. Each pixel in the label mask is assigned a classification label value, which corresponds to one of the multiple predefined color categories, such as using different values ​​to identify neutral gray, warm colors, and cool colors. The label mask serves as a ground truth supervision signal for model training, defining the correct category assignment of each pixel in the color category probability map that the neural network should output when receiving the image to be corrected.

[0063] The simulated original sensor image refers to a derived image based on the original sensor image, simulating more diverse lighting conditions while maintaining the same scene content as the original sensor image. Specifically, this may include: applying color temperature shifts in different directions to the original sensor image in the color space (e.g., simulating light source variations in the range of 5000K to 7000K), simulating color cast effects under different white balance presets (e.g., fluorescent lighting, shadow environments), or adding sensor noise specific to the device. This simulated original sensor image aims to expand the diversity of training data and enhance the model's generalization ability to different lighting scenarios.

[0064] Using the labeled mask as a supervisory signal to train a convolutional neural network (CNN) to obtain the white balance inference model involves constructing a CNN with the original sensor image or a simulated original sensor image as network input and the labeled mask as the expected ground truth output. The error between the network's predicted color category probability map and the labeled mask is calculated using a pixel-by-pixel classification loss function (such as cross-entropy loss), and the weight parameters of the CNN are iteratively updated using the backpropagation algorithm. Essentially, this training process enables the CNN to learn to extract deep features related to the color cast of the scene's lighting from the input image with color bias, and to establish a mapping relationship from the pixel's RGB response to its predefined color category, ultimately obtaining a white balance inference model capable of pixel-level color category inference for the image to be corrected.

[0065] For a specific scene, two image acquisitions are required: First, acquire a reference image whose white balance has been manually or using traditional algorithms. This reference image can be an image that has undergone full ISP processing, such as JPG or PNG format, whose color reproduction is considered accurate. Second, acquire a raw sensor image (i.e., a RAW image) of the same scene under the same lighting conditions to be processed. This raw sensor image retains the original linear data from the sensor without any white balance processing. This pair of images (the reference image and the raw sensor image) constitutes a complete scene-paired training sample.

[0066] For the color-accurate reference image obtained above, color reference region segmentation is performed. The purpose is to generate a pixel-level annotation mask with the same spatial resolution as the reference image. This annotation mask divides each pixel in the scene into one of the multiple predefined color categories, that is, it labels each pixel with a semantic tag indicating "which color category it belongs to".

[0067] In some embodiments, this segmentation process can be semi-automated: First, the red, green, and blue channel color values ​​of each pixel in the reference image are compared pixel-by-pixel with the red, green, and blue channel color values ​​of the standard color space (such as RGB values ​​converted from CIELAB space) corresponding to each color category in a predefined color chart (such as a 24-color ColorChecker color chart). The color category with the smallest difference value is selected as the initial classification result for that pixel. Then, spatial clustering is performed on all the initial classification results of pixels, and manual interactive correction instructions are used to correct any misclassified regions that the algorithm may produce, ultimately obtaining an accurate labeled mask.

[0068] By constructing a color semantic mapping relationship from the calibrated "perfect image" to the original sensor image, the system not only knows "what the original sensor image should be calibrated to" (reference image), but more importantly, it knows the true color identity of each pixel in the original sensor image (label mask).

[0069] In some embodiments, this segmentation process can be fully automated based on a semantic color segmentation model: First, the reference image is converted from the original RGB color space to a color space with perceptual uniformity such as CIELAB or HSV, and pixel-level color representation vectors are extracted. Then, these color representation vectors are input into a pre-trained lightweight segmentation network (such as U-Net), which performs multi-scale color feature extraction and classification inference for each pixel, outputting the probability value of the pixel belonging to one of the multiple predefined color categories. Finally, a pixel-by-pixel maximum probability category selection operation is performed on the probability values, and spatial consistency smoothing optimization is performed using conditional random field post-processing to obtain the final labeled mask. Specifically, the maximum probability category selection operation involves examining the corresponding N probability values ​​(N being the total number of predefined color categories) for each pixel in the image and directly selecting the category with the highest probability value as the initial category to which the pixel belongs. After this pixel-by-pixel selection operation, a preliminary segmentation result image is generated. Next, to ensure that the color reference points relied upon for subsequent gain calculations are highly reliable, the preliminary segmentation result is filtered. Specifically, a confidence threshold is preset. Each pixel in the initial segmentation result is iterated through, and the probability value at which it is classified into the highest probability category is checked against the preset threshold. If a pixel's highest probability value is lower than the preset threshold, it means the model is uncertain about its classification result, and the pixel needs to be filtered out (e.g., marked as an invalid pixel and excluded from subsequent gain calculations). Finally, pixels with both clear classification results and high confidence are retained. It should be understood that using a pixel filtering mechanism based on a confidence threshold effectively eliminates low-confidence, ambiguous classification noise generated by model inference, ensuring that each pixel participating in the white balance gain matrix solution is a highly reliable color reference point, thereby enhancing the accuracy of the final correction result.

[0070] After preparing the training pair consisting of the original sensor image and the labeled mask, a convolutional neural network is trained. The original sensor image is directly used as the network input. After forward propagation, the network outputs a predicted probability map. Each pixel in the map has a set of probability values, representing the likelihood that the pixel belongs to each predefined color category. This predicted probability map is compared with the labeled mask, the error between the two is calculated, and the network parameters are continuously updated through the backpropagation algorithm. Ultimately, the network is able to infer the color category of each pixel from an original sensor image. After training, this network is the white balance inference model described above.

[0071] Understandably, considering that in typical scenarios, ordinary white balance algorithms can generally output fairly accurate AWB gain values ​​because there are always enough gray patches in a typical scene for the white balance algorithm to make correct calculations, this embodiment primarily focuses on collecting problematic scenes: such as large areas of greenery, large areas of sky, large areas of beach, and other scenes with very few gray patches. After training the white balance inference model using such scene images, the model can learn which high-level objects correspond to which low-level color patch categories. AWB can then perform correct color correction based on this network output.

[0072] In this embodiment, a pixel-level color category label mask obtained from the segmentation of the reference image is used as a supervision signal to train the convolutional neural network. The model learns the mapping relationship from the original sensor response to the inherent color semantic composition of the scene, enabling it to "recall" the prior knowledge of the color composition of similar scenes during the inference stage.

[0073] Based on any of the above embodiments, the step of segmenting the reference image into color reference regions and generating a labeled mask includes: The red, green and blue three-channel color values ​​of each pixel in the reference image are calculated by difference with the red, green and blue three-channel color values ​​of the standard color space corresponding to each color category in the predefined color card. The color category whose difference value meets the preset requirements is selected as the initial classification result of the corresponding pixel. The initial classification results of each pixel in the reference image are clustered to obtain the annotation mask.

[0074] It should be noted that the red, green, and blue three-channel color values ​​refer to the three color components R (red), G (green), and B (blue) in the RGB color space. Each color category in the predefined color chart corresponds to a set of known RGB standard values ​​under standard light.

[0075] The preset requirement refers to calculating the Euclidean distance or other distance measure (such as Manhattan distance or Mahalanobis distance) between the RGB value of the current pixel in the reference image and the RGB standard values ​​of each color category in the predefined color chart, and selecting the color category with the smallest difference measure value as the initial classification label of the pixel (i.e., the nearest neighbor classification rule); or, the preset requirement can also refer to the difference value being less than or equal to a preset tolerance threshold - when the difference measure between the pixel color value and the standard value of a certain color category falls within the preset allowable deviation range, the color category is determined as the initial classification result of the pixel; if there are multiple color categories that meet the conditions, the one with the smallest difference value is selected; if the difference values ​​of all color categories do not meet the threshold conditions, the pixel is marked as unclassified, pending subsequent clustering processing or manual interactive correction.

[0076] Furthermore, for RGB color space difference calculation, each pixel in the reference image can be converted to the CIELAB color space, extracting only the A component (red-green axis) and B component (yellow-blue axis), ignoring the luminance L component. Then, the A and B component values ​​are compared with the standard A and B component values ​​of the standard LAB space corresponding to each color category in the predefined color chart (again ignoring the L component), and the initial classification result is selected according to the preset requirements. Since the luminance component is easily affected by changes in illumination and contributes little to color category discrimination, using the AB components of the LAB space for chromatic distance calculation can obtain more robust classification results in uneven lighting or shadow scenes.

[0077] Specifically, firstly, each pixel in the reference image is treated as a processing unit. For each pixel, its color value in the red, green, and blue channels is obtained. Simultaneously, the system pre-stores the standard red, green, and blue channel color values ​​for each color category in a predefined color chart within a specific standard color space (e.g., the display RGB space corresponding to the CIELAB space after gamma correction and color space transformation).

[0078] Next, a difference calculation is performed on the pixel: the color values ​​of its three channels are subtracted from the standard three-channel color values ​​of each color category in the color chart, and the total difference (e.g., Euclidean distance) is calculated. After traversing all color categories in the color chart, the color category whose difference value meets the preset requirements (e.g., the smallest difference) is selected as the initial classification result for the pixel.

[0079] Finally, after obtaining the initial classification results for all pixels in the entire reference image, these results are clustered. The clustering process aims to eliminate sporadic misclassified pixels and merge spatially connected pixel blocks belonging to the same category into complete and semantically coherent regions, ultimately generating a well-structured and accurately defined labeled mask.

[0080] In this embodiment, combining pixel-wise nearest neighbor matching based on standard color values ​​with clustering post-processing enables the efficient generation of high-quality pixel-level color category label masks for reference images. This solves the problem of the high cost and time consumption of manually creating dense pixel-level ground truth labels for supervised learning tasks, improving the efficiency and quality of training data preparation.

[0081] Based on any of the above embodiments, the step of training a convolutional neural network using the original sensor image or the simulated original sensor image as input and the labeled mask as a supervision signal to obtain the white balance inference model includes: The original sensor image or the simulated original sensor image is input into the convolutional neural network, and forward propagation is used to obtain a predicted probability map of each pixel belonging to each predefined color category; Based on the predicted probability map and the labeled mask, the loss value is calculated using an asymmetric loss function; the asymmetric loss function assigns unequal loss weights to different predefined color categories, so that the convolutional neural network imposes different penalties for classification errors of different color categories during training. Based on the loss value, backpropagation is performed on the convolutional neural network to update the parameters of the convolutional neural network, thereby obtaining the white balance inference model.

[0082] It should be noted that, in order to further optimize the training effect of the convolutional neural network and enable it to output highly reliable colored regions, this embodiment uses a special asymmetric loss function to perform supervised training.

[0083] First, forward propagation is performed: the original sensor image or its synthetic simulation version is input into the convolutional neural network to be trained. After the network calculates layer by layer, it outputs a prediction probability map. Each position in the probability map records the predicted probability of the pixel belonging to each predefined color category.

[0084] Next, the loss value is calculated: the predicted probability map is compared with the labeled mask. An asymmetric loss function is used to calculate the error between the two, i.e., the loss value. The special feature of this asymmetric loss function is that it implements a differentiated penalty strategy when backpropagating gradients and penalizing network errors: false positive errors (where colored pixels are misclassified into other color categories) are included in the loss value with a first penalty weight, while false negative errors (where colored pixels are classified into the neutral gray category) are included in the loss value with a second penalty weight, and the first penalty weight is greater than the second penalty weight. Specifically, it identifies false positive errors (where the network incorrectly classifies a pixel belonging to one color category into another). For this type of error, the function applies a large first penalty weight, strictly penalizing this confusion and forcing the network to finely distinguish between different color categories. Simultaneously, it identifies false negative errors (where the network incorrectly classifies a pixel belonging to one color category into the neutral gray category). For this type of error, the function applies a relatively small second penalty weight, showing some tolerance for this type of error. The rationale behind this design is as follows: During the gain calculation stage of white balance correction, pixels identified as colored are treated as strong constraint points with standard color values, and their accuracy is crucial, thus preventing confusion between different color classes. Pixels classified as neutral gray, on the other hand, are effective constraint points in the gain calculation (forcing them to represent neutral gray). Therefore, sacrificing some gray points (misclassifying colors as gray) has a far smaller impact on the final correction accuracy than the impact of misclassification in colored areas. This asymmetric optimization approach ensures high reliability in the classification of colored areas.

[0085] Finally, backpropagation and parameter updates are performed: based on the calculated loss value, an optimizer (such as Adam, SGD, etc.) is used to backpropagate and differentiate all parameters of the convolutional neural network, and the parameter values ​​are updated to minimize the loss. By iterating the above process repeatedly, the trained white balance inference model is finally obtained.

[0086] In this embodiment, color category-supervised training is performed on the convolutional neural network, and an asymmetric loss function is introduced. This allows the model to apply differentiated penalties for misclassification of neutral gray and color reference classes during training—especially enhancing the classification accuracy of neutral gray pixels, avoiding deviations in subsequent white balance gain calculations due to neutral gray being incorrectly classified into other color categories. At the same time, data augmentation is performed using simulated original sensor images, expanding the illumination diversity of the training data, enabling the trained white balance inference model to have good generalization ability for images to be corrected with different color temperatures and different light source types.

[0087] Based on any of the above embodiments, the original sensor image of the simulation is obtained through the following steps: The reference image is converted from a nonlinear color domain to a linear light domain to obtain a linear reference image; Unequal gain adjustments are applied to each color channel of the linear reference image to simulate the original channel response state of the image sensor under different light source conditions without white balance correction, thereby obtaining a linear image with channel imbalance. Based on the linear image, the original sensor image of the simulation is obtained.

[0088] It should be noted that during the model training phase, simulated original sensor images can be used as input to expand training data or adapt to specific scenarios. This synthetic simulation version aims to reverse engineer simulated original sensor images from a color-accurate reference image.

[0089] Since reference images are typically stored in non-linear color spaces such as sRGB, their pixel values ​​do not correspond linearly to the actual light intensity of the scene. Therefore, it is necessary to apply inverse gamma curves or other non-linearity removal processes to restore the pixel values ​​to their original values ​​that are linearly related to the physical light intensity. In other words, it is the process of converting the reference image from a non-linear color domain to a linear light domain to obtain a linear reference image.

[0090] Applying unequal gain adjustments to each color channel of the linear reference image to simulate the original channel response state of the image sensor under different light source conditions without white balance correction is essentially simulating the unequal offset caused by different color temperature light sources to the sensor's three-channel response, causing the originally color-accurate reference image to exhibit a color cast state, thereby approximating the original output of the real sensor without white balance correction.

[0091] Optionally, the linear image can be further processed by color space transformation, adding sensor noise model, etc., to obtain a more realistic simulation effect, thereby completing the linear image based on channel imbalance and obtaining the simulated original sensor image.

[0092] In the specific implementation, a degamma operation is performed on the reference RGB image to convert it to the linear optical domain. Then, a synthetic RAW Bayer format image is generated by simulating the channel imbalance (de-WB) before white balance of the sensor to obtain the simulated original sensor image, thereby expanding the diversity of training samples under different light source conditions.

[0093] In this embodiment, by simulating the unbalanced channel state of the sensor under different light sources without white balance correction, a simulated original sensor image is synthesized from a color-corrected reference image. This achieves the goal of eliminating the need to repeatedly capture paired data under different lighting conditions, reducing the cost of training data acquisition. Simultaneously, the paired data generated in this way achieves naturally noise-free registration on the supervisory signal, i.e., the labeled mask, which helps improve the accuracy and generalization ability of model training.

[0094] Based on any of the above embodiments, the method further includes: When the image to be corrected includes multiple illumination areas of light sources with different color temperatures, the image to be corrected is divided into multiple sub-regions based on the color category probability map. The local white balance gain is calculated for each color reference category within each sub-region. Spatially smooth interpolation is performed on the local white balance gain of each sub-region to generate a pixel-by-pixel gain field; The gain field is applied to the image to be corrected to complete continuous white balance correction for the image to be corrected.

[0095] It should be noted that, in response to the problem of poor correction performance of related technologies in multi-color-temperature mixed light source scenes, this embodiment proposes a regional white balance processing scheme. This scheme is triggered and executed when the image to be corrected is in a complex scene containing multiple light sources of different color temperatures. In addition, this scheme can also be manually triggered and executed by the user.

[0096] In the specific implementation, the trigger condition judgment is as follows: First, the system internally or through external input, it is determined whether the current image to be corrected is in a complex scene containing multiple lighting areas with different color temperature light sources.

[0097] After determining that a scene with mixed color temperatures exists, perform the following operations: Step 1: Divide the region based on the color category probability map: Based on the color category probability map, the probability distribution of each pixel belonging to different predefined color categories is analyzed, along with the spatial clustering characteristics of this probability distribution. Accordingly, the image to be corrected is spatially divided into multiple sub-regions. For example, one sub-region is mainly illuminated by high color temperature sunlight from outside a window, resulting in a color category distribution that is predominantly bluish-white; another sub-region is illuminated by low color temperature indoor lighting, resulting in a color category distribution that is predominantly yellow-orange. In this process, the color category affiliation tendency of each pixel in the probability map and its spatial continuity can be used to identify pixel clusters with similar illumination characteristics, serving as a basis for division.

[0098] Step 2: Solve the local white balance gain independently for each sub-region: Within each defined sub-region, the global gain calculation process is performed independently. Specifically, for a sub-region, the pixels belonging to each color reference category within that sub-region are determined based on the color category probability map. The actual sensor response values ​​of these pixels in the current frame are obtained, and combined with the standard color values ​​corresponding to each predefined color category, an optimal local white balance gain matrix is ​​solved separately for that sub-region using methods such as least squares optimization or linear constraint optimization.

[0099] Step 3: Spatial smoothing interpolation and gain field generation: After calculating the local white balance gain for each sub-region, directly applying the hard-switching gain of each sub-region to the image will result in abrupt color jumps at the region boundaries. To solve this color jump problem, this embodiment uses spatial smooth interpolation, using the local gain of each sub-region as control points, and performs continuous interpolation operations within the spatial range of the entire image, ultimately generating a pixel-by-pixel gain field with the exact same resolution as the image to be corrected.

[0100] Step 4: Gain Field Application and Continuous White Balance Correction: Finally, this smoothly transitioning gain field is applied to every pixel of the image to be corrected. In this way, each region of the image affected by different light sources will apply a gain that best suits its local lighting environment, while the transition regions between regions are seamlessly smoothed through interpolation gain. The final result is a corrected image with accurate colors in each region and natural color transitions throughout the image, thus achieving continuous white balance correction in multi-color temperature scenes.

[0101] In this embodiment, the image is divided into sub-regions corresponding to different color temperature light sources using a color category probability map, and the local gain of each region is solved independently. Then, a pixel-wise gain field is formed through spatial smooth interpolation, which effectively solves the problem of global white balance failure in mixed light source scenes. It can perform continuous, smooth, and accurate differentiated color correction on objects affected by different color temperatures in the same image, eliminate color banding between regions, and achieve high-quality white balance in multi-color temperature scenes.

[0102] Based on any of the above embodiments, the step of dividing the image to be corrected into regions based on the color category probability map includes: The maximum probability category of each pixel in the color category probability map is selected to obtain the preliminary segmentation result; Pixels with confidence levels below a preset threshold in the preliminary segmentation results are filtered out to obtain a deterministic segmentation result; Based on the deterministic segmentation results, the image to be corrected is divided into regions.

[0103] It should be noted that the maximum probability class is determined by taking the index of the class corresponding to the maximum probability among N classes for each pixel. For example, if a pixel has [0.1, 0.7, 0.05, ...], then the maximum probability class is class 2.

[0104] Preliminary segmentation results: The single-channel label map obtained after performing argmax on the probability map pixel by pixel, with each pixel assigned to a category.

[0105] Confidence score refers to the maximum probability value itself. A higher value (e.g., 0.95) indicates that the model is more confident in classifying the pixel; a lower value (e.g., 0.45) indicates that the judgment is ambiguous.

[0106] Preset threshold: for example, 0.6, used to filter low-confidence pixels.

[0107] Deterministic segmentation results: The remaining pixels after filtering have high-confidence category labels, forming a reliable segmentation map; low-confidence regions are left blank or labeled separately.

[0108] Based on the deterministic segmentation results, the retained high-confidence label maps are transformed into spatially connected components. For example, all pixels labeled red and connected to each other constitute a red region; another connected component labeled blue constitutes a blue region. If multiple non-adjacent connected components of the same category appear (such as two red objects in the image), they are divided into multiple sub-regions.

[0109] In this embodiment, by selecting the highest probability category for each pixel in the color category probability map and filtering out low-confidence pixels, a reliable deterministic segmentation result is generated, and the image to be corrected is divided into regions accordingly. This effectively avoids interference from low-confidence pixels on region boundaries, improving the accuracy and robustness of region division in multi-light source scenes. Simultaneously, region division based on the deterministic segmentation result makes subsequent local white balance gain calculations for each sub-region more accurate, thereby achieving smooth and natural continuous white balance correction under multi-color temperature mixed illumination and improving the color reproduction quality in complex lighting scenes.

[0110] Based on any of the above embodiments, the predefined color categories further include neutral gray; the step of solving the white balance gain matrix based on the color category probability map, using the correspondence between the standard color value corresponding to each predefined color category and the current frame sensor response of the image to be corrected, includes: Pixels belonging to the neutral gray class in the color category probability map are selected as candidate neutral points; The white balance gain matrix is ​​solved based on the sensor response of the candidate neutral point and the standard color value corresponding to the neutral gray class, as well as the sensor response and corresponding standard color value of each color reference class.

[0111] It should be noted that neutral gray is a category within predefined color categories, corresponding to gray patches in a standard color chart (such as Macbeth ColorChecker) (usually with 6 different shades of gray). The standard value is R=G=B (under standard light).

[0112] Candidate neutral points refer to the set of pixels in an image that are judged by the white balance inference model to belong to the neutral gray class. These pixels appear as neutral gray (without color cast) under ideal white balance conditions.

[0113] Sensor response: The RAW value of each pixel in the image to be corrected, reflecting the photon count of each channel before white balance correction.

[0114] For a given category (such as red), the mean sensor response (actual measured value) should satisfy a channel gain relationship with the standard color value: Gain_R × sensor_R ≈ standard_R. The white balance gain matrix is ​​obtained by solving for these gain values.

[0115] Optionally, the white balance gain matrix can be solved using the least squares method or linear constraint optimization. Specifically, based on equations provided by multiple categories, the global gain (GR, GG, GB) of the three channels can be solved using the least squares method or linear constraint optimization, such that all reference categories, after gain correction, are closest to their standard values.

[0116] In this embodiment, the sensor response of the candidate neutral point and its standard color value are used together with the correspondence between each color reference class to solve the white balance gain matrix. This fully utilizes the neutral gray areas that may exist in the scene as additional constraints, and works in synergy with the color reference classes to effectively suppress the gain calculation deviation caused by insufficient number or uneven distribution of color reference classes. At the same time, the introduction of neutral gray classes enhances the adaptability to non-color scenes (such as large-area gray cards, cement walls, etc.) and improves the generalization performance of the white balance algorithm in various scenes.

[0117] Furthermore, embodiments of this application also provide a white balance processing system, the system comprising: The inference module is used to input the image to be corrected into a pre-trained white balance inference model to obtain a color category probability map corresponding to the image to be corrected; wherein, the white balance inference model is trained by supervised learning based on scene pairing data, and is used to infer the predefined color category to which each pixel belongs from the original sensor image, the scene pairing data includes the original sensor image and a white balance corrected reference image in the same scene, and the predefined color category includes a color reference class; The calculation module is used to solve the white balance gain matrix based on the color category probability map and by utilizing the correspondence between the standard color value corresponding to each predefined color category and the current frame sensor response of the image to be corrected. The correction module is used to apply the white balance gain matrix to the image to be corrected, thereby completing the white balance correction for the image to be corrected.

[0118] In some embodiments, the white balance inference model is trained by the following steps: acquiring the scene pairing data; performing color reference region segmentation on the reference image to generate a label mask; the label mask dividing scene pixels into the plurality of predefined color categories; using the original sensor image or the simulated original sensor image as input and the label mask as a supervision signal to train a convolutional neural network to obtain the white balance inference model.

[0119] In some embodiments, the inference module is further configured to perform difference calculations on the red, green and blue three-channel color values ​​of each pixel in the reference image and the red, green and blue three-channel color values ​​of the standard color space corresponding to each color category in the predefined color chart, and select the color category whose difference value meets the preset requirements as the initial classification result of the corresponding pixel; and perform clustering processing on the initial classification results of each pixel in the reference image to obtain the annotation mask.

[0120] In some embodiments, the inference module is further configured to input the original sensor image or the simulated original sensor image into the convolutional neural network, and perform forward propagation to obtain a predicted probability map of each pixel belonging to each predefined color category; calculate a loss value using an asymmetric loss function based on the predicted probability map and the labeled mask; the asymmetric loss function assigns unequal loss weights to different predefined color categories so that the convolutional neural network imposes different penalties on classification errors of different color categories during training; and perform backpropagation on the convolutional neural network based on the loss value to update the parameters of the convolutional neural network and obtain the white balance inference model.

[0121] In some embodiments, the simulated original sensor image is obtained by the following steps: converting the reference image from a nonlinear color domain to a linear light domain to obtain a linear reference image; applying unequal gain adjustments to each color channel of the linear reference image to simulate the original channel response state of the image sensor under different light source conditions without white balance correction, to obtain a channel-imbalanced linear image; and obtaining the simulated original sensor image based on the linear image.

[0122] In some embodiments, the correction module is further configured to, when the image to be corrected includes illumination areas of multiple light sources with different color temperatures, divide the image to be corrected into multiple sub-regions based on the color category probability map; solve the local white balance gain for each color reference category in each sub-region; perform spatial smoothing interpolation on the local white balance gain of each sub-region to generate a pixel-wise gain field; and apply the gain field to the image to be corrected to complete continuous white balance correction for the image to be corrected.

[0123] In some embodiments, the correction module is further configured to select the maximum probability category of each pixel in the color category probability map to obtain a preliminary segmentation result; filter pixels with confidence scores lower than a preset threshold in the preliminary segmentation result to obtain a deterministic segmentation result; and divide the image to be corrected into regions based on the deterministic segmentation result.

[0124] In some embodiments, the predefined color category further includes a neutral gray category; the calculation module is further configured to use pixels belonging to the neutral gray category in the color category probability map as candidate neutral points; and to solve the white balance gain matrix based on the sensor response of the candidate neutral points and the standard color value corresponding to the neutral gray category, as well as the sensor response of each color reference class and the corresponding standard color value.

[0125] Based on the methods described in any of the above embodiments, this application also provides, as follows: Figure 2 The diagram shows the structure of an electronic device. Figure 2 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the methods described in any of the above embodiments.

[0126] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0127] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0128] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0129] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0130] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0131] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A white balance processing method, characterized in that, The method includes: The image to be corrected is input into a pre-trained white balance inference model to obtain a color category probability map corresponding to the image to be corrected; wherein, the white balance inference model is trained by supervised learning based on scene pairing data and is used to infer the predefined color category to which each pixel belongs from the original sensor image, the scene pairing data includes the original sensor image and a white balance corrected reference image in the same scene, and the predefined color category includes a color reference class; Based on the color category probability map, the white balance gain matrix is ​​solved by utilizing the correspondence between the standard color value corresponding to each predefined color category and the current frame sensor response of the image to be corrected. The white balance gain matrix is ​​applied to the image to be corrected to complete the white balance correction of the image.

2. The method according to claim 1, characterized in that, The white balance inference model is trained using the following steps: Obtain the scene pairing data; The reference image is segmented into color reference regions to generate a labeled mask; The label mask divides scene pixels into the multiple predefined color categories; Using the original sensor image or the simulated original sensor image as input, and the labeled mask as a supervision signal, a convolutional neural network is trained to obtain the white balance inference model.

3. The method according to claim 2, characterized in that, The step of segmenting the reference image into color reference regions and generating a labeled mask includes: The red, green and blue three-channel color values ​​of each pixel in the reference image are calculated by difference with the red, green and blue three-channel color values ​​of the standard color space corresponding to each color category in the predefined color card. The color category whose difference value meets the preset requirements is selected as the initial classification result of the corresponding pixel. The initial classification results of each pixel in the reference image are clustered to obtain the annotation mask.

4. The method according to claim 2, characterized in that, The process of training a convolutional neural network using the original sensor image or the simulated original sensor image as input, and the labeled mask as a supervision signal, to obtain the white balance inference model includes: The original sensor image or the simulated original sensor image is input into the convolutional neural network, and forward propagation is used to obtain a predicted probability map of each pixel belonging to each predefined color category; Based on the predicted probability map and the labeled mask, the loss value is calculated using an asymmetric loss function; the asymmetric loss function assigns unequal loss weights to different predefined color categories, so that the convolutional neural network imposes different penalties for classification errors of different color categories during training. Based on the loss value, backpropagation is performed on the convolutional neural network to update the parameters of the convolutional neural network, thereby obtaining the white balance inference model.

5. The method according to claim 2, characterized in that, The original sensor images of the simulation are obtained through the following steps: The reference image is converted from a nonlinear color domain to a linear light domain to obtain a linear reference image; Unequal gain adjustments are applied to each color channel of the linear reference image to simulate the original channel response state of the image sensor under different light source conditions without white balance correction, thereby obtaining a linear image with channel imbalance. Based on the linear image, the original sensor image of the simulation is obtained.

6. The method according to claim 1, characterized in that, The method further includes: When the image to be corrected includes multiple illumination areas of light sources with different color temperatures, the image to be corrected is divided into multiple sub-regions based on the color category probability map. The local white balance gain is calculated for each color reference category within each sub-region. Spatially smooth interpolation is performed on the local white balance gain of each sub-region to generate a pixel-by-pixel gain field; The gain field is applied to the image to be corrected to complete continuous white balance correction for the image to be corrected.

7. The method according to claim 6, characterized in that, The step of dividing the image to be corrected into regions based on the color category probability map includes: The maximum probability category of each pixel in the color category probability map is selected to obtain the preliminary segmentation result; Pixels with confidence levels below a preset threshold in the preliminary segmentation results are filtered out to obtain a deterministic segmentation result; Based on the deterministic segmentation results, the image to be corrected is divided into regions.

8. The method according to claim 1, characterized in that, The predefined color categories also include neutral gray; the step of solving the white balance gain matrix based on the color category probability map, using the correspondence between the standard color value corresponding to each predefined color category and the current frame sensor response of the image to be corrected, includes: Pixels belonging to the neutral gray class in the color category probability map are selected as candidate neutral points; The white balance gain matrix is ​​solved based on the sensor response of the candidate neutral point and the standard color value corresponding to the neutral gray class, as well as the sensor response and corresponding standard color value of each color reference class.

9. A white balance processing system, characterized in that, The system includes: The inference module is used to input the image to be corrected into a pre-trained white balance inference model to obtain a color category probability map corresponding to the image to be corrected; wherein, the white balance inference model is trained by supervised learning based on scene pairing data, and is used to infer the predefined color category to which each pixel belongs from the original sensor image, the scene pairing data includes the original sensor image and a white balance corrected reference image in the same scene, and the predefined color category includes a color reference class; The calculation module is used to solve the white balance gain matrix based on the color category probability map and by using the correspondence between the standard color value corresponding to each predefined color category and the current frame sensor response of the image to be corrected. The correction module is used to apply the white balance gain matrix to the image to be corrected, thereby completing the white balance correction for the image to be corrected.

10. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store processor-executable instructions; Wherein, when the processor invokes the executable instructions, it implements the method according to any one of claims 1-8.

Images