White balance processing method, device, equipment, storage medium and program product

Abstract

The present application discloses a white balance processing method, device, equipment, storage medium and program product, which relates to the field of image processing, including: inputting a color histogram with a mask image added into an inference model to obtain a white balance inference result and a model confidence of the inference model; if the white balance inference result output by the inference model is determined to be reliable based on the comprehensive index of the current frame image and the model confidence, white balance processing is performed on the current frame image through the white balance inference result; if the white balance inference result output by the inference model is determined to be unreliable based on the comprehensive index of the current frame image and the model confidence, the color difference between the current frame image and the previous frame image is compared to determine whether the color temperature between the color histogram of the current frame image and the previous frame image has changed, and whether to select a target white balance result from a memory pool to perform white balance processing on the current frame image according to the color temperature change result. The present application improves the accuracy and adaptability of existing white balance processing methods.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a white balance processing method, a white balance processing device, a display device, a storage medium and a computer program product. Background Art

[0002] In the image acquisition and processing process of cameras, digital cameras, smart phones and other devices, image white balance processing is a crucial technology. The main purpose of image white balance processing is to remove the color deviation of objects under different light source color temperatures and restore the color of objects, so as to improve the quality of images or videos. However, when processing complex scenes such as large areas of pure color blocks, existing white balance processing methods such as machine learning-based methods tend to restore pure colors to gray, and have low accuracy and poor adaptability, making it difficult to achieve ideal white balance effects. Summary of the invention

[0003] The main purpose of the present application is to provide a white balance processing method, a white balance processing device, a display device, a storage medium and a computer program product, aiming to solve the technical problems of low accuracy and poor adaptability of existing white balance processing methods.

[0004] To achieve the above object, the present application proposes a white balance processing method, the method comprising:

[0005] Input the color histogram with the mask image added into the inference model to obtain the white balance inference result and the model confidence of the inference model;

[0006] If the white balance inference result output by the inference model is determined to be reliable based on the comprehensive index and model confidence of the current frame image, white balance processing is performed on the current frame image using the latest white balance inference result, wherein the comprehensive index includes gray area landing point confidence, information entropy, and number of primary colors;

[0007] If the white balance inference result output by the inference model is determined to be unreliable based on the comprehensive indicators of the current frame image and the model confidence, the color difference between the current frame image and the previous frame image is compared to determine whether the color temperature between the color histograms of the current frame image and the previous frame image has changed, and based on the color temperature change result, it is determined whether to select the target white balance result from the memory pool to perform white balance processing on the current frame image.

[0008] In one embodiment, after the step of inputting the color histogram with the mask image added into the inference model to obtain the white balance inference result and the model confidence of the inference model, the method further includes:

[0009] When it is determined that the white balance inference result output by the inference model is reliable based on the comprehensive index of the current frame image and the model confidence, determining whether the number of main colors meets a preset number threshold;

[0010] If the number of main colors meets a preset threshold, a memory pool is updated, wherein the memory pool is used to store the color features of the scene where the current frame image is located and the historical white balance results of white balance processing on the color features.

[0011] In one embodiment, the step of determining whether to select a target white balance result from a memory pool to perform white balance processing on the current frame image according to the color temperature change result includes:

[0012] If the color temperature between the color histogram of the current frame image and the color histogram of the previous frame image is continuous, the white balance result of the previous frame image is used to perform white balance processing on the current frame image;

[0013] If the color temperature between the color histogram of the current frame image and the color histogram of the previous frame image is discontinuous, then searching the memory pool for a historical white balance result that matches the color feature of the current frame image as the target white balance result, and performing white balance processing on the current frame image using the target white balance result, wherein the memory pool is used to store the color feature of the scene in which the current frame image is located and the historical white balance result of performing white balance processing on the color feature;

[0014] If no historical white balance result matching the color feature of the current frame image is found in the memory pool, the white balance inference result is used as the target white balance result, and the current frame image is white balanced using the target white balance result.

[0015] In one embodiment, the step of determining whether the color temperature between the color histograms of the current frame image and the previous frame image has changed includes:

[0016] Calculate the intersection of the pixel quantity intervals of the color histogram of the current frame image and the previous frame image;

[0017] If the intersection of the pixel number intervals is less than or equal to the preset value, the color temperature between the color histograms of the current frame image and the previous frame image is discontinuous;

[0018] If the intersection of the pixel quantity intervals is greater than a preset value, the color temperature between the color histograms of the current frame image and the previous frame image is continuous.

[0019] In one embodiment, the method comprises:

[0020] According to the model confidence obtained by performing white balance strategy inference on the current frame image based on the inference model, the smoothing convergence coefficient of the inference model is adjusted to obtain a new inference model.

[0021] In one embodiment, the step of inputting the color histogram of the added mask image into the inference model includes:

[0022] Segment the initial mask image to obtain the target mask image;

[0023] Normalize the color histogram of the current frame image to obtain a uniformly distributed color histogram;

[0024] The target mask image is added to the uniformly distributed color histogram to obtain a color histogram with the mask image added for inputting into an inference model.

[0025] In addition, to achieve the above-mentioned purpose, the present application also proposes a white balance processing device, the white balance processing device comprising:

[0026] An inference module, used for inputting the color histogram with the mask image added into the inference model to obtain the white balance inference result and the model confidence of the inference model;

[0027] A processing module, configured to perform white balance processing on the current frame image according to the white balance reasoning result if it is determined based on the comprehensive index of the current frame image and the model confidence that the white balance reasoning result output by the reasoning model is reliable, wherein the comprehensive index includes the gray area landing point confidence, information entropy and the number of main colors;

[0028] The selection module is used to determine whether the color temperature between the color histograms of the current frame image and the previous frame image has changed if the white balance inference result output by the inference model is determined to be unreliable based on the comprehensive indicators of the current frame image and the model confidence. According to the color temperature change result, it is determined whether to select the target white balance result from the memory pool to perform white balance processing on the current frame image.

[0029] In addition, to achieve the above-mentioned purpose, the present application also proposes a display device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the white balance processing method described above.

[0030] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the white balance processing method described above are implemented.

[0031] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the white balance processing method described above are implemented.

[0032] Compared with the existing white balance processing methods, it is difficult to achieve the ideal effect, tends to restore pure colors to gray, and has the problems of low accuracy and poor adaptability. This application uses the color histogram input of the mask image to perform white balance inference on the inference model, and then combines the model confidence and gray area landing point confidence, information entropy, and the number of main colors to determine whether to use the white balance inference result obtained by the inference model. It can make full use of the color information of the image for analysis, and obtain the white balance inference result based on the specific features of the image, avoiding the situation where pure colors are simply misjudged as gray, thereby greatly improving the accuracy of white balance processing; in addition, this application adopts different white balance strategies according to different situations, specifically, based on the comprehensive indicators of the current frame image and the model confidence, it is determined that the white balance inference result output by the inference model is unreliable, and whether the color temperature has changed, and then according to the color temperature change result, it is determined whether to select the target white balance result for the current frame image from the memory pool for processing. This multi-strategy selection mechanism can adapt to various complex scenes and improve the adaptability of white balance processing of images. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 A schematic diagram of a flow chart of a first embodiment of a white balance processing method of the present application;

[0036] Figure 2 A schematic diagram of a flow chart of a second embodiment of a white balance processing method of the present application;

[0037] Figure 3 A brief flowchart of the white balance processing method provided in this application;

[0038] Figure 4 This is a schematic diagram of the module structure of the white balance processing device of the present application;

[0039] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the white balance processing method of this application.

[0040] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0041] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0042] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0043] The existing methods based on prior knowledge, sub-regions, and machine learning are all difficult to achieve ideal results and tend to restore pure colors to gray, which results in low accuracy and poor adaptability.

[0044] The method based on prior knowledge includes calibrating a variety of pure colors under different standard lighting conditions, such as skin color, sky, grass, desktop, light blue wall, etc. When the image is white balanced, the color falling in the relevant statistical area is judged, specifically: judging whether the pure color to be processed in the image belongs to the statistical pure color; if it does, it is processed according to the calibrated color; if it does not, the pure color is restored to gray. The method based on prior knowledge has the problem that the calibration method is complex and it is difficult to cover the pure colors in all scenes. Therefore, there are problems of inaccuracy and poor adaptability when performing white balance processing.

[0045] The sub-region based method includes using a segmentation model to segment the image according to features such as people, scenery, and buildings, and then adopting different white balance strategies for different segmented regions. Different white balance strategies are weighted according to the naturalness of the image, and the white balance strategy with the highest weight is selected for comprehensive processing of the entire image. The sub-region based method has the problems of low accuracy in sub-region segmentation and large amount of calculation for adopting different white balance strategies for each sub-region. At the same time, the segmented sub-regions do not necessarily have pure color features, and there are problems of inaccuracy and poor adaptability when performing white balance processing.

[0046] The machine learning-based method has a certain degree of robustness for pure color blocks, but the sample data features for training the neural network model are incomplete, which makes the neural network model inaccurate when performing white balance processing on images when facing large areas of pure color scenes, making it difficult to achieve ideal results and having poor adaptability.

[0047] Since the existing white balance processing methods have low accuracy and poor adaptability, the present application provides a solution: inputting the color histogram with the mask image added into the inference model to perform white balance result inference, and obtaining the white balance inference result and the model confidence of the inference model; then, if the white balance inference result output by the inference model is determined to be reliable based on the comprehensive indicators of the current frame image and the model confidence, the white balance inference result is used to perform white balance processing on the current frame image; if the white balance inference result output by the inference model is determined to be unreliable based on the comprehensive indicators of the current frame image and the model confidence, then determining whether to select the target white balance result from the memory pool to perform white balance processing on the current frame image is based on the color temperature change result between the color histogram of the current frame image and the previous frame image.

[0048] The white balance inference result obtained through the inference model can make full use of the color information of the image for analysis, and obtain the white balance inference result based on the specific features of the image, avoiding the situation where pure colors are simply misjudged as gray, thereby greatly improving the accuracy of white balance processing; in addition, different white balance strategies are adopted according to different situations. This multi-strategy selection mechanism can adapt to various complex scenes and improve the adaptability of white balance processing of images.

[0049] It should be noted that the execution subject of this embodiment may be a display device, wherein the display device may be a camera, a digital camera, a smart phone, etc., or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, processing, etc. The following takes the display device as an example to illustrate this embodiment and the following embodiments.

[0050] Based on this, the present application embodiment provides a white balance processing method, referring to Figure 1 , Figure 1 This is a flowchart diagram of the first embodiment of the white balance processing method of the present application.

[0051] In this embodiment, the white balance processing method includes steps S10 to S30:

[0052] Step S10, inputting the color histogram with the mask image added into the inference model to obtain the white balance inference result and the model confidence of the inference model;

[0053] It should be noted that the inference model is used to infer the white balance inference result based on the color histogram after preprocessing (i.e., adding the mask image), wherein the inference model is used to infer the white balance inference result of the current frame image based on the histogram. When the inference model performs white balance inference, the Softmax function is used to select the white balance result with the highest probability from multiple white balance results as the model inference result, i.e., the white balance inference result, wherein the color histogram is also called the loguv (loguv is a logarithmic histogram calculated in the uv color space, u and v represent the natural logarithm of g / r and g / b respectively) histogram. Model confidence (model_confidence) is a quantitative measure of the reliability of the results output by the inference model, indicating the degree of trust of the inference model in the output white balance inference result. The inference model is a neural network model, and the training method of the inference model is not explained here.

[0054] Optionally, before step S10, a comprehensive index is obtained based on a color histogram of the current frame image, wherein the comprehensive index includes gray area landing point confidence, information entropy, and the number of main colors;

[0055] It should be noted that how to calculate the color histogram of the current frame image is not explained here; when calculating the color histogram of the current frame image, the number of pixel colors in the image that fall within the gray area is counted to obtain the gray area landing point confidence (gray_confidence). The gray area landing point confidence provides a valid amount of data for white balance reasoning. The gray_confidence value is between 0 and 1. The higher the gray_confidence value, the more pixel colors are within the gray area.

[0056] After obtaining the color histogram of the current frame image, the complexity of the color distribution in the color histogram is measured. The quantitative standard of the complexity of the color distribution in the color histogram is called information entropy (H). Information entropy (H) is an indicator to measure the uncertainty or confusion of a data set. A high information entropy means that the color distribution in the color histogram is relatively uniform and difficult to predict, indicating that the image contains more different color information; on the contrary, a low information entropy indicates that the color distribution is more concentrated and the image color is relatively single.

[0057] After obtaining the color histogram of the current frame image, the peaks representing the dominant colors in the image can be obtained in the color histogram. By extracting the coordinates and peak values ​​corresponding to these peaks, the core features of the entire histogram can be obtained. At the same time, when the color histogram reaches the maximum value, assuming that the corresponding color is umax, vmax, the brightness histogram composed of similar umax, vmax pixels, the main peak number of the histogram represents the brightness level of the most dominant color. The total number of main colors (main_color_num) is:

[0058] main_color_num=main_chroma_num + main_y_num;

[0059] Among them, main_chroma_num is the main color number, and main_y_num is the main color peak of the brightness histogram.

[0060] In a feasible implementation manner, before step S10, steps A10 to A30 are included:

[0061] Step A10, segmenting the initial mask image to obtain a target mask image;

[0062] It should be noted that the initial mask image is obtained by inputting a uniformly distributed color histogram into the neural network model used for processing, which reflects the probability distribution of different areas of the image (high confidence areas are set to 1, and low confidence areas are set to 0); segmentation is performed to screen out the areas in the image that are most likely to affect the accuracy of the white balance algorithm, and then form a target mask image.

[0063] Optionally, the target mask map may also be generated by database distribution statistics. There are many ways to generate the target mask map, which are not specifically limited here.

[0064] Step A20, normalizing the color histogram of the current frame image to obtain a uniformly distributed color histogram;

[0065] It should be noted that a feasible method for normalizing the color histogram of the current frame image is to uniformly set all pixel values ​​of the color histogram to 1 to obtain a uniformly distributed color histogram to ensure that the data input to the model is homogenized.

[0066] Step A30, adding the target mask image to the uniformly distributed color histogram to obtain a color histogram with the added mask image for inputting into an inference model.

[0067] It should be noted that the target mask image is added to the uniformly distributed color histogram to obtain the color histogram with the added mask image for input into the inference model. This is so that the inference model can use the target mask image to filter out the highly saturated areas in the color histogram when performing white balance inference, and only retain the features in the gray area for white balance inference.

[0068] Optionally, the irrelevant features can be removed by freely adjusting the gray area features formed by the target mask image through drawing software, opencv, etc.

[0069] In this embodiment, by adding a target mask image to the uniformly distributed histogram, irrelevant features are accurately removed, and the interference of bright color blocks outside the gray area on the white balance result is effectively avoided. The accuracy and stability of the inference model when performing white balance inference is significantly improved, ensuring high-quality white balance effects even in complex scenes.

[0070] Step S20, if it is determined based on the comprehensive index of the current frame image and the model confidence that the white balance inference result output by the inference model is reliable, white balance processing is performed on the current frame image according to the white balance inference result, wherein the comprehensive index includes the gray area landing point confidence, information entropy and the number of main colors;

[0071] It should be noted that if the white balance inference result output by the inference model is determined to be reliable based on the comprehensive indicators of the current frame image and the model confidence, it is equivalent to that the comprehensive indicators and the model confidence satisfy the first threshold condition, wherein the color richness of the scene in which the current frame image is located is obtained based on the comprehensive indicators of the current frame image; the first threshold condition is a basis for judging whether the white balance inference result obtained by the inference model is reliable; when the white balance inference result of the inference model is relatively reliable and the scene is rich in color, the white balance inference result can be used to perform white balance processing on the current frame image.

[0072] Optionally, the first threshold condition may be set as model confidence>0.3 and gray area landing point confidence>0.4 and information entropy>0.4 and main color number>3.

[0073] Step S30: If it is determined based on the comprehensive indicators of the current frame image and the model confidence that the white balance inference result output by the inference model is unreliable, the color temperature change result between the color histogram of the current frame image and the previous frame image is determined by comparing the color difference between the current frame image and the previous frame image, and according to the color temperature change result, it is determined whether to select the target white balance result from the memory pool to perform white balance processing on the current frame image.

[0074] It should be noted that if the white balance inference result output by the inference model is determined to be unreliable based on the comprehensive indicators and model confidence of the current frame image, it is equivalent to that the comprehensive indicators and model confidence do not meet the first threshold condition, the white balance inference result obtained by model inference is unreliable or the color of the scene in the current frame is relatively poor. Since the purpose of white balance processing is to adjust the pure color balance of the image so that the pure color looks closer to the pure color in the real world, so that the color of the entire image looks more natural, when the white balance inference result obtained by the inference model is unreliable, the color temperature change result between the color histogram of the current frame image and the previous frame image can be obtained by comparing the color difference between the current frame image and the previous frame image, so as to judge whether the color temperature changes between frames, and then determine whether to select the corresponding white balance strategy from the memory pool to perform white balance processing on the current frame image according to the color temperature change result, so as to more effectively perform white balance processing on the image to obtain an image effect with more natural colors and improve the accuracy of white balance processing.

[0075] In a feasible implementation manner, step S30 includes steps B10 to B30:

[0076] Step B10, calculating the intersection of the pixel quantity intervals of the color histogram of the current frame image and the previous frame image;

[0077] It should be noted that to determine the color temperature change result between the color histogram of the current frame and the previous frame image, first calculate the intersection of the pixel number intervals of the color histogram of the current frame image and the previous frame image, and divide the value range of the pixel number in the histogram into several continuous intervals, which are called pixel number intervals.

[0078] Optionally, the following formula is used to calculate the intersection of the pixel number intervals of the color histogram of the current frame image and the previous frame image:

[0079] ;

[0080] Among them, INTERSECT represents the intersection of the pixel number intervals of the color histogram of the current frame image and the previous frame image; hist and pre_hist are the histograms of the current frame and the previous frame image respectively; cv::compareHist is a function in OpenCV that is used to compare two histograms and return a scalar value. It is a function in OpenCV that is used to calculate the index value representing the similarity between two histograms. This value represents the size of the intersection of the two histograms; M and N represent the length and width of the histogram of the current frame.

[0081] Step B20, if the intersection of the pixel quantity intervals is less than or equal to the preset value, the color temperature between the color histograms of the current frame image and the previous frame image is discontinuous;

[0082] Step B30: If the intersection of the pixel quantity intervals is greater than a preset value, the color temperatures between the color histograms of the current frame image and the previous frame image are continuous.

[0083] Optionally, the preset value may be set to 0.35, that is, when INTERSECT≤0.35, it indicates that the color temperature of the two frames of images has changed; when INTERSECT>0.35, it indicates that the color temperature of the two frames of images has not changed.

[0084] In this embodiment, by adding the color histogram of the mask image to the input of the inference model for white balance inference, and then combining the model confidence and the gray area landing point confidence, information entropy, and the number of main colors to decide whether to use the white balance inference result obtained by the inference model, the color information of the image can be fully utilized for analysis, and the white balance inference result is obtained by inference based on the specific features of the image, thereby avoiding the situation where pure colors are simply misjudged as gray, thereby greatly improving the accuracy of white balance processing; in addition, the present application adopts different white balance strategies according to different situations, specifically, based on the comprehensive indicators of the current frame image and the model confidence, it is determined that the white balance inference result output by the inference model is unreliable, it is determined whether the color temperature has changed, and then according to the color temperature change result, it is determined whether to select the target white balance result for the current frame image from the memory pool for processing. This multi-strategy selection mechanism can effectively respond to scene changes and adapt to various complex scenes, thereby improving the adaptability of white balance processing of images.

[0085] In some cases, using only color temperature continuity judgment has limitations and can easily lead to erroneous results. For example, in complex lighting environments, or when dealing with scenes with a mixture of multiple types of light sources, color temperature continuity judgment may not accurately reflect the actual situation. Therefore, in order to improve the accuracy and reliability of judgments, it is very necessary to use a memory pool strategy. The memory pool strategy can store previous related information, such as the main characteristics of scene colors under different lighting conditions and the corresponding correct judgment results, etc., and call these data in the memory pool to assist the current judgment, thereby avoiding errors caused by relying solely on color temperature continuity judgments.

[0086] Based on this, the second embodiment of the present application introduces a memory pool strategy to store the color features of the scene in which the current frame image is located and the historical white balance strategy for white balancing the color features. When the preset conditions are met, the memory pool is updated; when the preset conditions are not met, the corresponding white balance strategy is queried from the memory pool and applied, which can effectively utilize the stored historical white balance results and improve the accuracy and adaptability of white balance processing.

[0087] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can refer to the above introduction, and will not be repeated in the following. Figure 2 The white balance processing method further includes steps D10 to D20:

[0088] Step D10, when it is determined based on the comprehensive index of the current frame image and the model confidence that the white balance inference result output by the inference model is reliable, determining whether the number of main colors meets a preset number threshold;

[0089] It should be noted that while the model confidence of the comprehensive index meets the first threshold condition (the white balance inference result output by the inference model is reliable), determining whether the number of dominant colors in the comprehensive index meets the preset quantity threshold (i.e., the second threshold condition) is for updating the memory pool. The memory pool has good stability in white balance adjustment. The number of dominant colors in the image refers to the number of dominant colors in the image.

[0090] Optionally, the preset quantity threshold is set to 5, and the second threshold condition is that the number of main colors>5.

[0091] Step D20: if the number of main colors meets a preset threshold, the memory pool is updated, wherein the memory pool is used to store the color features of the scene in which the current frame image is located and the historical white balance results of white balance processing on the color features.

[0092] It should be noted that the memory pool is updated on the premise that the comprehensive index and the model confidence meet the first threshold condition, the number of dominant colors meets the second threshold condition, and the first threshold condition and the second threshold condition are in an "and" relationship. The memory pool is used to store the color features of the scene in which the current frame image is located and the historical white balance results of white balancing the color features, where the color features refer to the dominant colors of the scene, and the historical white balance results can be the white balance results of white balancing the dominant colors.

[0093] Updating the memory pool is to store the main color of the scene where the current frame image is located and the model inference result (that is, the white balance result of the main color) into the memory pool to update the memory pool data.

[0094] Optionally, the specific process of updating the memory pool is as follows: convert the color histogram into histogram coordinates, and then compare all the main color coordinates in the current scene with the color coordinates stored in the memory pool one by one, wherein the color matching method is to calculate the Euclidean distance between the coordinates of the two colors in the color histogram. If the Euclidean distance between the coordinates of the two colors is less than 1.5, it is considered a color match; if the matching condition is met, the corresponding color coordinates in the memory pool and the color inference result (the color white balance result) are moved to the end of the memory. If the calculated Euclidean distance is greater than or equal to 1.5, it is determined to be a mismatch. At this time, if there is no color matching the current main color in the memory pool, the new main color coordinates and the inference result (the new color white balance result) are added to the end of the memory pool memory.

[0095] Optionally, the capacity of the memory pool can be set to store 8 primary color coordinates and the white balance results of the primary colors. When the memory pool reaches its upper capacity, if new color coordinates need to be added, the elements at the head of the memory pool will be removed first, and then the color to be added will be placed at the end of the memory pool.

[0096] In this embodiment, when it is determined based on the comprehensive indicators of the current frame image that the scene in which the current frame image is located has rich color features and the inference model is determined to be reliable based on the model confidence, it is determined whether the number of primary colors meets the preset quantity threshold, so as to adjust the white balance result according to the color features of the image and improve the effect and efficiency of image processing; the memory pool is updated to store the color features of the current frame image and the historical processing results of white balance processing of the color features, which helps to quickly retrieve and apply the stored information when performing white balance processing on the color later, thereby improving the processing efficiency and accuracy of image white balance.

[0097] In a feasible implementation manner, the white balance processing method further includes steps E10 to E30:

[0098] Step E10, if the color temperatures of the color histograms of the current frame image and the previous frame image are continuous, the white balance result of the previous frame image is used to perform white balance processing on the current frame image;

[0099] It should be noted that if the color temperature between the color histograms of the current frame image and the previous frame image is continuous, it means that the color temperature of the current frame image and the previous frame image has not changed and the difference is small. When the inference result of the inference model is unreliable, the white balance result of the previous frame image can be used to perform white balance processing on the current frame image.

[0100] Step E20, if the color temperature between the color histograms of the current frame image and the previous frame image is discontinuous, searching a historical white balance result that matches the color feature of the current frame image from a memory pool as a target white balance result, and performing white balance processing on the current frame image using the target white balance result, wherein the memory pool is used to store the color feature of the scene in which the current frame image is located and the historical white balance results of performing white balance processing on the color feature;

[0101] It should be noted that, since the premise for judging whether the color temperature has changed is that the comprehensive index and the model confidence do not meet the first threshold condition, the white balance inference result output by the inference model is unreliable at this time, indicating that the color features of the scene in which the image is located are poor. If the color temperature between the color histograms of the current frame image and the previous frame image is discontinuous, it means that there is a difference between the current frame image and the previous frame image. At this time, the color inference result stored in the memory pool can be queried (that is, the Euclidean distance between the main color of the current frame image and the main color stored in the memory pool is calculated). If it is found (that is, the Euclidean distance is less than the preset value), it means that the color feature of the scene with low reliability matches the scene stored in the memory pool. At this time, the inference result (white balance result) corresponding to the color feature in the memory pool can be directly used as the target white balance result to perform white balance processing on the current frame image.

[0102] Step E30: If no historical white balance result matching the color feature of the current frame image is found in the memory pool, the white balance inference result is used as the target white balance result, and white balance processing is performed on the current frame image using the target white balance result.

[0103] It should be noted that if no historical white balance result matching the color feature of the current frame image is found in the memory pool (i.e., the Euclidean distance is greater than or equal to the preset value), it means that no matching result is found in the memory pool. In this case, it is determined that the scene of the current frame image may have a new color or the light source condition has changed. At this time, the white balance inference result of model inference will be used to perform white balance processing on the current frame image. Table 1 shows the above memory pool strategy.

[0104] Table 1

[0105]

[0106] In this embodiment, since the color temperature is continuous, the white balance strategy of the previous frame image is also applicable to the current frame image, and there is no need to re-search or determine a new white balance strategy. The consistency of the image sequence in white balance can be maintained, making the image sequence look smoother and more natural, without sudden color changes. In the case of discontinuous color temperature, using the white balance result of a color similar to that in the memory pool can use the existing empirical data to quickly and effectively perform white balance processing on the current frame image. Compared with re-determining a completely new white balance strategy, the accuracy of the white balance processing result can be guaranteed to a certain extent, so that the white balance effect of the image is more in line with the color feature requirements of the scene. When no matching historical white balance result is found in the memory pool, using the white balance inference result can ensure that the current frame image can improve the white balance effect of the image to a certain extent, making the image color more natural.

[0107] Since the inference model is very sensitive to small changes in the input data, when the scene changes continuously, this sensitivity will cause the decision results to change frequently and irregularly, resulting in a high error rate of the inference model.

[0108] In order to smoothly transition between consecutive images or video frames, avoid drastic changes in color and brightness, reduce the impact of wrong decisions on the effect, and provide a more natural and comfortable visual experience, in the third embodiment of the present application, the smooth convergence coefficient of the reasoning model is adjusted to improve the fault tolerance of the reasoning model, so that the reasoning model can maintain correct reasoning results in scenes with large areas of pure color blocks.

[0109] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description and will not be described in detail later. The white balance processing method also includes: adjusting the smoothing convergence coefficient of the inference model according to the model confidence obtained by performing white balance inference on the current frame image according to the inference model to obtain a new inference model.

[0110] It should be noted that the smoothing convergence coefficient of the inference model is adjusted according to the model confidence. The larger the smoothing convergence coefficient is, the faster the inference model converges.

[0111] When the model confidence is high (i.e., model confidence > 0.3), such as when a wide-angle lens is used to shoot scenes with many color features, a faster convergence speed is used to adjust the white balance;

[0112] When the model confidence is low (i.e., model confidence ≤ 0.3), for example, when a telephoto lens is facing a large area of ​​yellow cloth, the inference model is prone to inference errors. At this time, a smaller convergence speed needs to be set so that the white balance processing occurs over a longer period of time, such as 6 to 10 seconds. In this case, even if the white balance decision is wrong for a short time, the user will not feel the wrong white balance decision, thereby improving the user experience.

[0113] In this embodiment, different smoothing convergence coefficients are selected according to the confidence level to optimize the performance of the white balance algorithm, so that the inference model can maintain correct inference results when performing white balance strategy inference in large-area pure color fast scenes, thereby improving the accuracy of image white balance processing.

[0114] For example, to help understand the implementation process of the white balance processing method obtained by combining this embodiment with the above first embodiment, please refer to Figure 3 , Figure 3 A brief flowchart of a white balance processing method is provided, specifically:

[0115] Step 1: After inputting the color histogram of the current frame image with the mask image added into the inference model for inference, the white balance inference result of the current frame image and the model confidence of the inference model are obtained.

[0116] Step 2: Determine whether the model confidence and comprehensive indicators (gray area landing point confidence, information entropy and main color number) meet the first threshold condition (the white balance inference result output by the inference model is reliable). If the first threshold condition is met, use the white balance inference result to perform white balance processing on the current frame image.

[0117] Step 3: If the first threshold condition is not met, the color temperature continuity is judged (i.e., whether the color temperature has changed); if the color temperature between the color histogram of the current frame image and the previous frame image is continuous, the white balance result of the previous frame image is used to perform white balance processing on the current frame image.

[0118] Step 4: If the color temperature between the color histogram of the current frame image and the previous frame image is discontinuous, execute the step of querying the memory pool; if it is found in the memory pool, the historical white balance result that matches the color feature of the current frame image is used as the target white balance result, and the current frame image is white balanced through the target white balance result; if the historical white balance result that matches the color feature of the current frame image is not found in the memory pool, the white balance inference result inferred by the inference model is used as the target white balance result, and the current frame image is white balanced through the target white balance result.

[0119] This application can automatically select a suitable white balance strategy through reasonable threshold setting and judgment of threshold conditions, realize adaptive processing of different scenes, optimize the overall logic of selecting white balance strategy, and make the whole process of selecting white balance strategy smoother and more efficient.

[0120] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the white balance processing method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0121] This application also provides a white balance processing device, please refer to Figure 4 , the white balance processing device comprises:

[0122] The inference module 10 is used to input the color histogram with the mask image added into the inference model to obtain the white balance inference result and the model confidence of the inference model;

[0123] A processing module 20 is used to perform white balance processing on the current frame image according to the white balance reasoning result if it is determined that the white balance reasoning result output by the reasoning model is reliable based on the comprehensive index and model confidence of the current frame image, wherein the comprehensive index includes gray area landing point confidence, information entropy and main color number;

[0124] The selection module 30 is used to determine whether the color temperature between the color histograms of the current frame image and the previous frame image has changed if it is determined that the white balance inference result output by the inference model is unreliable based on the comprehensive indicators of the current frame image and the model confidence. According to the color temperature change result, it is determined whether to select the target white balance result from the memory pool to perform white balance processing on the current frame image.

[0125] Optionally, the inference module 10 is further configured to determine whether the number of main colors meets a preset number threshold when it is determined that the white balance inference result output by the inference model is reliable based on the comprehensive index of the current frame image and the model confidence;

[0126] If the number of main colors meets a preset threshold, a memory pool is updated, wherein the memory pool is used to store the color features of the scene where the current frame image is located and the historical white balance results of white balance processing on the color features.

[0127] Optionally, the selection module 30 is further configured to perform white balance processing on the current frame image using the white balance result of the previous frame image if the color temperatures of the color histograms of the current frame image and the previous frame image are continuous;

[0128] If the color temperature between the color histogram of the current frame image and the color histogram of the previous frame image is discontinuous, then searching the memory pool for a historical white balance result that matches the color feature of the current frame image as the target white balance result, and performing white balance processing on the current frame image using the target white balance result, wherein the memory pool is used to store the color feature of the scene in which the current frame image is located and the historical white balance result of performing white balance processing on the color feature;

[0129] If no historical white balance result matching the color feature of the current frame image is found in the memory pool, the white balance inference result is used as the target white balance result, and the current frame image is white balanced using the target white balance result.

[0130] Optionally, the selection module 30 is further used to calculate the intersection of the pixel quantity intervals of the color histogram of the current frame image and the previous frame image;

[0131] If the intersection of the pixel number intervals is less than or equal to the preset value, the color temperature between the color histograms of the current frame image and the previous frame image is discontinuous;

[0132] If the intersection of the pixel quantity intervals is greater than a preset value, the color temperature between the color histograms of the current frame image and the previous frame image is continuous.

[0133] Optionally, the reasoning module 10 segments the initial mask image to obtain a target mask image;

[0134] It is also used to normalize the color histogram of the current frame image to obtain a uniformly distributed color histogram;

[0135] The target mask image is added to the uniformly distributed color histogram to obtain a color histogram with the mask image added for inputting into an inference model.

[0136] Optionally, the white balance processing device further includes an adjustment module, which is used to adjust a smoothing convergence coefficient of the inference model according to a model confidence obtained by performing white balance inference on the current frame image based on the inference model, so as to obtain a new inference model.

[0137] The white balance processing device provided by the present application adopts the white balance processing method in the above embodiment, which can solve the technical problems of low accuracy and poor adaptability of the existing white balance processing method. Compared with the prior art, the beneficial effects of the white balance processing device provided by the present application are the same as the beneficial effects of the white balance processing method provided by the above embodiment, and other technical features in the white balance processing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0138] The present application provides a display device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the white balance processing method in the above-mentioned first embodiment.

[0139] Reference below Figure 5, which shows a schematic diagram of the structure of a display device suitable for implementing an embodiment of the present application. The display device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions: tablet computers), PMPs (Portable Media Players: portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The display device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0140] like Figure 5 As shown, the display device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the display device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the display device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a display device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0141] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0142] The display device provided by the present application adopts the white balance processing method in the above embodiment, which can solve the technical problems of low accuracy and poor adaptability of the existing white balance processing method. Compared with the prior art, the beneficial effects of the display device provided by the present application are the same as the beneficial effects of the white balance processing method provided by the above embodiment, and other technical features in the display device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0143] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0144] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0145] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the white balance processing method in the above-mentioned embodiment.

[0146] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0147] The computer-readable storage medium may be included in the display device; or may exist independently without being assembled into the display device.

[0148] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the display device, the display device: inputs the color histogram with the mask image added into the inference model to obtain the white balance inference result and the model confidence of the inference model, wherein the inference model is used to infer the white balance inference result of the current frame image according to the color histogram;

[0149] If the white balance inference result output by the inference model is determined to be reliable based on the comprehensive index and model confidence of the current frame image, white balance processing is performed on the current frame image according to the white balance inference result, wherein the comprehensive index includes gray area landing point confidence, information entropy and main color number;

[0150] If the white balance inference result output by the inference model is determined to be unreliable based on the comprehensive indicators of the current frame image and the model confidence, the color difference between the current frame image and the previous frame image is compared to determine whether the color temperature between the color histograms of the current frame image and the previous frame image has changed, and based on the color temperature change result, it is determined whether to select the target white balance result from the memory pool to perform white balance processing on the current frame image.

[0151] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0152] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0153] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0154] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned white balance processing method, and can solve the technical problems of low accuracy and poor adaptability of the existing white balance processing method. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the white balance processing method provided in the above-mentioned embodiment, and will not be described in detail here.

[0155] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned white balance processing method when executed by a processor.

[0156] The computer program product provided by the present application can solve the technical problems of low accuracy and poor adaptability of the existing white balance processing method. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the white balance processing method provided by the above embodiment, which will not be described in detail here.

[0157] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A white balance processing method, characterized in that: The white balance processing method comprises: Input the color histogram of the current frame image with the mask image added into the inference model to obtain the white balance inference result and the model confidence of the inference model; If the white balance inference result output by the inference model is determined to be reliable based on the comprehensive index and model confidence of the current frame image, white balance processing is performed on the current frame image according to the white balance inference result, wherein the comprehensive index includes the gray area landing point confidence of the current frame image, the information entropy and the number of main colors of the color histogram of the current frame image, the gray area landing point confidence is the distribution of the pixel color of the current frame image in the gray area range, and the number of main colors is the number of types of colors that play a dominant role in the current frame image; If the white balance inference result output by the inference model is determined to be unreliable based on the comprehensive indicators of the current frame image and the model confidence, the color difference between the current frame image and the previous frame image is compared to determine whether the color temperature between the color histograms of the current frame image and the previous frame image has changed, and based on the color temperature change result, it is determined whether to select the target white balance result from the memory pool to perform white balance processing on the current frame image.

2. The white balance processing method according to claim 1, wherein: After the step of inputting the color histogram of the current frame image with the mask image added thereto into the inference model to obtain the white balance inference result and the model confidence of the inference model, the method further includes: When it is determined that the white balance inference result output by the inference model is reliable based on the comprehensive index of the current frame image and the model confidence, determining whether the number of main colors meets a preset number threshold; If the number of main colors meets a preset threshold, a memory pool is updated, wherein the memory pool is used to store the color features of the scene where the current frame image is located and the historical white balance results of white balance processing on the color features.

3. The white balance processing method according to claim 1, wherein: The step of determining whether to select a target white balance result from a memory pool to perform white balance processing on the current frame image according to the color temperature change result comprises: If the color temperature between the color histogram of the current frame image and the color histogram of the previous frame image is continuous, the white balance result of the previous frame image is used to perform white balance processing on the current frame image; If the color temperature between the color histogram of the current frame image and the color histogram of the previous frame image is discontinuous, then searching the memory pool for a historical white balance result that matches the color feature of the current frame image as the target white balance result, and performing white balance processing on the current frame image using the target white balance result, wherein the memory pool is used to store the color feature of the scene in which the current frame image is located and the historical white balance result of performing white balance processing on the color feature; If no historical white balance result matching the color feature of the current frame image is found in the memory pool, the white balance inference result is used as the target white balance result, and the current frame image is white balanced using the target white balance result.

4. The white balance processing method according to claim 1, wherein: The step of determining whether the color temperature between the color histograms of the current frame image and the previous frame image has changed comprises: Calculate the intersection of the pixel quantity intervals of the color histogram of the current frame image and the previous frame image; If the intersection of the pixel number intervals is less than or equal to the preset value, the color temperature between the color histograms of the current frame image and the previous frame image is discontinuous; If the intersection of the pixel quantity intervals is greater than a preset value, the color temperature between the color histograms of the current frame image and the previous frame image is continuous.

5. The white balance processing method according to claim 1, wherein: The method comprises: According to the model confidence obtained by performing white balance inference on the current frame image based on the inference model, the smoothing convergence coefficient of the inference model is adjusted to obtain a new inference model.

6. The white balance processing method according to claim 1, wherein: The step of inputting the color histogram of the current frame image with the mask image added thereto into the inference model includes: Segment the initial mask image to obtain the target mask image; Normalize the color histogram of the current frame image to obtain a uniformly distributed color histogram; The target mask image is added to the uniformly distributed color histogram to obtain a color histogram of the current frame image to which the mask image is added for input into an inference model.

7. A white balance processing device, characterized in that: The white balance processing device comprises: An inference module, used for inputting a color histogram of the current frame image with the mask image added thereto into an inference model to obtain a white balance inference result and a model confidence of the inference model; A processing module, configured to perform white balance processing on the current frame image through the white balance inference result if it is determined based on the comprehensive index and model confidence of the current frame image that the white balance inference result output by the inference model is reliable, wherein the comprehensive index includes the gray area landing point confidence of the current frame image, the information entropy and the number of main colors of the color histogram of the current frame image, the gray area landing point confidence is the distribution of the pixel color of the current frame image in the gray area range, and the number of main colors is the number of types of colors that play a dominant role in the current frame image; The selection module is used to determine whether the color temperature between the color histograms of the current frame image and the previous frame image has changed if the white balance inference result output by the inference model is determined to be unreliable based on the comprehensive indicators of the current frame image and the model confidence. According to the color temperature change result, it is determined whether to select the target white balance result from the memory pool to perform white balance processing on the current frame image.

8. A display device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the white balance processing method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the white balance processing method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the white balance processing method according to any one of claims 1 to 6 are implemented.

Images