White balance adjustment method, device and storage medium based on motion judgment

Abstract

The present application discloses a white balance adjustment method, device and storage medium based on motion judgment, which includes: obtaining pixel information, speed information and mean speed information of each pixel point in the current frame image and the mean speed information of the current frame image; clustering each pixel point according to the speed information and preset clustering parameters to obtain the clustering value of each type of pixel area; traversing each pixel point and determining the target filtering weight and gain information of the current frame image according to the pixel information, clustering value, mean speed information and preset weight information; filtering the gain information according to the target filtering weight to obtain the target white balance gain. The above scheme can accurately perform white balance adjustment.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a white balance adjustment method, device and storage medium based on motion judgment. Background Art

[0002] In image and video acquisition scenarios, white balance is a very critical photography setting. Accurate white balance settings can capture images with true colors to ensure image quality.

[0003] Among the existing white balance adjustment methods, manual white balance is commonly used, for example, by using a white balance card or a white reference object to determine the correct white balance value. However, the manual white balance adjustment method has high limitations and is not suitable for all application scenarios.

[0004] In addition, there are also methods for automatically adjusting white balance in the prior art. However, the existing automatic adjustment methods may not be able to accurately adjust the white balance in some application scenarios. For example, when collecting vehicle images at night, the color temperature of different headlights varies greatly. In this application environment, most existing automatic white balance algorithms will have the problem of white balance fluctuation, which can usually only be weakened by increasing the filter and reducing the saturation, which is a temporary solution but not a fundamental solution. Summary of the invention

[0005] The present application at least provides a white balance adjustment method, apparatus, device and computer-readable storage medium based on motion judgment.

[0006] The first aspect of the present application provides a white balance adjustment method based on motion judgment, including: obtaining pixel information, speed information and mean speed information of each pixel point in a current frame image; clustering each pixel point according to the speed information and preset clustering parameters to obtain a clustering value for each type of pixel area; traversing each pixel point and determining a target filtering weight and gain information of the current frame image according to the pixel information, the clustering value, the mean speed information and preset weight information; filtering the gain information according to the target filtering weight to obtain a target white balance gain.

[0007] In one embodiment, the preset weight information includes a preset motion weight and a preset filtering weight, and the traversing each pixel point and determining the target filtering weight and gain information of the current frame image according to the pixel information, the clustering value, the mean velocity information and the preset weight information includes: determining the target filtering weight of the current frame image according to the mean velocity information and the preset filtering weight; determining the target motion weight of each pixel point according to the clustering value and the preset motion weight; determining the target pixel point in each pixel point according to the pixel information of each pixel point, and determining the Planck weight of each target pixel point according to the distance between the target pixel point and a preset Planck curve; determining the gain information according to the target motion weight, the Planck weight and the pixel information of each target pixel point.

[0008] In one embodiment, the preset filtering weight includes a first filtering weight and a second filtering weight, the first filtering weight is less than the second filtering weight, and determining the target filtering weight of the current frame image according to the mean speed information and the preset filtering weight includes: comparing the mean speed information with a first speed threshold and / or a second speed threshold to obtain a comparison result, the first speed threshold is greater than the second speed threshold; in response to the comparison result indicating that the mean speed information is greater than the first speed threshold, determining the first filtering weight as the target filtering weight; in response to the comparison result indicating that the mean speed information is less than the second speed threshold, determining the second filtering weight as the target filtering weight.

[0009] In one embodiment, the preset motion weight includes a first motion weight and a second motion weight, and the target motion weight of each pixel point is determined according to the cluster value and the preset motion weight, including: sorting each pixel area according to the cluster value and the regional speed average of each type of pixel area to obtain a sorted image, wherein the regional speed average is determined according to the speed information and the number of pixels in each type of pixel area; performing binary segmentation on the sorted image to obtain a high-speed motion area and a low-speed motion area in the sorted image; determining a high-speed motion average of the high-speed motion area according to the speed information of each pixel point in the high-speed motion area, and determining a low-speed motion average of the low-speed motion area according to the speed information of each pixel point in the low-speed motion area; determining a high-speed clustering threshold corresponding to the high-speed motion area according to the high-speed motion average and the cluster value, and determining a low-speed clustering threshold corresponding to the low-speed motion area according to the low-speed motion average and the cluster value; selecting the first motion weight and / or the second motion weight to determine the target motion weight according to the comparison result between the cluster value of each pixel point and the high-speed clustering threshold and / or the low-speed clustering threshold.

[0010] In one embodiment, the pixel information includes brightness information, chromaticity information and initial gain information, and the target pixel among the pixels is determined according to the pixel information of each pixel, and the Planck weight of each target pixel is determined according to the distance between the target pixel and a preset Planck curve, including: determining the pixel whose brightness information, chromaticity information and initial gain information among the pixels meet preset judgment conditions as the target pixel; and determining the Planck weight from a preset weight parameter according to a comparison result between the distance and a distance threshold.

[0011] In one embodiment, the pixel information includes color information, and the gain information is determined based on the target motion weight, the Planck weight, and the pixel information of each target pixel point, including: performing weighted summation processing on each color information of each target pixel point based on the target motion weight and the Planck weight to obtain each comprehensive color information; and determining the gain information of the current frame image based on each comprehensive color information.

[0012] In one embodiment, filtering the gain information according to the target filtering weight to obtain a target white balance gain includes: obtaining gain information and target weight information corresponding to at least one frame of image; and performing mean filtering on the sum of gain information of at least one frame of image according to the sum of target weight information of at least one frame of image to obtain the target white balance gain.

[0013] In one embodiment, the step of obtaining pixel information, speed information of each pixel in a current frame image and average speed information of the current frame image includes: obtaining the pixel information, and determining the optical flow speed of each pixel based on the pixel information and a preset optical flow algorithm; determining the speed information of each pixel based on the optical flow speed and resolution information of the current frame image; and determining the average speed information based on the speed information of each pixel and the number of pixels.

[0014] The second aspect of the present application provides a white balance adjustment device based on motion judgment, including: an acquisition module, used to obtain pixel information, speed information and mean speed information of each pixel point in a current frame image; a clustering module, used to cluster each pixel point according to the speed information and preset clustering parameters to obtain a clustering value for each type of pixel area; an information determination module, used to traverse each pixel point and determine the target filtering weight and gain information of the current frame image according to the pixel information, the clustering value, the mean speed information and preset weight information; a filtering processing module, used to filter the gain information according to the target filtering weight to obtain a target white balance gain.

[0015] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned white balance adjustment method based on motion judgment.

[0016] A fourth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, and when the program instructions are executed by a processor, the above-mentioned white balance adjustment method based on motion judgment is implemented.

[0017] The above scheme, by acquiring the pixel information, speed information and the mean speed information of the current frame image of each pixel point in the current frame image and the mean speed information of the current frame image, can cluster each pixel point according to the speed information and preset clustering parameters to obtain the clustering value of each type of pixel area; traverse each pixel point and determine the target filtering weight and gain information of the current frame image according to the pixel information, clustering value, mean speed information and preset weight information, thereby adaptively determining the weights corresponding to different pixel points according to the different degrees of motion of each type of pixel point in the current frame image; then filter the gain information according to the target filtering weight to obtain the target white balance gain, and complete the white balance setting according to the target white balance gain, thereby not only realizing automatic white balance adjustment, but also effectively removing the interference of moving objects on the white balance adjustment and reducing white balance jitter.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0020] Figure 1 is a flowchart of an exemplary embodiment of a white balance adjustment method based on motion judgment of the present application;

[0021] Figure 2 is an exemplary data conversion diagram in the exposure adjustment method based on motion judgment of the present application;

[0022] Figure 3 is a block diagram of a white balance adjustment device based on motion judgment according to an exemplary embodiment of the present application;

[0023] Figure 4 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0024] Figure 5 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0025] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0026] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0027] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0028] For ease of understanding, an exemplary description is now given of one of the applicable scenarios of the white balance adjustment method based on motion judgment of the present application. The technical principle of the existing automatic white balance is mainly to estimate the color temperature of the ambient light, and then calculate the gain and adjust it to correct the image color deviation. That is, the automatic white balance is that the camera (image sensor) automatically adjusts the white balance according to the current shooting environment.

[0029] However, in traffic detection application scenarios, there may be a large difference in color temperature between headlights of different vehicles. In the videos collected in this scenario, most of the existing white balance algorithms will have the problem of white balance fluctuations. The existing solutions can only weaken this problem by increasing the filter and reducing the saturation, which is a temporary solution but not a fundamental solution. Therefore, the present application proposes a white balance adjustment method based on motion judgment. In general, the present application can calculate the motion information of each pixel (or statistical block, which can include multiple pixel points) based on the previous and next two frames of image data or the previous and next multiple frames of image data, and can effectively judge the motion area. According to the different degrees of motion in the image, the weights between different pixels (statistical blocks) and the weights between the previous and next multiple frames of images can be dynamically adjusted to calculate the final white balance gain.

[0030] It should be noted that in the various embodiments described in the following examples of this application, unless the order of each execution step is clearly stated, or the order of each execution step can be determined according to the execution logic, the order of the execution steps is not limited. Some of the data acquisition steps, data calculation steps, etc. may be acquired in advance or acquired when the data is required, which is not limited here.

[0031] See also Figure 1 , Figure 1 This is a flowchart of an exemplary embodiment of the white balance adjustment method based on motion judgment of the present application. Specifically, it may include the following steps:

[0032] Step S110, obtaining pixel information, speed information of each pixel point in the current frame image and average speed information of the current frame image.

[0033] It should be noted that in the example description of the embodiments of the present application, the subsequent analysis and processing can be based on pixel points; or a certain number of pixel points can be used as statistical blocks, and the analysis and processing can be performed based on the statistical blocks. This is not limited here, and the following text mainly uses pixel points as an example for explanation.

[0034] The pixel information may be read from the image data, and the pixel information may include but is not limited to brightness information, chromaticity information, and initial gain information of each pixel, which is not limited here.

[0035] Methods for obtaining speed information may include but are not limited to optical flow method, frame difference method and background subtraction method, etc., which are not limited here. Preferably, in this application, the optical flow method can be used to obtain the speed information of each pixel in the current frame image. The optical flow method is a method that uses the change of pixels in the image sequence in the time domain and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, thereby calculating the motion information of the object between adjacent frames. For details, please refer to the existing explanation of the optical flow method, which will not be repeated here.

[0036] For example, the directional optical flow velocity flow_u of each pixel in the current frame image can be calculated according to the pyramid optical flow method. Dividing flow_u by the resolution information of the current frame image can obtain the motion percentage u (velocity information) of each pixel relative to the original image (current frame image). The mean velocity information of the current frame image can be obtained by calculating the mean of the sum of the motion percentage u of each pixel in the current frame image and the number of pixels in the current frame image. In a specific application scenario, the mean velocity information may be calculated after obtaining the u of each pixel point in the current frame image, or may be calculated when the mean velocity information is needed in a subsequent step, which is not limited here.

[0037] Step S120: clustering each pixel point according to the speed information and preset clustering parameters to obtain a clustering value for each type of pixel area.

[0038] The preset clustering parameter is a preset parameter used to characterize the number of categories that need to be obtained after clustering processing. The clustering method of the present application can refer to the existing clustering algorithm, which may include but is not limited to K-means clustering, DBSCAN clustering, etc. Exemplarily, the present application mainly uses the K-means clustering algorithm as an example for explanation, and the preset clustering parameter is the k value in the K-means clustering algorithm. During the clustering process, the data set (each pixel point) will be divided into predefined k clustering categories. For details, please refer to the principle of the K-means clustering algorithm, which will not be repeated here. Thus, the current frame image can be divided into k categories to obtain several pixel areas (image matrix mean_k), whose length and width dimensions can be consistent with u, and the value of the pixel area (clustering value) can be 0 to k-1, and different numerical values ​​represent different categories.

[0039] In conjunction with the above steps, the above steps obtain the speed information u of each pixel point, so after clustering processing, the regional mean speed mean_u of each type of pixel area can be calculated. For example, after clustering processing, the current frame image is clustered and divided into k types of image matrices mean_k. Since each type of image matrix mean_k includes at least one pixel point, the average value mean_u of u in each type of pixel area can be obtained according to the motion percentage u of the pixel points in the image matrix mean_k and the number of pixels in the image matrix, and the regional mean speed of each image matrix mean_k can be obtained.

[0040] Furthermore, after obtaining the clustering value of each type of pixel area, each type of image matrix mean_k can be sorted according to the regional mean speed mean_u of each type of image matrix mean_k, for example, the sorting process can be performed from small to large, so that the mean_u corresponding to the pixel area with a clustering value of 0 is the smallest, and the mean_u corresponding to the pixel area with a clustering value of k-1 is the largest. It should be noted that the sorting process can be performed in step S120, or in step S130, or between step S120 and step S130, which is not limited here.

[0041] Step S130, traverse each pixel point and determine the target filtering weight and gain information of the current frame image according to the pixel information, cluster value, mean velocity information and preset weight information.

[0042] The preset weight information refers to the weight information required for weighted comprehensive analysis of each pixel in the image. There are multiple preset weight information in this application, and the corresponding preset weight information can be determined by traversing each pixel and according to the relevant information of each pixel.

[0043] For example, the Planck weight can be determined according to the pixel information, the motion weight can be determined according to the cluster value, and the filter weight can be determined according to the mean velocity information, etc., which will not be described in detail here. In addition, the gain information (white balance gain) can also be determined according to the pixel information of each pixel point combined with the weight corresponding to each pixel point.

[0044] Step S140 , filtering the gain information according to the target filtering weight to obtain a target white balance gain.

[0045] In combination with the above steps, after the target filtering weight and gain information are input into the mean filter for filtering, the final target white balance gain can be obtained.

[0046] For example, the target white balance gain can be applied to the raw data to directly adjust the white balance effect of the image data, and / or it can be set in the white balance module of the ISP unit (image signal processor) to adjust the white balance effect during image acquisition.

[0047] It can be seen that the present application obtains the pixel information, speed information and mean speed information of each pixel in the current frame image, and can cluster each pixel according to the speed information and preset clustering parameters to obtain the clustering value of each type of pixel area; traverses each pixel and determines the target filtering weight and gain information of the current frame image according to the pixel information, clustering value, mean speed information and preset weight information, thereby adaptively determining the weights corresponding to different pixel points according to the different degrees of motion of each type of pixel in the current frame image; and then filters the gain information according to the target filtering weight to obtain the target white balance gain, and completes the white balance setting according to the target white balance gain, thereby not only realizing automatic white balance adjustment, but also effectively removing the interference of moving objects on the white balance adjustment and reducing white balance jitter.

[0048] Based on the above embodiment, the embodiment of the present application describes the steps of traversing each pixel point and determining the target filtering weight and gain information of the current frame image according to the pixel information, cluster value, mean velocity information and preset weight information. Among them, the preset weight information includes preset motion weight and preset filtering weight. Specifically, the method of this embodiment includes the following steps:

[0049] The target filtering weight of the current frame image is determined according to the mean velocity information and the preset filtering weight; the target motion weight of each pixel is determined according to the clustering value and the preset motion weight; the target pixel in each pixel is determined according to the pixel information of each pixel, and the Planck weight of each target pixel is determined according to the distance between the target pixel and the preset Planck curve; the gain information is determined according to the target motion weight, the Planck weight and the pixel information of each target pixel.

[0050] In conjunction with the above-mentioned embodiments, the preset weight information of the present application may include but is not limited to preset motion weight, preset filter weight and preset Planck weight, wherein the preset filter weight corresponds to the mean velocity information, and the preset motion weight corresponds to the clustering value.

[0051] Exemplarily, the preset filtering weights may include and , which are used to represent the maximum filter weight and the minimum filter weight respectively. The preset filter weight is mainly used for the subsequent mean filter processing. Therefore, according to the value of the mean velocity information, a preset filter weight can be selected as the target filter weight. .

[0052] Similarly, preset motion weights can include and , which are used to characterize the preset motion weights (also called white balance motion weights) required for high-speed motion and low-speed motion. The preset motion weights are mainly used to reduce the white balance impact of high-speed motion areas in the overall image. Therefore, a preset motion weight can be selected as the target motion weight according to the speed of different pixel areas. It should be noted that, since in the aforementioned embodiment, various pixel regions (image matrix mean_k) are reordered according to mean_u, a preset motion weight may be selected as the target motion weight according to the size of the clustering value (for example, the smaller the clustering value, the lower the speed of mean_k).

[0053] In addition, it is necessary to determine the gray point (target pixel) in each pixel through the traversal results of each pixel, and then determine the Planck weight of each target pixel from the preset Planck weight according to the point-line distance between the target pixel and the preset Planck curve. . Among them, the Planck curve can be obtained by pre-calibration, which plays a key role in the process of determining the color temperature. It is a curve drawn according to the black body radiation law, which represents the color of the light emitted by the black body at different temperatures. This curve appears as a continuous track from red to blue on the chromaticity diagram, reflecting the relationship between the black body temperature and the color of the light it emits. Then, the gain information can be determined based on the target motion weight determined in the previous steps, the Planck weight, and the pixel information of each target pixel.

[0054] It should be noted that the order of determining the weights in this embodiment may include but is not limited to the above example, for example, first obtaining the target motion weight, then obtaining the target filtering weight, and finally obtaining the Planck weight, etc., which is not limited here.

[0055] Based on the above embodiment, the embodiment of the present application describes the steps of determining the target filtering weight of the current frame image according to the mean velocity information and the preset filtering weight. The preset filtering weight includes a first filtering weight and a second filtering weight, and the first filtering weight is less than the second filtering weight. Specifically, the method of this embodiment includes the following steps:

[0056] The mean speed information is compared with the first speed threshold and / or the second speed threshold to obtain a comparison result, in which the first speed threshold is greater than the second speed threshold; in response to the comparison result indicating that the mean speed information is greater than the first speed threshold, the first filter weight is determined as the target filter weight; in response to the comparison result indicating that the mean speed information is less than the second speed threshold, the second filter weight is determined as the target filter weight.

[0057] Combined with the above-mentioned embodiment, the first filtering weight in this embodiment is equivalent to , the second filter weight is equivalent to , Greater than . A first speed threshold is also preset and the second speed threshold , used to evaluate mean velocity information The numerical size of .

[0058] For example, the mean speed information With the first speed threshold and / or a second speed threshold For comparison; if Greater than , then the first filter weight can be Determine the target filter weight ;like Less than , then the second filter weight can be Determine the target filter weight .like In [ , ], then we can combine and Determine the target filter weight . Its mathematical expression can be:

[0059]

[0060] in, Yes and The weights of the weighted sum, The mathematical expression of can be:

[0061]

[0062] Based on the above embodiment, the embodiment of the present application describes the steps of determining the target motion weight of each pixel point according to the cluster value and the preset motion weight. The preset motion weight includes a first motion weight and a second motion weight, and the first motion weight is less than the second motion weight. Specifically, the method of this embodiment includes the following steps:

[0063] According to the clustering value and the regional speed average of each type of pixel area, each pixel area is sorted to obtain a sorted image, and the regional speed average is determined according to the speed information and the number of pixel points in each type of pixel area; the sorted image is subjected to binary segmentation processing to obtain high-speed motion areas and low-speed motion areas in the sorted image; the high-speed motion average of the high-speed motion area is determined according to the speed information of each pixel point in the high-speed motion area, and the low-speed motion average of the low-speed motion area is determined according to the speed information of each pixel point in the low-speed motion area; the high-speed clustering threshold corresponding to the high-speed motion area is determined according to the high-speed motion average and the clustering value, and the low-speed clustering threshold corresponding to the low-speed motion area is determined according to the low-speed motion average and the clustering value; according to the comparison result between the clustering value of each pixel point and the high-speed clustering threshold and / or the low-speed clustering threshold, the first motion weight and / or the second motion weight is selected to determine the target motion weight.

[0064] The method for determining the target motion weight from the preset motion weight in this embodiment can still refer to the example description of the process of determining the target filtering weight in the aforementioned embodiment.

[0065] For example, the process of sorting each pixel region can refer to the description in the above embodiment, which will not be described here. After sorting, the mean_k average value in the sorted image can be determined according to the mean_k sum of each type of pixel region in the sorted image and the total number of pixel regions, which is used as the segmentation threshold of the binary segmentation process. ,Then the sorted image is subjected to binary segmentation processing to obtain the high-speed motion area and the low-speed motion area in the sorted image.

[0066] Among them, for each matrix mean_k, its corresponding value is compared with the segmentation threshold; if the corresponding point value of a matrix mean_k is greater than , then the image area where the matrix is ​​located is a high-speed motion area; if the corresponding point value of a matrix mean_k is less than or equal to , then the image area where the matrix is ​​located is a low-speed motion area.

[0067] Furthermore, after obtaining the high-speed motion area and the low-speed motion area, the high-speed motion mean value of the high-speed motion area can be determined according to the sum of the speed information u of each pixel in the high-speed motion area and the number of each pixel in the high-speed motion area. ; According to the sum of the speed information u of each pixel in the slow motion area and the number of each pixel in the slow motion area, determine the slow motion mean value of the slow motion area In addition, preset parameters can also be set and If the high-speed motion average Less than the preset parameter , then you can ; Otherwise, the high-speed motion mean Similarly, if the slow motion mean Greater than the preset parameter ,but ; Otherwise, the slow motion mean Remain unchanged.

[0068] Then, the high-speed clustering threshold corresponding to the high-speed motion area can be determined according to the high-speed motion mean and clustering value, and the low-speed clustering threshold corresponding to the low-speed motion area can be determined according to the low-speed motion mean and clustering value. Specifically, according to the above embodiment, the k value of each type of pixel area can be obtained, and then according to the average value mean_u of each type of pixel area u calculated in this embodiment, the table lookup method can be used to find the k value position corresponding to mean_u, that is, the k value corresponding to mean_u can be found. The corresponding k value is the high-speed clustering threshold , find The corresponding k value is the low-speed clustering threshold It should be noted that if the value in the table is between two k values, it can be calculated using the two-point interpolation method, which will not be described in detail here.

[0069] Therefore, each pixel in the current frame image can be traversed later, and the mean_k clustering value of each pixel is compared with the high-speed clustering threshold. and / or low velocity clustering threshold The target motion weight is determined from the preset motion weights according to the comparison result. The preset motion weights may include the first motion weight and the second motion weight , Less than , ( <1.0, the lower the value, the smaller the weight of the high-speed motion area; can be 1.0, that is, the weight representing the low-speed motion area can be 1.0).

[0070] Therefore, for each pixel, if mean_k is greater than , then Determine its target motion weight ; If mean_k is less than , then Determine its target motion weight ; If mean_k is in [ , ], then according to and Determine its target motion weight . Its mathematical expression can be:

[0071]

[0072] in,

[0073] Based on the above embodiment, the embodiment of the present application describes the steps of determining a target pixel among the pixels according to the pixel information of each pixel, and determining the Planck weight of each target pixel according to the distance between the target pixel and a preset Planck curve. The pixel information includes brightness information, chromaticity information, and initial gain information. Specifically, the method of this embodiment includes the following steps:

[0074] The pixel points whose brightness information, chromaticity information and initial gain information meet the preset judgment conditions are determined as target pixel points; and the Planck weight is determined from the preset weight parameters according to the comparison result between the distance and the distance threshold.

[0075] In conjunction with the above-mentioned embodiment, it is explained that before determining the Planck weight, it is necessary to determine the target pixel (gray point) among the pixels. This embodiment provides a triple judgment method to screen out the target pixel.

[0076] For example, for each pixel, first determine whether the brightness information Y of the current pixel is within a preset brightness threshold range. ; It is also necessary to determine whether the chromaticity information of the current pixel (blue chromaticity cb and red chromaticity cr) are both within the preset chromaticity threshold range [tcolorMax, tcolorMin]; Then it is also necessary to determine whether the initial gain information of the current pixel (red gain rgain and blue gain bgain) are both within the preset Planck curve range. Among them, each judgment process of the triple judgment can be carried out according to the above example description, or each judgment process of the above example can be recombined in other orders, which is not limited here.

[0077] If the results of the above three judgments are all yes, the pixel can be determined to be the target pixel (gray point). If at least one judgment result is no, it can be determined that the pixel is not the target pixel, and there is no need to continue to perform other subsequent judgment processes in the triple judgment on the pixel, but choose to judge other pixels.

[0078] Then, the present application may also pre-set the Planck weight and ( Less than ), which can be respectively compared with the distance thresholds preset according to the empirical constants and Corresponding. Among them, Can be set to 0.0, It can be set to 1.0. Therefore, the Planck weight can be determined based on the comparison result of the point-line distance d between the target pixel and the Planck curve and the distance threshold. Its mathematical expression can be:

[0079]

[0080] That is, when the distance d between the points is greater than When As Planck weight; when the distance d between points is less than When As Planck weight; at the point line distance d Within the interval, according to and Determine the Planck weight.

[0081] Based on the above embodiment, the embodiment of the present application describes the steps of determining gain information according to the target motion weight, the Planck weight and the pixel information of each target pixel. The pixel information includes color information. Specifically, the method of this embodiment includes the following steps:

[0082] The color information of each target pixel is weighted and summed according to the target motion weight and the Planck weight to obtain each comprehensive color information; and the gain information of the current frame image is determined according to each comprehensive color information.

[0083] Among them, color information can include RGB (red, green, and blue) statistical values ​​( , as well as ).

[0084] Specifically, according to the target motion weight of each target pixel Planck weight as well as , and The three components are weighted and summed separately to obtain the comprehensive color information SumR, and Then, according to the comprehensive color information SumR, and Determine global gain information and . Its mathematical expression can be:

[0085]

[0086] Based on the above embodiments, the embodiment of the present application describes the steps of filtering the gain information according to the target filtering weight to obtain the target white balance gain. Specifically, the method of this embodiment includes the following steps:

[0087] Obtain gain information and target weight information corresponding to at least one frame of image; perform mean filtering on the sum of gain information of at least one frame of image according to the sum of target weight information of at least one frame of image to obtain target white balance gain.

[0088] Combined with the above embodiment, the gain information of each frame of image data can be determined by referring to the method provided in the above embodiment. and .

[0089] In actual application, the present application can determine the target white balance gain ( and For example, the gain information corresponding to at least one frame of image ( and ) and target weight information (target filter weight ) is input into the mean filter for mean filtering, and its mathematical expression can be:

[0090]

[0091] The above formula represents the calculation method of final_Rgain and final_Bgain of image data of frames 0 to t and t+1 in total.

[0092] Furthermore, the final final_Rgain and final_Bgain can be applied to the raw data or the white balance module of the ISP to achieve white balance adjustment.

[0093] Based on the above embodiment, the embodiment of the present application describes the steps of obtaining pixel information, speed information and average speed information of each pixel point in the current frame image. Specifically, the method of this embodiment includes the following steps:

[0094] Obtain pixel information, and determine the optical flow speed of each pixel point based on the pixel information and a preset optical flow algorithm; determine the speed information of each pixel point based on the optical flow speed and the resolution information of the current frame image; determine the mean speed information based on the speed information of each pixel point and the number of each pixel point.

[0095] In conjunction with the above-mentioned embodiments, the method for obtaining pixel velocity information in the present application can be to use the pyramid optical flow method to calculate the raw data, or to use the statistical information obtained by ISP statistics (the information obtained by ISP statistics is equivalent to the information that has been downsampled) and the optical flow method to calculate the optical flow velocity (no need to perform multi-layer pyramid calculations, which can greatly reduce the amount of calculations), which is not limited here. However, the pyramid optical flow method has better accuracy. It can be determined based on actual needs (for example, the need to give priority to the amount of calculation or the accuracy of calculation, etc.) and the specifications of the equipment running this method (computing power, current load and other parameters), which will not be elaborated here.

[0096] By way of example, this embodiment uses the pyramid optical flow method as an example for explanation. In a specific implementation process, raw data of two adjacent frames of images may be used for calculation, and the steps may include but are not limited to:

[0097] Step S11, obtaining raw data of two adjacent frames of images and corresponding exposure information.

[0098] Step S12, calculate the multiples X1 and X2 of the exposure values ​​of the previous and next two frames and the preset exposure value. For example, the shutter speed of the current video frame is (unit is row), the gain is (Unit: db), the shutter speed corresponding to the preset exposure value is , the gain is , then the mathematical expression of the multiple X can be:

[0099]

[0100] It should be noted that if the camera uses an automatic aperture lens, the equivalent db number of the current aperture can also be obtained by pre-calibration or by approximate calculation based on the relationship between the lens step length and the light-passing area.

[0101] Step S13, convert the two frames of raw data into statistical data respectively to obtain , , , Four components; data conversion can refer to the prior art, which will not be described here. For example, it can be as follows Figure 2 As shown, Figure 2 This is an exemplary data conversion diagram in the exposure adjustment method based on motion judgment of this application. The raw data is a group of data represented by every 4 points, so Figure 2 Every four points in the , , , , the length and width of the resolution of the final statistical value are divided by 2 respectively. Its mathematical expression can be:

[0102]

[0103] Among them, BLC_R, BLC_Gr, BLC_Gb, and BLC_B are the black level values ​​corresponding to the raw data, which can generally be obtained through prior calibration and will not be described in detail here.

[0104] Step S14, dividing the Y component in the statistical information of the raw data of the two frames before and after by the calculated multiples X1 and X2 respectively, and then obtaining the raw statistical information of the two frames before and after at the same exposure level. and , .

[0105] Step S15: Statistical information of the raw images of the two frames before and after at the same exposure level The data is reduced m times according to the ratio k, and m layers of pyramid data are obtained respectively. Among them, k and m are both empirical values, which can be set according to the resolution of the original statistical information, and will not be described here.

[0106] Step S16, traverse all layers of the pyramid and collect statistics on the nth layer and Perform mean filtering to reduce noise and obtain filtered statistical information and Among them, the mean filtering process can use the convolution function, which will not be described here.

[0107] Step S17, using a preset gradient function to filter the statistical information and Extract the x and y direction gradients respectively and get the x direction gradient and , the gradient in the y direction and , and obtain the averaged x-direction gradient and the y-direction gradient ; Its mathematical expression can be:

[0108]

[0109] Step S18: The filtered statistical information and Subtract to get the frame difference , its mathematical expression can be:

[0110]

[0111] Step S19, determine whether the currently traversed pyramid level is at the highest level of the pyramid; if so, the optical flow speed in the x and y directions of the current level and The matrix is ​​initialized to all 0s; if not, the matrix calculated in the previous layer is used. and The matrix is ​​multiplied by the pyramid scaling factor k as the current layer and The initial value of the matrix. Its mathematical expression can be:

[0112]

[0113] Step S20: determine whether the current number of iterations i exceeds the iteration number threshold (empirical constant) or whether the difference delta calculated before and after is less than the preset threshold (an empirical constant that may vary with the resolution of the current pyramid); if so, exit the iteration and proceed to step S25; if not, proceed to step S21.

[0114] Step S21, determine whether the current layer is the first iteration of the pyramid; if so, keep and The matrix remains unchanged; if not, the result of the previous iteration is and Assign the matrix to and .

[0115] Step S22, the optical flow speed in the x and y directions of the current layer and The matrix is ​​mean filtered to obtain the filtered and matrix.

[0116] Step S23, based on the data matrix calculated above, use the local average and optical flow constraint method to calculate the optical flow speed in the x and y directions and , its mathematical expression can be:

[0117]

[0118]

[0119]

[0120] in, It is a preset compensation constant and cannot be 0.

[0121] Step S24, calculate the difference delta before and after, which can be expressed mathematically as:

[0122]

[0123] Then, the number of iterations i is increased by 1, and the process returns to step S20 for iteration.

[0124] Step S25, until all pyramids have been calculated, use the optical flow speed in the x and y directions of the bottom pyramid result and , calculate the directionless optical flow speed . Its mathematical expression can be:

[0125]

[0126] In summary, this application can use the optical flow method to calculate the motion of each pixel (statistical block) based on the raw data of the previous and next two frames, effectively determine the high and low speed motion areas in the image, dynamically adjust the filter weights between different pixels (statistical blocks) and the motion weights between the white balance values ​​of the previous and next multiple frames according to the different degrees of motion, and calculate the final white balance gain, which can effectively remove the interference of moving objects on white balance and reduce white balance jitter. And it is not limited to using only the raw data of the previous and next two frames, and can also directly use the statistical information of the ISP statistics, without the need for multi-layer pyramid calculations, which can greatly reduce the amount of calculations, but the accuracy of the pyramid optical flow method results may be better.

[0127] It should be further explained that the execution subject of the white balance adjustment method based on motion judgment may be a white balance adjustment device based on motion judgment, for example, the white balance adjustment method based on motion judgment may be executed by a terminal device or a server or other processing device, wherein the terminal device may be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the white balance adjustment method based on motion judgment may be implemented by a processor calling a computer-readable instruction stored in a memory.

[0128] Figure 3 FIG. 1 is a block diagram of a white balance adjustment device based on motion judgment, which is shown in an exemplary embodiment of the present application. Figure 3 As shown, the exemplary white balance adjustment device 300 based on motion judgment includes: an acquisition module 310, a clustering module 320, an information determination module 330 and a filtering processing module 340. Specifically:

[0129] The acquisition module 310 is used to acquire pixel information, speed information of each pixel point in the current frame image and average speed information of the current frame image.

[0130] The clustering module 320 is used to perform clustering processing on each pixel point according to the speed information and the preset clustering parameters to obtain the clustering value of each type of pixel area.

[0131] The information determination module 330 is used to traverse each pixel point and determine the target filtering weight and gain information of the current frame image according to the pixel information, cluster value, mean velocity information and preset weight information.

[0132] The filtering processing module 340 is used to filter the gain information according to the target filtering weight to obtain the target white balance gain.

[0133] In this exemplary white balance adjustment device based on motion judgment, by obtaining the pixel information, speed information and mean speed information of each pixel point in the current frame image, each pixel point can be clustered according to the speed information and preset clustering parameters to obtain the clustering value of each type of pixel area; each pixel point is traversed and the target filtering weight and gain information of the current frame image is determined according to the pixel information, clustering value, mean speed information and preset weight information, thereby the weights corresponding to different pixel points can be adaptively determined according to the different degrees of motion of each type of pixel point in the current frame image; then the gain information is filtered according to the target filtering weight to obtain the target white balance gain, and the white balance setting is completed according to the target white balance gain, thereby not only realizing automatic white balance adjustment, but also effectively removing the interference of moving objects on the white balance adjustment and reducing white balance jitter.

[0134] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0135] The functions of each module may be found in the embodiment of the white balance adjustment method based on motion judgment, which will not be described in detail here.

[0136] See also Figure 4 , Figure 4 1 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 100 includes a memory 101 and a processor 102, and the processor 102 is used to execute program instructions stored in the memory 101 to implement the steps in any of the above-mentioned white balance adjustment method embodiments based on motion judgment. In a specific implementation scenario, the electronic device 100 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 100 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0137] Specifically, the processor 102 is used to control itself and the memory 101 to implement the steps in any of the above-mentioned white balance adjustment method embodiments based on motion judgment. The processor 102 can also be called a CPU (Central Processing Unit). The processor 102 may be an integrated circuit chip with signal processing capabilities. The processor 102 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 102 can be implemented by an integrated circuit chip.

[0138] In this exemplary electronic device, by acquiring pixel information, speed information and mean speed information of each pixel in the current frame image, each pixel can be clustered according to the speed information and preset clustering parameters to obtain clustering values ​​for each type of pixel area; each pixel is traversed and the target filtering weight and gain information of the current frame image is determined according to the pixel information, clustering value, mean speed information and preset weight information, thereby adaptively determining the weights corresponding to different pixels according to the different degrees of motion of each type of pixel in the current frame image; and then filtering the gain information according to the target filtering weight to obtain the target white balance gain, and completing the white balance setting according to the target white balance gain, thereby not only realizing automatic white balance adjustment, but also effectively removing the interference of moving objects on the white balance adjustment and reducing white balance jitter.

[0139] See also Figure 5 , Figure 5 The computer-readable storage medium 110 stores program instructions 111 that can be executed by a processor, and the program instructions 111 are used to implement the steps in any of the above-mentioned white balance adjustment method embodiments based on motion judgment.

[0140] In this exemplary storage medium, by running the program instructions in the storage medium, the pixel information, speed information and mean speed information of each pixel point in the current frame image are obtained, and each pixel point can be clustered according to the speed information and preset clustering parameters to obtain the clustering value of each type of pixel area; each pixel point is traversed and the target filtering weight and gain information of the current frame image are determined according to the pixel information, clustering value, mean speed information and preset weight information, thereby the weights corresponding to different pixel points can be adaptively determined according to the different degrees of motion of each type of pixel point in the current frame image; then the gain information is filtered according to the target filtering weight to obtain the target white balance gain, and the white balance setting is completed according to the target white balance gain, thereby not only realizing automatic white balance adjustment, but also effectively removing the interference of moving objects on the white balance adjustment and reducing white balance jitter.

[0141] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0142] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0143] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0144] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A white balance adjustment method based on motion judgment, characterized in that: The method comprises: Obtain pixel information, speed information of each pixel point in the current frame image and average speed information of the current frame image; Clustering each pixel point according to the speed information and preset clustering parameters to obtain a clustering value for each type of pixel area; Traversing each pixel point and determining the target filtering weight and gain information of the current frame image according to the pixel information, the cluster value, the mean velocity information and the preset weight information; The step of traversing each pixel point and determining the target filtering weight and gain information of the current frame image according to the pixel information, the clustering value, the mean velocity information and the preset weight information includes: determining the target filtering weight of the current frame image according to the mean velocity information and the preset filtering weight; determining the target motion weight of each pixel point according to the clustering value and the preset motion weight; determining the target pixel point in each pixel point according to the pixel information of each pixel point, and determining the Planck weight of each target pixel point according to the distance between the target pixel point and the preset Planck curve; determining the gain information according to the target motion weight, the Planck weight and the pixel information of each target pixel point; The gain information is filtered according to the target filtering weight to obtain a target white balance gain.

2. The method according to claim 1, characterized in that: The preset filtering weight includes a first filtering weight and a second filtering weight, the first filtering weight is less than the second filtering weight, and determining the target filtering weight of the current frame image according to the mean velocity information and the preset filtering weight includes: Comparing the mean speed information with a first speed threshold and / or a second speed threshold to obtain a comparison result, wherein the first speed threshold is greater than the second speed threshold; In response to the comparison result indicating that the mean speed information is greater than the first speed threshold, determining the first filtering weight as the target filtering weight; In response to the comparison result indicating that the mean speed information is less than the second speed threshold, the second filtering weight is determined as the target filtering weight.

3. The method according to claim 1, characterized in that The preset motion weight includes a first motion weight and a second motion weight, and determining the target motion weight of each pixel point according to the cluster value and the preset motion weight includes: According to the cluster value and the regional velocity mean of each type of pixel area, each pixel area is sorted to obtain a sorted image, wherein the regional velocity mean is determined according to the velocity information and the number of pixel points in each type of pixel area; Performing binary segmentation processing on the sorted image to obtain a high-speed motion area and a low-speed motion area in the sorted image; Determine a high-speed motion mean value of the high-speed motion area according to the speed information of each pixel point in the high-speed motion area, and determine a low-speed motion mean value of the low-speed motion area according to the speed information of each pixel point in the low-speed motion area; Determine a high-speed clustering threshold corresponding to the high-speed motion area according to the high-speed motion mean and the clustering value, and determine a low-speed clustering threshold corresponding to the low-speed motion area according to the low-speed motion mean and the clustering value; According to the comparison result between the clustering value of each pixel point and the high-speed clustering threshold and / or the low-speed clustering threshold, the first motion weight and / or the second motion weight is selected to determine the target motion weight.

4. The method according to claim 1, characterized in that: The pixel information includes brightness information, chromaticity information and initial gain information, and the step of determining a target pixel among the pixels according to the pixel information of each pixel, and determining the Planck weight of each target pixel according to the distance between the target pixel and a preset Planck curve includes: Determine the pixel point whose brightness information, chromaticity information and initial gain information meet the preset judgment condition as the target pixel point; The Planck weight is determined from preset weight parameters according to a comparison result between the distance and a distance threshold.

5. The method according to claim 1, characterized in that The pixel information includes color information, and determining the gain information according to the target motion weight, the Planck weight, and the pixel information of each target pixel point includes: According to the target motion weight and the Planck weight, each color information of each target pixel is weighted and summed to obtain each comprehensive color information; The gain information of the current frame image is determined according to each comprehensive color information.

6. The method according to claim 1, characterized in that The filtering the gain information according to the target filtering weight to obtain a target white balance gain includes: Obtaining gain information and target weight information corresponding to at least one frame of image; The target white balance gain is obtained by performing mean filtering processing on the sum of gain information of at least one frame of image according to the sum of target weight information of at least one frame of image.

7. The method according to claim 1, characterized in that The obtaining of pixel information, speed information of each pixel point in the current frame image and average speed information of the current frame image includes: Acquire the pixel information, and determine the optical flow speed of each pixel point according to the pixel information and a preset optical flow algorithm; Determine the speed information of each pixel point according to the optical flow speed and the resolution information of the current frame image; The mean speed information is determined according to the speed information of each pixel point and the number of each pixel point.

8. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images