A method for correcting camera automatic white balance for FCN human skin segmentation

Abstract

The application provides a method for FCN human skin segmentation and correction camera automatic white balance, comprising the following steps: S1, collecting human skin segmentation data; S2, designing a convolutional neural network, which satisfies that the last layer of the convolutional neural network is a convolutional non-full connection layer; S3, designing a training strategy; S4, training a human skin model; S5, calibrating a camera white balance color temperature curve; and S6, correcting a white balance parameter. Before automatic white balance, the global information of an image is considered, a human skin is segmented by using a full convolutional neural network, the human skin area and the non-human skin area are distinguished, the image information is counted, and the skin area information is not considered when the white balance parameter is weighted, so that the white balance parameter is more accurate, and the visual effect is good.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image signal processing and the field of intelligent video technology, and particularly relates to a method for FCN human skin segmentation and rectification camera automatic white balance. BACKGROUND

[0002] The current existing automatic white balance method generally obtains a color temperature curve by calibrating a series of standard color temperatures, and the closer to the color temperature curve, the greater the weight, then the image information is blocked and counted, and the optimal parameters of the current automatic white balance are obtained through weighting.

[0003] The human body is a common scene in image videos, and the current existing automatic white balance is usually close to the color temperature curve weight when the human skin color information is close to the color temperature curve weight, which leads to the optimal parameters of the white balance of the human skin part color, so that the image is blue, which affects the visual effect. The current common method for detecting human skin is classification, but the detection effect is poor due to the lack of global information.

[0004] In addition, the common terms in the prior art are as follows:

[0005] Fully convolutional neural network FCN: is to restore the class to which each pixel belongs from abstract features. That is, to further extend from image-level classification to pixel-level classification.

[0006] Human skin segmentation: for an image, a method for distinguishing pixels belonging to human skin in the image by using an algorithm.

[0007] The fully convolutional neural network FCN is generally used for image segmentation, and the core problem to be solved is the classification of image pixels. The network output is generally a probability map with the same size as the image, such as the probability of a pixel being skin is 0.9 and the probability of a pixel being non-skin is 0.1.

[0008] Training: first, design a model, input corresponding data and labels, so that the model prediction result and the label are as close as possible. After training is completed, the model parameters converge, and the prediction label is obtained after inputting data. Human skin segmentation is based on the training method, and most of the data accounts for a small proportion, which may cause the problem of sample imbalance, which may cause the model to tend to segment skin or background, resulting in poor effect.

[0009] White balance: white balance is produced to restore the true color of an image.

[0010] Automatic white balance: the automatic white balance adjustment function is a function of most cameras. When the camera is directed at an object, the white balance of the camera is automatically adjusted as the color temperature of the lighting light changes. If the automatic white balance effect is poor, the image will be colored, which is generally red and blue. SUMMARY

[0011] In order to solve the above problems, the purpose of the present application is to consider the global information of the image before automatic white balance, to segment the human skin by using the full convolutional neural network, to classify the image pixel by pixel, to further distinguish the human skin area and the non-human skin area, to count the image information, and to not consider the skin area information when weighting the white balance parameter, so that the white balance parameter is more accurate and the visual effect is good.

[0012] Specifically, the present application provides a method for correcting camera automatic white balance by FCN human skin segmentation, which comprises the following steps:

[0013] S1. Collect human skin segmentation data;

[0014] S2. Design a convolutional neural network, which satisfies that the last layer of the convolutional neural network is a convolutional non-full connection layer;

[0015] S3. Design a training strategy

[0016] S3.1 Considering that the data size ratio is not the same, 512x512 resolution training is adopted during training, and 480x288 inference is adopted during actual use considering the real-time requirement;

[0017] S3.2 Since the area proportion of human skin data is small, the image is scaled in range with 512x512 and 800x800 as the reference resolution during training, and the input network size is 512x512 by using random cropping or padding;

[0018] S3.3 In view of the color cast caused by poor camera automatic white balance effect, random color cast data enhancement is adopted during training;

[0019] S4. Obtain a human skin segmentation model according to the designed full convolutional neural network and the training strategy, and design a confidence threshold T c and a skin area threshold T r ;

[0020] S5. Camera automatic white balance calibration:

[0021] S5.1 Take standard color card pictures under each standard color temperature;

[0022] S5.2 Correct the R, G and B channels based on the gray card of the standard color card to obtain the corresponding correction parameters;

[0023] S5.3 Interpolate and draw the color temperature curve based on the correction parameters, and assign the color temperature curve weight;

[0024] S6. Correct the camera automatic white balance according to the human skin segmentation result:

[0025] S6.1 Given human skin segmentation partition default initial value N 15x15 , 0 matrix of 15x15 size initial value, indicating the non-human skin area;

[0026] S6.2 Start automatic white balance calculation and full convolution neural network human skin segmentation model calculation;

[0027] S6.3 Automatic white balance divides the image into 15x15 equal size areas, respectively, and calculates the three color information luminance information of each block area, and calculates the current gain R gain , B gain ;

[0028] S6.4 If the model calculation is completed, the human skin segmentation result is divided into 15x15 equal size areas, and the confidence threshold T c is used to determine whether the current pixel is human skin, and the skin area threshold T r is used to judge whether each partition is a skin area,

[0029] If it is a skin area, the automatic white balance calculation parameter is 0, and if it is not a skin area, the current skin area weight is the original weight;

[0030] If the model has not been calculated, the last result is used, and the gain R gain , B gain of each partition is combined with the color temperature curve weight to calculate the global gain.

[0031] The step S2 further comprises:

[0032] The designed convolutional neural network, the first two layers of the network are basic convolution operation, the whole network uses jump connection to keep high resolution feature series after the first two layers of convolution, and the high resolution features at different network depths are down-sampled using convolution to generate low resolution branches, the low resolution branches are kept the same resolution and connected in series using jump connection, and the low resolution branches are up-sampled and fused with the high resolution branches to output in high resolution, and finally the high resolution is up-sampled four times to obtain the same size as the original input.

[0033] The step S3.3 further comprises:

[0034] In view of the poor camera automatic white balance effect and color deviation, random color deviation data enhancement is adopted in training, and the specific operation is as formula (1):

[0035]

[0036] Where I r , I g and Ib respectively the value of data RGB channel, r gain and b gain respectively the random generated color cast gain.

[0037] The step S6.3 respectively calculates the three color information luminance information of each block region, and calculates the current gain R gain , B gain , as shown in formula (2):

[0038]

[0039] Wherein Sum R , Sum G , Sum B respectively represent the sum of R, G, B color components of the current region.

[0040] In the step S6.4,

[0041] If the model is calculated, as shown in formula (3), according to the confidence threshold T c , it is determined whether the current pixel is human skin, and T c is 0.5 under normal circumstances, and in some scenes, in order to reduce the negative impact on the original automatic white balance, the threshold T c is 0.9 or more to be considered as human skin, and then according to the skin region threshold T γ to determine whether each subregion is a skin region, and T r is 288 under normal circumstances, and in some scenes, human skin has a large impact on white balance, so when the human skin pixel threshold reaches 144, it is judged as a skin region; if it is a skin region, the current skin region weight is 0 when calculating the automatic white balance parameters,

[0042] If it is not a skin region, the current skin region weight is the original weight;

[0043]

[0044] O'(i,j) = argmax(O'(i,j,k))

[0045]

[0046] Wherein i, j are the pixel positions of width and height, k is the model output category, O(i, j, k) is the result before adjusting the confidence, O'(i, j, k) is the result after adjusting the confidence, O'(i, j) is the final determined pixel category, and n is the number of pixels in the current region.

[0047] Therefore, the advantages of the present application are:

[0048] (1) using a full convolutional neural network for human skin segmentation, maintaining high resolution and low resolution features in the network, high resolution features focus on detailed information, low resolution considers global information, so that the skin segmentation effect is more accurate;

[0049] (2) According to the characteristics of human skin data, two kinds of reference resolution are used for image scaling with a range of equal proportion during training, which relieves the data balance problem and increases data diversity. In addition, for the case that the automatic white balance effect is poor, random color cast is used during training, so that the human skin segmentation is more accurate under color cast.

[0050] (3) Combined with the human skin segmentation model, the color temperature curve weight is modified to reduce the weight of the automatic white balance, so that the automatic white balance is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0051] The drawings described herein are used to provide a further understanding of the present application, constitute a part of this application, and do not constitute a limitation of the present application.

[0052] Figure 1 is a method flowchart of an embodiment of the present application.

[0053] Figure 2 is a flowchart of correcting the camera automatic white balance in step S6 of an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to more clearly understand the technical content and advantages of the present application, the present application will be further described in detail in conjunction with the drawings.

[0055] The present application provides a full convolutional neural network FCN human skin segmentation method for correcting camera automatic white balance, wherein the method preparation flow is as shown in Figure 1 The main implementation steps are as follows:

[0056] Step S1. Collect human skin segmentation data

[0057] Step S2. Design a convolutional neural network, which satisfies that the last layer of the convolutional neural network is a convolutional non-full connection layer;

[0058] The designed convolutional neural network has the following characteristics: the first two layers of the network are basic convolutional operations, the high resolution features are kept in series through jump connection after the first two layers of convolution in the whole network, and the low resolution branch is generated by using convolution to downsample the high resolution features at different network depths, the low resolution branch is kept in series by jump connection, and the low resolution branch is upsampled and fused with the high resolution branch to output in high resolution, and finally the high resolution is upsampled four times to obtain the same size as the original input.

[0059] Step S3. Design training strategy

[0060] S3.1 Considering the different data size ratios, 512x512 resolution training is used in training, and 480x288 inference is used in actual use considering real-time requirements;

[0061] S3.2 Since the area of human skin data accounts for a small proportion, the image is scaled in range with 512x512 and 800x800 as the reference resolution in training, and the input network size is 512x512 by random cropping or padding;

[0062] S3.3 In view of the poor color cast caused by the poor effect of camera automatic white balance, random color cast data enhancement is adopted in training, and the specific operation is as formula (1):

[0063]

[0064] Where I r , I g and I b are the values of the RGB channels of the data, r gain and b gain are the random generated color cast gains.

[0065] Step S4. Obtain the human skin segmentation model according to the designed full convolutional neural network and training strategy, and design the confidence threshold T c and the skin area threshold T γ according to the actual human skin segmentation effect.Step S5. Camera automatic white balance calibration

[0066] S5.1 Take standard color card pictures under each standard color temperature;

[0067] S5.2 Correct the R, G, and B channels based on the gray card of the standard color card to obtain the corresponding correction parameters;

[0068] S5.3 Interpolate and draw the color temperature curve based on the correction parameters, and assign the color temperature curve weight.

[0069] Step S6. Correct the camera automatic white balance according to the human skin segmentation result, and the flow is as shown in Figure 2

[0070] S6.1 Given the human skin segmentation partition default initial value N 15×15 , which is a 0 matrix with an initial value of 15x15, indicating a non-human skin area;

[0071] S6.2 Start automatic white balance calculation and full convolutional neural network human skin segmentation model calculation;

[0072] S6.3 Auto white balance divides the image into 15x15 equal size regions, as formula (2), respectively statistics each block area of three kinds of color information luminance information, and calculate the current gain R gain , B gain ;

[0073]

[0074] Where Sum R , Sum G , Sum B respectively represent the sum of R, G, B color components of the current region.

[0075] S6.4 If the model is calculated, the human skin segmentation result is divided into 15x15 equal size regions, as formula (3) shows, according to the confidence threshold T c determine whether the current pixel is human skin, the threshold is different in different scenes, can be adjusted, T c is 0.5 in normal scene, the algorithm model result may not be accurate in some scenes (such as dark light, motion condition, etc.), in order to reduce the negative impact on the original auto white balance (false detection is not necessarily negative impact), the threshold can be slightly larger, such as 0.9, that is, the human skin confidence threshold T c is 0.9 or more to be considered as human skin, and then according to the skin region threshold T γ judge whether each partition is a skin region, the threshold is different in different scenes, can be adjusted, considering the relationship between the actual use of 1920x1080 image and network inference image size 480x288, the value range of T r is between 0 and (480 / 15)*(270 / 15) = 576, 270 is 1080 scaled by 4 times, 15x15 is the partition size, so there are 576 pixels in a single partition, T r is 288 in normal scene, a small amount of human skin has a great impact on white balance in some scenes, the threshold can be slightly smaller, such as 144, that is, the human skin pixel is judged as a skin region when it reaches 144; if it is a skin region, the current skin region weight is 0 when calculating the auto white balance parameters, if it is not a skin region, the current skin region weight is the original weight; if the model is not calculated, the last result is used, the gain R gain , B gain combined with the color temperature curve weight to calculate the global gain, the combination here involves the principle and calculation of auto white balance, which is the prior art, and the present application mainly assists in correcting auto white balance, so it will not be described in detail here;

[0076]

[0077] O'(i,j) = argmax(O(i,j,k))

[0078]

[0079] where i, j are pixel positions of width and height, k is a model output category, O(i,j,k) is a result before adjusting confidence, O'(i,j,k) is a result after adjusting confidence, O'(i,j) is a final determined pixel category, and n is a number of pixels in a current region.

[0080] In summary, the above method can be simply summarized as:

[0081] Start;

[0082] S1, collect human skin segmentation data;

[0083] S2, design a full convolutional neural network;

[0084] S3, design a training strategy;

[0085] S4, train a human skin model;

[0086] S5, camera white balance color temperature curve calibration;

[0087] S6, white balance parameter correction;

[0088] Output result.

[0089] Further, as shown in Figure 2 , the method step S6 in the correction camera automatic white balance process is as follows:

[0090] Start, initialize human skin segmentation result; then automatically white balance statistical information; then determine whether the model is calculated? If not, then automatically white balance weighting parameters, then output the result; if yes, then calculate the human skin area; then modify the corresponding area weight; automatically white balance weighting parameters, then output the result.

[0091] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application can be variously changed and modified. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for FCN human skin segmentation rectification camera automatic white balance, characterized in that, The method comprises the following steps: S1. Collect human skin segmentation data; S2. Design a convolutional neural network, which satisfies that the last layer of the convolutional neural network is a convolutional non-full connection layer; S3. Design a training strategy: S3.

1. Considering that the data size ratio is not the same, 512x512 resolution training is adopted during training, and 480x288 resolution is adopted for inference during actual use considering the real-time requirement; S3.

2. Due to the small area proportion of human skin data, the image is scaled in range by 512x512 and 800x800 as the reference resolution during training, and the input network size is 512x512 by using random cropping or padding; S3.

3. In view of the poor color deviation of the camera automatic white balance effect, random color deviation data enhancement is adopted during training; S4. According to the designed full convolutional neural network and training strategy, the human skin segmentation model is trained, and the confidence threshold T is designed according to the actual human skin segmentation effect c And the skin area threshold T r ; S5. Camera automatic white balance parameter calibration: S5.

1. Take pictures of standard color cards under each standard color temperature; S5.

2. Correct the R, G and B channels based on the gray card of the standard color card to obtain the corresponding correction parameters; S5.

3. Interpolate and draw the color temperature curve based on the correction parameters, and assign the color temperature curve weight; S6. Correct the camera automatic white balance according to the human skin segmentation result: S6.

1. Given the initial value N of the human skin segmentation partition 15×15 is a 0 matrix of size 15x15 with initial value N, representing the non-human skin region; S6.

2. Start the automatic white balance calculation and the full convolutional neural network human skin segmentation model calculation; S6.

3. Auto white balance divides the image into 15x15 equal size regions, respectively counts the three color information luminance information of each region, and calculates the current gain R gain , B gain ; S6.4 If the model calculation is completed, the human skin segmentation result is divided into 15x15 equal size regions, and according to the confidence threshold T c determines whether the current pixel is human skin, and then according to the skin region threshold T r judges whether each partition is a skin region, If it is a skin area, the current skin area weight is 0 during automatic white balance calculation, If it is not a skin area, the current skin area weight is the original weight; If the model has not been calculated, use the last result, gain R for each partition gain , B gain The global gain can be calculated by combining the color temperature curve weight.

2. The method of claim 1, wherein the FCN human skin segmentation corrects the camera auto white balance. The step S2 further comprises: ​ The designed convolutional neural network, the first two layers of the network are basic convolutional operations, and the entire network uses skip connection to keep high-resolution features in series after the first two layers of convolution, and uses convolution to down-sample the high-resolution features at different network depths to generate low-resolution branches, the low-resolution branches are kept in series by skip connection, and the low-resolution branches are up-sampled and fused with the high-resolution branches to output in high resolution, and finally the high-resolution is up-sampled four times to obtain the same size as the original input.

3. The method of claim 1, wherein the FCN human skin segmentation corrects the camera auto white balance. The step S3.3 further comprises: In view of the poor color deviation of the camera automatic white balance effect, random color deviation data enhancement is adopted during training, and the specific operation is as shown in formula (1): Formula (1); where I r , I g , and I b are the values of the data RGB channels, respectively, and r gain and b gain are randomly generated color cast gains.

4. The method of claim 1, wherein the FCN human skin segmentation corrects the camera auto white balance. The step S6.3 respectively counts the three color information luminance information of each block region, and calculates the current gain R gain , B gain As shown in formula (2): Formula (2); where SumR, SumG, SumB R , SumR, SumG, SumB G , SumR, SumG, SumB B represent the sum of the R, G, B color components of the current region, respectively.

5. The method of claim 4, wherein the FCN human skin segmentation corrects the camera auto white balance. In the step S6.4, If the model calculation is completed, according to the confidence threshold T c , it is determined whether the current pixel is human skin, T c is 0.5 in a normal scene, and in some scenes, in order to reduce the negative impact on the original automatic white balance, the threshold T c is 0.9 or more before considering it as human skin, and then according to the skin area threshold T r , it is judged whether each partition is a skin area, T r is 288 in a normal scene, and in some scenes, human skin has a large impact on white balance, so when the human skin pixel threshold reaches 144, it is judged as a skin area; if it is a skin area, the current skin area weight is 0 when calculating the automatic white balance parameters, If it is not a skin area, the current skin area weight is the original weight; Formula (3); Where i and j are pixel positions of width and height, k is a model output category, O(i,j,k) is a result before adjusting confidence, O'(i,j,k) is a result after adjusting confidence, O'(i,j) is a final determined pixel category, and n is a current area pixel number.

Images