A method to solve the severe imbalance of semantic segmentation data for specific tasks

Abstract

The present invention provides a method for resolving severe imbalance in semantic segmentation data for a specific task. The method comprises the following steps: S1, collecting basic data; S2, converting basic data; S3, cleaning data; S4, preparing offline copy and paste data; S5, preparing online copy and paste data; S6, starting training; determining whether to copy paste online. If so, proceed to step S6.1, followed by step S6.2; if not, proceed directly to step S6.2; S6.1, copy paste online; S6.2, determining whether to mosaic online. If so, perform online mosaic, followed by step S7; if not, proceed directly to S7; S7, obtaining a training model: training iteration. This method uses offline copy paste to better combine human skin and blue sky data, making the data more balanced and improving the semantic segmentation effect. Through online copy paste and online mosaic, the data is further balanced and scene diversity is increased, making the semantic segmentation effect more stable.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent surveillance video processing, and in particular relates to a method for solving the serious imbalance of semantic segmentation data for a specific task. Background Art

[0002] Semantic segmentation is the process of classifying every pixel in an image. It is currently widely used in medical imaging and autonomous driving. Different tasks require different data categories. For example, in automatic white balance, we need to segment common human skin, blue sky, and other background categories to improve the accuracy of automatic white balance.

[0003] For specific semantic segmentation tasks, a common approach is to collect relevant data from public data. Unlabeled data needs to be manually labeled to create a dataset. However, there will be serious data imbalance during the data collection process, and labeling human skin, blue sky and other background-related data at the same time is a time-consuming and labor-intensive process. Severe data imbalance and data volume directly affect the effectiveness of the semantic segmentation algorithm, which may cause the model to only tend to segment one type of object.

[0004] In addition, the commonly used terms in the prior art are as follows:

[0005] Semantic segmentation is the process of classifying each pixel in an image.

[0006] Training: First, design the model and input the corresponding data and labels so that the model's predictions are as close as possible to the labels. After training, the model parameters converge, and the predicted labels are obtained after inputting data. Data imbalance refers to the uneven distribution of data in the semantic segmentation task. For example, if dataset A contains only a small amount of class B, the segmentation model may learn to segment only one class, resulting in poor results.

[0007] Semantic segmentation data is generally obtained manually through dot annotation. For example, human skin annotation requires connecting the human skin area with multiple points to form a polygon. Annotating data is time-consuming and labor-intensive.

[0008] Copypaste: It is a method similar to copy and paste, pasting the image of interest on another image. Offline copypaste is a preparation made before model training. The purpose is to balance the data. The enhanced images and labels will be saved offline. If there are too many, it will take up too much storage space. If there are too few, the data diversity is limited, so online copypaste will be introduced. Online copypaste is an operation performed during training. The purpose is to balance the data while expanding the data diversity and avoiding overfitting.

[0009] Online mosaic: This is a type of data augmentation during training. The enhanced image has four identical regions, each with data from a different image. For example, in tasks involving both human skin and blue sky, blue sky is included simultaneously, alleviating data imbalance and increasing data diversity. Summary of the Invention

[0010] In order to solve the above problems, the purpose of this application is: based on the characteristics of data-driven semantic segmentation target tasks, first obtain human skin category data and other background categories in the public human parsing dataset, use color segmentation to collect blue sky categories and other background categories in the public sky dataset, obtain other background categories on common public datasets, and then use offline copypaste, online mosaic, and copypaste data enhancement methods to integrate data, increase the data volume, and solve the problem of severe data imbalance.

[0011] Specifically, the present invention provides a method for solving the serious imbalance of semantic segmentation data for a specific task, the method comprising the following steps:

[0012] S1, collect basic data;

[0013] S2, basic data conversion;

[0014] S3, data cleaning;

[0015] S4, prepare for offline copy and paste data enhancement;

[0016] S5, prepare online copy and paste data enhancement;

[0017] S6, start training;

[0018] Determine whether to copypaste online? If yes, proceed to step S6.1, then proceed to step S6.2; if no, proceed directly to step S6.2;

[0019] S6.1, online copypaste;

[0020] S6.2, determine whether online mosaic data enhancement is required. If yes, perform online mosaic and proceed to step S7; if no, proceed directly to S7;

[0021] S7, obtain the training model: training iteration.

[0022] The step S1 further comprises:

[0023] Obtain human skin classes and other background classes from the public human parsing dataset, and obtain sky class data and other background classes from the public sky dataset;

[0024] The human skin class in the human body analysis data set is the part of the human skin including the face, hands, arms and legs in the human body analysis data set as the human skin class;

[0025] Other background classes in the human body analysis dataset include non-human skin parts such as hats, tops, and pants.

[0026] The blue sky class of the sky dataset is to perform color space conversion on the sky category image data portion in the sky dataset, convert RGB to HSV space, perform color segmentation in the HSV space, set the required blue sky color threshold range, and use the blue in the sky as the blue sky class; other colors are used as other background classes of the sky dataset.

[0027] The basic data conversion in step S2 further includes:

[0028] The human skin class and other background classes in the human body analysis dataset are specifically operated as shown in formula (1);

[0029]

[0030] Among them, i, j are the image pixel coordinates, L a (i, j) is the human body analysis and annotation data, L s (i, j) is the converted human skin data, 0 is other background classes, 1 is human skin classes, index_list is the index list of the human skin part annotations of the hands, arms, and legs, and k is the human skin part of the human body parsing annotation data;

[0031] The sky data set is used to obtain sky category data and other background categories. The specific operation is shown in formula (2):

[0032]

[0033] Among them L sky (i, j) is the sky label data, L b (i, j) is the blue sky label data after conversion, 0 is other background classes, 1 is blue sky class, I hsv (h, w, c) is HSV data, and thershold_blue is the selected blue HSV color threshold range.

[0034] Step S3 data cleaning:

[0035] In order to ensure data accuracy, the converted data needs to be cleaned to ensure that the existing human skin data does not contain blue sky, and the blue sky data does not contain human skin.

[0036] The step S3 further comprises:

[0037] S3.1. Use human skin data to train a separate human skin segmentation model skin , use the blue sky data to train a separate blue sky segmentation model model bluesky ;

[0038] S3.2. Using the model skin Model Screening Blue Sky Data I b (i,j) gets R b (i,j), define w, h as R b The width and height of (i,j) are b (i, j)) < = 0.9*w*h, it means that the current blue sky data image contains more than 10% of human skin, and the current blue sky data is directly removed. If sum(R b If (i,j))>0.9*w*h, the current blue sky data is manually screened to see if it contains human skin. If so, it is removed; otherwise, it is retained.

[0039] S3.3. Using the model bluesky Model Screening Human Skin Data I s (i,j) gets R s (i,j), define w, h as R s The width and height of (i,j) are s (i, j)) < = 0.9*w*h, it means that the current human skin data image contains more than 10% of blue sky, and the current human skin data is directly eliminated. If sum(R s (i,j))>0.9*w*h, the current skin data is manually screened to see if it contains blue sky. If so, it is removed; otherwise, it is retained.

[0040] The step S4 further comprises:

[0041] S4.1 For the existing human skin image L s (i, j) and label L s (i, j), firstly, a high-precision human body segmentation algorithm is used to obtain an image L containing only the human body part. p (i, j) and L p (i, j), there is the following relationship L p (i, j)∈L s (i, j), L s (i, j)∈L p (i, j);

[0042] S4.2 with blue sky image I b(i, j) is the background image used as offline copypaste, and the human body image I p (i, j) is randomly scaled and flipped and then pasted onto the blue sky image I b (i, j), and I p (i, j) performs corresponding operations to L b (i, j), we can get data that contains skin, blue sky and other backgrounds at the same time, making the data more balanced.

[0043] The step S5 further comprises:

[0044] For L in step S4 p (i, j) performs edge segmentation, divides the data into the minimum circumscribed rectangle of each person's body, and obtains several human body images I' p (i, j) and human body label L′ p (i, j), and also obtain several personal human skin labels L′ sp (i, j).

[0045] The training process in step S6 further includes:

[0046] S6.1 During training, perform online copypaste with a certain probability, randomly scale and flip a single human body and paste it onto the current data to expand data diversity;

[0047] During S6.2 training, perform online mosaicing with a certain probability. First, determine the mosaic base image size. Then, process the current data according to the mosaic base image size and fill it into the first position of the mosaic. The other three positions are randomly filled with data according to the corresponding processing.

[0048] The certain probability mentioned above is as follows: the iterative process generates a random number between 0 and 1, and copypaste is performed when the random number is less than 0.25, that is, the probability is 0.25. During the training process, online copypaste and online mosaic are all performed using a probability of 0.25 by default.

[0049] The step S6.2 further includes: determining the mosaic reference image size, the reference image size is 512x512, and the mosaic uses a 2x2 grid format by default, which means that four 256x256 image data should be filled in, but the original image data is not necessarily 256x256 in size, and needs to be randomly scaled and cropped to 256x256 in size and spliced ​​into the mosaic reference size.

[0050] The method is targeted at semantic segmentation, which requires segmenting specific tasks: human skin, blue sky, and other backgrounds.

[0051] Therefore, the advantages of this application are:

[0052] (1) Through offline copypaste, the human skin and blue sky data are better combined, making the data more balanced and improving the semantic segmentation effect;

[0053] (2) Through online replication and online mosaic, the data is further balanced and the scene diversity is increased, making the semantic segmentation effect more stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0055] Figure 1 It is a schematic flow diagram of the method of the present invention.

[0056] FIG2( a ) shows four randomly selected original images in the mosaic principle diagram of the method of the present invention.

[0057] FIG2( b ) is a mosaic image of FIG2( a ).

[0058] Figure 3 It is a schematic diagram showing the copypaste principle in the method of the present invention. DETAILED DESCRIPTION

[0059] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail with reference to the accompanying drawings.

[0060] This paper proposes a method to solve the serious imbalance of semantic segmentation data for specific tasks. Semantic segmentation requires segmenting specific tasks: human skin, blue sky and other backgrounds.

[0061] like Figure 1 As shown, the method includes:

[0062] start,

[0063] S1, collect basic data;

[0064] S2, basic data conversion;

[0065] S3, data cleaning;

[0066] S4, prepare to copypaste data offline;

[0067] S5, prepare online copypaste data;

[0068] S6, start training;

[0069] Determine whether to copypaste online? If yes, proceed to step S6.1, then proceed to step S6.2; if no, proceed directly to step S6.2;

[0070] S6.1, online copypaste;

[0071] S6.2, determine whether online mosaic is required. If yes, perform online mosaic and proceed to step S7. If no, proceed directly to step S7.

[0072] S7, training iteration.

[0073] Furthermore, the main implementation steps of the method are as follows:

[0074] Step S1. Collect basic data

[0075] Get human skin classes and other background classes in the public human parsing dataset, and get sky class data and other background classes in the public sky dataset.

[0076] Step S2. Basic data conversion

[0077] The human skin parts including face, hands, arms, legs, etc. in the human body analysis data set are regarded as human skin classes, and the non-human skin parts such as hats, tops, pants, etc. are regarded as other background classes. The human skin classes and other background classes in the human body analysis data set are specifically operated as shown in formula (1);

[0078]

[0079] Among them, i and j are the image pixel coordinates respectively. The task of this application is to solve the semantic segmentation problem: that is, to classify each pixel of the image, L a (i, j) is the human body analysis and annotation data, L s (i, j) is the converted human skin data, 0 is other background classes, 1 is the human skin class, index_list is the index list of the human skin parts of the hands, arms, and legs, and k is the human skin part of the human body parsing annotation data; k is in the table, and the public human body parsing data is a table like this: 1: Hat, ..., 13: Face, 14: Left-arm, 15: Right-arm, 16: Left-leg, 17: Right-leg..., the table number of each data set may be different, such as some Face numbers may be 1, and the classification may also be different, such as some Face parts include Neck and some do not. This application focuses on the human skin class, and the class belonging to human skin is index_list. In the above example, index_list is 13, 14, 15, 16);

[0080] The sky data set is used to obtain sky category data and other background categories. The specific operation is shown in formula (2):

[0081]

[0082] Among them L sky (i, j) is the sky label data, I b (i, j) is the blue sky label data after conversion, 0 is other background classes, 1 is blue sky class, I hsv (h, w, c) is HSV data, and thershold_blue is the selected blue HSV color threshold range.

[0083] Step S3. Data cleaning

[0084] To ensure data accuracy, the data after the conversion needs to be cleaned to ensure that the existing human skin data does not contain blue sky, and the blue sky data does not contain human skin. This further includes:

[0085] S3.1. Use human skin data to train a separate human skin segmentation model skin , use the blue sky data to train a separate blue sky segmentation model model bluesky ;

[0086] S3.2. Using the model skin Model Screening Blue Sky Data I b (i,j) gets R b (i,j), define w, h as R b The width and height of (i,j) are b (i, j)) < = 0.9*w*h, it means that the current blue sky data image contains more than 10% of human skin, and the current blue sky data is directly removed. If sum(R b If (i,j))>0.9*w*h, the current blue sky data is manually screened to see if it contains human skin. If so, it is removed; otherwise, it is retained.

[0087] S3.3. Using the model bluesky Model Screening Human Skin Data I s (i,j) gets R s (i,j), define w, h as R s The width and height of (i,j) are s (i, j)) < = 0.9*w*h, it means that the current human skin data image contains more than 10% of blue sky, and the current human skin data is directly eliminated. If sum(R s(i,j))>0.9*w*h, the current skin data is manually screened to see if it contains blue sky. If so, it is removed; otherwise, it is retained.

[0088] Step S4. Prepare offline copypaste data

[0089] S4.1 For the existing human skin image I s (i, j) and label L s (i, j), firstly, a high-precision human body segmentation algorithm is used. Here, Alibaba Cloud human body segmentation SegmentBody can be used to obtain an image I containing only the human body part. p (i, j) and L p (i, j), there is the following relationship I p (i, j)∈I s (i, j), L s (i, j)∈L p (i, j).

[0090] S4.2 with blue sky image I b (i, j) is the background image used as offline copypaste, and the human body image I p (i, j) is randomly scaled and flipped and then pasted onto the blue sky image I b (i, j), while L p (i, j) performs corresponding operations to L b (i, j), we can get data containing skin, blue sky and other backgrounds at the same time, making the data more balanced;

[0091] Step S5. Prepare online copypaste data

[0092] For L in step S4 p (i, j) performs edge segmentation, divides the data into the minimum circumscribed rectangle of each person's body, and obtains several human body images I' p (i, j) and human body label L′ p (i, j), and also obtain several personal human skin labels L′ sp (i, j).

[0093] Step S6. Training process

[0094] S6.1 During the training process, online copypsate is performed with a certain probability, randomly scaling and flipping a single human body and pasting it onto the current data to expand data diversity;

[0095] During S6.2 training, perform online mosaicing with a certain probability. First, determine the mosaic base image size. Then, process the current data according to the mosaic base image size and fill it into the first position of the mosaic. The other three positions are randomly filled with data according to the corresponding processing.

[0096] The certain probability mentioned above is as follows: the iterative process generates a random number between 0 and 1, and copypaste is performed when the random number is less than 0.25, that is, the probability is 0.25. During the training process, online copypaste and online mosaic are all performed using a probability of 0.25 by default.

[0097] The step S6.2 further includes: determining the mosaic reference image size. For example, if the reference image size is 512x512, the mosaic defaults to a 2x2 grid format, which means that four 256x256 image data should be filled in. However, the original image data is not necessarily 256x256 in size, and needs to be randomly scaled and cropped to 256x256 in size and then spliced ​​into the mosaic reference size.

[0098] Step S7: Obtain a training model.

[0099] The method is targeted at semantic segmentation, which requires segmenting specific tasks: human skin, blue sky, and other backgrounds.

[0100] In addition, as shown in Figures 2(a) and 2(b), there are schematic diagrams of the mosaic principle in the method of the present invention: Figure 2(a) shows four original images that were randomly selected, and Figure 2(b) shows the image after mosaic.

[0101] like Figure 3 The figure shows a schematic diagram illustrating the copypaste principle in the method of the present invention: from left to right, the original human skin image, the Alibaba Cloud human segmentation result, the background image, and the copypaste image.

[0102] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for solving the serious imbalance of semantic segmentation data for a specific task, characterized by: The method comprises the following steps: S1, collect basic data; S2, basic data conversion; S3, data cleaning; S4, prepare for offline copy and paste data enhancement; S5, prepare online copy and paste data enhancement; S6, start training; Determine whether to copypaste online? If yes, proceed to step S6.1, then proceed to step S6.2; If not, proceed directly to step S6.2; S6.1, online copypaste; S6.2, determine whether online mosaic data enhancement is required. If yes, perform online mosaic and proceed to step S7; if no, proceed directly to S7; S7, obtain the training model: training iteration.

2. A method for solving the serious imbalance of semantic segmentation data for a specific task according to claim 1, characterized in that: The step S1 further comprises: Obtain human skin classes and other background classes from the public human parsing dataset, and obtain sky class data and other background classes from the public sky dataset; The human skin class in the human body analysis data set is the part of the human skin including the face, hands, arms and legs in the human body analysis data set as the human skin class; Other background classes in the human body analysis dataset include non-human skin parts such as hats, tops, and pants. The blue sky class of the sky dataset is to perform color space conversion on the sky category image data portion in the sky dataset, convert RGB to HSV space, perform color segmentation in the HSV space, set the required blue sky color threshold range, and use the blue in the sky as the blue sky class; other colors are used as other background classes of the sky dataset.

3. The method for solving the serious imbalance of semantic segmentation data for a specific task according to claim 2, characterized in that: The basic data conversion in step S2 further includes: The human skin class and other background classes in the human body analysis dataset are specifically operated as shown in formula (1); Among them, i, j are the image pixel coordinates, L a (i, j) is the human body analysis and annotation data, L s (i, j) is the converted human skin data, 0 is other background classes, 1 is human skin classes, index_list is the index list of the human skin part annotations of the hands, arms, and legs, and k is the human skin part of the human body parsing annotation data; The sky data set is used to obtain sky category data and other background categories. The specific operation is shown in formula (2): Among them L sky (i, j) is the sky label data, L b (i, j) is the blue sky label data after conversion, 0 is other background classes, 1 is blue sky class, I hsv (h, w, c) is HSV data, and thershold_blue is the selected blue HSV color threshold range.

4. The method for solving the serious imbalance of semantic segmentation data for a specific task according to claim 3, characterized in that: The step S3 data cleaning: In order to ensure data accuracy, the converted data needs to be cleaned to ensure that the existing human skin data does not contain blue sky, and the blue sky data does not contain human skin.

5. The method for solving the serious imbalance of semantic segmentation data for a specific task according to claim 4, characterized in that: The step S3 further comprises: S3.1, use human skin data to train a separate human skin segmentation model model skin , use the blue sky data to train a separate blue sky segmentation model model bluesky ; S3.2, use model skin Model Screening Blue Sky Data I b (i, j) gets R b (i, j), define w, h as R b The width and height of (i, j) are b (i, j)) <= 0.9*w*h means that the current blue sky data image contains more than 10% of human skin, and the current blue sky data is directly removed. If sum(R b If (i, j) > 0.9*w*h, the current blue sky data is manually screened to see if it contains human skin. If so, it is removed; otherwise, it is retained. S3.3, use model bluesky Model Screening Human Skin Data I s (i, j) gets R s (i, j), define w, h as R s The width and height of (i, j) are s (i, j)) <= 0.9*w*h means that the current human skin data image contains more than 10% of blue sky, and the current human skin data is directly eliminated. If sum(R s (i, j))>0.9*w*h, the current skin data is manually screened to see if it contains blue sky. If so, it is removed; otherwise, it is retained.

6. The method for solving the serious imbalance of semantic segmentation data for a specific task according to claim 4, characterized in that: The step S4 further comprises: S4.1, for the existing human skin image I s (i, j) and label L s (i, j), firstly, a high-precision human body segmentation algorithm is used to obtain an image I containing only the human body part. p (i, j) and L p (i, j), there is the following relationship I p (i, j)∈I s (i, j), L s (i, j)∈L p (i, j); S4.2, with blue sky image I b (i, j) is used as the background image for offline copypaste, and the human body image I p (i, j) is randomly scaled and flipped and then pasted onto the blue sky image I b (i, j), while L p (i, j) performs corresponding operations to L b (i, j), that is, data containing skin, blue sky and other backgrounds are obtained, making the data more balanced.

7. The method for solving the serious imbalance of semantic segmentation data for a specific task according to claim 6, characterized in that: The step S5 further comprises: For L in step S4 p (i, j) performs edge segmentation, divides the data into the minimum circumscribed rectangle of each person's body, and obtains several human body images I' p (i, j) and human body label L′ p (i, j), and also obtain several personal human skin labels L′ sp (i, j).

8. The method for solving the serious imbalance of semantic segmentation data for a specific task according to claim 1, characterized in that: The training process in step S6 further includes: S6.1, during the training process, perform online copypaste with a certain probability, randomly scale and flip a single human body and paste it onto the current data to expand data diversity; S6.

2. During the training process, online mosaicing is performed with a certain probability. First, the mosaic base image size is determined. The current data is processed according to the mosaic base image size and then filled into the first position of the mosaic. The other three positions are randomly filled with data according to the corresponding processing. The certain probability mentioned above is as follows: the iterative process generates a random number between 0 and 1, and copypaste is performed when the random number is less than 0.25, that is, the probability is 0.25; during the training process, both online copypaste and online mosaic are performed using a probability of 0.25 by default.

9. The method for solving the serious imbalance of semantic segmentation data for a specific task according to claim 1, characterized in that: The step S6.2 further includes: determining that the mosaic reference image size is 512x512. Mosaic uses a 2x2 grid format by default, which means that four 256x256 image data should be filled in. However, the original image data is not necessarily 256x256 in size, and needs to be randomly scaled and cropped to 256x256 in size and spliced ​​into the mosaic reference size.

10. The method for solving the serious imbalance of semantic segmentation data for a specific task according to claim 1, characterized in that: The method is targeted at semantic segmentation, which requires segmenting specific tasks: human skin, blue sky, and other backgrounds.

Images