A data enhancement method and system suitable for small sample learning tasks of target detection algorithm

Abstract

The present application relates to the technical field of target detection based on deep learning, and more particularly to a data enhancement method and system suitable for small sample learning task of target detection algorithm. In order to solve the defect that the traditional target detection algorithm will gradually fail with the decrease of the number of data set in the small sample learning problem, the present application provides a data enhancement method and system suitable for small sample learning task of target detection algorithm. The method of the present application is based on the target detection data set expansion method of copy and paste algorithm and U2Net image segmentation tool. The instances of the detection target are cut, copied, enhanced, pasted and other operations to expand the training sample set. At the same time, the pre-training U2Net image segmentation algorithm is used to remove the background of the target instance, so that the target instance can be integrated into the new image scene, and finally the number of labeled instances and the diversity of samples are improved simultaneously.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection based on deep learning, and particularly relates to a data enhancement method and system suitable for small sample learning task of target detection algorithm. BACKGROUND

[0002] Data augmentation is the most direct way to improve data utilization efficiency and is a key component in various deep learning algorithms. In the field of target detection, commonly used data augmentation methods include basic algorithms such as image affine transformation and cutting, as well as Mosaic, MixUp and other methods integrated into the target detection algorithm framework. These algorithms are more general in nature and are not specifically designed for small sample learning problems, so these algorithms will gradually fail as the number of data sets decreases. For small sample learning scenarios, the traditional copy and paste algorithm cannot delete the background of the target instance, and the enhancement effect of the augmented data is not good, and it is difficult to achieve simultaneous improvement of the number of instances and sample diversity. SUMMARY

[0003] To solve the above problems, the present application provides a data enhancement method and system suitable for small sample learning task of target detection algorithm, which aims to expand the training sample set through cutting, copying, enhancing and pasting of detection target instances, and finally realize the simultaneous improvement of the number of labeled instances and sample diversity.

[0004] A data enhancement method suitable for small sample learning task of target detection algorithm, comprising the following steps:

[0005] S1, count the number of labeled instances of each class in the training data, and define the head class and the tail class according to the number;

[0006] S2, start a round of training, read the training data in order, define the current read training data as augmented data, and randomly select another training data as source data;

[0007] S3, change the image of the augmented data by random cropping, random rotation, random scaling and random cutting, and modify its label information;

[0008] S4, crop the labeled instances from the image of the source data using the label information of the source data, and record all the labeled instances as a source instance image set I, and according to the number of labeled instances of each class, record the instance set belonging to the head class in the source instance image set I as I L , and record the instance set belonging to the tail class as I S ;

[0009] S5, change the I Sexample images;

[0010] S6, randomly sampling the I L instances in the predetermined number of instances, and deleting the unselected instances;

[0011] S7, using U2Net to detect and identify the foreground and background of all instance images obtained in steps S5 and S6, and to remove the background of each instance image; S and I L

[0012] S8, pasting the instance images with the removed background into the data to be enhanced, and modifying the corresponding annotation information;

[0013] S9, repeating steps S2 to S8 until all training data in the current round are traversed;

[0014] S10, using the enhanced training data set to train the model, completing the current training round, and determining whether the training round reaches the preset number of times, if yes, ending the training, otherwise, returning to step S2.

[0015] Further, the method for defining the head class and the tail class in step S1 is to count the number of annotated instances of all classes in the training data, determine whether the number of annotated instances of a certain class is less than the preset threshold of the number of annotated instances of the class with the maximum number of annotated instances, if yes, define the class as a tail class, otherwise, define it as a head class.

[0016] Further, the training data includes training images, training image scales, training image formats, training image addresses, and annotation information. The annotation information includes: detection / segmentation task type, annotation attribute, annotation class, annotation box coordinate, and annotation box class.

[0017] Further, the random cropping step defines its cropping method by four random values R1, R2, R3, and R4, wherein R1 and R2 are Boolean values 0 or 1, R1=0,1 respectively represents that the rectangular instance starts to crop from the left and right direction; R2=0,1 respectively represents that the rectangular instance starts to crop from the top and bottom; R3 and R4 are random decimal values in (0,1), which are used to define the cropping range, wherein R3 defines the horizontal direction of the cropping range, and the cropping range is equal to the width of the current instance multiplied by R3; R4 defines the vertical direction of the cropping range, and the cropping range is equal to the height of the current instance multiplied by R4.

[0018] Further, the image enhancement sequence in step S5 is fixed, and is in turn random cropping, random rotation, random scale, and random cutting.

[0019] Further, in step S7, the I L and I S ​The foreground and background specific method of all instances is as follows: a U2Net model trained through a COCO data set is selected, a modified source instance image is input, the pre-trained U2Net is used to perform target segmentation on the modified source instance image, the background area in the image is marked as 0, and the rest is marked as 1, and the instance graph and the corresponding segmentation graph are saved, wherein the segmentation graph is a binary graph.

[0020] Further, the step S8 specifically comprises:

[0021] Before pasting, firstly, it is judged whether the current source instance image is overlapped on a certain instance image of the image of the data to be enhanced, if there is no overlap between the instance images, the corresponding region in the image of the data to be enhanced is replaced by the instance image;

[0022] If the current instance image is overlapped with the instance image in the image of the data to be enhanced, and the number of overlapped pixel points is greater than a preset proportion of the total number of pixel points of the current instance image, the above pasting operation is cancelled;

[0023] If the number of overlapped pixel points is less than or equal to the preset proportion of the total number of pixel points of the current instance image, only the non-overlapping region is replaced by the instance image;

[0024] If the current instance image exceeds the range of the image of the data to be enhanced, only the part of the current instance image located in the range of the image of the data to be enhanced is kept;

[0025] When pasting, the covered instance image is deleted, the source instance image is added, and the labeling information of the source instance and the image of the data to be enhanced is modified.

[0026] The application also provides a data enhancement system suitable for a small sample learning task of a target detection algorithm, comprising:

[0027] The traversal module is used for traversing the training data set, and dividing the training data into a head class and a tail class according to the number of labeled instances of each class;

[0028] The training round start module is used for starting the current training round;

[0029] The reading module is used for reading one training data as the data to be enhanced, and simultaneously randomly selecting another training data as the source data;

[0030] The enhancement module is used for enhancing the image of the data to be enhanced and the tail class instance image in the source data through an image enhancement method;

[0031] The clipping module is used for clipping the instance image in the source data;

[0032] The selection module is used for selecting the head class instance image in the source data;

[0033] A background elimination module identifies and removes the background of the head-class instance image and the tail-class instance image through a U2Net target segmentation algorithm;

[0034] A pasting module is configured to paste the head-class instance image and the tail-class instance image into the to-be-enhanced data image.

[0035] A training round ending module is configured to end the current training round and start the next training round.

[0036] Compared with the prior art, the present application has the following advantages:

[0037] 1. The present application can realize the synchronous improvement of the number of labeled instances and the diversity of samples by expanding the training sample set through the copy and paste algorithm based on the copy and paste algorithm for labeling instances, cutting, enhancing, segmenting, and pasting operations;

[0038] 2. The present application uses the U2Net image segmentation algorithm to remove the background of the target instance, so that the target instance can be integrated into a new scene, significantly improving the diversity of instances;

[0039] 3. The present application adopts different enhancement strategies for the training instances of the head class and the tail class according to the sample categories to realize the manual balancing of the training data set.

[0040] 4. The algorithm framework is only for the data enhancement link, so it can be embedded into any target detection framework, and has both innovation and practicality. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The present application is a statistical diagram for the number of labeled instances of each category in the training data of step S1.

[0042] Figure 2 The present application is an operation diagram for reading and randomly extracting training data in step S2.

[0043] Figure 3 The present application is a process diagram for using a pre-trained U2Net to identify the foreground and background of the instance in step S7. L Or the process diagram for removing the background of the instance in I S

[0044] Figure 4a The present application is a comparison diagram of the method and the original copy and paste method. DETAILED DESCRIPTION

[0045] The present application will be further described below in conjunction with the drawings and examples.

[0046] Example 1 ​

[0047] The application provides a data enhancement method suitable for a small sample learning task of a target detection algorithm, which expands a training sample set through operations such as copying, enhancing and pasting of detection target instances, and realizes synchronous improvement of the number of labeled instances and sample diversity.

[0048] A data enhancement method suitable for a small sample learning task of a target detection algorithm, comprising:

[0049] S1, traverse the training data set, and count the number of labeled instances of each class. The head class and the tail class in the training data set are defined by the number of instances, wherein the head class has more samples and the tail class has fewer samples.

[0050] Specifically, the class definition work in step 1 can be realized by setting a soft threshold. For example, after counting the number of labeled instances of all classes, if the number of labeled instances of a certain class is less than 5% of the number of labeled instances of the class with the largest number of labeled instances, then the class is defined as a tail class, otherwise it is defined as a head class. Figure 1 As shown in the figure, in this scenario, the number of instances of the box is the largest, with 712 instances. The class with the number of instances less than 5% is determined as a small sample class (tail class), and the others are determined as large sample classes (head classes), such as billboard, table, etc., which are all determined as small sample classes (tail classes).

[0051] S2, the model starts the current training round; read the training data in order, and take the current read training data as the data to be enhanced, and randomly select another training data in the training data set as the source data;

[0052] Specifically, each training data in step S2 includes a training image, a training image scale, a training image format, a training image address and labeling information, wherein the labeling information includes: a detection / segmentation task type, a labeling attribute, a labeling class, labeling box coordinates (in order: left upper corner, right upper corner, right lower corner, left lower corner) and a labeling box class. As shown in the figure, Figure 2 As shown in the figure, it is an operation diagram for reading training data and randomly extracting training data. The training network first reads the image of the training data and the corresponding labeling file, and saves it as a labeling information in the python dictionary. Each time the training data is read, another training data in the training data set is randomly selected.

[0053] S3, perform enhancement operations on the image of the data to be enhanced, including random cropping, random rotation, random scaling, random shear transformation, and modify the labeling information; wherein the order of each operation is not limited.

[0054] Specifically, the random clipping step S3 defines the clipping method by four random values R1, R2, R3, R4, wherein R1, R2 are Boolean values 0 or 1, R1 = 0, 1 respectively represents that the rectangular instance starts clipping from the left or right direction; R2 = 0, 1 respectively represents that the rectangular instance starts clipping from the top or bottom; R3, R4 are random decimal values within (0, 1) and are used to define the clipping range; wherein R3 defines the clipping range in the horizontal direction, and the clipping range is equal to the width of the current instance multiplied by R3; R4 defines the clipping range in the vertical direction of the instance, and the clipping range is equal to the height of the current instance multiplied by R4; the random rotation angle in the random rotation operation process is a random value within (-10°, 10°); the random scale value in the random scale transformation process is a random value within (0.5, 1.5), when the value is within (0, 1), it means that the instance is reduced; when the value is within (1, 1.5), it means that the instance is enlarged; the random shear parameter value in the random shear transformation process is a random value within (-2, 2).

[0055] S4, clip the labeled instances from the images of the source data respectively using the labeling information of the source data, and record all the labeled instances as a source instance image set I; and according to the number of labeled instances of each category, set the instance set I belonging to the head category as I L , and set the instance set I belonging to the tail category as I S .

[0056] S5, sequentially perform random clipping, random rotation, random scale, and random shear transformation on all instances in the tail category instance set I S in S4.

[0057] In particular, the image enhancement order in step S5 is fixed. The order is random clipping, random rotation, random scale, and random shear transformation; specifically, the parameter value ranges of random clipping, random rotation, random scale, and random shear transformation are the same as those in step S3.

[0058] S6, randomly draw a predetermined number of instances from the head category instance set I L in S4, and delete the instances not drawn.

[0059] Specifically, considering that the number of head category instances is much larger than that of tail category instances, a predetermined number of instances are drawn from the head category instance set I L . Wherein, the predetermined number is determined by the number of head category instances in the source data and a custom parameter N, wherein N is an integer and the value of N is freely set according to the capacity of the training data set, and the value of the predetermined number is the maximum of the number of head category instances in the current source data and the custom parameter N.

[0060] S7, detect and identify I L and IS foreground and background of all instance images, and remove the background of each instance image;

[0061] Since the target detection algorithm annotation information is a rectangular box, each instance image contains not only the instance target to be detected, but also a large amount of background; for example Figure 3 As shown in FIG. 5, a pre-trained U2Net is used to identify I L or I S The process of foreground and background of each instance and removing the background is shown in FIG. 6.

[0062] Specifically, the pre-trained U2Net can use a U2Net model trained by a COCO dataset; the input of the U2Net is the source instance image, and considering that part of the instance size is small, the input size of the U2Net during detection is adjusted to [200, 200]; the pre-trained U2Net is used to perform target segmentation on all instances obtained in steps S5 and S6, mark the foreground target and the background in the image; the target segmentation result will mark the background area in the image as 0, and the rest as 1; save the instance image and the corresponding segmentation image, wherein the segmentation image is a binary image composed of (0, 1), delete the 0 value part (i.e. the background part) of the segmentation image corresponding to each instance image;

[0063] Since the source instance image only contains the target to be detected and the background around it, the pre-trained model can basically identify the objects in the instance image, and the probability of misidentification is low.

[0064] S8, paste I L and I S to the data to be enhanced, and modify the annotation information of the data to be enhanced. The enhancement of the data to be enhanced is completed;

[0065] Specifically, a random position is defined for each source instance image to be pasted, assuming that the width and height of a specific instance are W and H respectively, an initial point (X1, Y1) is first randomly positioned in the data to be enhanced, and then the upper left corner point of the instance in the annotation information is modified to (X1, Y1), the upper right corner point is (X1+W, Y1), the lower right corner point is (X1+W, Y1+H), and the lower left corner point is (X1, Y1+H);

[0066] Specifically, before pasting the instance image to the data image to be enhanced, firstly, it is judged whether the current instance covers a certain instance of the data to be enhanced, if there is no overlap between the instances, the corresponding region in the data to be enhanced is replaced by the instance image (non-0 part); if the current instance image overlaps with the instance image of the data to be enhanced, and the number of overlapping pixel points is greater than 50% of the total number of pixel points of the current instance image, the above pasting operation is cancelled; if the number of overlapping pixel points is less than or equal to 50% of the total number of pixel points of the current instance image, only the non-overlapping region is replaced by the instance image (non-0 value part); the instance image exceeding the range of the data image to be enhanced is cropped, only the part of the current instance image located in the range of the data image to be enhanced is retained;

[0067] When pasting, the instance image covered is deleted, the source instance image is added, and the labeling information of the source instance and the data image to be enhanced is modified.

[0068] S9, repeat the image enhancement process of steps S2 to S8 until all training data of the current round are traversed to generate an enhanced training data set;

[0069] S10, training the model by using the enhanced training data set, completing the current training round, judging whether the training round reaches a preset number of times, which can be 3 times in this embodiment, if yes, ending the training, otherwise, returning to step S2.

[0070] As shown in Figure 4a , b is a comparison diagram of the method and the original copy and paste method, from the comparison diagram, it can be seen that the original copy and paste method does not change the background of each instance after enhancing the data image to be enhanced, except that the number of instances increases, the generalization is not enhanced. The method makes the background of each instance change, not only increases the number of instances, but also enriches the diversity of samples.

[0071] Embodiment 2

[0072] Based on the method in embodiment 1, the embodiment of the application further provides a data enhancement system suitable for a small sample learning task of a target detection algorithm, the system comprises:

[0073] A traversal module is configured to traverse the training data set and divide the training data into a head class and a tail class according to the number of labeled instances of each class.

[0074] A training round starting module is configured to start the current training round.

[0075] A reading module is configured to read one training data as data to be enhanced and simultaneously randomly select another training data as source data.

[0076] An enhancement module is configured to enhance the tail class instance images in the source data and the data image to be enhanced by an image enhancement method;

[0077] A cropping module is configured to crop the instance images in the source data;

[0078] A sampling module is configured to sample the head class instance images in the source data;

[0079] A background elimination module is configured to identify and delete the backgrounds of the head class instance images and the tail class instance images by a U2Net target segmentation algorithm;

[0080] A pasting module is configured to paste the head class instance images and the tail class instance images into the data image to be enhanced;

[0081] A training round ending module is configured to end the current training round and start the next training round.

[0082] Application case: In a certain municipal management business scenario, the customer side of a telecom company needs to detect whether a street vendor has cross-door business. The relevant target detection objects include tables, chairs, gas stoves, billboards, express boxes and express frames. Due to the requirement of customer data protection, the customer side provides less data, which belongs to a small sample scenario. Under the premise of using the same target detection model framework and other unchanged conditions, the mAP value of the model trained by using a traditional data enhancement algorithm is 89%. The data is enhanced by using the method and system provided by the present application, and the target detection model is trained, and the obtained mAP value is improved to 94.1%, which shows that the data enhancement algorithm can significantly improve the application ability of the target detection model.

[0083] The above-described embodiments only express the preferred embodiments of the present application, and the description is more specific and detailed, but it should not be understood as limiting the scope of the patent of the present application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications, improvements and substitutions can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A data augmentation method suitable for small sample learning tasks of object detection algorithms, characterized in that, The method comprises the following steps: S1, counting the number of labeled instances of each category in the training data, and defining the head class and the tail class according to the number; S2, starting a round of training, reading the training data in order, defining the currently read training data as the data to be enhanced, and randomly selecting another training data as the source data; S3, changing the image of the data to be enhanced by random cropping, random rotation, random scaling and random cutting, and modifying the label information thereof; S4, cutting the labeled instances from the images of the source data using the labeling information of the source data, recording all the labeled instances as a source instance image set I, and recording the instance set belonging to the head category in the source instance image set I as I L ; The instance set belonging to the tail class is denoted as I S ; S5, change the I by random cropping, random rotation, random scale and random cutout S example image; S6、Randomly select the I L instances, delete the unselected instances. S7, using U2Net to detect and recognize the I S and I L foreground and background of all instance images, and eliminate the background of each instance image; S8, pasting the instance image excluding the background into the data to be enhanced, and modifying the corresponding label information; S9, repeating steps S2 to S8 until all training data in the current round is traversed; S10, training the model using the enhanced training data set, completing the current training round, and judging whether the training round reaches the preset number, if yes, ending the training, otherwise, returning to step S2.

2. The data augmentation method suitable for small sample learning task of target detection algorithm according to claim 1, characterized in that, The method for defining the head class and the tail class in step S1 is to count the number of labeled instances of all categories in the training data, judge whether the number of labeled instances of a certain category is less than the preset threshold of the number of labeled instances of the category with the largest number of labeled instances, if yes, define the category as the tail class, otherwise, define it as the head class.

3. The data augmentation method for small sample learning task of target detection algorithm according to claim 1, characterized in that, The training data includes training images, training image scales, training image formats, training image addresses and label information, wherein the label information includes detection / segmentation task type, label attribute, label category, label box coordinates and label box category.

4. The data augmentation method suitable for small sample learning task of target detection algorithm according to claim 1, characterized in that, The random cropping step defines its cropping method by four random values R1, R2, R3 and R4, wherein R1 and R2 are Boolean values 0 or 1, R1=0,1 respectively represents that the rectangular instance starts to crop from the left or right direction; R2=0,1 respectively represents that the rectangular instance starts to crop from the top or bottom; R3 and R4 are random decimal values within (0,1) and are used to define the cropping range, wherein R3 defines the horizontal direction cropping range, and the cropping range is equal to the width of the current instance multiplied by R3; R4 defines the vertical direction cropping range, and the cropping range is equal to the height of the current instance multiplied by R4.

5. The data augmentation method suitable for small sample learning task of target detection algorithm according to claim 1, characterized in that, The image enhancement sequence in step S5 is fixed, and is in turn random cropping, random rotation, random scaling and random cutting.

6. The data augmentation method suitable for small sample learning task of target detection algorithm according to claim 1, characterized in that, The step S7 identifies I L and I S The specific method for foreground and background of all instances in the source instance image is as follows: a U2Net model trained through a COCO dataset is selected, the modified source instance image is input, the pre-trained U2Net is used to perform target segmentation on the modified source instance image, the background area in the image is marked as 0, and the rest is marked as 1, and the instance image and the corresponding segmentation image are saved, wherein the segmentation image is a binary image.

7. The data augmentation method suitable for small sample learning task of target detection algorithm according to claim 1, characterized in that, The step S8 specifically comprises: Before pasting, first judge whether the current source instance image covers a certain instance image in the image of the data to be enhanced, if there is no overlap between the instance images, replace the corresponding area in the image of the data to be enhanced with the instance image; If the current instance image overlaps with the instance image in the image of the data to be enhanced, and the number of overlapping pixels is greater than the preset proportion of the total number of pixels of the current instance image, cancel the above pasting operation; If the number of overlapping pixels is less than or equal to the preset proportion of the total number of pixels of the current instance image, only replace the non-overlapping area with the instance image; If the current instance image exceeds the image range of the data to be enhanced, only keep the part of the current instance image located within the image range of the data to be enhanced; When pasting, delete the covered instance image, add the source instance image, and modify the label information of the source instance and the data to be enhanced image.

8. A data augmentation system suitable for small sample learning task of object detection algorithm, for implementing the method of any one of claims 1-7, comprising: a traversal module for traversing the training dataset and dividing the training data into head classes and tail classes according to the number of labeled instances of each class; a training round start module for starting the current training round; a reading module for reading one training data as the data to be augmented and simultaneously randomly selecting another training data as the source data; an augmentation module for augmenting the image of the data to be augmented and the tail class instance image in the source data through image augmentation methods; a cropping module for cropping the instance image in the source data; a selection module for selecting the head class instance image in the source data; a background removal module for identifying and deleting the background of the head class instance image and the tail class instance image through a U2Net object segmentation algorithm; a pasting module for pasting the head class instance image and the tail class instance image into the data to be augmented image; a training round end module for ending the current training round and starting the next training round.

Images