A deep learning image enhancement method based on annotation box splicing

Abstract

A deep learning image enhancement method based on annotation frame splicing includes the following steps: step 1, select N pictures, scale the N pictures to the same size, and prepare a blackboard picture of the same size; step 2, randomly sort several pictures, and randomly determine a splicing point in the picture; step 3, crop the N pictures and the corresponding parts of each picture according to this ratio; step 4, filter the annotation frame; step 5, scale, transform, and rotate the filtered cropped area; step 6, repeat steps 1 to 5. The present invention introduces a method of semantic segmentation and size discrimination, and identifies the spliced ​​annotation frame of the noise generated by the cropping data enhancement algorithm to ensure the quality of the data generated by the splicing data enhancement method, and improve the benefits of the enhanced data for the model performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of image enhancement, and in particular relates to a deep learning image enhancement method based on annotation frame splicing. Background Art

[0002] At present, the splicing data augmentation algorithm has attracted much attention in the industry due to its obvious effect and low cost. The main idea is to crop a part of the target in multiple existing annotated frame images, and then splice them into a new state, and mark each cropped annotated frame as the original target type, and train the deep learning network to identify the difference between different targets. For example, algorithms such as Cutout, cutmix[3] and mosaic[4]. Cutout increases the complexity of the data by randomly deleting a part of the target frame, on the one hand, by simulating the occlusion problem. On the other hand, Cutout can make the model focus on the area where the target is difficult to distinguish, thereby improving the performance of the model. Cutmix is ​​improved on the basis of cutout, because the annotated frame without any information left after cutout will affect the training efficiency. Cutmix fills the empty part after cutting to improve the training efficiency.

[0003] Mosaic is a technique that randomly crops and scales multiple images and fuses them into a new image to form a new annotation box. Mosaic technology enriches the background information of the target by randomly cropping; scaling and transforming to simulate objects at different scales. Data enhancement algorithms are widely used in the field of deep learning target detection. Target detection is a hot topic in computer vision and digital image processing. It is widely used in many fields such as robot navigation, intelligent video surveillance, industrial inspection, aerospace, etc. It has important practical significance to reduce the consumption of human capital through computer vision. Target detection is also a challenging task. It not only needs to identify the category but also needs to find the location of the target. The production and annotation of data sets are costly. While expanding the data, data enhancement algorithms can generate a large amount of labeled data through a small amount of labeled data. It is of great significance in actual production.

[0004] In data enhancement, cropping methods such as cutout, cutmix, and mosaic inevitably introduce noise in actual operation (noise: the target annotation box contains a small amount of messy information or a small amount of target information, but is not enough to represent the target). Taking the mosaic algorithm as an example, during the splicing process, the target box is merged into the new image along with the cropped area. In the process of cropping the target box, it is inevitable that a small part will be regarded as the target (for example, a narrow strip in the picture is marked as a human type). Finally, this unrecognizable target box increases the difficulty of network training, and ultimately affects the poor performance of the network.

[0005] The selected larger cropped area is the non-main area where the target is located. Because the cropped area is too large and is within the target frame and does not contain the target body itself, the ordinary screening method cannot filter out the annotation box of such targets, resulting in background information being trained as a target, thereby introducing noise. Summary of the invention

[0006] The purpose of the present invention is to provide a deep learning image enhancement method based on annotation box splicing to solve the above problems.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A deep learning image enhancement method based on annotation frame splicing includes the following steps:

[0009] Step 1: select N images with masks of objects to be identified, and scale the N images and their masks to the same size;

[0010] Step 2, randomly sort the N pictures and select M pictures in sequence;

[0011] Step 3, randomly select one or more stitching points within the image area, and the stitching points divide the image area into M areas;

[0012] Step 4: Select one area from the M areas segmented in step 3, and stitch the selected areas of the M images selected in step 2 in order to form a complete new image;

[0013] Step 5: for the new image formed in step 4, the size, area, proportion, and whether the mask is included in the annotation box of the new image are used to select the annotation box in the new image;

[0014] Step 6: For the invalid annotation boxes, discard them, repeat step 3, or adjust the splicing area of ​​the selected pictures;

[0015] Step 7: Repeat steps 1 to 5 until all N images are enhanced.

[0016] Furthermore, the adjustment in step 6 is to randomly generate a new splicing area.

[0017] Furthermore, in step 5, the method of screening the annotation box in the new image includes one or more of the following methods:

[0018] (1) Determine whether the area of ​​the smallest region or the shortest side of all the annotation boxes is less than a threshold. If so, determine that the annotation box is an invalid one;

[0019] (2) Obtain the mask of each annotation box, select the part to be spliced ​​in each picture, cut out the same position on the mask corresponding to the picture, and judge the N annotation boxes in the cutout position in turn.

[0020] Furthermore, method (2) uses one or more of the following methods:

[0021] (1) Determine whether the annotation box contains the mask part of the new image. If it does not contain it at all, it is determined to be an invalid annotation box;

[0022] (2) The area of ​​the mask containing the new image in this annotation box, if the area is smaller than the threshold, it will be discarded or returned to step 3;

[0023] (3) Calculate the ratio between the new image mask contained in the annotation box and the total mask area, and determine that the annotation box with a value less than the threshold is an invalid annotation box;

[0024] (4) Set a threshold. When the shortest distance from the stitching point to the edge of the mask is less than a threshold, or it is considered that the cropped image cannot represent the characteristics of the object, it is judged as an invalid annotation box.

[0025] Furthermore, in step 6, the stitching area of ​​the selected image is adjusted by translating the position of the area in the original image to obtain a valid annotation box, and the moving method is to move the area in the original image in the opposite direction relative to the stitching point in the image, and the moving distance is the threshold.

[0026] Furthermore, in step 3, the selection range of the splicing points is limited to a ratio of the width to the height of the image within a range of 0.2 to 0.8.

[0027] Compared with the prior art, the present invention has the following technical effects:

[0028] The present invention introduces the method of semantic segmentation and mask discrimination to identify the spliced ​​annotation boxes of the noise generated by the cropping data enhancement algorithm to ensure the quality of the data generated by the splicing data enhancement method and improve the benefits of the enhanced data for model performance. By screening the noise formed in the splicing data enhancement algorithm, the enhanced data set is of higher quality and a higher performance model can be trained.

[0029] The principle is that, first of all, the outline of the target is usually irregular. Using the traditional rectangular box to mark the target will inevitably include a small amount of information outside the target, but the target mask only contains the target's own information. By comparing and calculating the coordinates of the target in the marked box with the coordinates in the mask, the real situation of the target in the marked box can be obtained, thereby realizing operations such as screening and moving the marked box. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is an example diagram for introducing a callout box.

[0031] Figure 2 To obtain a contour diagram of each target subject;

[0032] Figure 3 The figure is a schematic diagram for sequentially judging the N annotation boxes within the interception position. DETAILED DESCRIPTION

[0033] The present invention is further described below in conjunction with the accompanying drawings:

[0034] A deep learning image enhancement method based on annotation box splicing,

[0035] 1. Select N pictures, scale them to the same size, and obtain the mask picture described in the above picture at the same time. Each target in the mask picture has a clear mask area and its position information.

[0036] 2. Randomly sort several pictures, randomly determine a splicing point in the picture, and limit the selection range of the splicing point to a range of X and Y side lengths (for example, for N=4, this value can be selected between 0.2-0.8).

[0037] 3. According to this ratio, cut out the N pictures and the corresponding parts of each picture (for example, N=4, you can take the upper left, upper right, lower right, and lower left parts). Figure 1 shown.

[0038] 4. Filter the annotation box: Figure 1 The box is the annotation box. The annotation box is the outer rectangular box of all the objects to be detected in the picture, such as Figure 1In the figure, the rectangular box outside the character and the annotation box will surround the entire target.

[0039] (1) Determine whether the area of ​​the smallest region or the shortest side of all the annotation boxes is greater than a certain threshold, otherwise the annotation box is discarded

[0040] (2) Obtain the contour of each target subject (the subject's mask, such as Figure 2 The shaded area on the right side), the part to be spliced ​​in each picture is selected, the same position is intercepted on the mask corresponding to the picture, and the N labeled boxes in the intercepted position are judged in turn, such as Figure 3 As shown:

[0041] A. Determine whether the target box contains the mask part of the target, otherwise discard the annotation box. This is because the area selected during cropping is exactly the background area in the annotation box and is judged as an invalid box.

[0042] B. The area of ​​the mask containing the target in the target frame, if the area is smaller than a certain threshold, it will be discarded or returned to step 3, because the target contained in the target frame is too small to represent the characteristic properties of the target, and it is judged as an invalid frame;

[0043] C. Calculate the ratio between the mask area of ​​the target contained in the annotation box and the total mask area, and judge the annotation box that is less than a certain threshold as an invalid box;

[0044] D. Set a threshold. When the shortest distance from the stitching point to the mask is less than a threshold, it can be considered that the cropped image cannot represent the characteristics of the object and is judged as an invalid frame.

[0045] 5. For invalid boxes that do not meet the conditions, you can take the following approaches:

[0046] A. For the invalid annotation box mentioned above, you can directly discard the annotation box;

[0047] B. For the invalid annotation boxes mentioned above, return to step 2 to reselect new splicing points until there are no invalid annotation boxes;

[0048] C. Move the invalid frame area in the original image. When moving, according to the orientation of the area where the marked frame is located relative to the center point of the cropping area, move the cropping area along the direction.

[0049] 6. Scaling, transformation, rotation and other operations are performed on the filtered cropped area. The cropped area is placed on a new template of equal size according to the coordinates of its upper left vertex. At the same time, the same coordinate transformation is performed on the annotation box in each cropped area.

[0050] 7. Perform the above operation multiple times.

Claims

1. A deep learning image enhancement method based on annotation box splicing, characterized in that: The following steps are involved: Step 1: select N images with masks of objects to be identified, and scale the N images and their masks to the same size; Step 2, randomly sort the N pictures and select M pictures in sequence; Step 3, randomly select one or more stitching points within the image area, and the stitching points divide the image area into M areas; Step 4: Select one area from the M areas segmented in step 3, and stitch the selected areas of the M images selected in step 2 in order to form a complete new image; Step 5: for the new image formed in step 4, the size, area, proportion, and whether the mask is included in the annotation box of the new image are used to select the annotation box in the new image; Step 6: For the invalid annotation boxes, discard them, repeat step 3, or adjust the splicing area of ​​the selected pictures; Step 7, repeat steps 1 to 5 until all N images are enhanced; In step 5, the method of screening the annotation box in the new image includes one or more of the following methods: (1) Determine whether the area of ​​the smallest region or the shortest side of all the annotation boxes is less than a threshold. If so, determine that the annotation box is an invalid one; (2) Obtain the mask of each annotation box, select the part to be spliced ​​in each picture, cut out the same position on the mask corresponding to the picture, and judge the N annotation boxes in the cutout position in turn; The adjustment of the stitching area of ​​the selected image is to obtain a valid annotation frame by translating the position of the area in the original image, and the moving method is to move the area in the original image in the opposite direction relative to the stitching point in the image, and the moving distance is the threshold.

2. The deep learning image enhancement method based on annotation box splicing according to claim 1 is characterized in that: The adjustment in step 6 is to randomly generate a new stitching area.

3. The deep learning image enhancement method based on annotation frame splicing according to claim 2 is characterized in that: Method (2) in step 5 uses one or more of the following methods: (1) Determine whether the annotation box contains the mask part of the new image. If it does not contain it at all, it is determined to be an invalid annotation box; (2) The area of ​​the mask containing the new image in this annotation box, if the area is smaller than the threshold, it will be discarded or returned to step 3; (3) Calculate the ratio between the mask of the new image contained in the annotation box and the total mask area, and determine that the annotation box that is less than the threshold is an invalid annotation box; (4) Set a threshold. When the shortest distance from the stitching point to the edge of the mask is less than a threshold, or it is considered that the cropped image cannot represent the characteristics of the object, it is judged as an invalid annotation box.

4. The deep learning image enhancement method based on annotation box splicing according to claim 1, characterized in that: In step 3, the selection range of the stitching points is limited to a ratio of the width to the height of the image within a range of 0.2 to 0.8.

Images