Data enhancement method

Abstract

The invention relates to a data enhancement method. The method comprises the following steps: acquiring an original image and a background image; obtaining an annotation box OLabel of a target object in the original image; extracting an annotation box DLabel of a candidate target object in the background image and a depth map of the background image; dividing a plurality of sub candidate areas in the background image according to the annotation box OLabel and the annotation box DLabel; determining an optimal sub-candidate region by using the depth map; and combining the target object with the background image based on the optimal sub candidate area to generate enhanced image data. According to the method and the device, the training data set conforming to the reality semantic logic can be generated, so that data enhancement is better realized.

Description

Technical Field

[0001] The present invention relates to the field of computer image vision, and in particular to a data enhancement method. Background Art

[0002] The current data enhancement methods can be roughly divided into two categories: basic data enhancement and advanced data enhancement. Basic data enhancement includes 1. Image processing, namely geometric image processing (rotation, translation, shearing) and non-geometric image processing (flipping, cropping, injecting noise, perturbation) 2. Image erasure, namely Cutout, Random erasing and GridMask. The images generated by the above basic data enhancement methods based on image changes have the same semantic information as the original data set images, and have limited ability to improve the diversity and generalization of training data.

[0003] Object detection models based on convolutional neural networks (CNNs) often require a large amount of diverse data to avoid overfitting of the model and improve the generalization ability of the model. In real-world scenarios, vehicle detection is complex and requires information about different directions, positions, scales, and brightness of the vehicle. However, the scenarios of manually collected data sets are limited and it is difficult to meet actual requirements, which leads to poor training results for non-motor vehicle object detection models. Therefore, a data enhancement method is needed to generate training data sets that can conform to the semantic logic of reality, such as the complete fusion of the target and the background, the conformity of the target scale with the background semantics, the matching of the target pixel distribution with the background pixel distribution, etc., so as to better achieve data enhancement of small and rare targets. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a data enhancement method, which can generate a training data set that conforms to the real semantic logic, thereby better realizing data enhancement.

[0005] The technical solution adopted by the present invention to solve the technical problem is: to provide a data enhancement method, comprising the following steps:

[0006] Get the original image and background image;

[0007] Get the annotation box O of the target object in the original image Label ;

[0008] Extract the annotation box D of the candidate target area in the background image Label , and a depth map of the background image;

[0009] According to the marking box O Label and the marked box D Label Dividing a plurality of candidate sub-regions in the background image;

[0010] Determining an optimal sub-candidate region using the depth map;

[0011] The target object is combined with the background image based on the optimal sub-candidate region to generate enhanced image data.

[0012] Further, the marking box O Label and the marked box D Label A plurality of candidate sub-regions are divided in the background image, including:

[0013] Determining whether the target object is a small target or a large target;

[0014] If the target object is a small target, adjust the annotation box D Label The size enables it to be marked by several of the annotation boxes O Label Use the marking box O Label The adjusted annotation frame O Label Dividing into a plurality of sub-candidate regions, each of the sub-candidate regions has the same width and height as the target object;

[0015] If the target object is a large target, the annotation box O Label Slide horizontally according to the set division step length to make the marking box Label is divided into a plurality of said sub-candidate regions, and each of said sub-candidate regions is adjacent to said marked box O Label The width and height are the same.

[0016] Furthermore, the division step size is set by the following formula:

[0017]

[0018] Among them, μ is the partition step size, O Bwidth is the marking box O when the target object is a large target Label Width, N width and x width are the marked boxes D Label The width of the annotation box O is divided by Label The quotient and remainder of the width.

[0019] Furthermore, the marking box O Label Before the step of filling the candidate target area with the set step length, the method further includes:

[0020] If the target object is a large target, the marking box O Label Correction is made so that the marking box O Label The width is smaller than the label box D Label The width of the label box OLabel The height is less than the marked box D Label The height from the bottom edge of the background image to the top edge of the background image.

[0021] Further, the determining whether the target object is a small target or a large target includes:

[0022] Determine the marking box O Label With the marked box D Label Is the union of the corresponding area areas equal to the annotation box D? Label If the corresponding area is equal to , the target object is considered to be a small target; otherwise, the target object is considered to be a large target.

[0023] Furthermore, the adjustment of the marking frame D Label The sizes include:

[0024] Determine the marking box D Label Is the width of the label box O Label If it is not an integer multiple of the width, the label box D Label The width of the annotation box O is adjusted Label An integer multiple of the width;

[0025] Determine the marking box D Label Is the height of the annotation box O Label If it is an integer multiple of the height, then adjust the annotation box D Label The upper edge of the label box D Label The height of the annotation box O Label An integer multiple of the height of .

[0026] Furthermore, the marking box D Label The width is adjusted by:

[0027] Calculate the annotation box D Label The width of the annotation box D Label The remainder of the width divided by the width of

[0028] Calculate the annotation box D according to the width remainder Label The width expansion amount is then used to obtain the label box D Label The estimated width of

[0029] If the label box D Label If the estimated width of the background image does not exceed the range of the background image, the labeling box D is enlarged by using the width expansion amount. Label ;

[0030] If the label box D LabelIf the estimated width of the background image exceeds the range of the background image, the width remainder is used to reduce the annotation box D Label .

[0031] Furthermore, the marking box D Label The minimum ordinate of is adjusted by:

[0032] Calculate the annotation box D Label The height of the annotation box O Label The remainder of the height divided by the height of

[0033] Calculate the annotation box D according to the height remainder Label The height expansion amount of , thereby obtaining the annotation box D Label The estimated height of

[0034] If the label box D Label If the estimated height of the background image does not exceed the range of the background image, the height expansion amount is used to expand the annotation box D Label ;

[0035] If the label box D Label If the estimated height exceeds the range of the background image, the height remainder is used to reduce the annotation box D Label .

[0036] Further, the determining the optimal sub-candidate region by using the depth map includes:

[0037] Calculate an average depth value and a standard depth value of each of the sub-candidate regions based on the depth map;

[0038] The weights of the average depth value and the standard depth value are respectively set according to whether the target object is a large target or a small target;

[0039] The optimal sub-candidate region is determined according to a weighted sum of the average depth value and the standard depth value.

[0040] Furthermore, the weighted sum of the average depth value and the standard depth value Calculated by the following formula:

[0041]

[0042] in, is the average depth value of the k-th sub-candidate area, is the standard depth value of the kth sub-candidate area, and α is the weight parameter.

[0043] Further, the marking box O Label and the marked box D LabelBefore dividing a plurality of candidate sub-regions in the background image, the method further includes a step of increasing the similarity between the original image and the background image based on the frequency distribution of channel pixel values.

[0044] Furthermore, the increasing the similarity between the original image and the background image based on the frequency distribution of channel pixel values ​​includes:

[0045] Obtaining a cumulative histogram of the original image and the background image;

[0046] For any pixel value of the original image on any color channel, obtain all the corresponding pixel points of the original image as the pixel points to be updated, and calculate the minimum value of the difference between its cumulative histogram and the cumulative histogram of each pixel value of the background image on any color channel as the pixel update value, and update the pixel value of the pixel point to be updated to the pixel update value.

[0047] Beneficial Effects

[0048] Due to the adoption of the above-mentioned technical scheme, the present invention has the following advantages and positive effects compared with the prior art: the present invention extracts the annotation box of the target object in the original image and the annotation box of the candidate target area in the background image, and uses the annotation box of the target object to divide the candidate target area, and then selects the best pasting area according to the depth value of each sub-candidate target area to generate new image data, while enriching the semantic information of the original target, ensuring the consistency of the target scale with the background semantics. In addition, based on the histogram to increase the similarity between the original image and the background image, the style of the target and the new background can be unified, so that the target object and the background image are more integrated, and the pixel distribution is more matched with the background pixel distribution. Compared with other data enhancement methods for enhancing semantics, less noise is introduced, making the model training more effective. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of a flow chart of an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of large and small target judgment according to an embodiment of the present invention;

[0051] Figure 3 is a schematic diagram of a Pixel Correction algorithm according to an embodiment of the present invention;

[0052] Figure 4 is a schematic diagram of a method for correcting a labeling frame of a large target in an embodiment of the present invention;

[0053] Figure 5 Schematic diagram of a method for correcting a labeling frame of a candidate target area in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only intended to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art may make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0055] The embodiment of the present invention relates to a data enhancement method based on foreground target instance segmentation combined with background element semantic understanding, such as Figure 1 As shown, the following steps are included:

[0056] Step S1: prepare two sets of data sets, an original image data set (including target object annotations) and a background image data set; in this embodiment, the original image and the background image have the same size and shape.

[0057] Step S2: three deep learning models are used: the SAM model is used to segment the original image to obtain the Mask of the target object; the YOLO model is used to detect elements related to the target object (i.e., elements that can be combined with the target object) in the background image to obtain the annotation of the target candidate area; and the ZeoDepth model is used to analyze the depth of field properties of the background image and generate the corresponding DepthMap;

[0058] For the output of the model, we agree that the upper left corner of the image is the origin, the coordinate value x increases gradually to the right, and the coordinate value y increases gradually downward; the segmentation maps of the original image and the target area are I and I respectively. mask , the target objects in the original image and their corresponding annotation boxes are O, O Label , the background image is B, the drivable area (i.e., candidate target area) in the background image and its annotation box are D, D Label , the depth map of the background image is M;

[0059] like Figure 2 As shown, according to formula (1), O Label Is it a large target or a small target? If the area of ​​the union is equal to the area of ​​the drivable area D, it is recorded as a small target O. S , if it is greater than, it is recorded as a large target O B .

[0060] D Label ∪O Label (1)

[0061] Step S3: Three algorithms are used to reduce the noise generated when the target and the background are combined. The Pixel Correction algorithm is responsible for making the style of the target object consistent with that of the background image. The Label Correction algorithm is responsible for correcting the target object labeling box and the target candidate area and dividing the candidate area. The Location Selection algorithm is responsible for determining the optimal position of the target object in the target candidate area. The algorithm operation order is as follows:

[0062] Step S3.1: Use the Pixel Correction algorithm to perform pixel correction on the target object according to the histogram, such as Figure 3 As shown, the color style of the target image is unified with that of the background image, that is, the brightness and saturation of the target image and the background image are consistent. The specific operations are as follows:

[0063] Step S3.1.1: Calculate the frequency distribution histogram of each pixel value in each channel of the original image and the background, as shown in formula (1.1), c is the color channel of the image, j is the pixel value, j∈{0,1,2,…255}, and then calculate the global cumulative histogram as shown in formula (1.2),

[0064]

[0065] in, is the histogram of pixel value j of the original image on channel c, I (j,c) is the pixel point with pixel value j on channel c of the original image, is the histogram of pixel value j of the background image on channel c, B (j,c) is the pixel of the background image with pixel value j on channel c, is the cumulative histogram of pixel value j of the original image on channel c, It is the cumulative histogram of pixel value j of the background image on channel c.

[0066] Step S3.1.2: Then, the pixels of the original image are corrected according to the cumulative histogram. For any pixel value j on any channel c of the original image, its cumulative histogram is calculated. The cumulative histogram of all pixel values ​​of the background image on this channel The difference between them is calculated, and the pixel values ​​of all pixels with pixel value j on channel c of the original image are replaced with the pixel value n corresponding to the minimum difference, so as to obtain the new original image newI. The minimum difference is shown in formula (2):

[0067]

[0068] in, It is the cumulative histogram of pixel value n of the background image on channel c.

[0069] Step S3.2: Use the Label Correction algorithm to correct and divide the candidate area based on the background after the drivable area detection and the target labeling;

[0070] Step S3.2.1: Figure 4 As shown, for O Label Correction, for large targets, for O B Make further corrections and determine the scaling parameter λ to update O Label .

[0071]

[0072] Then by λ×O Bwidth,Bheight , so that O Bwidth ≤D width and O Bheight ≤D Label.bottomm ; Among them, O Bwidth is the width of the large target, D width is the width of the drivable area, O Bheight is the height of the large target, D Label.bottom is the distance from the bottom of the drivable area to the top of B.

[0073] Step S3.2.2: Figure 5 As shown, for D Label Make corrections and follow D Label Divide B into N and O s or b The same size area. For small targets O S According to formula (4.1), if x is 0, then according to formula (4.2), N = N width ×N height .

[0074] x (width,height) =D (width,height) mod O (width,height) (4.1)

[0075]

[0076] Among them, O (width,height) The width and height of the target object label box, D (width,height) N is the width and height of the drivable area label box. (width,height) After rounding down, it is the number of target object annotation boxes that can be divided by the width and height of the drivable area annotation box, x (width,height) According to N (width,height) The area that may remain after division is called the remainder.

[0077] Step S3.2.3: If x is not 0, first estimate whether the expanded D exceeds the range of B according to formula (5.1). If it does not exceed, update D according to formula (5.1); otherwise, update D according to formula (5.2).

[0078] newD (width,height) =D (width,height) +O (Swidth,Sheight) -x (width,height) (5.1)

[0079] newD (width,height) =D (width,height) -x (width,height) (5.2)

[0080] Step S3.2.4: Keep D Label.Bottom (lower edge of the drivable area) does not need to be corrected, D Label.left and D Label.right D is divided equally from both sides. Label Expand or compress it to update it to newD width ;D Label.top Then expand or compress upward to update it to newD height ; More specifically, when D is updated by expansion, the new coordinates are calculated by formula (6.1), and when D is updated by compression, the new coordinates are calculated by formula (6.2);

[0081]

[0082] Step S3.2.5: For large object O B , we need to introduce the step size μ first, and then according to formula (4.1) (4.2) we only consider the width to get x width and N width , if x width = 0, then the step size μ = O Bwidth , N=N width If it is not 0, the value of μ must be determined according to formula (7), and then the value of μ can be determined by using the marking box O. Label Slide along the x-axis according to the step size μ, and divide the drivable area into N = N width +1 sub-candidate region;

[0083]

[0084] Step S3.2.6: In the Label , D Label After correction, D is divided into candidate regions and recorded as List. (wide,height) The candidate area list List can be calculated. For a small target O SThe division method is shown in formula (8.1, 8.2), where List k represents the k=i×jth candidate region, k∈N, i,j means that the candidate region is located in the i-th column and j-th row of D;

[0085]

[0086] Step S3.2.7: For large object O B , which is different from the small target, the vertical coordinate division method is shown in formulas (9.1, 9.2), first determine the bottom edge coordinate of the candidate box and then calculate the top edge coordinate. The horizontal coordinate division method is shown in formulas (9.3, 9.4), where List k represents the k=i-th candidate region, i represents the candidate region located in the i-th column of D, and μ is the step length calculated in step S3.2.5

[0087]

[0088] Step S3.3: Use the Location Selection algorithm to perform pixel statistics on the depth map to determine the optimal location of the target in the candidate area.

[0089] Step S3.3.1: Position selection refers to combining the List obtained in step 3.2 with M, and determining the candidate position to paste newI by counting the pixels in each sub-candidate area. In the depth map M, each pixel records the distance from the viewpoint to the surface of the occluder, i.e., the shadow generating object. So for the small target O S , prefer to paste it to the average depth value M mean Larger location, and large target O B It should be pasted to the average depth value M mean Smaller location.

[0090] M mean The calculation formula is as follows:

[0091]

[0092] in, is the average depth value of the kth sub-candidate area, c is the number of channels, N k is the total number of pixels in the kth sub-candidate area, M (x,y) [i] is the pixel value of pixel point (x, y) on channel i.

[0093] Step S3.3.2: In addition, the pixel standard method represents the contrast degree of light and dark changes in the image area, reflecting the complexity of the object in the area. S Or O B , Mstd The smaller it is, the fewer objects there are in the area, which is more conducive to object pasting. std The calculation formula is as follows;

[0094]

[0095] in, is the standard depth value of the kth sub-candidate area.

[0096] Step S3.3.3: Finally, introduce the parameter α for each M k Calculate the final weight The choice of α reflects the priority of the target scale and the background position. The larger α is, the more accurate the scale semantic relationship of the target is. Conversely, the smaller α is, the more accurate the background semantic relationship of the target is. The correction and calculation of weights tend to choose The larger area is used to paste the target. Finally, the larger area on M is recorded. The corresponding area on B is B k .

[0097]

[0098] Step S4: Combining the target with the new background to generate a new data set, specifically including the following steps:

[0099] Step S4.1: In the background migration stage, the target that has been corrected by annotation and pixel correction needs to be pasted on the background image of the determined candidate area. First, after obtaining O, according to specific needs, Label In newI,I mask Crop and scale to get newI aug with I mask.aug Assumptions ψ are scaling and cropping parameters respectively, and the formula is as follows;

[0100]

[0101] Step S4.2: Then you need to use ⊙ to convert newI aug ,I mask.aug Perform a dot multiplication to achieve the split effect, and finally paste it to the selected B k area, as shown in formula (14), assuming that φ is the pasting parameter;

[0102] B BGM =f aug (newI aug ⊙I mask.aug +B k |B,φ) (14)

[0103] Step S4.3: Using the original image I, the target annotation O Label , and the background image B, combined with the segmentation mask I of the original image mask , marking of the background drivable area D Label And the background depth map M finally gets B BGM And obtained the corresponding annotations.

Claims

1. A data enhancement method, characterized in that: The following steps are involved: Get the original image and background image; Get the annotation box O of the target object in the original image Label ; Extract the annotation box D of the candidate target area in the background image Label , and a depth map of the background image; According to the marking box O Label and the marked box D Label Dividing a plurality of candidate sub-regions in the background image; Determining an optimal sub-candidate region using the depth map; The target object is combined with the background image based on the optimal sub-candidate region to generate enhanced image data.

2. The method according to claim 1, characterized in that The marking box O Label and the marked box D Label A plurality of candidate sub-regions are divided in the background image, including: Determining whether the target object is a small target or a large target; If the target object is a small target, adjust the annotation box D Label The size enables it to be marked by several of the annotation boxes O Label Divide into equal parts and use the marking box O Label The adjusted annotation frame D Label is divided into a plurality of said sub-candidate regions, and each of said sub-candidate regions is adjacent to said marked box O Label The width and height are the same; If the target object is a large target, the annotation box O Label Slide horizontally according to the set division step length, so that the marking box D Label is divided into a plurality of said sub-candidate regions, and each of said sub-candidate regions is adjacent to said marked box O Label The width and height are the same.

3. The method according to claim 2, characterized in that The division step size is set by the following formula: Among them, μ is the partition step size, O Bwidth is the marking box O when the target object is a large target Label Width, N width and x width are the marked boxes D Label The width of the annotation box O is divided by Label The quotient and remainder of the width.

4. The method according to claim 2, characterized in that: The marking box O Label Before the step of filling the candidate target area with the set step length, the method further includes: If the target object is a large target, the marking box O Label Correction is made so that the marking box O Label The width is smaller than the label box D Label The width of the label box O Label The height is less than the marked box D Label The height from the bottom edge of the background image to the top edge of the background image.

5. The method according to claim 2, characterized in that: The determining whether the target object is a small target or a large target comprises: Determine the marking box O Label With the marked box D Label Is the union of the corresponding area areas equal to the annotation box D? Label If the corresponding area is equal to , the target object is considered to be a small target; otherwise, the target object is considered to be a large target.

6. The method according to claim 2, characterized in that The adjustment of the marking frame D Label The sizes include: Determine the marking box D Label Is the width of the label box O Label If it is not an integer multiple of the width, the label box D Label The width of the annotation box O is adjusted Label An integer multiple of the width; Determine the marking box D Label Is the height of the annotation box O Label If it is an integer multiple of the height, then adjust the annotation box D Label The upper edge of the label box D Label The height of the annotation box O Label An integer multiple of the height of .

7. The method according to claim 1, characterized in that Determining the optimal sub-candidate region by using the depth map includes: Calculate an average depth value and a standard depth value of each of the sub-candidate regions based on the depth map; The weights of the average depth value and the standard depth value are respectively set according to whether the target object is a large target or a small target; The optimal sub-candidate region is determined according to a weighted sum of the average depth value and the standard depth value.

8. The method according to claim 7, characterized in that The weighted sum of the average depth value and the standard depth value Calculated by the following formula: in, is the average depth value of the k-th sub-candidate area, is the standard depth value of the kth sub-candidate area, and α is the weight parameter.

9. The method according to claim 1, characterized in that: The marking box O Label and the marked box D Label Before dividing a plurality of candidate sub-regions in the background image, the method further includes a step of increasing the similarity between the original image and the background image based on the frequency distribution of channel pixel values.

10. The method according to claim 9, characterized in that The increasing the similarity between the original image and the background image based on the frequency distribution of channel pixel values ​​comprises: Obtaining a cumulative histogram of the original image and the background image; For any pixel value of the original image on any color channel, obtain all the corresponding pixel points of the original image as the pixel points to be updated, and calculate the minimum value of the difference between its cumulative histogram and the cumulative histogram of each pixel value of the background image on any color channel as the pixel update value, and update the pixel value of the pixel point to be updated to the pixel update value.

Images