Manufacturing method of rectangular target detection data set under highly reflective background

Abstract

The invention discloses a manufacturing method of a rectangular target detection data set under a highly reflective background. The manufacturing method comprises the following steps: acquiring a target image and a first background image; carrying out distortion removal operation and perspective transformation on the first background image in sequence; obtaining a background profile curve; respectively placing target images at two positions in the second background image; judging whether the target image exceeds the background area; randomly selecting a plurality of points on the initial image to generate light spots, and generating images under different illumination conditions; and respectively combining each target image with each background image, and recording the left upper corner point and the right lower corner point of the minimum circumscribed regular rectangle in a label file. According to the method, under the condition that only a small number of data samples exist, a large number of data sets with more diversity are formed through target angle and position transformation and reflection light spot generation, and the generated data sets not only improve the diversity of the target in the position and angle level, but also improve the diversity of the reflection light spots in the high-reflection background.

Description

Technical Field

[0001] The present invention relates to the field of information engineering, and in particular to a method for preparing a rectangular target detection data set under a high-reflective background. Background Art

[0002] In the field of industrial inspection, the demand for automation and intelligence is growing, and inspection robots play an important role in this process. These robots need to have efficient inspection capabilities, including the ability to move along a predetermined route and locate and identify targets in the area along the route. Object detection technology enables robots to identify one or more objects in the image or video frame obtained from the camera and determine their location, which is crucial for industrial inspection.

[0003] Using deep learning target detection technology can help robots more accurately identify and locate target objects in the environment. The premise is that high-quality data sets are needed. The collection and annotation of data sets are key steps in building effective target detection models. Usually, a large number of data samples need to be prepared. These samples need to consider the state of the target in different situations. At the same time, accurate annotation also takes a lot of time. At present, the production of data sets mainly relies on manual annotation, which requires manually circling the targets on each picture one by one, which will greatly increase the cost and difficulty of data preparation. Summary of the invention

[0004] Purpose of the invention: In view of the above problems, the purpose of the present invention is to provide a method for producing a rectangular target detection data set under a highly reflective background.

[0005] Technical solution: A method for preparing a rectangular target detection data set under a highly reflective background of the present invention comprises the following steps: Step 1, respectively obtaining a target image and a first background image; Step 2, performing a dedistortion operation and a perspective transformation on the first background image in sequence, so as to convert the first background image at different angles into a second background image; Step 3, segmenting the background area in the second background image by thresholding to obtain a background contour curve; Step 4, placing two types of target images in the second background image, including a target image located in the background area and whose boundary falls on the background contour curve, and a target image located in the background area and whose boundary has no intersection with the background contour curve; Step 5, determine whether the target image exceeds the background area, place the target image that meets the requirements into the background area, generate a preliminary image, and record the upper left corner point and lower right corner point of the minimum circumscribed rectangle of the target image into the label file; Step 6, randomly selecting multiple points on the preliminary image to generate light spots, simulating the image captured by the camera device under the condition of uneven light in the real environment through the light gradient attenuation function, and generating images under different lighting conditions; Step 7: Repeat steps 1 to 6 to combine each target image with each background image to obtain an expanded data set.

[0006] Furthermore, step 2 includes: Using a checkerboard calibration method to obtain camera intrinsic parameters and distortion coefficients, and using a dedistortion function to dedistort the first background image; Obtain the perspective matrix, and use the perspective matrix to perform perspective transformation on the dedistorted image; The pixels of the perspective transformed image are filled in by bilinear interpolation to obtain a second background image.

[0007] Further, step 3 includes: Setting a color threshold, and using the color threshold to segment the second background image to obtain a binary background image; Perform opening operation on the binary background image; Using the area enclosed by the contour as the standard, find the outer contour corresponding to the maximum area enclosed by the contour on the binary image; Use the interpolation function to complete the missing points on the curve where the outer contour is located, and record the coordinates of the points on the curve, and record the curve as the background contour curve.

[0008] Further, step 4 includes: Adjust the target image to a square target image. For the target image in the first position, determine the concavity of the curve where the square target image falls on the corresponding point of the background contour curve L1. If it is a concave curve, make the lower left corner of the square target image Coincident with a point on the background contour curve, and rotate the square target image by an angle θ, , so that the lower right corner of the square target image Fall on another point of the background contour curve; if it is a convex curve, select the midpoint of the lower boundary of the square target image , through the point and Fit the straight line L2 where the lower boundary is located, and Taking the y-axis as the unit, gradually shift to the normal direction of the background contour curve until the straight line L2 and the background contour curve L1 have only one intersection point, and calculate the coordinates of the upper left corner of the target image at this time , record the coordinates of the upper left corner of the square target image at this time , side length r and the rotation angle θ; For the target image in the second position, randomly select a point in the background area, rotate the square target image by angle θ, use this point as the upper left corner of the target image, and record the coordinates of the upper left corner of the square target image. , side length r and the rotation angle θ.

[0009] Further, step 5 comprises: According to the coordinates of the upper left corner of the square target image , side length r And the rotation angle θ, calculate the upper left corner point of the target minimum circumscribed rectangle , bottom right corner and center point ; Perform a bitwise AND operation on the rectangular area where the minimum circumscribed regular rectangle is located and the second background image, and first determine whether the target image is in the background area. If not, the target image placed this time is abandoned. If it is in the background area, according to the width of the second background image w and high h , the side length of the square target image r And the coordinates of the center point of the minimum circumscribed regular rectangle Perform a second judgment. If it exceeds the boundary of the second background image, the target image placed this time is discarded. Otherwise, the target image is placed in the background area to generate a preliminary image containing the target image. The upper left corner point of the minimum circumscribed rectangle of the target image is and the lower right corner The coordinates are recorded in the label file.

[0010] Further, step 6 comprises: Randomly pick a point on the preliminary image As the center point of the light spot, the radius of the light spot is r 0 , calculate from the center of the spot To the surrounding radius r 0 Any point within The gradient of the decay g 0 , the gradient g 0 Mapped to the pixel value [0,255], get the point The pixel value P 0 ; The preliminary image and the light spot are weighted together to obtain the final combined image, where the weight of the preliminary image is set to 1 and the weight of the light spot is set to w. 0 , w 0 exist Floating within the range.

[0011] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: With only a small amount of data samples, the present invention forms a large number of more diverse data sets by transforming the target angle and position and generating reflected light spots. The generated data sets not only improve the diversity of the targets at the position and angle levels, but also improve the diversity when they have reflected light spots under a highly reflective background, which greatly reduces the time and cost of data set production and improves the accuracy of the trained model and its adaptability to lighting environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A flowchart of a method for preparing a rectangular target detection dataset under a highly reflective background; Figure 2 is a schematic diagram of a target image including a first position and a contour curve where the contour curve is a concave curve; Figure 3 is a schematic diagram of a target image including a first position and a contour curve where the contour curve is a convex curve; Figure 4 is a schematic diagram of a target image and a contour curve including a second position; Figure 5 is a schematic diagram of the background area containing the minimum circumscribed regular rectangle; Figure 6 Schematic diagram of the final combined image. DETAILED DESCRIPTION

[0013] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0014] The method for preparing a rectangular target detection data set under a high-reflective background described in this embodiment is as follows: Figure 1 As shown, the following steps are included: Step 1, respectively obtaining a target image and a first background image; Step 2, performing a dedistortion operation and a perspective transformation on the first background image in sequence, so as to convert the first background image at different angles into a second background image; Step 3, segmenting the background area in the second background image by thresholding to obtain a background contour curve; Step 4, placing two types of target images in the second background image, including a target image located in the background area and whose boundary falls on the background contour curve, and a target image located in the background area and whose boundary has no intersection with the background contour curve; Step 5, determine whether the target image exceeds the background area, place the target image that meets the requirements into the background area, generate a preliminary image, and record the upper left corner point and lower right corner point of the minimum circumscribed rectangle of the target image into the label file; Step 6, randomly selecting multiple points on the preliminary image to generate light spots, simulating the image captured by the camera device under the condition of uneven light in the real environment through the light gradient attenuation function, and generating images under different lighting conditions; Step 7: Repeat steps 1 to 6 above to combine each target image with each background image to obtain an expanded data set.

[0015] In this example, a camera can be used to collect a small number of images containing the background and the target image as the initial material for making a data set. The target image is set to a rectangle. Since the camera lens will cause image distortion and the camera has a certain angle when taking pictures, it is necessary to first dedistort the image, and then use perspective transformation to transform the image at any angle into a top view so that the image can be processed uniformly later.

[0016] Further, step 2 includes: Using a checkerboard calibration method to obtain camera intrinsic parameters and distortion coefficients, and using a dedistortion function to dedistort the first background image; Obtain the perspective matrix, and use the perspective matrix to perform perspective transformation on the dedistorted image; The pixels of the perspective transformed image are filled in by bilinear interpolation to obtain a second background image.

[0017] The perspective matrix is ​​a The matrix M ,in ,matrix M There are 8 unknown parameters in , and 8 equations need to be solved. Suppose the coordinates of any point on the image taken at any angle are , the coordinates of the point on the transformed image are , then both satisfy: Since what is involved here is the conversion from a two-dimensional image to a two-dimensional image, the image needs to be restricted to a two-dimensional plane. The coordinates of the final output are recorded as ,but , then a pair of coordinates can provide two equations to calculate the matrix M ,Right now: .

[0018] The perspective matrix has 8 unknown quantities, and four pairs of coordinate points are needed to calibrate and solve the perspective matrix. M, place a rectangular calibration target of known size in the camera field of view, obtain the coordinates of the four corner points of the calibration target in the camera field of view, and the corresponding coordinates of the four corner points in the top view are known. The perspective matrix can be obtained by solving the linear equation system M .

[0019] Using the obtained matrix M Perform perspective transformation on the image. Since images appear larger when near and smaller when far away at different angles, some pixels in the distance of the transformed image will be missing. Bilinear interpolation is used to fill in the missing pixels to complete the preliminary processing of the background image, which is recorded as the second background image.

[0020] Further, step 3 includes: Setting a color threshold, and using the color threshold to segment the second background image to obtain a binary background image; Perform an opening operation on the binary background image to remove noise; Using the area enclosed by the contour as the standard, find the outer contour corresponding to the maximum area enclosed by the contour on the binary image; Use the interpolation function to complete the missing points on the curve where the outer contour is located, and record the coordinates of the points on the curve, and record the curve as the background contour curve.

[0021] For example, the color threshold can be set according to specific needs. The threshold needs to ensure that the background contour of the segmented image is complete and clear. In order to achieve the ideal effect, the binary image can be opened to remove noise. The image opening operation includes erosion and dilation processing of the image in sequence. The binary image is opened by the kernel to filter out the burrs on the image. After the binary image is obtained, a rectangular coordinate system xoy can be established with the binary image, where the horizontal axis x is parallel to the horizontal edge of the binary image and the vertical axis y is parallel to the vertical edge of the binary image. For the contour with the largest area, its area A It can be calculated by the following formula: , where It is the first i The coordinates of the points, It is the first i +1 point coordinates, n is the total number of points. This contour is the background contour to be found. Select a curve in the contour, fit the curve and use the interpolation function to fill in the missing points on the curve where the contour is located, so that the contour curve is evenly distributed, and record the coordinates of the points on the curve.

[0022] Further, step 4 includes: Adjust the target image to a square target image. For the target image in the first position, determine the concavity of the curve where the square target image falls on the corresponding point of the background contour curve L1. If it is a concave curve, make the lower left corner of the square target image Coincident with a point on the background contour curve, and rotate the square target image by an angle θ, , so that the lower right corner of the square target image Falls on another point of the background contour curve, such as Figure 2 As shown, at this time, you can use Verify that r is the side length of the square target image; if it is a convex curve, select the midpoint of the lower boundary of the square target image ,in , through the point and Fit the straight line L2 where the lower boundary is located, and The y-axis is taken as a unit, and the normal direction of the background contour curve is gradually offset until the straight line L2 has only one intersection with the background contour curve L1, such as Figure 3 As shown, calculate the coordinates of the upper left corner of the target image at this time ,in , record the coordinates of the upper left corner of the square target image at this time , side length r and the rotation angle θ; For the target image in the second position, it is also adjusted to a square target image and rotated by a random angle θ. , randomly select a point in the background area, use this point as the upper left corner of the directional target image, and record the coordinates of the upper left corner of the square target image , side length r And the rotation angle θ, such as Figure 4 shown.

[0023] Further, step 5 includes: Since the target detection model needs to obtain the location of the target image on an image through the target's bounding box, the target's bounding box is usually represented by It consists of four parameters, where the target bounding box is the minimum circumscribed regular rectangle of the target. is the upper left corner of the target bounding box, so the upper left corner of the target's minimum circumscribed rectangle can be calculated and the lower right corner To obtain the coordinates of these four parameter records, ,by Figure 5For example, the solid line is a square target at any angle, and the dotted line is the bounding box of the target. Therefore, when making a dataset, in addition to generating images containing the target, the location of the target must also be recorded. The file that records the target location is called a label file. The label file records the upper left corner of the target. and the lower right corner When obtaining the target position, the upper left corner of the target box should be calculated first. , and the center of the target box, which is the center of the target ,pass Calculate the upper right corner , and the coordinates and Log in to the tag file.

[0024] According to the coordinate information of the upper left corner of the square target image , side length r And the rotation angle θ, calculate the minimum circumscribed rectangle of the square target image, that is, the coordinates of the upper left corner of the bounding box of the target image ,in: ; Next, calculate the minimum bounding rectangle, which is the center point coordinate of the bounding box of the target image. ,in: ; Then, according to the point and Point , calculate the coordinates of the lower right corner of the bounding box of the target image ,in: , record the location information and ; Perform a bitwise AND operation on the rectangular area where the minimum circumscribed regular rectangle, i.e., the bounding box of the target image, is located with the second background image, and first determine whether the target image is in the background area. If not, the target image placed this time is abandoned; If it is in the background area, according to the width of the second background image w and high h , the side length of the square target image r And the coordinates of the center point of the minimum circumscribed regular rectangle Make a second judgment and calculate the parameters ,parameter ,parameter And parameters If any of these four parameters is greater than , it means that the target image has exceeded the boundary of the background image; where: ; If it exceeds the boundary of the second background image, the target image placed this time is discarded. Otherwise, the target image is placed in the background area, where the upper left corner of the target image is , the rotation angle is θ, and the upper left corner of the bounding box of the target image is recorded in the corresponding label file and the lower right corner , complete the preliminary generation of the image; record the coordinates of the upper left corner and lower right corner of the bounding box of the target image in the label file to complete the image position annotation.

[0025] Further, step 6 comprises: Randomly pick a point on the preliminary image As the center point of the light spot, the radius of the light spot is r 0 , calculated from the center of the light spot through the light gradient attenuation function To the surrounding radius r 0 Any point within The gradient of the decay g 0 , the gradient g 0 Mapped to the pixel value [0,255], get the point The pixel value P 0 ; This function passes and r 0 Established gradient g 0 and radius r 0 Any point within the range Pixel value P 0 The mapping relationship between them, where the gradient g 0 and pixel value P 0 The calculation formulas are: , , The preliminary image and the light spot are weighted together to obtain the final combined image, where the weight of the preliminary image is set to 1 and the weight of the light spot is set to w. 0 , w 0 exist Floating within the range, such as Figure 6 shown.

[0026] The final image containing the light spot is saved in the same directory as the label file to facilitate the target detection model to read the data set. Repeat steps 1 to 6 above, combine each target image with each background image, and each combination will try to generate the first type of target and the second type of target in turn until all combinations are generated. This round is called a batch. Since the placement of the target is random and each generation may not be valid, 4 to 5 batches can be set as needed to ensure that enough data is generated.

[0027] In order to verify the application effect of the method for preparing a rectangular target detection data set under a highly reflective background described in the present invention, the generated images and the actually taken images are made into different training sets and added to the same validation set. The YOLOv3 model is used for training. The model trained based on the data set generated by the present invention is recorded as the first model, and the target recognition model trained based on the manually collected and labeled data set is recorded as the second model. The evaluation indicators of the two models are verified under the same validation set.

[0028] The verification results show that the precision rate of the first model is 6.1% higher than that of the second model. It can be seen that the precision rate of the second model is lower and it is more inclined to take the light spot as the target. In practical applications, misjudgment of the light spot may cause a large jump in the target position between the previous and next frame images, which will cause the robot to lose the correct target position. Therefore, in practical applications, more attention is paid to the precision rate. From this perspective, the first model is more suitable as a model used in the robot inspection process, which further shows that the expanded data set of the present invention is more in line with practical needs.

[0029] Taking the training set of only 2,000 images as an example, the traditional method requires taking 2,000 images and annotating them. The sampling time of each frame is set to 0.5s, so it takes 0.27 hours to take 2,000 images, and it takes 2.7 hours to annotate the data set, with a total time of nearly 3 hours. Under the method of the present invention, only 200 background images and 10 target images need to be taken to generate a data set of more than 2,000 images containing light spot noise, and the label file is automatically generated without manual annotation. It only takes about 5 minutes, which greatly improves the efficiency of sample generation.

[0030] The present invention proposes a method for producing a rectangular target detection data set under a highly reflective background. By using a relatively small amount of data sets, the data sets at different positions of the target object in a specific background are expanded, thereby increasing the diversity of the target at the position coordinate level. In addition, the light spots formed by reflection are added to the original finite samples through the fading gradient function, and the data samples under different lighting conditions are expanded, thereby solving the problem of insufficient sample diversity under a highly reflective background and avoiding a large amount of collection and labeling work.

Claims

1. A method for preparing a rectangular target detection dataset under a highly reflective background, characterized in that: The steps include: Step 1, respectively obtaining a target image and a first background image; Step 2, performing a dedistortion operation and a perspective transformation on the first background image in sequence, so as to convert the first background image at different angles into a second background image; Step 3, segmenting the background area in the second background image by thresholding to obtain a background contour curve; Step 4, placing two types of target images in the second background image, including a target image located in the background area and whose boundary falls on the background contour curve, and a target image located in the background area and whose boundary has no intersection with the background contour curve; Step 5, determine whether the target image exceeds the background area, place the target image that meets the requirements into the background area, generate a preliminary image, and record the upper left corner point and lower right corner point of the minimum circumscribed rectangle of the target image into the label file; Step 6, randomly selecting multiple points on the preliminary image to generate light spots, simulating the image captured by the camera device under the condition of uneven light in the real environment through the light gradient attenuation function, and generating images under different lighting conditions; Step 7: Repeat steps 1 to 6 to combine each target image with each background image to obtain an expanded data set.

2. The method for preparing a rectangular target detection dataset under a highly reflective background according to claim 1, characterized in that: Step 2 includes: Using a checkerboard calibration method to obtain camera intrinsic parameters and distortion coefficients, and using a dedistortion function to dedistort the first background image; Obtain the perspective matrix, and use the perspective matrix to perform perspective transformation on the dedistorted image; The pixels of the perspective transformed image are filled in by bilinear interpolation to obtain a second background image.

3. The method for preparing a rectangular target detection dataset under a high-reflective background according to claim 2, characterized in that: Step 3 includes: Setting a color threshold, and using the color threshold to segment the second background image to obtain a binary background image; Perform opening operation on the binary background image; Using the area enclosed by the contour as the standard, find the outer contour corresponding to the maximum area enclosed by the contour on the binary image; Use the interpolation function to complete the missing points on the curve where the outer contour is located, and record the coordinates of the points on the curve, and record the curve as the background contour curve.

4. The method for preparing a rectangular target detection dataset under a high-reflective background according to claim 3, characterized in that: Step 4 includes: Adjust the target image to a square target image. For the target image in the first position, determine the concavity of the curve where the square target image falls on the corresponding point of the background contour curve L1. If it is a concave curve, make the lower left corner of the square target image Coincident with a point on the background contour curve, and rotate the square target image by an angle θ, , so that the lower right corner of the square target image Fall on another point of the background contour curve; if it is a convex curve, select the midpoint of the lower boundary of the square target image , through the point and Fit the lower boundary straight line L2, and Taking the y-axis as the unit, gradually shift to the normal direction of the background contour curve until the straight line L2 and the background contour curve L1 have only one intersection point, and calculate the coordinates of the upper left corner of the target image at this time , record the coordinates of the upper left corner of the square target image at this time , side length r and the rotation angle θ; For the target image in the second position, randomly select a point in the background area, rotate the square target image by angle θ, use this point as the upper left corner of the target image, and record the coordinates of the upper left corner of the square target image. , side length r and the rotation angle θ.

5. The method for preparing a rectangular target detection dataset under a high-reflective background according to claim 4, characterized in that: Step 5 includes: According to the coordinates of the upper left corner of the square target image , side length r And the rotation angle θ, calculate the upper left corner point of the target minimum circumscribed rectangle , bottom right corner and center point ; Perform a bitwise AND operation on the rectangular area where the minimum circumscribed regular rectangle is located and the second background image, and first determine whether the target image is in the background area. If not, the target image placed this time is abandoned. If it is in the background area, according to the width of the second background image w and high h , the side length of the square target image r And the coordinates of the center point of the minimum circumscribed regular rectangle Perform a second judgment. If it exceeds the boundary of the second background image, the target image placed this time is discarded. Otherwise, the target image is placed in the background area to generate a preliminary image containing the target image. The upper left corner point of the minimum circumscribed rectangle of the target image is and the lower right corner The coordinates are recorded in the label file.

6. The method for preparing a rectangular target detection dataset under a high-reflective background according to claim 5, characterized in that: Step 6 includes: Randomly pick a point on the preliminary image As the center point of the light spot, the radius of the light spot is r0, and the radius of the light spot is r0. To any point within the radius r0 The gradient of the decay g0 is mapped to the pixel value [0,255] to obtain the point The pixel value P0; The preliminary image and the light spot are weighted together to obtain the final combined image, where the weight of the preliminary image is set to 1 and the weight of the light spot is set to w0. Floating within the range.

Images