Redundant object detection data set manufacturing method

Abstract

The invention belongs to the technical field of data processing, and provides a method for making a redundancy detection data set, which comprises the following steps of: 1, collecting a redundancy sample; step 2, primary sampling of samples; step 3, performing intermediate sampling on the sample; 4, creating a pre-training data set; step 5, targeted sampling: screening out unrecognized redundant object samples, and performing targeted sampling in a multi-view and multi-background mixed sampling mode; step 6, negative sample sampling: screening out misrecognized normal samples, and carrying out negative sample sampling; and step 7, creating a final data set: mixing the primary sampling data, the intermediate sampling data, the targeted sampling data and the negative sampling data according to a set proportion to form a final redundancy detection data set.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular to a method for preparing a redundant object detection data set, which is suitable for application scenarios such as machine learning and artificial intelligence that require data set support. Background Art

[0002] With the advent of the big data era, the production of data sets has become an important part of machine learning and artificial intelligence model training. For some specific application scenarios, such as closed packaging and visual occlusion, there may be metal excess (usually from metal parts in the processing or assembly process, such as screws, nails, metal chips, etc.) and non-metal excess (may come from the processing of packaging materials, insulating materials or other non-metallic materials, such as plastic fragments, rubber particles, fiber filaments, etc.). Since they cannot be directly identified by the human eye, they often need to use endoscopes to collect images and use relevant detection algorithms of artificial intelligence for identification. However, these scenarios often do not have the required public data sets, and data sets need to be manually produced. Traditional data set production methods usually rely on manual collection and organization of data, which has problems such as low efficiency, high cost, and easy errors. These problems are particularly prominent when dealing with large-scale and diverse data.

[0003] In the prior art, the method of making a data set mainly includes the following steps: first, collect a large amount of raw data; second, clean and preprocess the data to remove noise and outliers; finally, divide the data into training set, validation set and test set as needed. However, this method has the following shortcomings:

[0004] 1. Inefficiency: Manual data processing is time-consuming and labor-intensive, especially when the amount of data is huge, the processing cycle is significantly extended.

[0005] 2. High cost: It requires a lot of human resources investment, which increases the cost of data set production.

[0006] 3. Error-prone: Manual operations cannot avoid problems such as data overlap, duplication or omission, which affects the quality of the data set. Summary of the invention

[0007] In view of this, the present invention provides a method for preparing a redundant object detection dataset, which can greatly improve the richness and diversity of the redundant object detection dataset.

[0008] The technical solution for implementing the present invention is as follows:

[0009] A method for preparing a redundant object detection data set comprises the following steps:

[0010] Step 1, collecting redundant samples: for the redundant detection target object, collecting redundant samples that need to be detected;

[0011] Step 2: Primary sampling of samples: Use a pure color background to sample each redundant object sample from multiple perspectives and multiple backgrounds;

[0012] Step 3: Intermediate sampling of samples: In the created redundant object detection environment, the collected samples are sampled from multiple perspectives and multiple backgrounds;

[0013] Step 4, creating a pre-training data set: the primary sampling data and the intermediate sampling data are mixed in a set ratio to form a pre-training data set;

[0014] Step 5: Targeted sampling: After training the model with the pre-training data set, use the model to detect the redundant objects in the actual detection environment, filter out the unrecognized redundant object samples, and perform targeted sampling. The sampling method is mixed sampling under multiple perspectives and multiple backgrounds.

[0015] Step 6: Negative sample sampling: After training the model with the pre-training data set, use the model to detect redundant objects in the actual detection environment, filter out normal samples that are misidentified, and perform negative sample sampling, that is, sampling of non-redundant objects. The sampling method is mixed sampling under multiple perspectives and multiple backgrounds.

[0016] Step 7: Create the final data set: Mix the primary sampling data, intermediate sampling data, targeted sampling data, and negative sample sampling data in a set ratio to form the final redundant object detection data set.

[0017] Furthermore, when performing primary sampling of samples in step 2 and intermediate sampling of samples in step 3 of the present invention, it is necessary to perform separate sampling of samples from multiple perspectives and multiple backgrounds at different photographing distances.

[0018] Furthermore, when the primary sampling of samples is performed in step 2 and the intermediate sampling of samples is performed in step 3 of the present invention, perspective transformation and / or scaling transformation are further performed.

[0019] Furthermore, the perspective transformation of the present invention is:

[0020]

[0021] Among them, T p is the 3rd-order perspective transformation matrix, Represents the two-dimensional homogeneous coordinates obtained by transforming the two-dimensional pixel coordinates of the image, Represents the three-dimensional space coordinates obtained by perspective transformation of the two-dimensional homogeneous coordinates of the image. It means mapping the three-dimensional space coordinates obtained after the perspective transformation of the image to new pixel coordinates in the two-dimensional plane.

[0022] Furthermore, the scaling transformation of the present invention:

[0023]

[0024] Among them, T s is the 2nd order scaling transformation matrix, Represents the two-dimensional pixel coordinates before the image scaling transformation. Represents the two-dimensional pixel coordinates after image scaling transformation.

[0025] Furthermore, in step 2, the present invention also uses a gradient-based clarity algorithm to remove the blurred samples.

[0026] The gradient-based image sharpness algorithm is as follows:

[0027] D(f ′ )=∑ y ∑ x |G(x,y)|(G(x,y)>T)

[0028] Where T is the given edge detection threshold and G(x,y) represents the gradient at the pixel point (x,y).

[0029] Furthermore, in step three, the present invention also uses an image clarity algorithm based on information entropy to remove blurry samples.

[0030] The image clarity algorithm based on information entropy is as follows:

[0031]

[0032] Among them, information entropy D(f ′ ) represents the distribution of image gray levels, p i It represents the probability of occurrence of a pixel with gray value i in the image, and L is the total number of gray levels.

[0033] Furthermore, the pre-training data set in step 4 of the present invention is obtained by mixing the primary sampling data and the intermediate sampling data in a ratio of 1:4, and then randomly dividing them into a training set, a validation set and a test set in a ratio of 3:1:1.

[0034] Furthermore, in step seven of the present invention, the primary sampling data, intermediate sampling data, targeted sampling data and negative sample sampling data are mixed in a ratio of 2:7:1:1, and then randomly divided into a training set, a validation set and a test set in a ratio of 7:2:2 to form a final redundant object detection data set.

[0035] Furthermore, the model described in the present invention is a Yolo, Faster or R-CNN model.

[0036] Beneficial Effects

[0037] First, the present invention adopts targeted sampling and negative sample sampling, which greatly improves the richness and diversity of the data set.

[0038] Second, the present invention uses an image clarity algorithm to clean the blurred images in the data set, further improving the efficiency of data set production while also increasing the reliability of the data set.

[0039] Third, the present invention greatly improves the efficiency of data set production and saves time costs by utilizing the graphic transformation technology in computer vision to replace manual sampling. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 Create pipelines for datasets;

[0042] Figure 2 For some of the excess samples;

[0043] Figure 3 It is the primary sampling of a single sample (same sampling distance but different viewing angles);

[0044] Figure 4 It is the primary sampling of a single sample (same viewing angle but different sampling distances);

[0045] Figure 5 is the SIFT feature extracted from the image;

[0046] Figure 6 Blurred images are filtered out using a gradient-based image sharpness algorithm;

[0047] Figure 7 Mixed sampling of multiple samples from different perspectives;

[0048] Figure 8 Blurred images are filtered out using the image clarity algorithm based on information entropy;

[0049] Fig. 9 These are samples that were missed during targeted sampling;

[0050] Fig.10 It is the false positive sample in negative sample sampling. DETAILED DESCRIPTION

[0051] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0052] It should be noted that the following embodiments and features in the embodiments may be combined with each other in the absence of conflict; and, based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making any creative work are within the scope of protection of the present disclosure.

[0053] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0054] The present application embodiment provides a method for preparing a redundant object detection data set, comprising the following steps:

[0055] Step 1: Collecting redundant samples: For the redundant detection target object, collect the redundant samples that need to be detected.

[0056] Collecting samples of excess objects: According to the specific target object of excess object detection, determine and collect the samples of excess objects that need to be detected. The samples of excess objects should be combined with actual needs. For excess objects with fixed shapes (such as fallen nuts, metal washers, etc.), at least 1 sample should be collected; for excess objects with no fixed shape (such as discarded iron wire, scraps of paper, etc.), at least 3 to 5 samples should be collected.

[0057] Step 2: Primary sampling of samples: Use a pure color background to sample each redundant object sample from multiple perspectives and backgrounds.

[0058] When sampling, each redundant object sample needs to be sampled separately from multiple perspectives and backgrounds. The number of samples for a single sample should not be less than 50, and the image resolution should not be less than 640*640. The multiple perspectives can be divided into sampling from different perspectives such as the upper left, upper right, lower left, lower right, and directly above (simulated by the perspective transformation of the image), and the multiple backgrounds can be divided into different color backgrounds such as red, yellow, blue, green, black, and white (simulated by python and Pillow third-party libraries). In addition, the samples need to be sampled at different shooting distances (according to the principle of "near is big and far is small", the image scaling transformation can effectively simulate the sampling of samples at different shooting distances).

[0059] Perspective transformation is to project an image onto a new viewing plane, also known as projection mapping. The effect after perspective transformation is equivalent to the image taken when the sampling angle changes. The principle of perspective transformation is as follows:

[0060]

[0061] Among them, T p is the 3rd-order perspective transformation matrix, Represents the two-dimensional homogeneous coordinates obtained by transforming the two-dimensional pixel coordinates of the image, Represents the three-dimensional space coordinates obtained by perspective transformation of the two-dimensional homogeneous coordinates of the image. It means mapping the three-dimensional space coordinates obtained after the perspective transformation of the image to new pixel coordinates in the two-dimensional plane.

[0062] T p is a 3rd-order perspective transformation matrix, as shown below:

[0063]

[0064] Scaling transformation belongs to affine transformation, which is a linear transformation from two-dimensional coordinates to two-dimensional coordinates. It only involves the linear transformation of two-dimensional graphics in a plane. The principle of scaling transformation is as follows:

[0065]

[0066] Among them, T s is the 2nd order scaling transformation matrix, Represents the two-dimensional pixel coordinates before the image scaling transformation. Represents the two-dimensional pixel coordinates after image scaling transformation.

[0067]

[0068] After sampling is completed, a gradient-based clarity algorithm needs to be used to remove blurred samples (under a pure color background, the edge information noise of the image is relatively small, and the error of removing blurred images using a gradient-based image clarity algorithm is low). This separate sampling under a pure color background is helpful for the detection model to obtain the texture features of the sample.

[0069] The gradient-based image sharpness algorithm is as follows:

[0070] D(f ′ )=∑ y ∑ x |G(x,y)|(G(x,y)>T)

[0071] Where T is the given edge detection threshold, G(x,y) represents the gradient at the pixel point (x,y), and its form is as follows:

[0072]

[0073] Among them, G x (x,y) and G y (x, y) represents the horizontal gradient and vertical gradient at the pixel point (x, y) respectively.

[0074] Step 3: Intermediate sampling of samples: In the created redundant object detection environment, the collected samples are sampled from multiple perspectives and multiple backgrounds.

[0075] The actual redundant object detection environment created restores the actual detection scene as much as possible to ensure the authenticity of sample collection. When sampling, it is necessary to conduct mixed sampling of multiple perspectives and multiple backgrounds for the redundant object samples (sampling multiple samples at the same time), and the number of samples is not less than 200*n (n is the number of redundant object samples). As with primary sampling, samples need to be collected in the simulated redundant object detection environment from different perspectives.

[0076] In addition, intermediate sampling requires mixed sampling of samples to simulate actual detection conditions. Similarly, intermediate sampling can also use the perspective and scaling transformation technology of graphics instead of manual sampling. Finally, the collected samples also need to be cleaned. The image clarity algorithm based on information entropy is used for data cleaning of intermediate sampling (the sample background of intermediate sampling is more complex, the edge information noise in the image is larger, and the error of the gradient-based image clarity algorithm is larger).

[0077] The image clarity algorithm based on information entropy is as follows:

[0078]

[0079] Among them, p i Indicates the probability of a pixel with gray value i appearing in the image. L is the total number of gray levels (the default value is 256). Information entropy D(f ′ ) represents the distribution of the grayscale levels of the image. A high information entropy indicates that the grayscale distribution of the image is uniform and the image contains more information. A low information entropy indicates that the grayscale distribution of the image is single, the image contains less information, and the image may be blurred.

[0080] Step 4: Create a pre-training data set: Mix the primary sampling data with the intermediate sampling data in a set ratio to form a pre-training data set.

[0081] The pre-training data set needs to mix the primary sampling data and the intermediate sampling data in a ratio of 1:4, and then randomly divide them into training set, validation set and test set in a ratio of 3:1:1. The random division of the data set can be achieved through the Sklearn tool, which is a machine learning tool based on the Python language. After the data set and category labels are given, the data set can be randomly divided into a certain ratio.

[0082] Step 5: Targeted sampling: In the actual detection environment, after training the model with the pre-trained data set, perform target detection on the redundant objects, filter out samples with poor detection results (unidentified redundant object samples), and perform targeted sampling. The sampling method is also mixed sampling under multiple perspectives and multiple backgrounds. When using the pre-trained model to detect redundant objects, it is necessary to use more mature target detection models such as Yolo and Faster R-CNN for redundant object detection.

[0083] Step 6: Negative sample sampling: In the actual detection environment, after training the model with the pre-training data set, perform target detection on redundant objects, filter out samples with poor detection results (normal samples that are misidentified), and sample negative samples (samples that are not redundant objects). The negative samples in the data set will improve the generalization ability of the detection model and reduce the false detection of redundant objects in the actual detection process.

[0084] Step 7: Create the final dataset: Mix the primary sampling data, intermediate sampling data, targeted sampling data, and negative sample sampling data in a ratio of 2:7:1:1, and then randomly divide them into training set, validation set, and test set in a ratio of 7:2:2 to form the final redundant object detection dataset.

[0085] The dataset is annotated manually using tools such as MakeSence and LabelImg. MakeSence is an online image annotation tool that provides a simple and intuitive interface to mark objects in images. It is easy to operate and supports multiple export types. LabelImg is an open source image annotation tool that also provides a user-friendly graphical interface and supports multiple annotation formats.

[0086] Examples:

[0087] like Figure 1 FIG. 1 is a schematic diagram of the process steps of a basic implementation method of a method for preparing a redundant object detection data set of the present invention, and the method includes:

[0088] Step 1: Collect the redundant samples: According to the specific redundant detection target object, determine and collect the redundant samples that need to be detected. Figure 2These are some samples of excess materials (two fallen nuts, two fallen metal washers, three discarded iron wires and four pieces of paper scraps).

[0089] Step 2: Sample primary sampling: Use a simple solid color background to sample the redundant samples separately. Figure 3 The image is a normal distance sampling of a single sample wire under a pure color background, including 9 different viewing angles, and the resolution of each image is 1280*720. Among these 9 images, the middle image (the viewing angle taken from directly above) is used as the benchmark to calculate the projection transformation matrix between the images and the other images. The subsequent samples only need to be taken from directly above, and then the other viewing angles are collected using projection transformation instead of manual photography. Figure 4 For samples collected at different sampling distances from the same viewing angle, the middle image is used as a reference to calculate the scaling transformation matrix between the other two images. Subsequent samples can be scaled to replace manual sampling at different distances. The calculation method of the projection transformation matrix and the scaling transformation matrix is ​​to first extract feature points from the two images, and then calculate the projection matrix based on the pixel coordinates of the extracted feature points. There are many methods for extracting feature points (SURF, SIFT, ORB and Harris corner points, etc.). This embodiment selects the SIFT method to extract feature points, such as Figure 5 After all samples are collected, a script is written in Python to implement a gradient-based clarity algorithm to remove the blurred samples. Figure 6 Shown is a blurred image filtered using a gradient-based sharpness algorithm.

[0090] Step 3: Create a simulated detection environment: Create a simulated redundant object detection environment to restore the actual detection scene as much as possible;

[0091] Step 4: Intermediate sampling of samples: In a simulated redundant object detection environment, the collected samples are mixed and sampled. Figure 7 The figure shows the mixed sampling of some samples under different viewing angles in a simulated detection environment. The mixed sampling is the same as the primary sampling, using the perspective transformation and scaling transformation of the image to replace the manual sampling of different viewing angles and distances. After the sampling is completed, a script is written in Python to implement an image clarity algorithm based on information entropy to remove the blurred samples. Figure 8 Shown is a blurred image filtered using a gradient-based sharpness algorithm.

[0092] Step 5 Create a pre-training data set: Mix the primary sampling data and the intermediate sampling data in a ratio of 1:4. The random mixing of the two sampling data can be achieved by writing a script file in Python; After the two sampling data are randomly mixed, the sampling data needs to be annotated with the help of the MakeSence annotation tool (data set annotation can also be borrowed from other tools. This embodiment uses the online annotation tool MakeSence); After the annotation is completed, it is randomly divided into a training set, a validation set, and a test set in a ratio of 3:1:1. The division of the data set requires the use of the Sklearn tool, and the data set is randomly divided into a training set, a validation set, and a test set in proportion twice (divided twice because the Sklearn tool can randomly divide the data set into a training set and a test set in a certain ratio. Here, the entire data set is randomly divided into an intermediate set and a test set at a ratio of 4:1 for the first time, and the intermediate set is divided into a training set and a validation set for the second time at a ratio of 3:1);

[0093] Step 6 Targeted sampling: In the actual detection environment, use the pre-trained dataset to train the detection models Yolo v5, Yolo v8 and Faster R-CNN to detect redundant objects, such as Fig. 9 As shown, samples with poor detection results (missed detection) are screened out, and targeted sampling is performed. The sampling method is also mixed sampling under multiple perspectives and multiple backgrounds (when performing targeted sampling, other mature detection models can also be used to screen samples. This embodiment uses Yolo v5, Yolo v8 and Faster R-CNN for mixed screening);

[0094] Step 7 Negative sample sampling: In the actual detection environment, use the pre-trained dataset to train the detection models Yolo v5, Yolo v8 and Faster R-CNN to detect redundant objects, such as Fig.10 As shown, samples with poor detection results (false detection) are screened out, and negative sample sampling is performed. The sampling method is also mixed sampling under multiple perspectives and multiple backgrounds (when sampling negative samples, other mature detection models can also be used to screen samples. This embodiment uses Yolo v5, Yolo v8 and Faster R-CNN mixed screening);

[0095] Step 8 creates the final data set: randomly mix the primary sampling data, intermediate sampling data, targeted sampling data and negative sample sampling data in a ratio of 2:7:1:1 (implemented by writing a Python script), then use the MakeSence tool to annotate the sampling data, and finally use the Sklearn tool to randomly divide the data set into training set, validation set and test set in a ratio of 7:2:2 to form the final redundant object detection data set.

[0096] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for preparing a redundant object detection data set, characterized in that: The steps include: Step 1, collecting redundant samples: for the redundant detection target object, collecting redundant samples that need to be detected; Step 2: Primary sampling of samples: Use a pure color background to sample each redundant object sample from multiple perspectives and multiple backgrounds; Step 3: Intermediate sampling of samples: In the created redundant object detection environment, the collected samples are sampled from multiple perspectives and multiple backgrounds; Step 4, creating a pre-training data set: the primary sampling data and the intermediate sampling data are mixed in a set ratio to form a pre-training data set; Step 5: Targeted sampling: After training the model with the pre-training data set, use the model to detect the redundant objects in the actual detection environment, filter out the unrecognized redundant object samples, and perform targeted sampling. The sampling method is mixed sampling under multiple perspectives and multiple backgrounds. Step 6: Negative sample sampling: After training the model with the pre-training data set, use the model to detect redundant objects in the actual detection environment, filter out normal samples that are misidentified, and perform negative sample sampling, that is, sampling of non-redundant objects. The sampling method is mixed sampling under multiple perspectives and multiple backgrounds. Step 7: Create the final data set: Mix the primary sampling data, intermediate sampling data, targeted sampling data, and negative sample sampling data in a set ratio to form the final redundant object detection data set.

2. The method for preparing a redundant object detection data set according to claim 1, characterized in that: When performing primary sampling of samples in step 2 and intermediate sampling of samples in step 3, it is necessary to sample the samples from multiple perspectives and multiple backgrounds at different shooting distances.

3. The method for preparing a redundant object detection data set according to claim 1, characterized in that: When performing primary sampling of samples in step 2 and intermediate sampling of samples in step 3, perspective transformation and / or scaling transformation are further performed.

4. The method for preparing a redundant object detection data set according to claim 3, characterized in that: The perspective transformation is: Among them, T p is the 3rd-order perspective transformation matrix, Represents the two-dimensional homogeneous coordinates obtained by transforming the two-dimensional pixel coordinates of the image, Represents the three-dimensional space coordinates obtained by perspective transformation of the two-dimensional homogeneous coordinates of the image. It means mapping the three-dimensional space coordinates obtained after the perspective transformation of the image to new pixel coordinates in the two-dimensional plane.

5. The method for preparing a redundant object detection data set according to claim 3, characterized in that: The scaling transformation: Among them, T s is the 2nd order scaling transformation matrix, Represents the two-dimensional pixel coordinates before the image scaling transformation. Represents the two-dimensional pixel coordinates after image scaling transformation.

6. The method for preparing a redundant object detection data set according to claim 1, characterized in that: In step 2, the gradient-based clarity algorithm is used to remove the blurred samples. The gradient-based image sharpness algorithm is as follows: D(f ′ )=∑ y ∑ x |G(x,y)|(G(x,y)>T) Where T is the given edge detection threshold and G(x,y) represents the gradient at the pixel point (x,y).

7. The method for preparing a redundant object detection data set according to claim 1, characterized in that: In step three, the image clarity algorithm based on information entropy is used to remove the blurred samples. The image clarity algorithm based on information entropy is as follows: Among them, information entropy D(f ′ ) represents the distribution of image gray levels, p i It represents the probability of occurrence of a pixel with gray value i in the image, and L is the total number of gray levels.

8. The method for preparing a redundant object detection data set according to claim 1, characterized in that: The pre-training data set in step 4 is obtained by mixing the primary sampling data and the intermediate sampling data in a ratio of 1:4, and then randomly dividing them into a training set, a validation set, and a test set in a ratio of 3:1:

1.

9. The method for preparing a redundant object detection data set according to claim 1, characterized in that: In the step seven, the primary sampling data, intermediate sampling data, targeted sampling data and negative sample sampling data are mixed in a ratio of 2:7:1:1, and then randomly divided into a training set, a validation set and a test set in a ratio of 7:2:2 to form the final redundant object detection data set.

10. The method for preparing a redundant object detection data set according to claim 1, characterized in that: The model is a Yolo, Faster or R-CNN model.

Images