A method for making a superfluous detection dataset

Abstract

The application belongs to the technical field of data processing, and provides a manufacturing method of redundant object detection dataset, comprising the following steps: step one, collecting redundant object samples; step two, primary sampling of samples; step three, intermediate sampling of samples; step four, creating a pre-training dataset; step five, targeted sampling: screening out un-identified redundant object samples and performing targeted sampling, and the sampling mode is mixed sampling under multiple perspectives and multiple backgrounds; step six, negative sample sampling: screening out mis-identified normal samples and performing negative sample sampling; step seven, creating a final dataset: mixing the primary sampling data, the intermediate sampling data, the targeted sampling data and the negative sample sampling data according to a set proportion to form a final redundant object detection dataset.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data processing, in particular to a method for making a foreign matter detection dataset, which is applicable to application scenarios such as machine learning and artificial intelligence that require dataset support. BACKGROUND

[0002] With the advent of the big data era, the making of datasets has become an important link in the training of machine learning and artificial intelligence models. For some specific application scenarios, such as environments with closed packaging and visual obstruction, there may be metal foreign matters (usually derived from metal parts such as screws, nails, metal shavings, etc. in the processing or assembly process) and non-metal foreign matters (possibly from the processing of packaging materials, insulating materials or other non-metal materials, such as plastic fragments, rubber particles, fiber filaments, etc.), which cannot be directly identified by the human eye and often need to be collected by an endoscope and identified using artificial intelligence-related detection algorithms. However, these scenarios often do not have the required public datasets, and human-made datasets are needed. Traditional dataset making methods usually rely on manual collection and organization of data, which has the problems of low efficiency, high cost, and easy errors. Especially when dealing with large-scale and diversified data, these problems are particularly prominent.

[0003] In the prior art, the method for making a dataset mainly includes the following steps: first, collecting a large amount of raw data; second, cleaning and preprocessing the data to remove noise and outliers; and finally, dividing the data into training, validation, and test sets as needed. However, this method has the following disadvantages:

[0004] 1. Low efficiency: manual data processing is time-consuming and labor-intensive, especially when the data volume is large, the processing period is significantly prolonged.

[0005] 2. High cost: requires a large amount of human resources, increasing the cost of making the dataset.

[0006] 3. Error-prone: manual operations are prone to data crossing, duplication, or omission, affecting the quality of the dataset. SUMMARY

[0007] Therefore, the present application provides a method for making a foreign matter detection dataset, which can greatly improve the richness and diversity of the foreign matter detection dataset.

[0008] The technical solution of the present application is as follows:

[0009] A method for making a foreign matter detection dataset includes the following steps:

[0010] Step 1: Collect foreign matter samples: For the target object of foreign matter detection, collect the foreign matter samples that need to be detected;

[0011] Step two, sample primary sampling: using a pure color background to sample each excess sample from multiple perspectives and multiple backgrounds;

[0012] Step three, sample intermediate sampling: in the created excess detection environment, the collected samples are mixed sampled from multiple perspectives and multiple backgrounds;

[0013] Step four, creating a pre-training data set: the primary sampling data and the intermediate sampling data are mixed at a set ratio to form a pre-training data set;

[0014] Step five, targeted sampling: after training the model using the pre-training data set, the model is used to detect the excess in the actual detection environment, and the unrecognized excess samples are screened out and sampled, and the sampling method is mixed sampling from multiple perspectives and multiple backgrounds;

[0015] Step six, negative sample sampling: after training the model using the pre-training data set, the model is used to detect the excess in the actual detection environment, and the misidentified normal samples are screened out and sampled, that is, non-excess sample sampling, and the sampling method is mixed sampling from multiple perspectives and multiple backgrounds

[0016] Step seven, creating a final data set: the primary sampling data, the intermediate sampling data, the targeted sampling data and the negative sample sampling data are mixed at a set ratio to form a final excess detection data set.

[0017] Further, in the step two of the present application, the sample primary sampling and the step three of the present application, the sample intermediate sampling, different photographing distances are used to sample the sample from multiple perspectives and multiple backgrounds.

[0018] Further, in the step two of the present application, the sample primary sampling and the step three of the present application, the sample intermediate sampling, further perspective transformation and / or scaling transformation are performed.

[0019] Further, the perspective transformation of the present application is:

[0020]

[0021] Wherein, T p is a 3-order perspective transformation matrix, represents the two-dimensional homogeneous coordinates obtained by converting the two-dimensional pixel coordinates of the image, represents the three-dimensional space coordinates obtained by performing perspective transformation on the two-dimensional homogeneous coordinates of the image, represents the new pixel coordinates obtained by mapping the three-dimensional space coordinates of the image obtained after perspective transformation to a two-dimensional plane.

[0022] Further, the scaling transformation of the application:

[0023]

[0024] Wherein, T s is a 2-order scaling transformation matrix, represents the two-dimensional pixel coordinates before image scaling transformation, represents the two-dimensional pixel coordinates after image scaling transformation.

[0025] Further, the application also uses the gradient-based definition algorithm to remove the fuzzy samples in step two,

[0026] The gradient-based image definition algorithm is as follows:

[0027] D(f ′ )=∑ y ∑ x |G(x,y)|(G(x,y)>T)

[0028] Wherein, T is a given edge detection threshold, and G(x,y) represents the gradient at pixel point (x,y).

[0029] Further, the application also uses the information entropy-based image definition algorithm to remove the fuzzy samples in step three,

[0030] The information entropy-based image definition algorithm is as follows:

[0031]

[0032] Wherein, the information entropy D(f ′ ) represents the distribution of image gray levels, p i represents the probability of the occurrence of pixels with gray value i in the image, and L is the total number of gray levels.

[0033] Further, the pre-training data set in step four of the application is obtained by mixing the primary sampling data and the intermediate sampling data in a ratio of 1:4, and then randomly dividing them in a ratio of 3:1:1 into a training set, a validation set and a test set.

[0034] Further, the primary sampling data, the intermediate sampling data, the targeted sampling data and the negative sample sampling data in step seven of the application are mixed in a ratio of 2:7:1:1, and then randomly divided into a training set, a validation set and a test set in a ratio of 7:2:2 to form the final excess detection data set.

[0035] Further, the model of the application is Yolo, Faster or R-CNN model.

[0036] Beneficial effects

[0037] Firstly, the application adopts targeted sampling and negative sample sampling, greatly improving the richness and diversity of the data set.

[0038] Secondly, the application uses the image sharpness algorithm to clean the fuzzy images in the data set, further improving the efficiency of data set production, and also increasing the reliability of the data set.

[0039] Thirdly, the application uses the image transformation technology in computer vision to replace manual sampling, greatly improving the efficiency of data set production and saving time and cost. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0041] Figure 1 For data set production process;

[0042] Figure 2 For some redundant sample;

[0043] Figure 3 For primary sampling of single sample (different viewing angles at the same sampling distance);

[0044] Figure 4 For primary sampling of single sample (different sampling distances at the same viewing angle);

[0045] Figure 5 For SIFT features extracted from pictures;

[0046] Figure 6 For fuzzy pictures screened out by gradient-based image sharpness algorithm;

[0047] Figure 7 For mixed sampling of multiple samples at different viewing angles;

[0048] Figure 8 For fuzzy pictures screened out by information entropy-based image sharpness algorithm;

[0049] Figure 9 For missed samples in targeted sampling;

[0050] Figure 10 For false positive samples in negative sample sampling. DETAILED DESCRIPTION

[0051] Embodiments of the present application will be described in detail below with reference to the drawings.

[0052] It should be noted that the following embodiments and features in the embodiments can be combined with each other in the case of no conflict; and all other embodiments obtained by those skilled in the art based on the embodiments in the present disclosure without creative labor shall fall within the scope of the present disclosure.

[0053] It should be noted that various aspects of the embodiments described below are within the scope of the appended claims. As will be apparent, the aspects described herein can be implemented in various forms and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, one of ordinary skill in the art will appreciate that one aspect described herein can be implemented independently of any other aspects and that two or more of these aspects can be combined in various ways. For example, an apparatus can be implemented and / or a method can be practiced using any number of the aspects set forth herein. In addition, such an apparatus can be implemented and / or such a method can be practiced using other structure and / or functionality in addition to or other than one or more of the aspects set forth herein.

[0054] The embodiment of the present application is a method for making a redundant object detection dataset, comprising the following steps:

[0055] Step one, collecting redundant object samples: for the redundant object detection target object, collect the redundant object samples that need to be detected.

[0056] Collecting redundant object samples: according to the specific redundant object detection target object, determine and collect the redundant object samples that need to be detected, which need to be combined with the actual demand. For redundant objects with fixed shape (such as fallen nuts, metal gaskets, etc.), at least one sample is collected; for redundant objects without fixed shape (such as discarded iron wire, paper scraps, etc.), at least 3-5 samples are collected.

[0057] Step two, sample primary sampling: using a pure color background to sample each redundant object sample from multiple angles and multiple backgrounds.

[0058] When sampling, each redundant object sample needs to be sampled individually from multiple angles and multiple backgrounds, and the number of samples for a single sample is not less than 50 pictures, and the picture resolution is not less than 640*640. The multiple angles can be divided into different angles such as left upper, right upper, left lower, right lower and directly above (realized by perspective transformation of image), and the multiple backgrounds can be divided into different color backgrounds such as red, yellow, blue, green, black and white (realized by python and Pillow third-party library), in addition, different shooting distances are needed to sample the samples (according to the principle of "near big and far small", the zooming transformation of image can effectively simulate the sampling of samples under different shooting distances).

[0059] Perspective transformation is to project a picture to a new view plane, also called projection mapping, the effect of perspective transformation is equivalent to the picture taken when the sampling view angle changes, the principle of perspective transformation is as follows:

[0060]

[0061] Wherein, T p is a 3-order perspective transformation matrix, represents the two-dimensional homogeneous coordinates converted from the two-dimensional pixel coordinates of the image, represents the three-dimensional space coordinates obtained by performing perspective transformation on the two-dimensional homogeneous coordinates of the image, represents the new pixel coordinates obtained by mapping the three-dimensional space coordinates obtained by performing perspective transformation on the image to a two-dimensional plane.

[0062] T p is a 3-order perspective transformation matrix, as follows:

[0063]

[0064] Scaling transformation belongs to affine transformation, which is a linear transformation between two-dimensional coordinates, only involving linear transformation of two-dimensional graphics in a plane, the principle of scaling transformation is as follows:

[0065]

[0066] Wherein, T s is a 2-order scaling transformation matrix, represents the two-dimensional pixel coordinates of the image before scaling transformation, represents the two-dimensional pixel coordinates of the image after scaling transformation.

[0067]

[0068] After sampling, the blurred samples need to be removed by using the gradient-based sharpness algorithm (in a pure color background, the edge information noise of the image is relatively small, and the error of removing blurred images by using the gradient-based image sharpness algorithm is low), through this separate sampling in a pure color background, it is beneficial to the texture features of the samples obtained by the detection model.

[0069] The gradient-based image sharpness algorithm is as follows:

[0070] D(f ′ ) = ∑ y ∑ x |G(x,y)|(G(x,y)>T)

[0071] where T is a given edge detection threshold, G(x, y) represents the gradient at pixel point (x, y) and has the following form:

[0072]

[0073] where G x (x, y) and G y (x, y) represent the horizontal and vertical gradients at pixel point (x, y), respectively.

[0074] Step three, intermediate sampling of samples: in the created foreign matter detection environment, the collected samples are mixed sampled in multiple perspectives and multiple backgrounds.

[0075] The created actual foreign matter detection environment restores the actual detection scene as much as possible to ensure the authenticity of the sample collection. When sampling, the foreign matter samples need to be mixed sampled in multiple perspectives and multiple backgrounds (multiple samples are sampled at the same time), and the sampling quantity is not less than 200*n (n is the number of foreign matter samples). Like the primary sampling, the samples need to be collected in the simulated foreign matter detection environment using different perspectives.

[0076] In addition, the intermediate sampling needs to mix sample the samples to simulate the actual detection working condition. Similarly, the intermediate sampling can also use the perspective and scaling transformation technology of graphics to replace manual sampling. Finally, the collected samples also need to be cleaned, and the image definition algorithm based on information entropy is used for data cleaning of the intermediate sampling (the background of the intermediate sampling sample is relatively complex, the edge information noise in the image is large, and the error of the image definition algorithm based on gradient is large).

[0077] The image definition algorithm based on information entropy is as follows:

[0078]

[0079] where p i represents the probability of the occurrence of a pixel with a gray value of i in the image. L is the total number of gray levels (the default value is 256). The information entropy D(f ′ ) represents the distribution of the image gray levels. A high information entropy indicates that the image gray distribution is uniform, the image contains more information, a low information entropy indicates that the image gray distribution is single, the image information content is less, and the image may be blurred.

[0080] Step four, creating a pre-training data set: the primary sampling data and the intermediate sampling data are mixed in a set proportion to form a pre-training data set.

[0081] The pre-training data set needs to mix the primary sampling data and the intermediate sampling data in a ratio of 1:4, and then randomly divide them into a training set, a validation set and a test set in a ratio of 3:1:1. The random division of the data set can be realized by the Sklearn tool, which is a machine learning tool based on the Python language. Given a data set and a class label, the data set can be randomly divided according to a certain ratio.

[0082] Step five, targeted sampling: in the actual detection environment, the pre-trained data set is used to train the model, and the redundant objects are detected, and the samples with poor detection results (unidentified redundant object samples) are screened out and targeted sampling is performed. The sampling method is also mixed sampling in multiple perspectives and multiple backgrounds. When detecting the redundant objects by using the pre-trained model, a mature target detection model such as Yolo or Faster R-CNN is selected to detect the redundant objects.

[0083] Step six, negative sample sampling: in the actual detection environment, the pre-trained data set is used to train the model, and the redundant objects are detected, and the samples with poor detection results (misidentified normal samples) are screened out and negative sample (non-redundant object sample) sampling is performed. The negative samples in the data set can improve the generalization ability of the detection model, and reduce the misidentification of redundant objects in the actual detection process.

[0084] Step seven, creating a final data set: the primary sampling data, the intermediate sampling data, the targeted sampling data and the negative sample sampling data are mixed in a ratio of 2:7:1:1, and then randomly divided into a training set, a validation set and a test set in a ratio of 7:2:2 to form a final redundant object detection data set.

[0085] The data set is labeled by using labeling tools such as MakeSence and LabelImg for manual labeling. MakeSence is an online image labeling tool, which provides a simple and intuitive interface to mark objects in an image, is easy to operate, and supports multiple export types. LabelImg is an open source image labeling tool, which also provides a user-friendly graphical interface and supports multiple labeling formats.

[0086] Example:

[0087] As shown in Figure 1 , it is a flow step schematic diagram of the basic embodiment of the method for manufacturing a redundant object detection data set, and the method comprises the following steps:

[0088] Step 1, collecting redundant object samples: according to the specific redundant object detection target object, determining and collecting the redundant object samples to be detected, Figure 2For part of the redundant sample (2 dropped nut, 2 dropped metal gasket, 3 discarded wire and 4 pieces of paper scraps).

[0089] Step 2: Primary sampling of samples: using a simple solid color background to sample the redundant sample alone, Figure 3 For a single sample of iron wire under a solid color background, the normal distance sampling contains 9 different perspectives, and the resolution of each picture is 1280*720. Among the 9 pictures, the middle picture (directly above the shooting perspective) is used as the reference to calculate the projection transformation matrix between the other images. Subsequent samples only need to be taken from directly above, and other perspectives can use projection transformation instead of manual photographing. Figure 4 For samples collected at different sampling distances under the same perspective, the middle picture is used as the reference to calculate the scaling transformation matrix between the other two pictures. Subsequent samples can be scaled by the scaling transformation matrix instead of manual sampling at different distances. The calculation method of the projection transformation matrix and the scaling transformation matrix is to extract feature points from two images first, and then calculate the projection matrix according to the pixel coordinates of the extracted feature points. There are many methods for extracting feature points (SURF, SIFT, ORB, and Harris corner points, etc.), and the SIFT method is selected to extract feature points in this embodiment, as shown in Figure 5 After all sample collection is completed, a script is written using python to implement a gradient-based clarity algorithm to remove blurred samples, Figure 6 As shown in the figure, the blurred pictures selected by the gradient-based clarity algorithm are shown.

[0090] Step 3: Create a simulated detection environment: create a simulated redundant object detection environment to restore the actual detection scene as much as possible;

[0091] Step 4: Intermediate sampling of samples: in the simulated redundant object detection environment, the collected samples are mixed sampled, Figure 7 As shown in the figure, part of the sample is mixed sampled under different perspectives in the simulated detection environment; like the primary sampling, the mixed sampling uses perspective transformation and scaling transformation of images to replace manual photographing sampling at different perspectives and distances. After sampling is completed, a script is written using python to implement an information entropy-based image clarity algorithm to remove blurred samples, Figure 8 As shown in the figure, the blurred pictures selected by the gradient-based clarity algorithm are shown.

[0092] Step 5: Create a pre-training dataset: Mix the primary sampling data and the intermediate sampling data at a ratio of 1:4. Random mixing of the two sampling data can be achieved by writing a script file in Python. After random mixing of the two sampling data, label the sampling data with the help of the MakeSence labeling tool (the data set labeling can also use other tools, and the online labeling tool MakeSence is selected in this embodiment). After labeling, randomly divide the data set into training set, validation set and test set at a ratio of 3:1:1. The data set division needs to use the Sklearn tool to randomly divide the data set into training set, validation set and test set at a ratio of 3:1:1 (because the Sklearn tool can randomly divide the data set into training set and test set at a certain ratio, here the entire data set is first randomly divided into intermediate set and test set at a ratio of 4:1, and then the intermediate set is divided into training set and validation set at a ratio of 3:1);

[0093] Step 6: Targeted sampling: In the actual detection environment, use the pre-training dataset to train the detection models Yolo v5, Yolo v8 and Faster R-CNN to detect the excess material, as shown in Figure 9 , select samples with poor detection results (missed detection), and perform targeted sampling. The sampling method is also mixed sampling in multiple perspectives and multiple backgrounds (when targeted sampling, other mature detection models can also be used to select samples, and Yolo v5, Yolo v8 and Faster R-CNN are selected in this embodiment);

[0094] Step 7: Negative sample sampling: In the actual detection environment, use the pre-training dataset to train the detection models Yolo v5, Yolo v8 and Faster R-CNN to detect the excess material, as shown in Figure 10 , select samples with poor detection results (false detection), and perform negative sample sampling. The sampling method is also mixed sampling in multiple perspectives and multiple backgrounds (when negative sample sampling, other mature detection models can also be used to select samples, and Yolo v5, Yolo v8 and Faster R-CNN are selected in this embodiment);

[0095] Step 8: Create a final data set: Randomly mix the primary sampling data, intermediate sampling data, targeted sampling data and negative sample sampling data at a ratio of 2:7:1:1 (Python script implementation), then label the sampling data with the help of the MakeSence tool, and finally divide the data set into training set, validation set and test set at a ratio of 7:2:2 with the help of the Sklearn tool, forming the final excess material detection data set.

[0096] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of creating a set of excess detection data, characterized by, It comprises the following steps: Step one, collecting excess sample: for excess detection target object, collect the excess sample to be detected; Step two, sample primary sampling: use pure color background to sample each excess sample in multiple perspectives and multiple backgrounds; Step three, sample secondary sampling: in the created excess detection environment, mix sample collection in multiple perspectives and multiple backgrounds; Step four, creating pre-training data set: mix the primary sampling data and the secondary sampling data in a set proportion to form a pre-training data set; Step five, targeted sampling: after training the model with the pre-training data set, use the model to detect the excess in the actual detection environment, screen out the excess samples that are not recognized, and conduct targeted sampling in multiple perspectives and multiple backgrounds; Step six, negative sample sampling: after training the model with the pre-training data set, use the model to detect the excess in the actual detection environment, screen out the misidentified normal samples, and conduct negative sample sampling, i.e. non-excess sample sampling, in multiple perspectives and multiple backgrounds; Step seven, creating final data set: mix the primary sampling data, secondary sampling data, targeted sampling data and negative sample sampling data in a set proportion to form the final excess detection data set; In step two, the primary sampling and step three, the secondary sampling, different shooting distances are needed to sample the samples in multiple perspectives and multiple backgrounds, and further perspective transformation and / or scaling transformation are needed; In step two, the gradient-based clarity algorithm is also used to remove the blurred samples, The gradient-based image clarity algorithm is as follows: wherein, is a given edge detection threshold, denotes the gradient at pixel point . In step three, the information entropy-based image clarity algorithm is also used to remove the blurred samples, The information entropy-based image clarity algorithm is as follows: wherein the information entropy represents the distribution of image gray levels, represents the probability of the occurrence of pixels with gray values in the image, is the total number of gray levels.

2. The method of claim 1, wherein the step of generating the excess detection dataset is performed by a computer system. The perspective transformation is: ， wherein, is a 3rd order perspective transformation matrix, is a two-dimensional homogeneous coordinate converted from the two-dimensional pixel coordinate of the image, represents a three-dimensional space coordinate obtained by performing perspective transformation on the two-dimensional homogeneous coordinate of the image, represents a new pixel coordinate obtained by mapping the three-dimensional space coordinate obtained by performing perspective transformation on the image to a two-dimensional plane.

3. The method of claim 1, wherein the step of generating the data set of extraneous objects is performed by a computer system. The scaling transformation is: wherein is a 2nd order scaling transformation matrix, denotes the two-dimensional pixel coordinates before the image scaling transformation, denotes the two-dimensional pixel coordinates after the image scaling transformation.

4. The method of claim 1, wherein the step of generating the excess detection dataset is performed by a computer system. The pre-training data set in step four is to mix the primary sampling data and the secondary sampling data in a ratio of 1:4, and then randomly divide them into a training set, a validation set and a test set in a ratio of 3:1:

1.

5. The method of claim 1, wherein the step of generating the excess detection dataset comprises: In step seven, the primary sampling data, the secondary sampling data, the targeted sampling data and the negative sample sampling data are mixed in a ratio of 2:7:1:1, and then randomly divided into a training set, a validation set and a test set in a ratio of 7:2:2 to form the final excess detection data set.

6. The method of claim 1, wherein the step of generating the excess detection dataset is performed by a computer system. The model is Yolo, Faster or R-CNN model.

Images