Small sample target detection data enhancement method

Abstract

The invention relates to the field of target detection, in particular to a small sample target detection data enhancement method, which comprises the following steps of: dividing small sample data to form a training set, a verification set and a test set, and performing target detection labeling on the data set; using the center point position of the target detection frame in the training set as input, using the width and height of the target detection frame as labels, and training a neural network; performing two-dimensional Gaussian distribution conversion on the target detection frame in the training set; utilizing the image targets in the training set, and adopting a stable diffusion model to train an image to generate a new target picture; coordinates are randomly selected in a visual range, corresponding coordinates are generated through the previously trained large and small prior neural networks, and a new target detection frame is obtained; according to the invention, the problem of small sample target detection is effectively solved, the precision of target detection is remarkably improved, and various benefits, including reduction of false detection rate, improvement of production efficiency and reduction of production cost, are brought to manufacturing enterprises.

Description

Technical Field

[0001] The present invention relates to the field of target detection, and in particular to a small sample target detection data enhancement method. Background Art

[0002] In industrial production, target detection algorithms are widely used in automated production lines to achieve production optimization. By detecting and identifying target objects or defects in the production process, production efficiency can be improved, human errors and costs can be reduced. Target detection algorithms can quickly and accurately identify defects or errors in products, thereby adjusting production processes in a timely manner, reducing scrap rates and production time, and improving production quality and mass production capabilities. Through real-time monitoring and feedback, companies can quickly identify problems and make adjustments, improve the stability and reliability of production lines, and reduce resource waste and losses.

[0003] Image object detection has great potential in industrial defect detection, but in practical applications, since the probability of industrial defects is usually relatively low, there will be a lack of sufficient defect samples for training models, which may lead to the following problems: sample imbalance and insufficient generalization ability. For the above problems of small sample object detection, data enhancement is one of the effective solutions. The principle of data enhancement is to manually introduce some prior knowledge so that the model can generalize better in the case of small samples. The means of data enhancement include cropping, translation, rotation, etc. Through these operations, the model can learn more prior knowledge, such as translation invariance and rotation invariance, thereby improving generalization ability.

[0004] The data augmentation method of random copy and paste is used to improve the detection ability of deep learning models for targets. This method expands the data by randomly copying samples and pasting them to other pictures to improve the generalization ability of the model. However, the random copy and paste method interferes with the learning of target position and target size, and in the case of small samples, it will increase the probability of detection errors. Summary of the invention

[0005] In order to solve the above technical problems, an embodiment of the present invention provides a small sample target detection data enhancement method.

[0006] To achieve the above object, the present invention provides a small sample target detection data enhancement method, comprising the following steps:

[0007] Divide the small sample data into training set, validation set and test set, and annotate the data set for target detection;

[0008] Use the center point position of the target detection box in the training set as input and the width and height of the target detection box as labels to train the neural network;

[0009] Convert the target detection box in the training set into a two-dimensional Gaussian distribution;

[0010] Using the image targets in the training set, the stable diffusion model is used to train the images to generate new target images;

[0011] Randomly select coordinates within the visible range, generate corresponding coordinates through the previously trained size prior neural network, and obtain a new target detection frame;

[0012] The new target detection frame is converted into a two-dimensional Gaussian distribution, and the distance between the input target detection frame and the target detection frame in the training set is measured by similarity. The minimum distance is output, and if the minimum distance is lower than a given threshold, the detection frame is retained.

[0013] After obtaining a new target detection frame that meets the conditions, match and fill it with the target image generated by the stable diffusion model to obtain a new target that satisfies both the regional distance and the priors of near large and far small;

[0014] The original target is extracted from the training set, and is subjected to position constraints and shape constraints together with the image generated by the diffusion model to constrain the original target and the newly generated target. The target is randomly pasted into the randomly extracted original image of the training set to form an extended image to achieve constrained copy and paste data enhancement. The enhanced data is trained in the target detection model.

[0015] The LoRA method is used to fine-tune the stable diffusion model to generate a new target image. The process can be described as follows: first, the resolution of the target E in the original data set is reduced to construct a target image D with a lower resolution. The process can be described as follows:

[0016] D=E0⊙W+n

[0017] Where D∈R (H×W×3) is the extracted low-resolution image of size H×W, E0∈R H×W×3 is the original data image, W∈R (H×W×3) Represents the mapping from real high-quality images to low-resolution images, N∈R (H×W) is the noise term introduced due to sensor limitations and other factors,

[0018] By incorporating the low-resolution image D as a conditional input into the diffusion model, and then constraining the distance between the original image E and the resolution image D, the diffusion model learns the mapping from low-resolution restoration to high-resolution images in the reverse process.

[0019] The distance between the original image E and the resolution image D is constrained by the regularization strategy. Specifically, the original image and the low-resolution image are split into multiple small blocks, and these small blocks are clustered to form a clustering matrix by clustering. Each matrix has inherent image semantic information, which should not be changed during the diffusion model denoising process. The rank of this matrix can be used as an important measure of the global structure of the image. The matrix formed by the original image and the low-resolution image is regularized. The regularization function method can be expressed as:

[0020]

[0021] Where S W is the regularization method, L1 is the Manhattan distance Represents the output of singular value decomposition. By adjusting the adaptive factor K, the regularization function S is gradually applied during the training process to ensure that the model can effectively handle the characteristics of low-resolution images.

[0022] The target detection frame is transformed into a two-dimensional Gaussian distribution, specifically:

[0023]

[0024] The mean of a two-dimensional Gaussian distribution is represented by a two-dimensional vector of center coordinates, The covariance matrix is ​​represented as a diagonal matrix with w and h as traces,

[0025] The normalized bulldozer distance is used to measure the similarity between the input target detection box and the Gaussian distribution in the training set. The bulldozer distance is:

[0026]

[0027] Among them, m represents the mean, ∑ represents the variance, and the target detection frame is brought into the bulldozer distance:

[0028]

[0029] Among them, cx, cy, w, and h represent the center point coordinates and width and height of the target detection box.

[0030] Training a neural network is as follows:

[0031] All targets in the statistical data set are counted, and the center point position of the target is input into the neural network as a feature. The width and height information of the target are input, and the distance is measured by comparing the width and height information of the original sample. The gradient of the neural network is back-propagated to minimize the distance, so that the neural network can learn the mapping relationship between position and size.

[0032] A neural network is used to learn the prior relationship between distance and size. A floating-point data stream is input and a floating-point data stream is output. A perceptron with the most layers is used as the learning network.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention proposes a small sample target detection data enhancement method, which constrains the pasting position of the model and the pasting size of different types of targets. This needs to be customized according to different scenarios. In the transmission belt anomaly detection scenario, the following method is proposed.

[0035] To constrain the target position, we use the method of statistical sample distribution. First, we convert the sample target box into a two-dimensional Gaussian distribution, count the two-dimensional Gaussian distribution of all samples, calculate the mixed Gaussian distribution, set a threshold, and constrain the area outside the threshold. In simple terms, we use the original sample distribution to unify the location where the sample may need to be pasted. For example, the original samples all appear on the conveyor belt, and the sample statistics method can automatically obtain the conveyor belt area, and will not be pasted in the area outside the conveyor belt. This method of statistical sample distribution is applicable to all fixed scenarios with location information.

[0036] Regarding the problem of target size, in the case of conveyor belt anomaly detection, in addition to the size regularity of the sample itself, there is also prior knowledge that objects that are close together are larger and objects that are far away are smaller. This is done by introducing a neural network to learn the mapping relationship between the target area and the target size. In the process of copying and pasting data enhancement, the target size is adjusted through the neural network to force learning of the prior knowledge that objects that are close together are larger and objects that are far away are smaller, that is, the size of the target changes with the distance from the conveyor belt. By learning from a large amount of data, the model can better understand this prior knowledge and avoid confusing fuzzy targets in the distance with small targets nearby.

[0037] In order to solve the problem of insufficient diversity of pasted targets under small samples, a diffusion model is used to regenerate the targets. The diffusion model in the deep learning generative network is a generative model based on random walks, which generates data with complex structures by simulating the particle diffusion process. The generated samples and the original samples are used as the target library together. The target library has a certain diversity, thereby preventing the overfitting problem caused by the copy-paste method under constraints.

[0038] Through the present invention, the problem of small sample target detection in industry is effectively solved, and the accuracy of target detection is significantly improved. In practical applications, this method brings many benefits to manufacturing companies, including reducing false positive rates, improving production efficiency, and reducing production costs, thereby enhancing the competitiveness and sustainability of companies. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1It is the overall flow chart of the present invention;

[0040] Figure 2 This is a sample example picture;

[0041] Figure 3 Convert the target box to a Gaussian distribution visualization;

[0042] Figure 4 Visualize the distribution of sample frame constraints;

[0043] Figure 5 It is a performance comparison diagram of the present invention applied to the vfnet target detection model;

[0044] Figure 6 A performance comparison diagram of the present invention applied to a deformable target detection model;

[0045] Figure 7 This is a performance comparison chart of the present invention applied to the tood target detection model;

[0046] Figure 8 It is a performance comparison chart of the present invention applied to the dino target detection model. DETAILED DESCRIPTION

[0047] The following will be combined with the accompanying drawings to clearly and completely describe the technical solutions in some embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments provided by the present disclosure, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present disclosure.

[0048] In existing industrial production, target detection algorithms are widely used in automated production lines to achieve production optimization. By detecting and identifying target objects or defects in the production process, production efficiency can be improved, and human errors and costs can be reduced. Target detection algorithms can quickly and accurately identify defects or errors in products, thereby adjusting production processes in a timely manner, reducing scrap rates and production time, and improving production quality and mass production capabilities. Through real-time monitoring and feedback, companies can quickly identify problems and make adjustments, improve the stability and reliability of production lines, and reduce resource waste and losses.

[0049] With the rapid development of deep learning, object detection has made significant breakthroughs. Although image object detection has great potential in industrial defect detection, in practical applications, the probability of industrial defects is usually relatively low, resulting in a lack of sufficient defect samples for training models, which may lead to the following problems:

[0050] Sample imbalance: Since the number of defect samples is far less than that of normal samples, the model cannot fully learn the defect features during the training process, which can easily lead to inaccurate defect detection. Insufficient sample size can easily lead to overfitting of the model, and only a small number of defect sample features can be learned, which makes it impossible to effectively identify defects of different types or in different scenarios.

[0051] Insufficient generalization ability: Due to the lack of diverse training data, the generalization ability of the model will be limited, resulting in poor performance in actual production environments.

[0052] Data enhancement is one of the effective solutions to the above problems of small sample object detection. Driven by big data, the model can learn various forms of data, while in the case of small samples, the lack of various forms of data can easily lead to overfitting. The principle of data enhancement is to manually introduce some prior knowledge so that the model can generalize better in the case of small samples. The means of data enhancement include cropping, translation, rotation, etc. Through these operations, the model can learn more prior knowledge, such as translation invariance and rotation invariance, thereby improving generalization ability.

[0053] Traditional target detection data enhancement methods may not be sufficient to cover these complex situations. The data enhancement method of random copy and paste can be used to improve the detection ability of deep learning models for targets. This method expands the data by randomly copying samples and pasting them to other pictures to improve the generalization ability of the model. However, the random copy and paste method interferes with the learning of target position and target size. In the case of small samples, it will increase the probability of detection errors. For example, in the scenario of conveyor belt foreign body detection, the random copy and paste method will paste the target outside the conveyor belt, so that the model cannot learn the location information where the foreign body should be, and thus detect the objects outside the conveyor belt as targets, resulting in detection errors. The random copy and paste method will also randomly scale the size of the target, which will also cause the model to fail to learn the size information of the object, and the recognition accuracy of some small targets will be reduced. In the case of small samples, simply pasting existing targets is likely to lead to overfitting. In the case of destroying the target position information and size information, the random copy and paste method can easily have the opposite effect.

[0054] In order to solve the above technical problems, refer to Figure 1 - Figure 8 , the present application embodiment provides a method comprising the following steps:

[0055] Divide the small sample data into training set, validation set and test set, and annotate the data set for target detection;

[0056] The neural network is trained using the center point position of the target detection box in the training set as input and the width and height of the target detection box as labels.

[0057] Training a neural network is as follows:

[0058] All targets in the statistical data set are counted, and the center point position of the target is input into the neural network as a feature. The width and height information of the target are input, and the distance is measured by comparing the width and height information of the original sample. The gradient of the neural network is back-propagated to minimize the distance, so that the neural network can learn the mapping relationship between position and size.

[0059] See also Figure 2 Taking the conveyor belt scene as an example, this scene has the prior characteristic that things that are near are large and things that are far are small. In order to effectively retain this prior knowledge, a neural network is used to learn the prior relationship between distance and size. The input is a floating-point data stream, and the output is a floating-point data stream. The simplest multi-layer perceptron can be used as the learning network.

[0060] The target detection frame in the training set is transformed into a two-dimensional Gaussian distribution, specifically:

[0061]

[0062] The mean of a two-dimensional Gaussian distribution is represented by a two-dimensional vector of center coordinates, The covariance matrix is ​​represented as a diagonal matrix with w and h as traces,

[0063] Using the image targets in the training set, the stable diffusion model is used to train the images to generate new target images;

[0064] The LoRA method is used to fine-tune the diffusion model to generate a new target image. The process can be described as follows:

[0065] First, reduce the resolution of the target E in the original data set and construct a lower resolution target image D. The process can be described as:

[0066] D=E0⊙W+n,

[0067] Where D∈R (H×W×3) is the extracted low-resolution image of size H×W, E0∈R H×W×3 is the original data image, W∈R (H×W×3) Represents the mapping of real high-quality images to low-resolution images, taking into account hardware imaging errors, N∈R (H×W)It is a noise term introduced due to sensor limitations and other factors. By incorporating the low-resolution image D into the diffusion model as a conditional input, and then constraining the distance between the original image E and the resolution image D through a regularization strategy, the diffusion model learns the mapping from low-resolution restoration to high-resolution images in the reverse process. Regular constraint method, specifically splitting the original image and the low-resolution image into multiple small blocks of images, clustering these small blocks by clustering to form a clustering matrix. Each matrix has inherent image semantic information, which should not change during the denoising process of the diffusion model. The rank of this matrix can be used as an important measure of the global structure of the image. The matrix formed by the original image and the low resolution is regularized. This ensures that the learned samples have a similar global structure to the original image, which helps to improve model performance and generalization ability. The regularization function method can be expressed as:

[0068]

[0069] Where S W is the regularization method, L1 is the Manhattan distance, Represents the output of singular value decomposition. By adjusting the adaptive factor K, the regularization function S is gradually applied during the training process to ensure that the model can effectively handle the characteristics of low-resolution images;

[0070] Randomly select coordinates within the visible range, generate corresponding coordinates through the previously trained size prior neural network, and obtain a new target detection frame.

[0071] The new target detection box is converted into a two-dimensional Gaussian distribution, see Figure 3 , measure the distance between the input target detection box and the target detection box in the training set by similarity, output the minimum distance, if the minimum distance is lower than a given threshold, retain the detection box, given a threshold p, if the minimum distance is lower than p, retain the detection box.

[0072] Among them, the normalized bulldozer distance is used to measure the similarity between the input target detection box and the Gaussian distribution in the training set, wherein the bulldozer distance is calculated (bulldozer distance is a special term):

[0073]

[0074] Among them, m represents the mean, ∑ represents the variance, and the target detection frame is brought into the bulldozer distance:

[0075]

[0076] Among them, cx, cy, w, and h represent the center coordinates and width and height of the target detection box. The figure below shows the visualization of the global distance metric after calculation through sample distribution.

[0077] After obtaining a new target detection frame that meets the conditions, it is matched and filled with the target image generated by the stable diffusion model to obtain a new target. Multiple generated detection targets can be repeatedly obtained; the target simultaneously meets the prior of regional distance and near large and far small.

[0078] The original target is extracted from the training set, and is subjected to position constraints and shape constraints together with the image generated by the diffusion model to constrain the original target and the newly generated target. The target is randomly pasted into the randomly extracted original image of the training set to form an extended image, avoiding the target overlap problem and realizing constrained copy and paste data enhancement.

[0079] This task is a target detection task. The newly expanded data is the target object that needs to be detected. The main information includes location information and image information.

[0080] The enhanced data is trained in the target detection model, such as Faster R-CNN, YOLO, vfnet, deformable, tood, or dino.

[0081] The data obtained in the above examples are compared experimentally. Figures 4 to 8 Among them, the green indicator is this method, and the orange indicator is the original copy-paste method. In the training process of target detection models such as vfnet, deformable, tood and dino, this method is compared with the original copy-paste method. This method can improve the target detection accuracy by about two points.

[0082] The small sample target detection data enhancement method proposed in the present invention constrains the pasting position of the model and the pasting size of different types of targets, which needs to be customized according to different scenarios. In the transmission belt anomaly detection scenario, the following method is proposed.

[0083] To constrain the target position, we use the statistical sample distribution method. First, we convert the sample target box into a two-dimensional Gaussian distribution, count the two-dimensional Gaussian distribution of all samples, calculate the mixed Gaussian distribution, set a threshold, and constrain the area outside the threshold. In simple terms, we use the original sample distribution to unify the location where the sample may need to be pasted. For example, the original samples all appear on the conveyor belt, and the sample statistics method can automatically obtain the conveyor belt area, and will not be pasted in the area outside the conveyor belt. This statistical sample distribution method is applicable to all fixed scenarios with location information.

[0084] Regarding the problem of target size, in the case of conveyor belt anomaly detection, in addition to the size regularity of the sample itself, there is also prior knowledge that objects that are close together are larger and objects that are far away are smaller. This is done by introducing a neural network to learn the mapping relationship between the target area and the target size. In the process of copying and pasting data enhancement, the target size is adjusted through the neural network to force learning of the prior knowledge that objects that are close together are larger and objects that are far away are smaller, that is, the size of the target changes with the distance from the conveyor belt. By learning from a large amount of data, the model can better understand this prior knowledge and avoid confusing fuzzy targets in the distance with small targets nearby.

[0085] In order to solve the problem of insufficient diversity of pasted targets under small samples, a diffusion model is used to regenerate the targets. The diffusion model in the deep learning generative network is a generative model based on random walks, which generates data with complex structures by simulating the particle diffusion process. The target library has a certain diversity, thus preventing the overfitting problem caused by the copy-paste method under constraints.

[0086] Through this comprehensive solution, the problem of small sample target detection in industry has been effectively solved, and the accuracy of target detection has been significantly improved. In practical applications, this method has brought many benefits to manufacturing companies, including reducing false positives, improving production efficiency, and reducing production costs, thereby enhancing the competitiveness and sustainability of enterprises.

[0087] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0088] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0089] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0090] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0091] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

Claims

1. A small sample target detection data enhancement method, characterized in that: The following steps are involved: Divide the small sample data into training set, validation set and test set, and annotate the data set for target detection; The neural network is trained using the center point of the target detection box in the training set as input and the width and height of the target detection box as labels; Convert the target detection box in the training set into a two-dimensional Gaussian distribution; Using the image targets in the training set, the stable diffusion model is used to train the images to generate new target images; Randomly select coordinates within the visible range, generate corresponding coordinates through the previously trained size prior neural network, and obtain a new target detection frame; The new target detection frame is converted into a two-dimensional Gaussian distribution, and the distance between the input target detection frame and the target detection frame in the training set is measured by similarity. The minimum distance is output, and if the minimum distance is lower than a given threshold, the detection frame is retained. After obtaining a new target detection frame that meets the conditions, match and fill it with the target image generated by the stable diffusion model to obtain a newly generated target that satisfies both the regional distance and the priors of near large and far small; The original target is extracted from the training set, and the image generated by the diffusion model is subjected to position constraints and shape constraints together with the newly generated target. It is randomly pasted into the randomly extracted original image of the training set to form an extended image to achieve constrained copy and paste data enhancement. The enhanced data is then trained in the target detection model.

2. The small sample target detection data enhancement method according to claim 1, characterized in that: The LoRA method is used to fine-tune the stable diffusion model to generate a new target image. The process can be described as follows: first, the resolution of the target E in the original data set is reduced to construct a target image D with a lower resolution. The process can be described as follows: D=E0⊙W+n, Where D∈R (H×W×3) is the extracted low-resolution image of size H×W, E0∈R H×W×3 is the original data image, W∈R (H×W×3) Represents the mapping from real high-quality images to low-resolution images, N∈R (H×W) is the noise term introduced due to sensor limitations and other factors, By incorporating the low-resolution image D as a conditional input into the diffusion model, and then constraining the distance between the original image E and the resolution image D, the diffusion model learns the mapping from low-resolution restoration to high-resolution images in the reverse process.

3. The small sample target detection data enhancement method according to claim 2, characterized in that: The distance between the original image E and the resolution image D is constrained by the regularization strategy. Specifically, the original image and the low-resolution image are split into multiple small blocks, and these small blocks are clustered to form a clustering matrix by clustering. Each matrix has inherent image semantic information, which should not be changed during the diffusion model denoising process. The rank of this matrix can be used as an important measure of the global structure of the image. The matrix formed by the original image and the low-resolution image is regularized. The regularization function method can be expressed as: Where S W is the regularization method, L1 is the Manhattan distance, U∈R m×m , Represents the output of singular value decomposition. By adjusting the adaptive factor K, the regularization function S is gradually applied during the training process to ensure that the model can effectively handle the characteristics of low-resolution images.

4. The small sample target detection data enhancement method according to claim 1, characterized in that: The target detection frame is transformed into a two-dimensional Gaussian distribution, specifically: The mean of a two-dimensional Gaussian distribution is represented by a two-dimensional vector of center coordinates, The covariance matrix is ​​represented as a diagonal matrix with w and h as traces, 5. The small sample target detection data enhancement method according to claim 1, characterized in that: The normalized bulldozer distance is used to measure the similarity between the input target detection box and the Gaussian distribution in the training set. The bulldozer distance is: Among them, m represents the mean, ∑ represents the variance, and the target detection frame is brought into the bulldozer distance: Among them, cx, cy, w, and h represent the center point coordinates and width and height of the target detection box.

6. The small sample target detection data enhancement method according to claim 1, characterized in that: Training a neural network is as follows: All targets in the statistical data set are counted, and the center point position of the target is input into the neural network as a feature. The width and height information of the target are input, and the distance is measured by comparing the width and height information of the original sample. The gradient of the neural network is back-propagated to minimize the distance, so that the neural network can learn the mapping relationship between position and size.

7. The small sample target detection data enhancement method according to claim 1, characterized in that: A neural network is used to learn the prior relationship between distance and size. A floating-point data stream is input and a floating-point data stream is output. A perceptron with the most layers is used as the learning network.

Images