Sample generation method, target detection model training, target detection method and system
By automatically generating training samples for the object detection model and using unlabeled candidate sample sets for augmentation and iterative learning, the problem of low efficiency of manual annotation in existing technologies is solved, and efficient object detection result updates and accuracy improvement are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SPEEDBOT ROBOTICS CO LTD
- Filing Date
- 2023-03-07
- Publication Date
- 2026-08-04
AI Technical Summary
Existing target detection models have low accuracy in detecting continuously changing data, requiring a large number of manually labeled samples for retraining, resulting in high consumption of human and material resources and low labeling efficiency, which affects the accuracy of detection results.
By automatically generating training samples for the object detection model, using unlabeled candidate sample sets for augmentation and iterative learning of the object detection model, target samples are automatically selected and the model is updated.
It significantly reduces labor costs, continuously improves the accuracy of target detection results, and enhances the model's ability to adapt to data in online workflows.
Smart Images
Figure CN116152606B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of machine learning technology, and in particular to a sample generation method, object detection model training, object detection method and system. Background Technology
[0002] In real-world scenarios, it is often necessary to perform target detection on various objects, such as steel plates and components on factory assembly lines, continuous vehicle detection on highways, and pedestrian detection on sidewalks.
[0003] Existing object detection methods primarily focus on optimizing static datasets, with object detection models trained on fixed datasets. Therefore, existing object detection models suffer from low accuracy with continuously changing data, still requiring manual annotation of collected images to generate training samples, which are then retrained and deployed to improve the detection accuracy.
[0004] The aforementioned method of manually identifying and labeling samples requires significant manpower and resources, resulting in low sample labeling efficiency and low model training efficiency. Furthermore, the manual labeling process inevitably involves omissions and subjectivity, affecting the accuracy of the target detection results. Summary of the Invention
[0005] To address the need for manual annotation of large numbers of samples in existing technologies to train new object detection models, this invention provides a sample generation method, object detection model training, object detection method, and system. This system can continuously and automatically generate training samples for the object detection model, significantly reducing manual costs and continuously improving the accuracy of object detection results.
[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:
[0007] A sample generation method, characterized by including the following steps:
[0008] Step 1: Train the target detection model based on the original sample set. The original sample set includes multiple original sample images, and the output of the target detection model is the target detection information at each location on each original sample image.
[0009] Step 2, obtain a candidate sample set, wherein the candidate sample set includes N candidate sample images to be detected;
[0010] Step 3: Based on the candidate sample set and the target detection model, output the target information at each location on each candidate sample image;
[0011] Step 4: Based on the output information of Step 3, select target sample images from the candidate sample set and determine the target location and category information at each location on each target sample image;
[0012] Step 5: Output the target sample image and the target location and category information at various points in the image as training samples.
[0013] By using the above method, the present invention utilizes an unlabeled candidate sample set to automatically generate training samples for the target detection model, greatly reducing manual costs and continuously improving the accuracy of target detection results.
[0014] As a preferred approach, the target detection information at various locations on the image includes whether there is an object to be detected at each location on the image and the probability of the object category; in step 3, the process of outputting the target information at various locations on each candidate sample image includes:
[0015] Step 301: Perform various reversible transformations on the candidate sample images (e.g., adjust lighting, rotate, scale, flip, etc.) to obtain m different images P;
[0016] Step 302: Input the m different images P into the target detection model and output whether there is a target to be detected at each location on each image P and the probability of the target category;
[0017] Step 303: Based on the output of step 302, calculate the target location and target category probability at each location on the candidate sample image corresponding to m images P;
[0018] Step 304: Based on the calculation results of step 303, obtain the final target location and target category probability information at each location on the candidate sample image.
[0019] By employing the above method, this invention obtains multiple images by augmenting candidate sample images during the sample annotation process. It then obtains target detection information corresponding to each image through inference using an existing detection model. Finally, by comprehensively considering the target detection information of each image, it determines the target detection information on the original candidate sample image, thereby improving the accuracy of sample annotation.
[0020] As a preferred embodiment, step 4 includes:
[0021] Step 401: Use the object category probabilities at each location in the candidate sample image output in step 303 as the confidence score;
[0022] Step 402: Set a first threshold and a second threshold, where the first threshold is much smaller than the second threshold. Candidate samples with confidence levels between the first and second preset thresholds are considered useless and temporarily discarded. Candidate sample images with confidence levels below the first preset threshold or above the second preset threshold are retained as target sample images for output. Positions on the candidate sample images with confidence levels below the first preset threshold are set as background, and positions on the candidate sample images with confidence levels above the second preset threshold are set as target objects and their bounding boxes are set.
[0023] Step 403: Obtain the object class probability at the corresponding position of the object bounding box on the candidate sample image for each image P, and take the maximum value as the object class probability at the object bounding box.
[0024] As a preferred approach, in step 303, the average probability of object categories at each location on each image P is calculated and used as the probability of object categories at each location on the candidate sample image.
[0025] As a preferred method, in step 2, the N candidate sample images of the region to be detected are obtained through continuous real-time image acquisition of the region to be detected. Alternatively, the N candidate sample images of the region to be detected are obtained from a historical image set of the region to be detected.
[0026] As a preferred method, the processing includes position flipping, image brightness and contrast variation processing, shrinking or enlarging processing.
[0027] Based on the same inventive concept, this invention also provides a method for training an object detection model. Its characteristic is that the object detection model is trained using training samples generated by the aforementioned sample generation method to obtain an updated object detection model. Through continuous iterative learning, this invention can continuously improve the object detection model's adaptability to data in online workflows, thereby increasing object detection accuracy.
[0028] Preferably, the present invention also provides another object detection model training method, characterized in that the object detection model is trained using the original sample set and training samples generated by the sample generation method to obtain an updated object detection model.
[0029] Based on the same inventive concept, the present invention also provides an object detection model, characterized in that the object detection model is continuously self-learned and updated through the object detection model training method.
[0030] Based on the same inventive concept, the present invention also provides a target detection method, characterized by using the aforementioned target detection model to perform target detection on the image to be detected.
[0031] Based on the same inventive concept, the present invention also provides a target detection system, characterized by comprising an image acquisition unit, a model training unit, and the aforementioned target detection model, wherein:
[0032] Image acquisition unit: used to acquire images to be detected. A portion of the images to be detected is used as candidate sample images to generate a training sample set for training and updating the object detection model. The other portion of the images to be detected is used by the object detection model to identify and output the object detection result.
[0033] Model training unit: Used to train the object detection model based on the generated training sample set in order to update the object detection model.
[0034] Compared with existing technologies, this invention utilizes an unlabeled candidate sample set to automatically generate training samples for the target detection model, greatly reducing manual costs. At the same time, through continuous iterative learning, this invention continuously improves the target detection model's ability to adapt to data in online workflows, thereby increasing the accuracy of target detection results. Attached Figure Description
[0035] Figure 1 This is a layout diagram of the image acquisition unit according to an embodiment of the present invention.
[0036] Figure 2 This is a schematic diagram of the target detection model training method according to an embodiment of the present invention.
[0037] Figure 1 In the diagram, 1 is the image acquisition unit, 101 is the online acquisition camera, 102 is the local storage, 103 is the data storage center, and 104 is the data transmission module. Detailed Implementation
[0038] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments are clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0039] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, in the specification and claims of this invention are intended to cover without excluding their inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0040] This invention achieves continuous self-learning from a continuous data stream by automatically generating detection bounding boxes from continuously input image data and automatically training and updating them.
[0041] In one embodiment, the sample generation method provided by the present invention includes the following steps:
[0042] Step 1, based on the original sample set D A Training to obtain object detection model M A The model is then deployed to the production environment. The original sample set consists of multiple original sample images, and the output of the object detection model is the object detection information for each location on each original sample image. The original sample set can be an initial manually labeled sample set. The original sample set can also be obtained through other methods, which are not limited here.
[0043] Step 2, obtain the candidate sample set D B The candidate sample set includes N candidate sample images P to be detected. i Let i be an integer and 1 ≤ i ≤ N, P i Let i represent the i-th candidate sample image to be detected;
[0044] Step 3: Based on the candidate sample set and the object detection model, predict and output each candidate sample image P. i Target information in various locations;
[0045] Step 4: Based on the prediction output information from Step 3, select from the candidate sample set D B The target sample images are selected and the target location and category information of each target sample image are determined.
[0046] Step 5: Output the target sample image and the target location and category information at various points in the image as training samples.
[0047] This invention utilizes an unlabeled candidate sample set to automatically generate training samples for the target detection model, greatly reducing manual costs and continuously improving the accuracy of target detection results.
[0048] In some embodiments, the target detection information at various locations on the image includes whether there is a target to be detected at each location on the image and the probability of the target category; in step 3, for each candidate sample image P i Output candidate sample image P i The process of obtaining target information at various locations includes:
[0049] Step 301, process the candidate sample image P i Perform augmentation processing to obtain m different images. j is an integer and 1≤j≤m. This represents the image under the j-th transformation corresponding to the i-th candidate sample image to be detected;
[0050] Step 302, transform the m images with different transformations Input target detection model M A Using model M A Predict each image The detection results are output for each image. We obtain the probability of whether there is an object to be detected at each location and the object category. Object category probability matrix at various locations and object position matrix
[0051] Step 303: Based on the output of step 302, process m images with different transformations. Perform the inverse processing operation of step 301 to restore the original image position. After averaging the prediction results of m images, output the candidate sample image P. i Are there any objects to be detected at various locations, and what is the probability of each object category? The position of the object's outline is To obtain the most accurate prediction results.
[0052] In the sample annotation process, this invention processes candidate sample images to obtain images with different transformations of the candidate sample images, obtains target detection information at various locations on the different transformed images of the candidate sample images, and finally determines the target detection information on the original candidate sample images by comprehensively considering the target detection information at various locations on each transformation, thereby improving the accuracy of sample annotation.
[0053] In some embodiments, step 4 includes:
[0054] Step 401: Use the object category probability Tcls^ of each location on the candidate sample image output in step 303 as the confidence score, and set the first preset threshold β and the second preset threshold α, where α > β;
[0055] Step 402: Retain candidate sample images with confidence levels below the first preset threshold β or above the second preset threshold α as target sample images for output; set the positions on the candidate sample images with confidence levels below the first preset threshold as background, set the positions on the candidate sample images with confidence levels above the second preset threshold as target objects and set object bounding boxes, and discard candidate sample images with confidence levels between the first and second preset thresholds; specifically, for Tcls^, if any position exists... confidence level If a sample is not found in the sample, it is considered a confusing sample and is removed. Among the remaining candidate samples, each position below the second preset threshold α is set as background, and each position above the first preset threshold β is set as foreground. The bounding boxes are set as object bounding boxes, and the prediction results of all samples are finally fused to generate a self-labeled dataset D. LB .
[0056] Step 403: Obtain the object class probability at the corresponding position of the object bounding box on the candidate sample image for each image P, and take the maximum value as the object class probability at the object bounding box.
[0057] In some embodiments, in step 303, the average of the object category probabilities at each location on each image P is calculated as the object category probabilities at each location on the candidate sample image.
[0058] In some embodiments, in step 2, the N candidate sample images of the region to be detected are obtained through continuous real-time image acquisition of the region to be detected. In other embodiments, the N candidate sample images of the region to be detected are obtained from a historical image set of the region to be detected.
[0059] In some embodiments, the processing includes position flipping, image brightness and contrast adjustment, shrinking, or enlarging. For example, for each candidate sample image P i After performing various augmentations, including horizontal flipping, vertical flipping, scaling down to 0.5 times the original image, and scaling up to 1.5 times the original image, multiple images, including the original candidate sample image, are obtained. Then, model M is used. A The method involves parallel prediction of object detection results for multiple images to obtain the presence and class probability of an object at each location in the images. In this implementation, it is necessary to subsequently calculate the presence and class probability of an object at each location in the original images corresponding to the multiple images, and then calculate the average of the object class probabilities at corresponding locations in the multiple images to obtain the most accurate average prediction probability.
[0060] In some embodiments, the present invention provides a method for training an object detection model, which uses training samples generated by the sample generation method to train the object detection model and obtain an updated object detection model. Through continuous iterative learning, the present invention can continuously improve the object detection model's adaptability to data in online workflows and enhance object detection accuracy.
[0061] In a more preferred embodiment, such as Figure 2 As shown, this invention also provides another method for training an object detection model, which utilizes a manually labeled dataset (the original sample set D). A ) and the self-labeled training samples D generated by the aforementioned sample generation method LB For target detection model M A Training is performed to obtain an updated object detection model. In some embodiments, the manually labeled dataset and the self-labeled training sample set are fused at a 1:1 ratio and used together to continue training the object detection model. Simultaneously, methods such as cropping, scaling, color variation, brightness variation, mosaicking, and image blending can be used to enhance the diversity of images, strengthen the fusion of the original sample dataset and the self-labeled new sample dataset, and improve the comprehensive learning ability of the two datasets. The fused sample set is then used to train the object detection model M. A Continue training until the model converges, resulting in a new model M. B Using the new model M B Update the original model M A Then, model M A Once deployed to the production environment, steps 2 through 5 can be executed again to obtain automatically labeled samples, retrain the model, and update it, enabling the model to continuously learn itself until the model's loss converges.
[0062] In this invention, methods such as cropping, scaling, color changing, brightness changing, mosaic, and image blending are used to fuse images, which are existing technologies and will not be described in detail here. However, this does not affect the understanding and implementation of this invention by those skilled in the art.
[0063] In some embodiments, the present invention also provides an object detection model, which is continuously updated through the object detection model training method described above. After the object detection model is trained and updated, the updated object detection model is deployed to the production environment.
[0064] In some embodiments, the present invention also provides a target detection method, which utilizes the target detection model to perform target detection on the image to be detected.
[0065] In some embodiments, the present invention also provides a target detection system, which includes an image acquisition unit, a model training unit, and the target detection model, wherein:
[0066] Image acquisition unit: used to acquire images to be detected. A portion of the images to be detected is used as candidate sample images to generate a training sample set for training and updating the object detection model. Another portion of the images to be detected is used by the object detection model to identify and output the object detection result.
[0067] Model training unit: Used to train the object detection model based on the generated training sample set in order to update the object detection model.
[0068] In some embodiments, the object detection system further includes an online model deployment module for deploying the image acquisition unit, the model training unit, and the object detection model at the desired locations.
[0069] In some embodiments, the image acquisition unit employs a data retrieval unit, which is responsible for retrieving image data from a continuous data stream. For example... Figure 1 As shown, in this embodiment, the image acquisition unit 1 includes an online acquisition camera 101, a local memory 102, a data storage center 103, and a data transmission module 104. Figure 1 The direction indicated by the hollow arrow is the direction in which the workpiece is continuously fed during production, and the diamond and cross shapes represent different types of target workpieces to be identified.
[0070] The working process of image acquisition unit 1 is as follows:
[0071] First, the online acquisition camera 101 takes pictures of the area to be detected. After taking pictures, the output images are stored in the local storage 102 (such as a local disk).
[0072] Then, the data transmission module 104 reads the image from the local storage 102 and transmits the image back to the data storage center 103 via the network, etc. The data storage center 103 stores the received image data according to the receiving time, denoted by D.
[0073] The image data D stored in data storage center 103 can be used as candidate sample images D. B After automatic annotation, it is ultimately used as a training sample; it can also be used as the initial manually annotated dataset D. A .
[0074] This invention can be applied to object detection scenarios with continuous samples. It can fully utilize unlabeled samples on the production line to generate labels, automatically generate samples for target detection, and continuously update the sample set and target detection model in a loop. This significantly reduces labor costs, continuously improves the target detection model's adaptability to data in online workflows, and continuously optimizes the model to adapt to new data, thereby improving the accuracy of target detection results. This invention is particularly suitable for scenarios with continuous data, such as workpiece inspection in factory production lines (e.g., steel plate and component inspection), vehicle inspection on highways, and pedestrian detection on roads.
[0075] As shown in Table 1, the experimental results are as follows:
[0076] Using production line data from a factory as the experimental subject, and 5000 manually labeled historical data points as the initial sample set, a portion of the online data from March, April, and May was used as the test set. For the initial object detection model (Baseline model), the mean average accuracy (MAP) was only 94.1%. However, after incorporating manually labeled data, the MAP improved to 95%. After continuously training the object detection model using the method of this invention, the MAP increased to 97.87, significantly outperforming the original base model and even surpassing the model trained with a large number of manually labeled samples. The results demonstrate that this invention is highly effective in improving the mean average accuracy (MAP).
[0077] Table 1. Comparison of Experimental Results (MAP represents the average accuracy rate across all classes)
[0078]
[0079] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not limiting. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the scope of protection of the present invention.
Claims
1. A sample generation method, characterized in that, Includes the following steps: Step 1: Train the target detection model based on the original sample set. The original sample set includes multiple original sample images, and the output of the target detection model is the target detection information at each location on each original sample image. Step 2, obtain a candidate sample set, wherein the candidate sample set includes N candidate sample images to be detected; Step 3: Based on the candidate sample set and the object detection model, output the target information for each candidate sample image, including whether there is a target object at each location in the image and the probability of the object category. Step 3, for each candidate sample image, outputs the target information for each location, including: Step 301: Performing various reversible transformations on the candidate sample images to obtain m different images P; Step 302: Inputting the m different images P into the object detection model to output whether there is a target object at each location in each image P and the probability of the target category; Step 303: Based on the output of Step 302, calculating the target location and target category probability at each location in the candidate sample images corresponding to the m images P; Step 304: Based on the calculation results of Step 303, obtaining the final target location and target category probability information at each location in the candidate sample images. Step 4: Based on the output information of Step 3, select target sample images from the candidate sample set and determine the target location and category information at each location on each target sample image; including: Step 401: Use the object category probability at each location on the candidate sample image output in Step 303 as the confidence score; Step 402: Retain all candidate sample images with confidence scores lower than the first preset threshold or higher than the second preset threshold as target sample images; Set the locations on the candidate sample images with confidence scores lower than the first preset threshold as background, set the locations on the candidate sample images with confidence scores higher than the second preset threshold as target objects and set object bounding boxes, and discard candidate sample images with confidence scores between the first and second preset thresholds; Step 403: Obtain the object category probability at the location corresponding to the object bounding box on the candidate sample image for each image P and take the maximum value as the object category probability at the object bounding box. Step 5: Output the target sample image and the target location and category information at various points in the image as training samples.
2. The sample generation method according to claim 1, characterized in that, In step 303, the average of the object category probabilities at each location on each image P is calculated and used as the object category probabilities at each location on the candidate sample image.
3. The sample generation method according to claim 1, characterized in that, In step 2, the N candidate sample images of the region to be detected are obtained by continuously acquiring real-time images of the region to be detected; or, the N candidate sample images of the region to be detected are obtained from the historical image set of the region to be detected.
4. The sample generation method according to any one of claims 1 to 3, characterized in that, In step 301, the processing includes position flipping processing, image brightness and contrast change processing, shrinking processing, or magnification processing.
5. A method for training an object detection model, characterized in that, The target detection model is trained using training samples generated by the sample generation method according to any one of claims 1 to 4 to obtain an updated target detection model; Alternatively, the target detection model can be trained using the original sample set and training samples generated by the sample generation method according to any one of claims 1 to 4 to obtain an updated target detection model.
6. A target detection method, characterized in that, include: The target detection model is obtained by continuous self-learning and updating through the target detection model training method described in claim 5. The object detection model described above is used to perform object detection on the image to be detected.
7. A target detection system, characterized in that, It includes an image acquisition unit, a model training unit, and an object detection model, wherein: Image acquisition unit: used to acquire images to be detected. A portion of the images to be detected is used as candidate sample images to generate a training sample set for training and updating the object detection model. The other portion of the images to be detected is used by the object detection model to identify and output the object detection result. Model training unit: used to train the object detection model based on the generated training sample set to update the object detection model; the object detection model is continuously self-learned and updated through the object detection model training method as described in claim 5.