Method for training pattern collection for object detection model, and device and computer-readable storage medium for executing the above method

Abstract

The present invention aims to solve the problems of difficult pattern collection, high labeling cost and long model training time when training object detection models using deep learning techniques. The main technique is to use different pattern collection methods depending on whether there is an existing object detection model. In the case where no model exists yet, an image processing method is utilized for pattern collection. In the case where a model already exists, valuable rare patterns are collected based on mobile object detection, comparison of different models, and object tracking. Next, automatic labeling processes are used, so that the collected patterns are quickly used for training to obtain the optimal model. This technique can effectively reduce the cost of labeling and model training time after a large number of patterns are collected, and improve efficiency and accuracy of object detection models.

Description

[Technical Field]

[0001] This invention relates to the fields of artificial intelligence, data extraction, image processing methods, and object tracking. Its main objective is to retrain the required object detection model by combining image processing methods with artificial intelligence models, particularly deep learning models, to detect the number of moving objects, compare the detection results of different deep learning models, and track objects. Then, it extracts images with changed detection results and those that were not detected, using them as training samples for the model. This method requires only a small amount of manually checked and labeled data to generate standard answers for subsequent model training and to extract rare samples for object detection. [Previous Technology]

[0002] In the prior art, when it is necessary to use deep learning technology to improve existing vehicle detection models or to add vehicle detection models that do not yet exist, it is usually impossible to obtain valuable samples efficiently, and it requires a huge amount of image annotation time and manpower. Therefore, there are many problems and challenges, such as:

[0003] First, a large number of images need to be extracted from the video archives from which the image data is sourced, and the images of existing vehicle types and objects need to be manually selected. This will consume a lot of manpower and time, especially for images from new perspectives or new categories. More images need to be collected for annotation and training, and appropriate images need to be obtained manually from the video archives, which increases operating costs and manpower costs.

[0004] Second, since it is impossible to confirm whether the images selected by manual screening are valuable for improving the model, if the detection effect of the corresponding vehicle type detection model obtained after deep learning model training is not as expected, it is necessary to screen and label the data again from the source of the image data. This process is repeated until the model meets the expected results, which will take a lot of time.

[0005] Third, manual screening cannot confirm whether the selected images are those with poor model detection performance. It may select images that the model already has good results, which may lead to overfitting during training.

[0006] Previous manual image screening methods have problems such as consuming a lot of manpower and time, and the screened images may not be helpful to the model or may even cause overfitting. Further improvements are needed to accelerate model output and promote efficient model training and learning. [Summary of the Invention]

[0007] To solve the above-mentioned problems of the prior art, the present invention provides a method for extracting training samples for an object detection model, comprising: extracting multiple images from a video file; and for each image, detecting the number of at least one moving object in the image using an existing first object detection model or an image processing method, wherein if the number is greater than a threshold, the image is retained as a training sample for a second object detection model.

[0008] In one embodiment, the first object detection model and the second object detection model are different object detection models, and the object detection model training sample extraction method further includes: for each image, using the first object detection model and the second object detection model respectively to detect the category of moving objects in the image; if the first object detection model and the second object detection model respectively detect multiple different categories of the same moving object in the image, then the category of the moving object is set as a combination of the multiple different categories, so as to be the standard answer labeled in the training sample.

[0009] In one embodiment, the object detection model training sample extraction method further includes: using the first object detection model to detect the category of each moving object in each of the images; and if the same moving object is detected as having different categories in two images of the plurality of images by the first object detection model, then the two images are retained as the training samples.

[0010] In one embodiment, the object detection model training sample extraction method further includes: using the first object detection model to detect each of the moving objects in each of the images; and if, among the at least one moving object, the same moving object is detected by the first object detection model both before and after a time interval, and is not detected by the first object detection model during the time interval, then all images in the plurality of images corresponding to the time interval are retained as the training samples.

[0011] The present invention further provides an object detection model training sample extraction device for performing the object detection model training sample extraction method.

[0012] The present invention further provides an apparatus for performing a method for extracting training samples for an object detection model, comprising: an acquisition module for acquiring multiple images from a video file; a detection module for detecting the number of at least one moving object in each image using an existing first object detection model or an image processing method; a judgment module for judging whether the number is greater than a threshold; and a retention module for retaining the image as a training sample for a second object detection model when the number is greater than the threshold.

[0013] The present invention also provides a computer-readable storage medium storing a plurality of instructions, which are read by a processor, computer or server to execute the object detection model training sample extraction method.

[0014] This invention aims to provide an automatic and efficient method for extracting training samples for object detection models. This addresses the problem that extracting valuable samples for existing models and during model refinement is difficult and time-consuming, resulting in lengthy model training times and suboptimal model performance when using deep learning technology for object detection model training. The data extraction method used in this invention extracts images that best suit the model's training needs to improve accuracy. Furthermore, by using different extraction methods, existing annotation standards can be obtained, requiring only minimal manual review for deep learning model training. Therefore, compared to blindly investing in a large number of unfiltered training samples, the training samples selected in this invention can specifically improve the shortcomings of existing models, effectively enhancing the efficiency and results of model training. [Simplified Explanation of the Diagram]

[0063] Figure 1 is a flowchart illustrating the moving object detection method of the present invention.

[0064] Figure 2 is a flowchart of the image processing algorithm in the moving object detection method of the present invention.

[0065] Figure 3 is an example diagram of the image processing algorithm in the moving object detection method of the present invention.

[0066] Figures 4A to 4C are schematic diagrams illustrating examples of the different model comparison methods of the present invention.

[0067] Figure 5 is a flowchart illustrating the object tracking method of the present invention.

[0068] Figure 6 is a flowchart illustrating the category change method in the object tracking method of the present invention.

[0069] Figure 7 is a flowchart illustrating the undetected mode in the object tracking method of the present invention.

[0070] Figure 8 is a schematic diagram of the apparatus of the present invention for performing the object detection model training sample extraction method.

Implementation Method

[0015] The present invention provides an automatic and efficient method for extracting training samples for object detection models, which can extract and label rare training samples from video archives that are sources of image data.

[0016] The object detection model training sample extraction method of the present invention includes three parts: a moving object detection method, a method for comparing different models, and an object tracking method, which are described below.

[0017] Part One, Moving Object Detection Method:

[0018] Please refer to Figure 1, which is a flowchart of the moving object detection method of the present invention.

[0019] First, in step 11, an image (frame) is automatically captured from the video file of the image data source at regular intervals. The image data source may be multiple surveillance cameras deployed along multiple roads.

[0020] The sampling time can be set by the user or automatically according to the traffic flow in the video file. If the traffic flow is large, a shorter sampling time can be set, and if the traffic flow is small, a longer sampling time can be set.

[0021] After obtaining the image, there are two ways to find the vehicle type object. The first way is to detect the vehicle type object in the image through the existing vehicle type detection model in step 12.

[0022] The aforementioned existing vehicle type detection models refer to previously trained models or publicly available open-source models. If multiple existing vehicle type detection models are available for detecting vehicle objects in an image, the model with the highest mean average precision (mAP) can be selected. The higher the mean average precision of a vehicle type detection model, the more accurately it can select vehicle objects in the image and the more accurately it can detect the category of the vehicle objects.

[0023] The second method is to detect vehicle objects in the image in step 13 by means of image processing methods (such as algorithms) if there is no existing vehicle detection model or the average accuracy (mAP) of the existing vehicle detection model is lower than the threshold value.

[0024] The aforementioned image processing method (such as the algorithm) is shown in Figure 2.

[0025] First, since there is no existing vehicle type detection model to detect vehicle type objects, in step 21, all images captured from the video file within the most recent rewind time (e.g., 10 seconds) are averaged by adding each pixel to obtain the background image.

[0026] Then, in step 22, the background image is subtracted from the most recently captured image to obtain the foreground image.

[0027] Next, in step 23, various image processing techniques are used to adjust the binarization threshold and the core size required for morphology.

[0028] Finally, in step 24, each moving vehicle object is selected in the foreground image using the eight-way connectivity of connected components, and the number of vehicle objects is calculated.

[0029] For example, as shown in Figure 3, although the vehicle type objects in the image cannot be detected using existing models, each vehicle type object in the image can still be selected by image processing methods (such as algorithms), such as vehicle type object 30.

[0030] Regardless of whether an existing vehicle type detection model (such as various open-source models or self-trained models) is used in step 12, or an image processing method (such as an algorithm) is used in step 13, the number of each vehicle type object and the number of vehicle type objects in the image can be obtained.

[0031] Then, in step 14, according to the threshold number of vehicle objects to be retained for training the vehicle detection model set by the user, if the number of vehicle objects obtained in step 12 or 13 is greater than the threshold, then in step 15, this image is retained as the extracted image for training the vehicle detection model.

[0032] The vehicle type detection model to be trained here can be the vehicle type detection model used in step 12 or another vehicle type detection model, and the aforementioned vehicle type detection models are all deep learning models. If the number of vehicle type objects obtained in step 12 or 13 is less than or equal to the threshold, this image will not be used for training the vehicle type detection model, and the process will return to step 11.

[0033] Part Two, Comparison Methods of Different Models:

[0034] By comparing different vehicle detection models, the standard answer for labeled images can be obtained more quickly in images where there is no standard answer yet. Furthermore, the accuracy of different vehicle detection models in detecting vehicle types for the required scene or viewpoint can be compared, thereby determining whether the image needs to be extracted for training.

[0035] In detail, after the target image is input, each vehicle type detection model can produce its own detection results based on the image. Then, the detection results generated by multiple different vehicle type detection models are compared by using the intersection over union (IOU) comparison method.

[0036] If the detection result is the same object of the same category (e.g., vehicle), then ignore this object. If the detection result is the same object but the detected categories are different, then change the object category to a combination of the categories of the detection results of the multiple vehicle type detection models, as the standard answer for labeling.

[0037] If no object is matched through IOU, then keep the object and manually check and label the kept object.

[0038] This method can accelerate the generation of labeled standard answers for training vehicle detection models, without having to manually select all objects as in traditional labeling methods.

[0039] In one embodiment, Figures 4A and 4B represent the detection results of two different vehicle type detection models A and B, respectively. Both models A and B detect vehicle object 31 as a car, indicating that the detection results of the two models are consistent, and no additional training is needed for this object; therefore, the detection result is retained unchanged. Vehicle object 32 is detected as a car by model A but as a van by model B, indicating that the detection results of the two models are inconsistent. Therefore, the labeling of its detection result is revised to a combination of car and van, i.e., car@van, as shown in Figure 4C, for subsequent user revision. Vehicle object 33 is detected by model A but not by model B, so the detection result of model A, i.e., bicycle, is directly retained. The final labeling result is shown in Figure 4C.

[0040] The above-mentioned multiple vehicle type detection models may include existing vehicle type detection models and / or vehicle type detection models trained according to the process of steps 11 to 15.

[0041] Part Three, Object Tracking Methods:

[0042] Please refer to Figure 5, which is a flowchart of the object tracking method of the present invention. This method utilizes the different categories of detection results or the images that are missed during the continuous image detection process to extract data, which is used as training samples to improve the model and supplement the detection effect of the model.

[0043] First, in step 41, after the video file from which the image data is sourced is input, multiple images (frames) are captured from the video file, and an existing vehicle type detection model is used to perform object tracking on each captured image, that is, to detect each moving object (such as a person or vehicle) and its category in each image. After object tracking, each object is assigned its own sequence number.

[0044] Next, through two methods, namely category change and non-detection, it is determined whether the current image should be extracted as a rare sample that is valuable for fine-tuning the vehicle type detection model.

[0045] Step 42 corresponds to the category change method mentioned above.

[0046] As shown in Figure 6, in step 51, multiple images are input into the vehicle type detection model to obtain the sequence number and category of each object in each image. In step 52, the object sequence number and category in each image are matched to determine whether the same object (i.e., the object with the same sequence number) has a different category detected in different images, such as changing from a sedan to a truck. If so, in step 53 (i.e., step 43), both the currently changed image and the previous unchanged image are retained as images to be used for fine-tuning the model training, in order to supplement the training and detection effects of the model.

[0047] Step 44 corresponds to the above-mentioned undetected method.

[0048] As shown in Figure 7, in step 61, multiple images are input into the vehicle detection model to obtain the image number of each image and the sequence number of each object within it. In step 62, the object sequence numbers in each image are matched to determine if an object was first detected and then not detected (i.e., the object's sequence number is no longer present in the image), but after a period of time, the object is detected again and calculated as having the same sequence number (i.e., the same object). This indicates that the object was not detected by the model within the image segment where it was not detected. If such an object exists, in step 63 (i.e., step 45), all images within this segment are retained as images to be used for fine-tuning the model's training, thereby supplementing the model's training and detection effects.

[0049] The present invention is not limited to vehicle detection and tracking. For example, the vehicle detection model in the above embodiments can be replaced with a more widely used object detection model, and the above-mentioned vehicle objects can be replaced with other moving objects to detect or track the categories of various moving objects in the image, such as the category of motorcycles or the category of pedestrian clothing.

[0050] In addition, in one embodiment, the aforementioned object detection model is a deep learning model.

[0051] The present invention also provides an object detection model training sample extraction device, such as a processor, computer or server, which can be used to perform the above-described object detection model training sample extraction method.

[0052] As shown in FIG8, the apparatus 80 for performing the object detection model training sample extraction method of the present invention includes at least: an acquisition module 801, a detection module 802, a judgment module 803 and a retention module 804.

[0053] The capturing module 801 performs at least step 11, the detection module 802 performs at least steps 12-13, the judgment module 803 performs at least step 14, and the retention module 804 performs at least step 15, but is not limited thereto.

[0054] In one embodiment, each module of the present invention may be software, hardware or firmware; if it is hardware, it may be a processing unit, processor or computer host with data processing and computing capabilities; if it is software or firmware, it may include instructions executable by a processing unit, processor, computer or computer host, and may be installed on the same hardware device or distributed in different multiple hardware devices.

[0055] The present invention further provides a computer-readable storage medium, such as memory, magnetic tape, magnetic disk, or optical disk. This computer-readable storage medium can be used to store multiple instructions, which can be read by a processor, computer, or server to execute the above-described object detection model training sample extraction method. In one embodiment, this computer-readable storage medium is a non-transitory computer-readable storage medium.

[0056] In summary, this invention provides an automatic and efficient object detection model training sample extraction technique, including a moving object detection method, a method for comparing different models, and an object tracking method. Its main feature lies in the extraction technique for rare training samples. These methods effectively reduce the time and cost of manual image screening and improve the efficiency and accuracy of object detection model training.

[0057] Furthermore, through automated image processing and annotation technology, the present invention can quickly obtain valuable training samples from a large amount of data, thereby improving the accuracy and adaptability of the object detection model.

[0058] Furthermore, using the technology of the present invention, valuable training samples that can be used for fine-tuning training of the object detection model to improve its average accuracy can be quickly extracted from a large number of images. These training samples may be few in number, but by using these few training samples, the object detection model can be quickly fine-tuned and its average accuracy can be effectively improved, thereby improving the object detection model.

[0059] Conversely, without using the technology of this invention, it is impossible to accurately extract valuable training samples, and all images must be used to fine-tune the object detection model. However, only a very small portion of these images may help improve the accuracy of the model, while the rest of the images can already be correctly detected by the object detection model and are therefore of no value.

[0060] According to the prior art, a very small number of valuable images and a large number of worthless images are mixed as training samples. Therefore, compared with the present invention, the prior art not only takes a long time, but its fine-tuning effect is also not good.

[0061] For example, in one embodiment, the time spent by the technology of the present invention is reduced by 2 / 3 compared with the prior art that does not use the technology of the present invention, and the object detection model trained using the technology of the present invention can improve the average accuracy by 37.26%.

[0062] The above embodiments are used to illustrate the principles and effects of the present invention, and are not intended to limit the present invention. Any person skilled in the art can modify the above embodiments without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be as set forth in the following claims.

Claims

1. A method for extracting training samples for an object detection model, comprising: Extracting multiple images from a video file; For each image, the number of at least one moving object in the image is detected by using an existing first object detection model or by using an image processing method. If the number is greater than a threshold, the image is retained as a training sample for the second object detection model. The first object detection model is used to detect the category of each moving object in each of the images; and if the same moving object is detected as having a different category in two images of the plurality of images by the first object detection model, then the two images are retained as the training samples.

2. The object detection model training sample extraction method as described in claim 1, wherein, The step of capturing the plurality of images from the video file includes: capturing one of the plurality of images from the video file at sampling intervals, wherein the sampling interval is set according to the flow of moving objects in the video file.

3. The object detection model training sample extraction method as described in claim 1, wherein, The image processing method includes: averaging the images captured from the video file in the most recent retrospective time to obtain a background image; subtracting the background image from the most recent image captured from the video file in the multiple images to obtain a foreground image; and calculating the number of the at least one moving object based on the foreground image using binarization, imagery, and connected elements.

4. The object detection model training sample extraction method as described in claim 1 further includes: For each image, if the number of at least one moving object is less than or equal to the threshold, then the image is not retained as a training sample for the second object detection model.

5. The object detection model training sample extraction method as described in claim 1, wherein, The first object detection model and the second object detection model are the same object detection model.

6. The object detection model training sample extraction method as described in claim 1, wherein, The first object detection model and the second object detection model are different object detection models, and the object detection model training sample extraction method further includes: for each image, using the first object detection model and the second object detection model respectively to detect the category of moving objects in the image; if the first object detection model and the second object detection model respectively detect multiple different categories of the same moving object in the image, then the category of the moving object is set as a combination of the multiple different categories, as the standard answer labeled in the training sample.

7. The object detection model training sample extraction method as described in claim 1 further includes: The first object detection model is used to detect each moving object in each of the images; And if, among the at least one moving object, the same moving object is detected by the first object detection model both before and after a time interval, but is not detected by the first object detection model during the time interval, then all images corresponding to the time interval in the plurality of images are retained as the training samples.

8. An apparatus for performing a method for extracting training samples for an object detection model, comprising: The capturing module captures multiple images from the video file; the detection module detects the number of at least one moving object in each image using an existing first object detection model or an image processing method; the judgment module determines whether the number is greater than a threshold. And a retention module, when the number is greater than a threshold, the image is retained as a training sample for the second object detection model. The device uses the first object detection model to detect the category of each moving object in each image. If the same moving object is detected as a different category by the first object detection model in two images of the plurality of images, the device retains the two images as the training sample.

9. A computer-readable storage medium storing a plurality of instructions which are read by a processor, computer or server to perform an object detection model training sample extraction method as described in any one of claims 1 to 7.