A two-stage pedestrian and vehicle detection method

Abstract

The present invention provides a two-stage pedestrian and vehicle detection method, which specifically includes: S1, collection and annotation of pedestrian and vehicle data sets; S2, training of pedestrian and vehicle detection models; S3, application on the pedestrian and vehicle detection device side, and other steps. The present invention can effectively combine the advantages of single-stage and two-stage models, optimize some inherent problems existing in single-stage detection models. Through the serial combination of two single-stage detection models, it can significantly improve the detection accuracy while maintaining real-time performance, reduce false alarms and missed detections, and at the same time has strong robustness and flexibility. Compared with existing single high-performance models, two-stage detection models, and single lightweight models, this method provides a better balance in terms of performance and resource occupancy in complex backgrounds, especially in scenarios containing small targets, and is particularly suitable for scenarios that require both speed and accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of security monitoring, and particularly relates to a two-stage pedestrian and vehicle detection method. Background Art

[0002] With the rapid development of deep learning technology, intelligent monitoring, intelligent warning and other intelligentizations have become the focus of today's society. Among them, the pedestrian and vehicle detection technology based on cameras has also been widely applied. Especially in public security monitoring and intelligent security systems, the pedestrian and vehicle detection technology based on cameras plays a crucial role.

[0003] Existing pedestrian and vehicle detection technologies can be divided into single-stage and double-stage according to the principle of the target detection algorithm and its detection method. The single-stage target detection algorithm directly predicts the position and category of the target in the input image during the detection process, which is an end-to-end process. Since it is necessary to optimize the two tasks of classification and positioning in one network at the same time, in order to achieve better detection effects, the model is usually too complex to be trained and optimized. The lightweight single-stage network not only has poor detection effects on some small targets, but also is difficult to distinguish the background and the target in some complex background environments, resulting in low detection accuracy and many false alarms. The double-stage target detection algorithm needs to generate additional candidate regions during the detection process, which has a large demand for computing resources, resulting in slow detection speed and being difficult to meet the real-time requirement, and is not very suitable for embedded and mobile devices.

[0004] The current target detection algorithms have good detection effects on large and significantly featured targets. However, in the actual environment, there are often other vehicles or pedestrians blocking, and the vehicles and pedestrian targets in the distance are small, which easily causes the loss or non-obviousness of target features, not only leading to an increase in the missed detection rate and false detection rate, but also reducing the detection accuracy. Summary of the Invention

[0005] Aiming at the deficiencies of the existing technology, the present invention proposes a two-stage pedestrian and vehicle detection method. By serially combining two single-stage detection models, it can significantly improve the detection accuracy while maintaining real-time performance, reduce false alarms and missed detections, and at the same time has strong robustness and flexibility.

[0006] To implement the above technical solution, the present invention provides a two-stage pedestrian and vehicle detection method, which specifically includes the following steps:

[0007] S1. Collection and annotation of pedestrian and vehicle data sets

[0008] S11. Collection of pedestrian and vehicle data sets, obtaining picture data of complex and diverse scenes from publicly available data sets on the market or crawling from the Internet;

[0009] S12. First-stage pedestrian and vehicle detection data annotation: First, use annotation software to annotate data for special scenarios, train a data annotation model, then use the trained model to detect unannotated image data to obtain unannotated data labels, and then manually correct these labels annotated by the model and add them to the previously trained dataset, and train the data annotation model again. Repeat this process multiple times until all data annotation and correction work is completed;

[0010] S13. Second-stage pedestrian and vehicle detection data annotation: Directly put the data crawled from the network into the first-stage data annotation process for annotation and backup; for the data obtained in the first stage, while obtaining the image, calculate the current image label information according to the position information of the target in the original image;

[0011] S2. Training of pedestrian and vehicle detection models

[0012] S21. First-stage model training: Train pedestrian and vehicle pictures based on the yolov5 framework and replace the original Iou with the NWD loss function;

[0013] S22. Collect data for various scenarios without the detected target type as background pictures, and merge these pictures with the previously annotated pedestrian and vehicle dataset for training;

[0014] S23. Use the pedestrian and vehicle detection model trained in the first stage to detect datasets with a large scene coverage such as segAnything, verify the pictures with detected pedestrians and vehicles, extract the pictures with detection errors, and then merge the above data into the previous data for training;

[0015] S24. Second-stage detection model training: Train the model based on the yolov8 framework. During the training process, use image preprocessing operations to reduce the training size of the image, and combine the extracted pictures with detection errors for optimized training of the model;

[0016] S3. Application of pedestrian and vehicle detection equipment

[0017] S31. First-stage detection: Obtain the original data in yuv format from the video stream, convert and scale the yuv data to the format and size supported by the AI operation module, convert the previously trained model to the board model, use the operation module for operation to obtain the output features of the model, and perform post-processing on the output features to obtain the confidence levels and position information of pedestrian and vehicle targets;

[0018] S32. Pedestrian and vehicle target tracking. Use the SORT target tracking algorithm to track the detected pedestrian and vehicle targets, process consecutive frames, calculate the correlation of target information in two frames, including the distance information of the target and the overlapping ratio of the target. The same target forms the same ID. At the same time, according to the tracking trajectory information obtained from the previous tracking, extract the image data at the corresponding position in the current image for caching, and set the initial state of each target ID to True.

[0019] S33. Pedestrian and vehicle target screenshot. Map the target position information obtained from the first-stage detection to the corresponding position of the target in the original image, then expand the pixels around it to make it a square and the size is closest to the input size of the second-stage model. Finally, use the resize operation to scale the image to the input size of the model.

[0020] S34. Second-stage detection. First, perform a screening operation on the captured images, and only send the captured images obtained from the targets with a size ratio or confidence score lower than the set threshold into the second-stage model for detection. Otherwise, do not perform the second-stage detection and directly retain the image. Then, update the target status according to the detection results. If a target of the same detection type as in the first stage appears in the detection results, the target status remains unchanged. Otherwise, it is changed to False, and the image is filtered. Finally, the captured images and target information that are not sent into the second-stage detection and those remaining after the second-stage detection are used as the final pedestrian and vehicle image detection results.

[0021] Preferably, in the step S11, the publicly available datasets include COCO2017, VOC2007, VOC2012, Object365, and Widerperson.

[0022] Preferably, in the step S12, the semi-automatic annotation method is adopted for the first-stage pedestrian and vehicle detection data annotation. First, use the publicly available datasets on the market and the data of some special scenarios manually annotated with annotation software to train a model with a larger size, more parameters, and stronger generalization ability. Then, use the trained model to detect the unannotated image data to obtain the unannotated data labels.

[0023] Preferably, in the step S12, the annotation software is labelimg, and the annotation content is the position of each target and the corresponding type.

[0024] Preferably, the corresponding types are divided into 15 categories. In addition to the 6 categories of pedestrian and vehicle targets to be detected, 9 categories of false alarm categories are also annotated for training.

[0025] Preferably, in the step S24, the image preprocessing operations include color change, brightness change, rotation angle, and Gaussian blur processing.

[0026] Preferably, in the step S24, in order to improve the detection speed and reduce the model size, the training size is changed from 640x640 to 256x256.

[0027] The beneficial effects of a two-stage pedestrian and vehicle detection method provided by the present invention are as follows:

[0028] (1) The present invention can effectively combine the advantages of single-stage and two-stage models, optimize some inherent problems existing in the single-stage detection model. By serially combining two single-stage detection models, it can significantly improve the detection accuracy while maintaining real-time performance, reduce false alarms and missed detections, and at the same time has strong robustness and flexibility. Compared with existing single high-performance models, two-stage detection models, and single lightweight models, this method provides a better balance in terms of performance and resource occupancy in complex backgrounds, especially in scenarios containing small targets, and is particularly suitable for scenarios that require both speed and accuracy.

[0029] (2) The present invention uses the trained model to detect data in various scenarios. In particular, for misreported pictures, the misreported pictures are used as hard examples and merged into the existing dataset for training again to improve the adaptability of the model.

[0030] (3) Aiming at the problem of easy missed detections and false alarms of small-sized and blurred targets in complex environments, the present invention uses a method of serially combining two single-stage models for detection. Among them, in the first stage, candidate regions are quickly screened out to determine the target position and type, and in the second stage, refined detection is carried out, which can effectively improve the detection effect while maintaining real-time performance.

[0031] (4) In the acquisition of the input picture of the second-stage model of the present invention, a method of first mapping to the original picture and then performing extended picture grabbing is adopted to obtain the picture, which can make the features of the detection picture more obvious and can effectively optimize the problems of missed detections and false alarms of small targets to a certain extent.

[0032] (5) Through the application situation in the actual scenario, the present invention collects the dataset and trains the model in combination with the actual scenario, and then integrates it into the existing intelligent products, which can effectively improve the performance and applicability of the model and better meet the needs of actual applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is the flowchart of the steps of the present invention.

[0034] Figure 2 is the flowchart of the first-stage model training.

[0035] Figure 3 It is a flowchart for capturing images of pedestrian and vehicle targets.

[0036] Figure 4 It is a flowchart for two-stage pedestrian and vehicle detection. Specific implementation manners

[0037] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0038] Embodiment: A two-stage pedestrian and vehicle detection method.

[0039] Referring to Figures 1 to 4 As shown, for the two-stage pedestrian and vehicle detection method provided by the present invention, aiming at the situation that the two-stage object detection algorithm is slow and cannot meet the real-time requirement, the present invention uses a single-stage network for detection and directly outputs the position information and category of the detected object. Aiming at the situation that the single-stage object detection algorithm has poor detection effect in complex environments, especially in images containing complex backgrounds and small targets, the present invention adopts a serial combination method of two single-stage networks. The first-stage network quickly locates the position of the target, and then the output result of the first-stage network is processed and sent to the second-stage network for more accurate classification and positioning. By combining the detection results of the two times, the detection effect can be optimized to a certain extent.

[0040] The two-stage pedestrian detection method provided by the present invention mainly includes: collection of pedestrian and vehicle datasets, training of pedestrian and vehicle detection models, and application on the device side of pedestrian and vehicle detection. For the collection of pedestrian and vehicle datasets, in addition to using existing large datasets, some picture data corresponding to the scenarios are collected from the Internet as positive and negative samples for model training, and the semi-automatic annotation method is used to improve the annotation speed for the collected images. The first-stage detection model and the second-stage detection model are respectively trained using the deep learning framework with the data collected previously, and then the trained models are converted to the corresponding device side using the corresponding conversion tools. On the device side, after the picture data is input in the first stage, the AI operation module is used to load the pedestrian and vehicle detection model to perform operations on the data to obtain the position information and category of the pedestrian and vehicle targets in the first stage. The target tracking algorithm is used to track the pedestrian and vehicle targets, and the corresponding target pictures are cropped from the original image according to the tracked target information and sent to the second-stage detection model for detection to obtain the position and category of the pedestrian and vehicle targets again. Finally, the target information detected twice is combined to obtain the final detection result.

[0041] The detailed process of the two-stage pedestrian detection method provided by the present invention is as follows:

[0042] (1) Collection and annotation of pedestrian and vehicle detection datasets:

[0043] The images for the first-stage pedestrian and vehicle detection are mainly obtained from some existing large datasets, such as COCO2017, VOC2007, VOC2012, Object365, Widerperson, etc., as well as the image data of some complex and diverse scenarios crawled from the Internet. For the second-stage pedestrian detection images, part of them are obtained from the first-stage images using an extended method, and part of them are obtained separately from the Internet.

[0044] For the annotation of the first-stage pedestrian and vehicle detection data, a semi-automatic annotation method is adopted. First, a model with a relatively large size, more parameters, and stronger generalization ability is trained using the publicly available datasets on the market and the data of some special scenarios manually annotated using annotation software. Then, the trained model is used to detect the unannotated image data to obtain the unannotated data labels, and the labels of this part annotated by the model are corrected manually and added to the previously trained dataset, and the model is trained again, and the cycle is repeated until all data annotation and correction work are completed.

[0045] For the annotation of the second-stage pedestrian and vehicle detection data, the data crawled from the network is directly put into the first-stage data annotation process for annotation and standby; for the data obtained from the first stage, when the image is obtained, the current image label information can be calculated according to the position information of the target in the original image, and no separate annotation is required.

[0046] The tool used for the annotation of pedestrian and vehicle detection training data is labelimg, and the content of the annotation is the position of each target and the corresponding type. There are a total of 15 types. In addition to the 6 types of pedestrian and vehicle targets to be detected, 9 types of false alarm categories are also annotated for training.

[0047] (2) Model training

[0048] 1. First-stage model training:

[0049] Based on the yolov5 framework, the pedestrian and vehicle images are trained. To ensure the detection speed and facilitate subsequent deployment and application, in the first stage of this method, a more lightweight structure of yolov5s is adopted, and the depth and width of the network structure are appropriately reduced. To reduce the situation of small target missed detection, the NWD loss function is used to replace the original Iou, which can better reduce the sensitivity of Iou to the deviation of tiny objects.

[0050] To improve the generalization ability of the pedestrian and vehicle detection model, in addition to using common data augmentation operations during training, a part of the data of each scenario without the detection target type is collected as background images. These images are merged with the previously labeled pedestrian and vehicle dataset for training to improve the model's adaptability to various scenarios and reduce false alarms in unknown scenarios. The trained model will be tuned again to reduce false alarms. Specifically, the trained pedestrian and vehicle detection model is used to detect a dataset with a large scene coverage such as segAnything, and the images with detected pedestrians and vehicles are verified. In particular, the images with detection errors are extracted. Since there are false alarm data misrecognized by the model in these images, they are called hard examples. The hard example data is merged into the previous data for training, and the false alarm situation for such hard examples will be reduced after training. For the specific process, see Figure 3 as shown.

[0051] 2. Training of the second-stage detection model:

[0052] In the second stage, the model is trained based on the yolov8 framework. Compared with yolov5, yolov8 has a certain improvement in detection effect with less increase in computing resources. This stage mainly focuses on some scenarios with complex environments and small targets. Due to factors such as the detection size, size, and real-time requirements of the model, there are many misdetection cases when only using a single-stage detection model, especially for small-sized objects in the distance and some objects with low clarity. To improve the detection speed and reduce the model size, the training size is changed from 640x640 to 256x256, and image preprocessing operations such as color change, brightness change, and rotation angle are used during the training process; to reduce false alarms of blurred targets, Gaussian blur processing with a certain probability is added to the data during training; to improve the generalization ability of the model, this stage also adopts the method of training in combination with hard examples to tune the model.

[0053] (3) Application on the device side

[0054] The application of the two-stage pedestrian and vehicle detection device mainly includes: first, the image is sent to the first-stage model for detection to determine the specific position and category of the pedestrian and vehicle, then the target tracking algorithm is used to track the detected target, and the image is captured on the original image according to the tracked target information. Finally, the captured image is sent to the second stage for detection, and the final pedestrian and vehicle detection result is obtained by integrating the target information of the two detections.

[0055] For the specific process, refer to Figure 4 the two-stage pedestrian and vehicle detection flowchart:

[0056] 1. First-stage detection

[0057] The original data is obtained from the video stream in the YUV format, and the YUV data is converted and scaled to the format and size supported by the AI operation module. The previously trained model is converted to the board-side model, and the operation module is used for operation to obtain the output features of the model. The output features are post-processed to obtain the confidence levels and position information of pedestrian and vehicle targets. The confidence threshold set in the post-processing stage needs to be appropriately reduced to prevent some detection targets with low scores due to blurriness or small size from being filtered out, and these targets may be correctly detected in the second stage, which can reduce the missed detection phenomenon to a certain extent.

[0058] 2. Pedestrian and Vehicle Target Tracking and Image Capture

[0059] For pedestrian and vehicle target tracking, the SORT target tracking algorithm is used to track the detected pedestrian and vehicle targets. The consecutive frames are processed to calculate the correlation of target information in two frames, including the distance information of the targets and the overlapping ratio of the targets. The same targets form the same ID. At the same time, according to the tracking trajectory information obtained previously, the image data at the corresponding positions in the current image is captured and cached, and the initial state of each target ID is set to True.

[0060] For pedestrian and vehicle target image capture, in order to make the captured images directly usable for the second-stage model detection and improve the detection accuracy, some special processing is required for the image capture operation, rather than directly capturing images according to the target position information obtained in the first stage. First, since the input image size of the AI operation module of the embedded device needs to be fixed, it is necessary to ensure that the size of the captured image is the same as the size of the input of the second-stage detection model. Then, since the input image size of the first-stage detection model is generally smaller than the original image and has been scaled down, directly capturing images from the first-stage input image will result in unclear targets and unobvious features, affecting the detection effect. Here, we first map the target position information obtained in the first stage detection to the corresponding position of the target in the original image, and then expand a certain number of pixels around it to make it a square and the size is closest to the input size of the second-stage model. Finally, the image is scaled to the input size of the model using the resize operation. The images obtained through such an image capture process have targets that are not easily deformed, larger in size, and higher in clarity. Compared with the targets detected after scaling down the original image in the first stage, the features are more obvious, which is more conducive to the second-stage model for feature extraction, accurate classification, and positioning. The image capture process is as Figure 3 shown.

[0061] 3. Second-stage Detection

[0062] To improve the real-time performance of the entire process detection and reduce computing resources, the captured images are first screened. Only the images obtained from the targets with a size ratio or confidence score lower than a certain threshold are captured and sent to the second-stage model for detection. Otherwise, the second-stage detection is not performed, and the images are directly retained. Then, to prevent false detections, the filtering threshold for the target scores detected by the second-stage model needs to be set slightly higher. At the same time, the target status needs to be updated according to the detection results. If the targets consistent with the detection types in the first stage appear in the detection results, the target status remains unchanged. Otherwise, it is changed to False, and the image is filtered. Finally, the images that are not sent to the second-stage detection and the images remaining after the second-stage detection, along with the target information, are used as the final pedestrian and vehicle image detection results.

[0063] Compared with the prior art, the present invention can effectively combine the advantages of the two models, optimize some inherent problems existing in the single-stage detection model. Through the serial combination of two single-stage detection models, the detection accuracy can be significantly improved while maintaining real-time performance, reducing false alarms and missed detections, and having strong robustness and flexibility at the same time. Compared with the existing single high-performance model, two-stage detection model, and single lightweight model, this method provides a better balance in terms of performance and resource occupancy in complex backgrounds, especially in scenarios containing small targets, and is particularly suitable for scenarios that require both speed and accuracy. In addition, according to the actual application test situation, the method of training the model with hard examples is adopted, continuously adding the data set that causes false alarms to the previous training data set, and using a semi-automatic method to label the data, which can improve the accuracy of the model while reducing the consumption of labor costs.

[0064] Compared with the prior art, the present invention uses the trained model to detect the data in various scenarios. In particular, for the images with false alarms, the images with false alarms are used as hard examples and merged into the existing data set for training again to improve the adaptability of the model. Moreover, for the problem of easy missed detections and false alarms of small-sized and blurred targets in complex environments, the present invention uses the serial combination of two single-stage models for detection. Among them, the first stage quickly screens out the candidate regions, determines the target position and type, and the second stage performs refined detection, which can effectively improve the detection effect while maintaining real-time performance. In addition, for the acquisition of the input images of the second-stage model, the present invention adopts the method of first mapping to the original image and then performing extended image capture to obtain the images, which can make the features of the detected images more obvious and can effectively optimize the problems of missed detections and false alarms of small targets to a certain extent. In addition, through the application situation in the actual scenario, the present invention collects the data set and trains the model in combination with the actual scenario, and then integrates it into the existing intelligent products, which can effectively improve the performance and applicability of the model and better meet the needs of actual applications.

[0065] The above are the preferred embodiments of the present invention, but the present invention should not be limited to the content disclosed in these embodiments and the accompanying drawings. Therefore, all equivalent or modified implementations completed without departing from the spirit disclosed by the present invention fall within the protection scope of the present invention.

Claims

1. A two-stage pedestrian and vehicle detection method, characterized in that Specifically, it includes the following steps: S1. Collection and annotation of pedestrian and vehicle datasets S11. Collection of pedestrian and vehicle datasets, obtaining from publicly available datasets on the market or crawling image data of complex and diverse scenarios from the Internet; S12. First-stage annotation of pedestrian and vehicle detection data. First, use annotation software to annotate data in special scenarios, train a data annotation model, then use the trained model to detect unannotated image data to obtain unannotated data labels, and then manually correct this part of the labels annotated by the model and add them to the previously trained dataset, and train the data annotation model again. Repeat this process multiple times until all data annotation and correction work is completed; S13. Second-stage annotation of pedestrian and vehicle detection data. Directly put the data crawled from the network into the first-stage data annotation process for annotation and backup; For the data obtained in the first stage, while obtaining the image, calculate the current image label information according to the position information of the target in the original image; S2. Training of pedestrian and vehicle detection models S21. First-stage model training. Train pedestrian and vehicle pictures based on the yolov5 framework and replace the original Iou with the NWD loss function; S22. Collect data of various scenarios without the detected target type as background pictures, and merge these pictures with the previously annotated pedestrian and vehicle datasets for training; S23. Use the pedestrian and vehicle detection model trained in the first stage to detect datasets with a large scene coverage such as segAnything, verify the pictures with pedestrians and vehicles detected, extract the pictures with detection errors, and then merge the above data into the previous data for training; S24. Second-stage detection model training. Train the model based on the yolov8 framework. During the training process, use image preprocessing operations to reduce the training size of the images, and combine the extracted pictures with detection errors for optimized training of the model; S3. Application of pedestrian and vehicle detection on the device side S31. First-stage detection. Obtain the original data in yuv format from the video stream, convert and scale the yuv data into the format and size supported by the AI operation module, convert the previously trained model into a board-side model, use the operation module for operation to obtain the output features of the model, and perform post-processing on the output features to obtain the confidence levels and position information of pedestrian and vehicle targets; S32. Tracking of pedestrian and vehicle targets. Use the sort target tracking algorithm to track the detected pedestrian and vehicle targets, process consecutive frames, calculate the correlation of target information in two frames, including the distance information of the target and the overlap ratio of the target, form the same id for the same target, and at the same time, according to the previously obtained tracking trajectory information, extract and cache the image data at the corresponding position in the current image, and set the initial state of each target id to True; S33. Capture images of pedestrian and vehicle targets. Map the target position information obtained from the first-stage detection to the corresponding position of the target in the original image, then expand the pixels around it to make it a square and closest to the input size of the second-stage model. Finally, use the resize operation to scale the image to the input size of the model. S34. Second-stage detection. First, perform a screening operation on the captured images. Only send the captured images obtained from targets with a size ratio or confidence score lower than the set threshold into the second-stage model for detection. Otherwise, do not perform the second-stage detection and directly retain the images. Then, update the target status according to the detection results. If the same type of target as in the first-stage detection appears in the detection results, the target status remains unchanged. Otherwise, change it to False and filter the image. Finally, use the captured images and target information that were not sent into the second-stage detection and those remaining after the second-stage detection as the final pedestrian and vehicle image detection results.

2. The two-stage pedestrian and vehicle detection method according to claim 1, wherein In the step S11, the publicly available datasets include COCO2017, VOC2007, VOC2012, Object365, and Widerperson.

3. The two-stage pedestrian and vehicle detection method according to claim 1, characterized in that, In the step S12, the semi-automatic annotation method is adopted for the first-stage pedestrian and vehicle detection data annotation. First, train a model with a larger size, more parameters, and stronger generalization ability using the publicly available datasets on the market and the data of some special scenarios manually annotated with annotation software. Then, use the trained model to detect the unannotated image data to obtain the unannotated data labels.

4. The two-stage pedestrian and vehicle detection method according to claim 1, wherein, In the step S12, the annotation software is labelimg, and the annotation content is the position of each target and the corresponding type.

5. The two-stage pedestrian and vehicle detection method according to claim 4, characterized in that, The corresponding types are divided into 15 categories. In addition to the 6 types of pedestrian and vehicle targets to be detected, 9 types of false alarm categories are also annotated for training.

6. The two-stage pedestrian and vehicle detection method according to claim 1, wherein, In the step S24, the image preprocessing operations include color change, brightness change, rotation angle, and Gaussian blur processing.

7. The two-stage pedestrian and vehicle detection method according to claim 1, characterized in that, In the step S24, in order to improve the detection speed and reduce the model size, the training size is changed from 640x640 to 256x256.

Images