Two-stage pedestrian and vehicle detection method

Abstract

The invention provides a two-stage pedestrian and vehicle detection method. The two-stage pedestrian and vehicle detection method specifically comprises the following steps: S1, collecting and marking a pedestrian and vehicle data set; s2, training a pedestrian and vehicle detection model; and S3, applying the pedestrian and vehicle detection equipment end. According to the method, the advantages of a single-stage model and a double-stage model can be effectively combined, some inherent problems existing in a single-stage detection model are optimized, through serial combination of the two single-stage detection models, the detection precision can be remarkably improved on the premise that the real-time performance is kept, false alarm and missing alarm are reduced, and meanwhile high robustness and flexibility are achieved. Compared with a single high-performance model, a two-stage detection model and a single lightweight model in the prior art, the method provides better balance in performance and resource occupation in a complex background, especially in a scene containing a small target, and is especially suitable for a scene needing to consider both speed and precision.

Description

Technical Field

[0001] The present invention relates to the technical field of security monitoring, and in particular to a two-stage pedestrian and vehicle detection method. Background Art

[0002] The rapid development of deep learning technology has made intelligent monitoring, intelligent alarm and other intelligent technologies the focus of today's society. Among them, camera-based pedestrian and vehicle detection technology has also been widely used, especially in public safety monitoring and intelligent security systems. Camera-based pedestrian and vehicle detection technology plays a vital role.

[0003] Existing pedestrian and vehicle detection technologies can be divided into single-stage and dual-stage according to the principles of target detection algorithms and their detection methods. The single-stage target detection algorithm directly predicts the position and category of the target in the input image during the detection process, which is an end-to-end process. Since it is necessary to optimize both classification and positioning tasks in one network, in order to achieve better detection results, the model is usually too complex and difficult to train and optimize. The lightweight single-stage network not only has poor detection results for some small targets, but also has difficulty distinguishing between background and target in some complex background environments, resulting in low detection accuracy and more false alarms. The dual-stage target detection algorithm needs to generate additional candidate areas during the detection process, which requires a large amount of computing resources, resulting in a slow detection speed, making it difficult to meet real-time requirements, and is not suitable for embedded and mobile devices.

[0004] The current target detection algorithms are effective for detecting large targets with obvious features. However, in actual environments, there are often other vehicles or pedestrians that cause obstruction, and vehicles and pedestrians in the distance are small, which can easily cause target features to be missing or unclear. This will not only increase the missed detection rate and false detection rate, but also reduce the detection accuracy. Summary of the invention

[0005] In view of the shortcomings of the existing technology, the present invention proposes a two-stage pedestrian and vehicle detection method. Through the serial combination of two single-stage detection models, the detection accuracy can be significantly improved while maintaining real-time performance, and false alarms and missed alarms can be reduced. At the same time, it has strong robustness and flexibility.

[0006] To implement the above technical solution, the present invention provides a two-stage pedestrian and vehicle detection method, which specifically includes the following steps:

[0007] S1. Collection and annotation of pedestrian and vehicle datasets

[0008] S11. Collection of pedestrian and vehicle datasets, obtained from publicly available datasets on the market or by crawling complex and diverse scene image data from the Internet;

[0009] S12, the first stage of pedestrian and vehicle detection data labeling, first use the labeling software to label the data of special scenes, train a data labeling model, then use the trained model to detect the unlabeled image data, obtain the unlabeled data label, and then manually correct the part of the label labeled by the model, add it to the previously trained data set, and train the data labeling model again, and repeat it multiple times until all data labeling and correction work is completed;

[0010] S13, in the second stage of pedestrian and vehicle detection data labeling, the data crawled from the network is directly put into the first stage data labeling process for labeling and standby; while the data obtained from the first stage, while acquiring the image, the current image label information is calculated according to the position information of the target in the original image;

[0011] S2. Pedestrian and vehicle detection model training

[0012] S21, the first stage of model training, training pedestrian and vehicle images based on the yolov5 framework, and using the NWD loss function to replace the original Iou;

[0013] S22, collecting a portion of scene data that does not contain the detection target type as background images, and merging these images with the previously labeled pedestrian and vehicle data sets for training;

[0014] S23, use the pedestrian and vehicle detection model trained in the first stage to detect a data set with a large scene coverage such as segAnything, verify the images with pedestrians and vehicles detected, extract the images with detection errors, and then merge the above data into the previous data for training;

[0015] S24, the second stage detection model training, the model training is based on the yolov8 framework, the image preprocessing operation is used in the training process to reduce the training size of the image, and the model optimization training is carried out together with the extracted detection error pictures;

[0016] S3, Pedestrian and vehicle detection equipment application

[0017] S31, the first stage of detection, obtains the original data from the video stream in yuv format, converts and scales the yuv data to the format and size supported by the AI ​​computing module, converts the previously trained model to the board-side model, uses the computing module to calculate the output features of the model, and post-processes the output features to obtain the confidence and position information of pedestrians and vehicle targets;

[0018] S32, pedestrian and vehicle target tracking, using the sort target tracking algorithm to track the detected pedestrian and vehicle targets, process the continuous frames, calculate the correlation of the target information in the two frames, including the distance information of the target and the overlap ratio of the target, the same target forms the same ID, and at the same time, according to the tracking trajectory information obtained by the previous tracking, the image data of the corresponding position in the current image is extracted and cached, and the initial state of each target ID is set to True;

[0019] S33, capturing pedestrian and vehicle targets, mapping the target position information obtained by the first stage detection to the corresponding position of the target in the original image, and then expanding the pixels around to make it a square with a size closest to the input size of the second stage model, and finally resizing the image to the input size of the model;

[0020] S34, the second stage detection, first filter the captured images, and only send those captured images obtained from targets with size ratio or confidence score lower than the set threshold to the second stage model for detection. Otherwise, the second stage detection is not performed, and the image is directly retained. Then, the target status is updated according to the detection results. If a target consistent with the detection type of the first stage appears in the detection results, the target status remains unchanged, otherwise it is changed to False and the image is filtered. Finally, the captured images and target information that are not sent to the second stage detection and remain after the second stage detection are used as the final pedestrian and vehicle image detection results.

[0021] Preferably, in step S11, the publicly available data sets include COCO2017, VOC2007, VOC2012, Object365, and Widerperson.

[0022] Preferably, in step S12, the first-stage pedestrian and vehicle detection data labeling adopts a semi-automatic labeling method, firstly using the publicly available data sets on the market and the data of some special scenes manually labeled using labeling software to train a model with a larger size, more parameters and stronger generalization ability, and then using the trained model to detect unlabeled image data to obtain unlabeled data labels.

[0023] Preferably, in step S12, the labeling software is labelimg, and the labeled content is the position of each target and the corresponding type.

[0024] Preferably, the corresponding types are divided into 15 categories. In addition to the 6 types of pedestrian and vehicle targets that need to be detected, 9 types of false alarm categories are also marked for training.

[0025] Preferably, in step S24, the image preprocessing operation includes color change, brightness change, rotation angle and Gaussian blur processing.

[0026] Preferably, in step S24, in order to improve the detection speed and reduce the model size, the training size is changed from 640x640 to 256x256.

[0027] The beneficial effects of the two-stage pedestrian and vehicle detection method provided by the present invention are:

[0028] (1) The present invention can effectively combine the advantages of the single-stage and two-stage models, optimize some inherent problems of the single-stage detection model, and through the serial combination of two single-stage detection models, can significantly improve the detection accuracy while maintaining real-time performance, reduce false positives and missed detections, and have strong robustness and flexibility. Compared with the existing single high-performance model, two-stage detection model and single lightweight model, this method provides a better balance in performance and resource usage in complex backgrounds, especially in scenes containing small targets, and is particularly suitable for scenes that require both speed and accuracy.

[0029] (2) The present invention uses the trained model to detect data from various scenarios. In particular, for false positive pictures, the false positive pictures are taken as difficult examples and merged into the existing data set for training again, thereby improving the adaptability of the model.

[0030] (3) The present invention aims to solve the problem of small-sized and fuzzy targets being easily missed or misreported in complex environments by using two single-stage models in series for detection. In the first stage, candidate areas are quickly screened to determine the target location and type, and in the second stage, refined detection is performed, which can effectively improve the detection effect while maintaining real-time performance.

[0031] (4) In acquiring the input image of the second-stage model, the present invention adopts a method of first mapping to the original image and then performing extended image capture to acquire the image, which can make the features of the detected image more obvious and effectively optimize the problem of underreporting and false alarm of small targets to a certain extent.

[0032] (5) The present invention can effectively improve the performance and applicability of the model and better meet the needs of practical applications by collecting data sets and training models in combination with actual scenarios and then integrating them into existing smart products. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of the steps of the present invention.

[0034] Figure 2 This is the first stage model training flowchart.

[0035] Figure 3 Flowchart for capturing images of pedestrian and vehicle targets.

[0036] Figure 4 The flowchart of two-stage pedestrian and vehicle detection. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary persons in the art without creative work are within the scope of protection of the present invention.

[0038] Embodiment: A two-stage pedestrian and vehicle detection method.

[0039] Reference Figures 1 to 4 As shown, the present invention provides a two-stage pedestrian and vehicle detection method. In view of the fact that the two-stage target detection algorithm is slow and cannot meet the real-time requirements, the present invention uses a single-stage network for detection and directly outputs the location information and category of the detected target. In view of the fact that the single-stage target detection algorithm has poor detection effect in complex environments, especially in images containing complex backgrounds and smaller targets, the present invention adopts a serial combination of two single-stage networks. The first-stage network quickly locates the position of the target, and then the output result of the first-stage network is processed and sent to the second-stage network for more accurate classification and positioning. The combined detection results of the two times can optimize the detection effect to a certain extent.

[0040] The two-stage pedestrian detection method provided by the present invention mainly includes: the collection of pedestrian and vehicle data sets, the training of pedestrian and vehicle detection models, and the application of pedestrian and vehicle detection equipment. In the collection of pedestrian and vehicle data sets, in addition to using the existing large data sets, some picture data of corresponding scenes are collected from the Internet as positive and negative samples for model training, and the semi-automatic annotation method is used to improve the annotation speed of the collected images. The first-stage detection model and the second-stage detection model are trained respectively using the previously collected data using a deep learning framework, and then the trained models are converted to the corresponding device end using corresponding conversion tools. On the device end, after the image data is input in the first stage, the pedestrian and vehicle detection model is loaded using the AI ​​computing module to calculate the data to obtain the position information and category of the pedestrian and vehicle targets in the first stage, and the pedestrian and vehicle targets are tracked using the target tracking algorithm, and the corresponding target image is cut from the original image according to the tracked target information, and sent to the second-stage detection model for detection to obtain the pedestrian and vehicle target position and category again, and finally the target information of the two detections is combined to obtain the final detection result.

[0041] The detailed process of the two-stage pedestrian detection method provided by the present invention is as follows:

[0042] 1. Collection and annotation of pedestrian and vehicle detection datasets:

[0043] In the first phase, pedestrian and vehicle detection images are mainly obtained from some existing large-scale datasets, such as COCO2017, VOC2007, VOC2012, Object365, Widerperson, etc., as well as some complex and diverse scene image data crawled from the Internet. In the second phase, some pedestrian detection images are obtained from the first phase images using the extended method, and some are obtained separately from the Internet.

[0044] In the first stage, a semi-automatic labeling method is used for pedestrian and vehicle detection data labeling. First, a model with larger size, more parameters and stronger generalization ability is trained using publicly available data sets on the market and data of some special scenes manually labeled using labeling software. Then the trained model is used to detect unlabeled image data to obtain unlabeled data labels. The labels annotated by the model are manually corrected and added to the previously trained data set. The model is trained again and the cycle continues until all data are labeled and corrected.

[0045] In the second stage, pedestrian and vehicle detection data are labeled, in which the data crawled from the Internet is directly put into the first stage data labeling process for labeling and standby; while the data obtained from the first stage, while acquiring the image, can calculate the current image label information based on the position information of the target in the original image, and no separate labeling is required.

[0046] The tool used to label the training data for pedestrian and vehicle detection is labelimg. The labeled content is the location of each target and its corresponding type. There are a total of 15 types. In addition to the 6 types of pedestrian and vehicle targets that need to be detected, 9 types of categories that are prone to false alarms are also labeled for training.

[0047] (II) Model training

[0048] 1. The first stage of model training:

[0049] Pedestrian and vehicle images are trained based on the yolov5 framework. In order to ensure the detection speed and facilitate subsequent deployment and application, the first stage of this method uses the lighter structure of yolov5s and appropriately reduces the depth and width of the network structure. In order to reduce the omission of small targets, the original Iou is replaced with the NWD loss function, which can better reduce the sensitivity of Iou to the deviation of small objects.

[0050] In order to improve the generalization ability of the pedestrian and vehicle detection model, in addition to the commonly used data enhancement operations during training, a portion of scene data that does not contain the detection target type will be collected as background images, and these images will be merged with the previously labeled pedestrian and vehicle data sets for training to improve the model's adaptability in various scenes and reduce false alarms in unknown scenes. The trained model will be tuned again to reduce false alarms. Specifically, the trained pedestrian and vehicle detection model will be used to detect data sets with a large scene coverage such as segAnything, and the images with pedestrians and vehicles will be verified. In particular, the images with incorrect detection will be extracted, because these images contain false alarm data that the model recognizes incorrectly, which are called difficult examples. The difficult example data will be merged into the previous data for training. After training, the false alarms of such difficult examples will be reduced. For the specific process, see Figure 3 shown.

[0051] 2. Second stage detection model training:

[0052] The second stage is to train the model based on the yolov8 framework. Compared with yolov5, yolov8 has a certain improvement in detection effect with less increase in computing resources. This stage is mainly aimed at some scenes with complex environments and small targets. Due to the limitations of model detection size, size and real-time requirements, there are many false detections using only a single-stage detection model, especially for small-sized objects in the distance and some objects with low clarity. In order to increase the detection speed and reduce the model size, the training size is changed from 640x640 to 256x256. The training process uses image preprocessing operations such as color change, brightness change, rotation angle, etc.; in order to reduce false positives of blurred targets, a certain probability of Gaussian blur processing is added to the data during training; in order to improve the generalization ability of the model, this stage also uses the method of combining difficult examples with training to tune the model.

[0053] 3. Device-side applications

[0054] The two-stage pedestrian and vehicle detection device-side application mainly includes: first, sending the image to the first-stage model for detection to determine the specific location and category of pedestrians and vehicles, then using the target tracking algorithm to track the detected target, and capturing the image on the original image based on the tracked target information, and finally sending the captured image to the second stage for detection, and combining the target information of the two detections to obtain the final pedestrian and vehicle detection result.

[0055] Specific process reference Figure 4 Two-stage pedestrian and vehicle detection flow chart:

[0056] 1. First stage detection

[0057] Get the original data from the video stream in yuv format, convert and scale the yuv data to the format and size supported by the AI ​​computing module, convert the previously trained model to the board model, use the computing module to calculate the output features of the model, post-process the output features, and obtain the confidence and location information of pedestrians and vehicle targets. The confidence threshold set in the post-processing stage needs to be appropriately lowered to prevent some detection targets with low scores due to blur or small size from being filtered out. These targets may be correctly detected in the second stage, which can reduce missed detections to a certain extent.

[0058] 2. Pedestrian and vehicle target tracking and image capture

[0059] Pedestrian and vehicle target tracking uses the sort target tracking algorithm to track the detected pedestrian and vehicle targets. For continuous frames, the correlation of target information in two frames is calculated, including the distance information of the target and the overlap ratio of the target. The same target forms the same ID. At the same time, according to the tracking trajectory information obtained by the previous tracking, the image data of the corresponding position in the current image is captured and cached, and the initial state of each target ID is set to True.

[0060] Pedestrian and vehicle target capture, in order to allow the captured images to be directly used for the second-stage model detection and improve the accuracy of detection, the image capture operation needs some special processing, and it is no longer directly based on the target position information obtained in the first stage to capture the image. First, since the input image size of the embedded device AI computing module needs to be fixed, it is necessary to ensure that the captured image size is consistent with the input size of the second-stage detection model. Then, since the input image size of the first-stage detection model is generally smaller than the original image, and the image has been thumbnailed, directly capturing the image from the first-stage input image will result in unclear targets and unclear features, affecting the detection effect. Here, we first map the target position information obtained from the first-stage detection to the corresponding position of the target in the original image, and then expand a certain number of pixels around to make it a square and the size is closest to the input size of the second-stage model. Finally, the image is scaled to the input size of the model using the resize operation. The images obtained through such an image capture process are not easily deformed, have larger sizes, and higher clarity. Compared with the targets detected after thumbnails from the original images in the first stage, the features are more obvious, which is more conducive to feature extraction, accurate classification and positioning by the second-stage model. The image capture process is as follows: Figure 3 shown.

[0061] 3. Second stage detection

[0062] In order to improve the real-time detection of the entire process and reduce computing resources, the captured images are first filtered, and only those images obtained from targets with size ratios or confidence scores below a certain threshold are captured and sent to the second-stage model for detection. Otherwise, the second-stage detection is not performed and the images are directly retained. Then, in order to prevent false detection, the filtering threshold of the target score of the second-stage model detection needs to be set slightly higher, and the target status needs to be updated according to the detection results. If a target of the same type as the first-stage detection appears in the detection results, the target status remains unchanged, otherwise it is changed to False and the image is filtered. Finally, the captured images and target information of the images that are not sent to the second-stage detection and the images left after the second-stage detection are used as the final pedestrian and vehicle image detection results.

[0063] Compared with the prior art, the present invention can effectively combine the advantages of the two models, optimize some inherent problems existing in the single-stage detection model, and through the serial combination of two single-stage detection models, it can significantly improve the detection accuracy while maintaining real-time performance, reduce false positives and missed detections, and at the same time have strong robustness and flexibility. Compared with the existing single high-performance model, two-stage detection model and single lightweight model, this method provides a better balance in terms of performance and resource usage in complex backgrounds, especially in scenes containing small targets, and is particularly suitable for scenes that need to take both speed and accuracy into account. In addition, according to the actual application test situation, the method of training the model with difficult examples is adopted, and the data sets that cause false positives are continuously added to the previous training data sets, and the data are annotated in a semi-automatic manner, which improves the accuracy of the model while also reducing the consumption of labor costs.

[0064] Compared with the prior art, the present invention uses a trained model to detect data from various scenes. In particular, for false alarm pictures, the false alarm pictures are merged into the existing data set as difficult examples for training again, thereby improving the adaptability of the model. Moreover, the present invention uses two single-stage models in series to detect the problem that small-sized and fuzzy targets are prone to missed reports and false alarms in complex environments. Among them, the first stage quickly screens out candidate areas, determines the target location and type, and the second stage performs refined detection, which can effectively improve the detection effect while maintaining real-time performance. In addition, in the acquisition of the input picture of the model in the second stage, the present invention adopts the method of first mapping to the original picture and then performing extended picture capture to obtain the picture, which can make the characteristics of the detected picture more obvious, and can effectively optimize the problem of missed reports and false alarms of small targets to a certain extent. In addition, the present invention can effectively improve the performance and applicability of the model by combining the application in actual scenes, collecting data sets and training models in combination with actual scenes, and then integrating them into existing smart products, so as to better meet the needs of actual applications.

[0065] The above description is only a preferred embodiment of the present invention, but the present invention should not be limited to the contents disclosed in the embodiment and the drawings, so any equivalent or modification completed without departing from the spirit disclosed in the present invention shall fall within the scope of protection of the present invention.

Claims

1. A two-stage pedestrian and vehicle detection method, characterized in that The specific steps include: S1. Collection and annotation of pedestrian and vehicle datasets S11. Collection of pedestrian and vehicle datasets, obtained from publicly available datasets on the market or by crawling complex and diverse scene image data from the Internet; S12, the first stage of pedestrian and vehicle detection data labeling, first use the labeling software to label the data of special scenes, train a data labeling model, then use the trained model to detect the unlabeled image data, obtain the unlabeled data label, and then manually correct the part of the label labeled by the model, add it to the previously trained data set, and train the data labeling model again, and repeat it multiple times until all data labeling and correction work is completed; S13, in the second phase of pedestrian and vehicle detection data labeling, the data crawled from the network is directly put into the first phase of data labeling process for labeling and standby; The data obtained from the first stage, while acquiring the image, calculates the current image label information based on the position information of the target in the original image; S2. Pedestrian and vehicle detection model training S21, the first stage of model training, training pedestrian and vehicle images based on the yolov5 framework, and using the NWD loss function to replace the original Iou; S22, collecting a portion of scene data that does not contain the detection target type as background images, and merging these images with the previously labeled pedestrian and vehicle data sets for training; S23, use the pedestrian and vehicle detection model trained in the first stage to detect a data set with a large scene coverage such as segAnything, verify the images with pedestrians and vehicles detected, extract the images with detection errors, and then merge the above data into the previous data for training; S24, the second stage detection model training, the model training is based on the yolov8 framework, the image preprocessing operation is used in the training process to reduce the training size of the image, and the model optimization training is carried out together with the extracted detection error pictures; S3, Pedestrian and vehicle detection equipment application S31, the first stage of detection, obtains the original data from the video stream in yuv format, converts and scales the yuv data to the format and size supported by the AI ​​computing module, converts the previously trained model to the board-side model, uses the computing module to calculate the output features of the model, and post-processes the output features to obtain the confidence and position information of pedestrians and vehicle targets; S32, pedestrian and vehicle target tracking, using the sort target tracking algorithm to track the detected pedestrian and vehicle targets, process the continuous frames, calculate the correlation of the target information in the two frames, including the distance information of the target and the overlap ratio of the target, the same target forms the same ID, and at the same time, according to the tracking trajectory information obtained by the previous tracking, the image data of the corresponding position in the current image is extracted and cached, and the initial state of each target ID is set to True; S33, capturing pedestrian and vehicle targets, mapping the target position information obtained by the first stage detection to the corresponding position of the target in the original image, and then expanding the pixels around to make it a square with a size closest to the input size of the second stage model, and finally resizing the image to the input size of the model; S34, the second stage detection, first filter the captured images, and only send those captured images obtained from targets with size ratio or confidence score lower than the set threshold to the second stage model for detection. Otherwise, the second stage detection is not performed, and the image is directly retained. Then, the target status is updated according to the detection results. If a target consistent with the detection type of the first stage appears in the detection results, the target status remains unchanged, otherwise it is changed to False and the image is filtered. Finally, the captured images and target information that are not sent to the second stage detection and remain after the second stage detection are used as the final pedestrian and vehicle image detection results.

2. The two-stage pedestrian and vehicle detection method according to claim 1, characterized in that: In step S11, publicly available datasets include COCO2017, VOC2007, VOC2012, Object365, and Widerperson.

3. The two-stage pedestrian and vehicle detection method according to claim 1, characterized in that: In step S12, the first stage pedestrian and vehicle detection data labeling adopts a semi-automatic labeling method, firstly, a model with larger size, more parameters and stronger generalization ability is trained together with the publicly available data sets on the market and the data of some special scenes manually labeled using labeling software, and then the trained model is used to detect the unlabeled image data to obtain the unlabeled data labels.

4. The two-stage pedestrian and vehicle detection method according to claim 1, characterized in that: In step S12, the labeling software is labelimg, and the labeling content is the position of each target and the corresponding type.

5. The two-stage pedestrian and vehicle detection method as claimed in claim 4, characterized in that: The corresponding types are divided into 15 categories. In addition to the 6 types of pedestrian and vehicle targets that need to be detected, 9 types of false alarm categories are also marked for training.

6. The two-stage pedestrian and vehicle detection method according to claim 1, characterized in that: In step S24, the image preprocessing operations include color change, brightness change, rotation angle and Gaussian blur processing.

7. The two-stage pedestrian and vehicle detection method according to claim 1, characterized in that: In step S24, in order to improve the detection speed and reduce the model size, the training size is changed from 640x640 to 256x256.

Images