An adaptive detection and tracking method based on crowd density

Abstract

The application discloses a kind of adaptive detection and tracking method based on crowd density, comprising the following steps: monitoring video data preprocessing and data set generation;Pedestrian data set and head data set are used to train target detection algorithm respectively;Build target detection and tracking algorithm based on deep learning;Distance estimation is carried out using target position information;According to distance estimation result, crowd density is evaluated;According to the result of density evaluation, adaptive detection and tracking method switching is carried out.The application combines target detection and target tracking algorithm, uses distance estimation method to evaluate density to select appropriate detection and tracking target, realizes adaptive detection and tracking method, can adapt to any density scene, to a certain extent, solve the problem such as dense crowd shielding, improve target detection capability, improve pedestrian detection and tracking effect.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and computer vision, and is an adaptive target detection and tracking algorithm for surveillance images, involving a series of methods such as target detection, target tracking, monocular distance estimation, density assessment, and adaptive switching. Technical Background

[0002] In recent years, modern infrastructure construction has been gradually developing and improving. Video surveillance facilities have also formed a huge ubiquitous network, and intelligent security systems have been deployed in large numbers. They receive and perceive video image signals from IoT terminals and detect and track pedestrians in a large range of visual groups on the server side. This will effectively save manpower and financial resources, realize effective information mining, monitor abnormal emergencies, coordinate and dispatch resources, and provide early intelligent warnings.

[0003] However, the current security system is not intelligent enough, which leads to a prominent contradiction between the current growing massive video big data and the corresponding processing methods. For pedestrian targets in massive video data, they are still mainly recorded by online patrols and manual marking with the naked eye. Due to factors such as limited attention and decreased concentration, some important information will inevitably be missed and neglected.

[0004] Due to the ever-changing density of crowds caused by crowd mobility, pedestrians often block each other in densely populated scenes, which can severely impact pedestrian detection and tracking. Existing methods struggle to meet the demands of complex scenarios. Therefore, accurately detecting and tracking large groups of targets in real time from surveillance video big data, while also enabling intelligent analysis of multiple targets, has become an increasingly pressing and critical challenge facing the field of intelligent video surveillance. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings of the existing technology and propose an adaptive detection and tracking method based on crowd density. It uses a suitable distance estimation method to evaluate the crowd density, and adaptively switches the detection and tracking method according to the evaluated crowd density. It can adapt to any density scene, solve the problem of occlusion caused by dense crowds to a certain extent, enhance the target detection capability, and improve the pedestrian detection and tracking effect.

[0006] The object of the present invention is achieved through the following technical solutions:

[0007] An adaptive detection and tracking method based on crowd density includes the following steps:

[0008] Step S1: Obtain surveillance video data, perform preprocessing and dataset generation to obtain pedestrian datasets and head datasets;

[0009] Step S2: The pedestrian dataset and the head dataset are each trained by a target detection algorithm to obtain respective target detection models, that is, a pedestrian detection model and a head detection model;

[0010] Step S3: Build a target tracking model based on deep learning, cascade the target detection model and the target tracking model to obtain a cascade network, and the cascade network tracks pedestrians or head targets;

[0011] Step S4: Use the target position information output by the cascade network to estimate the distance between any two targets to obtain the distance D between any two targets;

[0012] Step S5: Evaluate the crowd density based on the estimated result of the distance D to obtain a density evaluation result of whether the crowd is dense or not dense;

[0013] Step S6: adaptively switch detection and tracking methods according to the density evaluation result to achieve target tracking.

[0014] In the above technical solution, S1 specifically includes the following steps: obtaining relevant surveillance video data, then extracting frames from the video at 1 frame per second and unifying the resolution, and finally obtaining a total of S m*n-sized high-quality pixel-level surveillance image datasets, and using labelImg software to annotate the pedestrian frames and head frames of the above surveillance image datasets to generate label data, thereby obtaining pedestrian datasets and head datasets.

[0015] In the above technical solution, in S2, a target detection network YOLOv5 network is trained using the pedestrian label dataset and the head label dataset respectively, that is, a pedestrian detection model and a head detection model are obtained to respectively realize pedestrian detection and head detection.

[0016] In the above technical solution, S3 specifically includes the following steps:

[0017] Build a DeepSORT target tracking algorithm based on deep learning and train a DeepSORT network to obtain a target tracking model;

[0018] The YOLOv5 network is cascaded with the DeepSORT network to obtain a cascade network. That is, after passing through the target detection network YOLOv5, the position information of the pedestrian or head is output, and then the pedestrian or head position information of consecutive frames is input into the DeepSORT network to track the pedestrian or head target.

[0019] In the above technical solution, in S4, the distance D between any two targets is obtained by the following method:

[0020] The target position information output by the cascade network is used to make appropriate distance estimates by setting a small number of parameters. Specifically, the following steps are included:

[0021] The target position information on the image is expanded from two-dimensional coordinates to three-dimensional coordinates, that is, each detected target is represented as a three-dimensional coordinate ( x , y , d ),in x , y is the plane coordinate of the uppermost midpoint of the target frame, d is the depth information of the target, and the expression is as follows:

[0022]

[0023] in w is the width of the target box, h is the height of the target frame. The depth is inversely proportional to the sum of the width and height of the target frame. The smaller the detection frame, the farther the pedestrian is from the surveillance camera and the greater the depth. Then, the L2 norm between the two targets is calculated and fine-tuned to estimate the distance. The formula is as follows:

[0024]

[0025] Among them, a, b, and c are hyperparameters and need to be adjusted according to the detection target and specific scenario. represent and The mean of the two targets, the three-dimensional coordinates of target 1 , target 2 three-dimensional coordinates .

[0026] In the above technical solution, in S5, the crowd density assessment adopts a close target pair threshold method based on target distance estimation, which specifically includes the following steps:

[0027] The distance D between any two targets and the distance threshold set in advance according to the specific scenario Compare, if the distance D is less than the threshold , then these two target pairs are set as close contact target pairs; when the number of close contact target pairs is greater than or equal to the close contact number threshold, it is determined that the crowd in this scene is dense; when the number of close contact target pairs is less than the set close contact number threshold, it is determined that the crowd in this scene is sparse.

[0028] In the above technical solution, in S6, the density assessment result obtained by the crowd density assessment method is judged. If the crowd is dense and it is in the pedestrian detection and tracking state at this time, it will switch to the head detection and tracking method to achieve target tracking. If the crowd is not dense and it is in the head detection and tracking state at this time, it will switch to the pedestrian detection and tracking method to achieve target tracking.

[0029] The present invention adopts the above technical solution, so it has the following beneficial effects:

[0030] 1. Real-time detection and tracking of crowds under monitoring perspective can adapt to scenes of any density, and the false detection and missed detection caused by occlusion and other problems are significantly improved. As a result, the average accuracy of target detection in scenes of any density is significantly improved, and the target detection and tracking capabilities are greatly enhanced. In special scenes such as hospitals, the demarcation of crowded dangerous areas and the search for suspected close special targets can be completed through the monitoring and analysis of group targets.

[0031] 2. To overcome some of the problems of common distance estimation methods for monocular images, including direct estimation methods, which are convenient and fast but suffer from multiple factors and low accuracy, and distance estimation methods based on inverse perspective transformation or 2D to 3D conversion, which are relatively accurate but technically complex and require many parameters, a new method for object distance estimation with fewer parameters is developed. This method uses a small number of parameters to accurately estimate the distance to the target position information output by the cascaded network. This method is fast and convenient, enabling real-time accurate distance estimation of targets in monocular images, with significantly improved accuracy compared to direct estimation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 、 Figure 2 This is an example of a surveillance image dataset;

[0033] Figure 3 Generate examples for labeled data;

[0034] Figure 4 This is an example diagram of pedestrian distance estimation;

[0035] Figure 5 Adaptive detection and tracking algorithm process based on crowd density;

[0036] Figure 6 An example of an adaptive detection and tracking algorithm based on crowd density. DETAILED DESCRIPTION

[0037] The following is a detailed description of the embodiments of the present invention. Although the present invention will be described and illustrated in conjunction with certain specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, modifications or equivalent substitutions of the present invention are intended to fall within the scope of the claims of the present invention.

[0038] In addition, in order to better illustrate the present invention, numerous specific details are given in the following detailed description. It will be understood by those skilled in the art that the present invention can also be implemented without these specific details.

[0039] An adaptive detection and tracking method based on crowd density includes the following steps:

[0040] Step S1: Obtain surveillance video data, perform preprocessing and dataset generation to obtain pedestrian datasets and head datasets;

[0041] Step S2: using the pedestrian dataset and the head dataset to train the target detection algorithm to obtain a target detection model;

[0042] Step S3: Build a target tracking model based on deep learning, cascade the target detection model and the target tracking model to obtain a cascade network, and the cascade network tracks pedestrians or head targets;

[0043] Step S4: Use the target position information output by the cascade network to estimate the distance between any two targets to obtain the distance D between any two targets;

[0044] Step S5: Evaluate the crowd density based on the estimated result of the distance D to obtain a density evaluation result indicating whether the crowd is relatively dense or not dense.

[0045] Step S6: adaptively switch detection and tracking methods according to the density evaluation result to achieve target tracking.

[0046] In the above technical solution, S1 specifically includes the following steps: obtaining relevant surveillance video data, then extracting frames from the video at 1 frame per second and unifying the resolution, and finally obtaining a total of S m*n-sized high-quality pixel-level surveillance image datasets, and using labelImg software to annotate the pedestrian frames and head frames of the above surveillance image datasets to generate label data, thereby obtaining pedestrian datasets and head datasets.

[0047] In the above technical solution, S2 specifically includes the following steps: using the pedestrian label dataset and the head label dataset to train a YOLOv5 network respectively, that is, obtaining a pedestrian detection model and a head detection model to respectively realize pedestrian detection and head detection.

[0048] The specific prior art YOLOv5 is further described as follows:

[0049] Compared with other frameworks, the YOLO series has advantages such as balanced speed and accuracy, high flexibility, and strong operability. In addition, YOLOv5 has four different models with strong selectivity. Therefore, we built the deep learning-based target detection algorithm YOLOv5.

[0050] YOLOv5 is an improvement over YOLOv4. Its overall detection strategy is to use a feature extraction network with a residual structure to extract features from the input image, then perform feature fusion and output three feature layers for classification and regression prediction.

[0051] The backbone network mainly adopts the Focus structure and CSP structure.

[0052] The Focus structure slices the input image and expands some information to the dimension level;

[0053] The CSP structure draws on the design ideas of YOLOv4's CSPNet and designs two CSP structures for application in different structural layers.

[0054] In addition, YOLOv5 introduces FPN and PANet structures in the detection branch, aggregates parameters of different detection layers from different backbone layers, and enhances the fusion of features of different layers. In the loss function part, CIOU_Loss is used as the loss function of the target Bounding Box, and the calculation method is:

[0055]

[0056] Among them, IOU is the ratio of the intersection area of ​​two boxes to the union area; it represents the Euclidean distance between the center points of the predicted box and the true box; Represents the diagonal distance of the minimum closure area that can contain both the predicted box and the true box; and The relationship is as follows:

[0057]

[0058]

[0059] in, and Denotes the width and height of the corresponding bounding box. The confidence loss and classification loss follow traditional methods. A YOLOv5 network is trained using the pedestrian label dataset and the head label dataset to perform pedestrian detection and head detection, respectively.

[0060] In the above technical solution, S3 specifically includes the following steps:

[0061] Among the multi-target tracking algorithms, DeepSort has the advantages of strong real-time performance and good occlusion tracking effect. Therefore, a target tracking algorithm DeepSORT based on deep learning is built to train a DeepSORT network to obtain a target tracking model.

[0062] Afterwards, the YOLOv5 network is cascaded with the DeepSORT network to obtain a cascade network. That is, after passing through the target detection network YOLOv5, the position information of the pedestrian or head can be output, and then the pedestrian or head position information of the continuous frames is input into the DeepSORT network of the target tracking algorithm DeepSORT, thereby tracking the pedestrian or head target.

[0063] In the above technical solution, S4 specifically includes the following steps:

[0064] To avoid some problems of common distance estimation methods for monocular images, including direct estimation methods that are convenient and fast but have many influencing factors and low accuracy, and distance estimation methods based on inverse perspective transformation or 2D to 3D conversion that are more accurate but technically complex and have many parameters, a target distance estimation method with fewer parameters and higher accuracy is applied. The target position information output by the cascade network is appropriately estimated by setting a small number of parameters.

[0065] like Figure 3 Specifically, the target information (pedestrians or heads) on the image is expanded from two-dimensional coordinates to three-dimensional coordinates, that is, each detected target is represented as a three-dimensional coordinate ( x , y , d ),in x , y is the plane coordinate of the uppermost midpoint of the target frame, d is the depth information of the target, and the expression is as follows:

[0066]

[0067] in w is the width of the target box, h The height of the target frame is inversely proportional to the sum of the width and height of the target frame. This means that the smaller the detection frame, the farther the pedestrian (head) is from the surveillance camera and the greater its depth. The L2 norm between the two targets is then calculated and fine-tuned to estimate the distance. The formula is as follows:

[0068]

[0069] Among them, a, b, and c are hyperparameters and need to be adjusted according to the detection target and specific scenario. represent and The mean of the two targets, the three-dimensional coordinates of target 1 , target 2 three-dimensional coordinates .

[0070] In the above technical solution, S5 specifically includes the following steps:

[0071] In order to reduce the complexity of density assessment and improve the speed, the crowd density assessment adopts the close target pair threshold method based on target distance estimation.

[0072] Specifically, the distance between all objects in the image is obtained by the above distance estimation method, and the distance threshold set in advance according to the specific scene is Comparison, if the distance D between the two targets is less than the threshold , then this target pair is set as a close contact target pair; when the number of close contact target pairs is greater than or equal to the close contact number threshold set in advance, it is determined that the crowd in this scene is dense; when the number of close contact target pairs is less than the set number threshold, it is determined that the crowd in this scene is sparse.

[0073] In the above technical solution, S6 specifically includes the following steps:

[0074] Most surveillance cameras today are located on indoor rooftops or high up outdoors, using a downward-looking angle for video surveillance. Therefore, when crowds are dense and obstructions occur, the proportion and probability of pedestrians below their heads being obstructed is much greater than the proportion and probability of pedestrians above their heads being obstructed. Therefore, in densely populated scenes, the performance of pedestrian head detection is stronger than the overall pedestrian detection performance, and the effect of pedestrian head detection and tracking is far better than that of detecting and tracking the entire pedestrian. However, in less crowded scenes, the proportion of occlusions decreases significantly, resulting in a higher probability of missed detection and lower detection accuracy due to the smaller size of the pedestrian head area compared to the entire pedestrian area, which is more influential than the impact of occlusion. Therefore, in sparsely populated scenes, the performance of pedestrian head detection is weaker than the overall pedestrian detection performance, and the effect of detecting and tracking the entire pedestrian is better than that of detecting and tracking the pedestrian head.

[0075] Therefore, if Figure 4 As shown, the density obtained by the above density assessment method is used to judge. If the crowd is dense and the pedestrian detection and tracking state is at this time, the head detection and tracking method will be switched. If the crowd is not dense and the head detection and tracking state is at this time, the pedestrian detection and tracking method will be switched. Figure 5 shown.

Claims

1. An adaptive detection and tracking method based on crowd density, characterized in that: The following steps are involved: Step S1: Obtain surveillance video data, perform preprocessing and dataset generation to obtain pedestrian datasets and head datasets; Step S2: The pedestrian dataset and the head dataset are each trained by a target detection algorithm to obtain respective target detection models, that is, a pedestrian detection model and a head detection model; Step S3: Build a target tracking model based on deep learning, cascade the target detection model and the target tracking model to obtain a cascade network, and the cascade network tracks pedestrians or head targets; Step S4: Use the target position information output by the cascade network to estimate the distance between any two targets to obtain the distance D between any two targets; Step S5: Evaluate the crowd density based on the estimated result of the distance D to obtain a density evaluation result of whether the crowd is dense or not dense; Step S6: adaptively switch detection and tracking methods according to the density evaluation result to achieve target tracking.

2. The method for adaptive detection and tracking based on crowd density according to claim 1, characterized in that: The S1 specifically includes the following steps: obtaining relevant surveillance video data, then extracting frames from the video at 1 frame per second and unifying the resolution, and finally obtaining a total of S m*n-sized high-quality pixel-level surveillance image datasets, and using labelImg software to annotate the above surveillance image datasets with pedestrian frames and head frames to generate label data, thereby obtaining pedestrian datasets and head datasets.

3. The method of adaptive detection and tracking based on crowd density according to claim 1, characterized in that: In S2, a target detection network YOLOv5 network is trained using the pedestrian label dataset and the head label dataset, respectively, to obtain a pedestrian detection model and a head detection model, so as to realize pedestrian detection and head detection respectively.

4. The method of adaptive detection and tracking based on crowd density according to claim 3, characterized in that: The S3 specifically includes the following steps: Build a DeepSORT target tracking algorithm based on deep learning and train a DeepSORT network to obtain a target tracking model; The YOLOv5 network is cascaded with the DeepSORT network to obtain a cascade network. That is, after passing through the target detection network YOLOv5, the position information of the pedestrian or head is output, and then the pedestrian or head position information of consecutive frames is input into the DeepSORT network to track the pedestrian or head target.

5. The method of adaptive detection and tracking based on crowd density according to claim 1, characterized in that: In S4, the distance D between any two targets is obtained by the following method: The target position information output by the cascade network is used to make appropriate distance estimates by setting a small number of parameters. Specifically, the following steps are included: The target position information on the image is expanded from two-dimensional coordinates to three-dimensional coordinates, that is, each detected target is represented as a three-dimensional coordinate ( x , y , d ),in x , y is the plane coordinate of the uppermost midpoint of the target frame, d is the depth information of the target, and the expression is as follows: in w is the width of the target box, h is the height of the target frame. The depth is inversely proportional to the sum of the width and height of the target frame. The smaller the detection frame, the farther the pedestrian is from the surveillance camera and the greater the depth. Then, the L2 norm between the two targets is calculated and fine-tuned to estimate the distance. The formula is as follows: Among them, a, b, and c are hyperparameters and need to be adjusted according to the detection target and specific scenario. represent and The mean of the two targets, the three-dimensional coordinates of target 1 , target 2 three-dimensional coordinates .

6. The method of adaptive detection and tracking based on crowd density according to claim 1, characterized in that: In S5, the crowd density assessment adopts a close target pair threshold method based on target distance estimation, which specifically includes the following steps: The distance D between any two targets and the distance threshold set in advance according to the specific scenario Compare, if the distance D is less than the threshold , then these two target pairs are set as close contact target pairs; when the number of close contact target pairs is greater than or equal to the close contact number threshold, it is determined that the crowd in this scene is dense; when the number of close contact target pairs is less than the set close contact number threshold, it is determined that the crowd in this scene is sparse.

7. The method of adaptive detection and tracking based on crowd density according to claim 6, characterized in that: In S6, the density assessment result obtained by the crowd density assessment method is judged. If the crowd is dense and the pedestrian detection and tracking state is in progress, the head detection and tracking method will be switched to achieve target tracking. If the crowd is not dense and the head detection and tracking state is in progress, the pedestrian detection and tracking method will be switched to achieve target tracking.

Images