Pedestrian wandering retention detection method, device, equipment and medium

Abstract

The embodiment of the invention provides a pedestrian wandering and detention detection method and device, equipment and a medium, and aims to solve the problem of inaccurate pedestrian wandering or detention detection results in related technologies. According to the embodiment of the invention, the electronic equipment determines each target in the video frame images according to the target detection model, determines the identifier of each target in the plurality of video frame images according to the tracking model, and determines that the target has a wandering or retention problem when the occurrence frequency of the target in the plurality of video frame images is high. Therefore, the accuracy of the wandering or retention detection result is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, equipment and medium for detecting pedestrian loitering and lingering. Background Technology

[0002] With the development of technology and the progress of society, pedestrian loitering and lingering detection algorithms have been widely used in many fields, especially in public safety, traffic management, commercial applications, and health and welfare. These algorithms typically combine computer vision and machine learning techniques, using cameras to collect video data and analyze pedestrian movement trajectories and behavioral patterns to identify loitering or lingering behavior.

[0003] In related technologies, pedestrian loitering or lingering is usually identified by users. This method has high labor costs and relies heavily on subjective opinions, resulting in low accuracy. Summary of the Invention

[0004] This application provides a method, apparatus, device, and medium for detecting pedestrian loitering or lingering, which aims to solve the problem of inaccurate results in the detection of pedestrian loitering or lingering in related technologies.

[0005] In a first aspect, embodiments of this application provide a method for detecting pedestrian loitering or lingering, the method comprising: The method involves acquiring video streams captured by image acquisition devices in public places, including airports, high-speed rail stations, stadiums, and other places where pedestrians gather; extracting multiple video frame images from the video stream at preset time intervals; and determining each target in the video frame image based on a pre-trained target detection model, where the target is a pedestrian or a preset body part of a pedestrian. Based on the pre-trained tracking model, the same target in the multiple video frame images is tracked, and the same identifier is used to label the same target in the multiple video frame images, thereby obtaining the labeled multiple video frame images output by the tracking model; If a target with a certain identifier appears a preset number of times in the multiple video frame images, then it is determined that the target with that identifier is abnormal.

[0006] The above technical solution has the following advantages or beneficial effects: In the embodiments of this application, the electronic device determines each target in the video frame image according to the target detection model, and determines the identifier of each target in multiple video frame images according to the tracking model. When a certain target appears more frequently in multiple video frame images, it is determined that the target has a problem of wandering or lingering, thereby improving the accuracy of the wandering or lingering detection results.

[0007] In one possible implementation, after determining each target in the video frame image according to a pre-trained target detection model, and before tracking the same target in the multiple video frame images according to a pre-trained tracking model and labeling the same target in the multiple video frame images with the same identifier, the method further includes: If the number of targets in a preset number of video frame images exceeds the preset number, then for multiple video frame images, a clustering algorithm is used to cluster the position of each target in the video frame image to obtain each target group; For each target group, the tracking model is pre-trained to track the same target in the multiple video frames, and the same identifier is used to label the same target in the multiple video frames.

[0008] The above technical solution has the following advantages or beneficial effects: In the embodiments of this application, when there are a large number of targets, clustering algorithms are used to cluster the targets to obtain multiple target groups, thereby improving the tracking effect.

[0009] In one possible implementation, the clustering algorithm is used to cluster the location of each target in the video frame image to obtain each target group, including: Using a preset number of cluster groups and clustering algorithm, the location of each target in the video frame image is clustered to obtain each target group; or, Based on the average number of targets in the multiple video frame images, the number of cluster groups is determined. Using the number of cluster groups and the clustering algorithm, the location of each target in the video frame image is clustered to obtain each target group.

[0010] In one possible implementation, the target detection model is a real-time target detection (RT-DETR) model.

[0011] In one possible implementation, determining each target in the video frame image based on a pre-trained target detection model includes: Based on the FasterNet network of the pre-trained object detection model, determine the global features in the video frame image; The global features are processed by the network layer of the target detection model to obtain each target in the video frame image.

[0012] The above technical solution has the following advantages or beneficial effects: In this embodiment of the application, global features in video frame images are determined by FasterNet, thereby improving the feature extraction capability and thus accurately and effectively performing target recognition.

[0013] In one possible implementation, the tracking model is the DeepSort model, which is a deep, simple, online, real-time tracking model.

[0014] In one possible implementation, the step of tracking the same target in the multiple video frames based on a pre-trained tracking model, and labeling the same target in the multiple video frames using the same identifier, includes: Based on the Yolov11-cls of the pre-trained tracking model, the features of each target in the multiple video frame images are determined; The features of each target are processed according to the network layer of the tracking model, the same target in the multiple video frame images is tracked, and the same label is used to annotate the same target in the multiple video frame images.

[0015] The above technical solution has the following advantages or beneficial effects: In the embodiments of this application, the features of each target in the video frame image are determined by YOLOv11-cls, thereby improving the feature extraction capability and thus accurately and effectively tracking the target.

[0016] Secondly, embodiments of this application provide a pedestrian loitering and lingering detection device, the device comprising: The processing module is used to acquire video streams captured by image acquisition devices in public places, including airports, high-speed rail stations, stadiums, and other places where pedestrians gather. It extracts multiple video frame images from the video stream at preset time intervals. For each of the multiple video frame images, it determines each target in the video frame image based on a pre-trained target detection model; the target is a pedestrian or a preset body part of a pedestrian. Based on a pre-trained tracking model, it tracks the same target in the multiple video frame images and labels the same target in the multiple video frame images using the same identifier, acquiring the labeled multiple video frame images output by the tracking model. The determination module is used to determine that if a target with a certain identifier appears a preset number of times in the multiple video frame images, the target with that identifier is abnormal.

[0017] In one possible implementation, the processing module is further configured to, if the number of targets in a preset number of video frame images all exceed a preset number, then, for the multiple video frame images, use a clustering algorithm to cluster the positions of each target in the video frame images to obtain each target group; for each target group, execute subsequent tracking based on a pre-trained tracking model to track the same target in the multiple video frame images, and use the same identifier to label the same target in the multiple video frame images.

[0018] In one possible implementation, the processing module is specifically used to cluster the position of each target in the video frame image using a preset number of clustering groups and a clustering algorithm to obtain each target group; or, based on the average number of targets in the multiple video frame images, determine the number of clustering groups, and use the number of clustering groups and the clustering algorithm to cluster the position of each target in the video frame image to obtain each target group.

[0019] In one possible implementation, the processing module is specifically configured to determine global features in the video frame image based on a pre-trained FasterNet target detection model; and process the global features according to the network layers of the target detection model to obtain each target in the video frame image.

[0020] In one possible implementation, the processing module is specifically configured to determine the features of each target in the plurality of video frame images based on the Yolov11-cls of the pre-trained tracking model; process the features of each target according to the network layer of the tracking model; track the same target in the plurality of video frame images; and label the same target in the plurality of video frame images using the same identifier.

[0021] Thirdly, this application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, used to execute a program stored in memory, implements the method described.

[0022] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in any of the preceding claims. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A schematic diagram illustrating the process of the pedestrian loitering detection method provided in this application embodiment; Figure 2 A schematic diagram illustrating each target determined before clustering, as provided in an embodiment of this application; Figure 3 A schematic diagram of each target group determined after clustering, provided in an embodiment of this application; Figure 4 A schematic diagram of the structure of an RT-DETR model provided in an embodiment of this application; Figure 5 A detailed process diagram of a pedestrian loitering detection method provided in this application embodiment; Figure 6 A schematic diagram of a pedestrian loitering and stopping device provided in an embodiment of this application; Figure 7 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0025] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0026] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0027] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0028] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0029] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0030] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0031] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

[0032] To improve the accuracy of pedestrian loitering or lingering detection, embodiments of this application provide a method, apparatus, device, and medium for pedestrian loitering or lingering detection.

[0033] The pedestrian loitering detection method includes: electronic devices acquiring video streams captured by image acquisition devices in public places, including airports, high-speed rail stations, stadiums, and other places where pedestrians gather; extracting multiple video frame images from the video stream at preset time intervals; for each video frame image, determining each target in the video frame image based on a pre-trained target detection model; the target is a pedestrian or a preset body part of a pedestrian; tracking the same target in the multiple video frame images based on a pre-trained tracking model, and labeling the same target in the multiple video frame images with the same identifier, acquiring multiple labeled video frame images output by the tracking model; if a target with a certain identifier appears a preset number of times in the multiple video frame images, it is determined that the target with that identifier is abnormal.

[0034] Figure 1 This is a schematic diagram of the process of the pedestrian loitering detection method provided in the embodiments of this application. The process includes the following steps: S101: Acquire video streams captured by image acquisition devices in public places, wherein the public places include airports, high-speed rail stations, stadiums and other places where pedestrians gather, and extract multiple video frame images from the video stream at preset time intervals; for the multiple video frame images, determine each target in the video frame image according to a pre-trained target detection model; the target is a pedestrian or a preset body part of a pedestrian.

[0035] The pedestrian loitering detection method provided in this application is applied to an electronic device, which can be a smart device such as a PC or server.

[0036] In real-world scenarios, pedestrians often linger or loiter in the detection area for extended periods, such as outside airports or concert venues. Therefore, electronic devices can detect pedestrians lingering or loitering based on long-term video streams.

[0037] Specifically, the electronic device can acquire video streams. These streams can be real-time video streams captured by image acquisition devices located in public places, or video streams captured by these devices over a historical period. These public places include airports, high-speed rail stations, stadiums, and other areas where pedestrians gather. In one example, the electronic device can access the image acquisition device via Real-Time Streaming Protocol (RTSP) or a video gateway to acquire the real-time video stream. The image acquisition device has good low-light performance and a wide dynamic range to ensure clear images are captured under various lighting conditions. The video stream is transmitted to the electronic device over a network. The electronic device decodes the acquired video stream using a dedicated hardware decoder, such as a Graphics Processing Unit (GPU) hardware decoding chip. This hardware can efficiently process video data, decoding the compressed video stream into frame images, providing raw data for subsequent processing.

[0038] Electronic devices can extract multiple video frames from a video stream at preset time intervals. In one example, this preset time interval could be 1 second, 10 seconds, etc. The specific time interval is not limited here. After acquiring the multiple video frames, preprocessing can be performed. Preprocessing involves using image processing algorithms to improve image quality. This includes using 3D filters to remove noise, histogram equalization to enhance contrast, and adaptive brightness adjustment. These techniques ensure clear images of non-motorized vehicles under various environmental conditions.

[0039] To ensure accurate and effective detection, a pre-trained object detection model is stored. In one example, this model can be stored locally on the electronic device. The electronic device can input multiple video frame images into the object detection model, which then identifies each object within the video frame. In one example, the model can annotate the bounding boxes that completely contain the object within the video frame and output the annotated video frame image. The object can be a pedestrian or a predefined body part of a pedestrian, such as the head or torso.

[0040] S102: Based on the pre-trained tracking model, track the same target in the multiple video frame images, and label the same target in the multiple video frame images with the same identifier, and obtain the multiple labeled video frame images output by the tracking model.

[0041] To determine if there is a problem of pedestrians loitering or lingering, electronic devices can identify the same target in different video frames. Specifically, to identify the same target in different video frames, a pre-trained tracking model is stored. In one example, this tracking model can be stored locally on the electronic device. The electronic device can track the same target in multiple video frames based on the pre-trained tracking model. The tracking model uses the same label to annotate the same target in multiple video frames, where the label is the same for the same target in multiple video frames, and different labels for different targets. The tracking model outputs the annotated multiple video frames.

[0042] S103: If a target with a certain identifier appears a preset number of times in the multiple video frame images, then it is determined that the target with that identifier is abnormal.

[0043] After acquiring multiple labeled video frame images output by the tracking model, the electronic device determines whether a certain target appears more than a preset number of times in the multiple video frame images. If a certain target appears more than a preset number of times in the multiple video frame images, it can be determined that the target has been staying in the detection area for a long time, and it can be determined that the target is abnormal.

[0044] In one example, the electronic device can record identified loitering targets in a database. The database employs a highly reliable storage solution, such as a cloud database, to ensure data security and accessibility.

[0045] Electronic devices can also be configured according to upper-level business settings. For example, if the same target appears in sensitive areas or arrival areas of the airport for multiple days, they can automatically enter suspected scalper areas and alert management personnel through user interface prompts or automatic email and SMS notifications to handle the detected anomalies. This step ensures that pedestrian loitering and lingering behaviors can be responded to and managed in a timely manner.

[0046] This application provides a deep learning-based pedestrian loitering detection algorithm. This method automatically detects pedestrian loitering behavior through a video surveillance system, records the detected behavior, and alerts management personnel for intervention. The solution includes steps such as video access, hardware decoding, preprocessing, pedestrian loitering detection, and data storage.

[0047] With the development of technology and the progress of society, pedestrian loitering detection algorithms have been widely used in many fields, especially in public safety, traffic management, commercial applications, and health and welfare. These algorithms typically combine computer vision and machine learning techniques, using cameras to collect video data and analyze pedestrian movement trajectories and behavioral patterns to identify loitering or lingering behavior. Existing pedestrian loitering detection algorithms mainly include three steps: object detection, trajectory tracking, and behavior recognition. Object detection typically uses deep learning models to detect pedestrians in videos; trajectory tracking uses techniques such as Kalman filters or optical flow to track pedestrian movement trajectories; and behavior recognition uses machine learning algorithms to identify pedestrian behavior patterns and determine whether loitering or lingering behavior exists. Although existing pedestrian loitering detection algorithms have achieved significant results in many scenarios, some problems and shortcomings still exist. First, existing algorithms have poor robustness in complex environments. For example, in scenes with significant changes in lighting, complex backgrounds, or dense crowds, the accuracy of target detection will drop significantly, resulting in high false positive and false negative rates. Due to limitations in feature extraction capabilities, existing tracking algorithms may experience tracking identifier (ID) swapping in areas with high pedestrian traffic, leading to a certain degree of false alarms.

[0048] In airport ground traffic centers (GTCs), pedestrian loitering and lingering detection systems are crucial for improving overall security and operational efficiency. By monitoring pedestrian flow in real time and identifying abnormal behavior or unusual loitering patterns, the system can promptly detect potential security risks, such as scalping, lost luggage, or possible terrorist activities. This not only helps maintain daily security management but also assists in the rapid evacuation of crowds during emergencies, reducing casualties. For example, based on spatiotemporal correlation, setting up upper-level detection of individuals appearing across multiple days can create a list of suspected scalpers for airport scalping management. Simultaneously, it supports understanding and predicting peak-hour passenger density by analyzing pedestrian flow data, optimizing service resource allocation, such as increasing security checkpoints or adjusting shuttle bus frequencies, thereby improving passenger processing speed and service quality, ensuring the smooth operation of the entire GTC. Furthermore, the application of intelligent monitoring technology can, to some extent, replace manual patrols, reducing operating costs. In addition, the smooth and orderly passage environment provided by this method reduces passenger waiting time, improves satisfaction, and allows for rapid response in special circumstances, providing more timely assistance and service information, enhancing passengers' sense of security. Finally, such a detection system meets or even exceeds industry standards and government regulations for the safety management of public places, demonstrating corporate social responsibility. Furthermore, the recorded video data can be used as evidence when necessary to assist law enforcement agencies in their investigations. Overall, pedestrian loitering detection systems play an indispensable role in the modern air transport system.

[0049] In this embodiment, the electronic device determines each target in the video frame image according to the target detection model and determines the identifier of each target in multiple video frame images according to the tracking model. When a target appears more frequently in multiple video frame images, it is determined that the target has a wandering or lingering problem, thereby improving the accuracy of the wandering or lingering detection results.

[0050] To improve detection accuracy, based on the above embodiments, in this embodiment, after determining each target in the video frame image according to the pre-trained target detection model, and before tracking the same target in the multiple video frame images according to the pre-trained tracking model and labeling the same target in the multiple video frame images with the same identifier, the method further includes: If the number of targets in a preset number of video frame images exceeds the preset number, then for multiple video frame images, a clustering algorithm is used to cluster the position of each target in the video frame image to obtain each target group; For each target group, the tracking model is pre-trained to track the same target in the multiple video frames, and the same identifier is used to label the same target in the multiple video frames.

[0051] In traffic scenarios, especially at traffic light intersections, there is a large flow of people, making tracking more difficult and increasing the likelihood of tracking IDs being exchanged. To address this issue, this application proposes a pedestrian cluster algorithm. This algorithm assumes that when the target is crowded, the relative displacement between individual targets remains almost constant, meaning that the overall features in the image do not change significantly.

[0052] In one example, if the number of targets in a preset number of video frames all exceed a preset number, it can be determined that there is a problem of high pedestrian traffic. The electronic device can then use a clustering algorithm to cluster the location of each target in the multiple video frames, obtaining each target group. In one possible implementation, this clustering algorithm can be K-means clustering. Targets and target groups can be identified using bounding boxes. In this embodiment, K-means clustering merges multiple bounding boxes into several, reducing the density of the bounding boxes, and then tracking is performed, improving the accuracy of the pedestrian loitering and lingering algorithm.

[0053] After obtaining each target group, the pre-trained tracking model tracks the same target group in multiple video frames and uses the same label to annotate the same target group in multiple video frames.

[0054] Figure 2 This is a schematic diagram of each target determined before clustering, as provided in an embodiment of this application.

[0055] Figure 3 This is a schematic diagram of each target group determined after clustering, as provided in an embodiment of this application.

[0056] Depend on Figure 2 and Figure 3 It can be seen that before clustering, a target is labeled with a rectangle, and after clustering, a target group is labeled with a rectangle.

[0057] To improve detection accuracy, based on the above embodiments, in this embodiment, a clustering algorithm is used to cluster the location of each target in the video frame image to obtain each target group, including: Using a preset number of cluster groups and clustering algorithm, the location of each target in the video frame image is clustered to obtain each target group; or, Based on the average number of targets in the multiple video frame images, the number of cluster groups is determined. Using the number of cluster groups and the clustering algorithm, the location of each target in the video frame image is clustered to obtain each target group.

[0058] In one possible implementation, the electronic device can use a preset number of clustering groups and a clustering algorithm to cluster the position of each target in the video frame image, and cluster targets with similar positions into targets within the same target group, thereby obtaining multiple target groups.

[0059] In another possible implementation, the electronic device can also determine the number of cluster groups based on the average number of targets in multiple video frame images. In one example, the ratio of this average to a preset value can be determined, and the ratio can be rounded down or up to obtain the number of cluster groups. After obtaining the number of cluster groups, the electronic device can use the number of cluster groups and a clustering algorithm to cluster each target in the video frame image, grouping targets that are close in location into the same target group, thus obtaining multiple target groups.

[0060] To improve the accuracy of detection, based on the above embodiments, in this embodiment of the application, the target detection model is the RT-DETR model.

[0061] To improve detection accuracy, based on the above embodiments, in this embodiment, determining each target in the video frame image according to the pre-trained target detection model includes: Based on the pre-trained object detection model FasterNet, determine the global features in the video frame image; The global features are processed by the network layer of the target detection model to obtain each target in the video frame image.

[0062] In this embodiment, the improved FasterNet can be used to replace the original Residual Network (ResNet) as the backbone network, thereby enhancing the global feature extraction capability and reducing computational complexity.

[0063] FasterNet determines global features in a video frame image, and processes these global features according to the network layers to obtain each target in the video frame image.

[0064] In this embodiment, the network structure of the RT-DETR model is optimized. Re-parameterized convolution (RepConv) is used to improve the FasterNet block, and an Efficient Multi-scale Attention Module (EMA) is introduced to enhance spatial feature extraction capabilities and reduce computational redundancy. In the encoder part, a Dynamic Adaptive Feature Fusion Structure (DyASF) is used to replace the Cross-Scale Feature Fusion Module (CCFM). The Dynamic Scale Sequence Feature Fusion (DySSFF) module and the Triple Path Encoder (TPE) module are utilized to avoid the loss of small target feature information caused by upsampling and downsampling, and to enrich the detailed information of small target detection, thereby enhancing the network's scale feature fusion capability. For the loss function, Focaler Intersection over Union (Focaler-IoU) and Shape Intersection over Union (SIoU) are combined. To leverage the advantages of Shape-IoU (overUnion), Focaler-Shape Intersection overUnion (Focaler-Shape-IoU) was proposed to replace the original model Generalized Intersection overUnion (GIOU). This model injects shape and scale information of the bounding box, focuses on difficult samples, and enhances the bounding box regression performance.

[0065] Figure 4 This is a schematic diagram of the structure of an RT-DETR model provided in an embodiment of this application.

[0066] Depend on Figure 4As can be seen, the RT-DETR model includes a re-parameterized Faster Efficient Multi-scale Attention Backbone (Rep-Faster EMABackbone) and a DyASF Encoder. The Rep-Faster EMA Backbone employs a multi-stage feature extraction design, with stages P1-P5 and F5 constituting gradient enhancement channels. In stage P1, three sets of 3×3 convolution-batch normalization-activation (CBA) modules are cascaded together using max pooling downsampling to achieve primary feature compression. In stages P2-P5, a RepFasterEMA block is deployed, enhancing feature extraction efficiency and reducing computational redundancy through re-parameterized convolution and EMA. In stage F5, after adjustment using 1×1 convolution channels, an attention-based intra-scale feature interaction module (AIFI) is connected to optimize high-order semantic feature representation. The DyASF Encoder comprises DySSFF and DyASF. The output of DyASF's P1 stage is dynamically upsampled (Dysample unqueeze) and then concatenated with the 1×1 Conv features from the P2 stage. Cross-scale feature fusion is achieved through 3D Convolution (3D Conv) + Batch Normalization (BatchNorm) + Sigmoid Linear Unit (SiLU) activation. The P3-P5 stages of DySSFF employ a TFE and a re-parameterized convolution 3×3 module (RepC3) structure, combining upsampling and convolution operations to complete multi-level feature fusion, avoiding the loss of information from small targets.

[0067] To improve the accuracy of detection, based on the above embodiments, in this embodiment, the tracking model is the DeepSort model, which is a simple online real-time tracking model.

[0068] To improve detection accuracy, based on the above embodiments, in this embodiment, the step of tracking the same target in the multiple video frame images according to the pre-trained tracking model and labeling the same target in the multiple video frame images with the same identifier includes: Based on the Yolov11-cls of the pre-trained tracking model, the features of each target in the multiple video frame images are determined; The features of each target are processed according to the network layer of the tracking model, the same target in the multiple video frame images is tracked, and the same label is used to annotate the same target in the multiple video frame images.

[0069] The feature extractor used by DeepSort currently has low feature extraction capability and cannot effectively distinguish similar features. Therefore, we use YOLOv11-cls as the feature extractor, remove the classification head of the model, and use the image features extracted by it for tracking optimization. This can better distinguish similar features and reduce ID swapping.

[0070] In this embodiment, the original Deepsort feature extractor can be replaced with Yolov11-cls to enhance feature extraction capabilities. Yolov11-cls determines the features of each target in multiple video frames, and the network layer processes these features to track the same target across multiple video frames, labeling the same target with the same identifier.

[0071] Figure 5 This is a detailed process diagram of a pedestrian loitering and lingering detection method provided in an embodiment of this application.

[0072] Depend on Figure 5 It can be seen that the video stream can be acquired first, and the population density can be determined based on the target detection algorithm. If the population density is high, K-means clustering is used to obtain pedestrian clusters (i.e., the target groups described in the embodiments of this application), and DeepSort model is used for tracking. If the population density is low, DeepSort model is used for tracking. If the dwell time of a target or a target group reaches a threshold, an event is reported. If the dwell time does not reach the threshold, target detection continues.

[0073] In this embodiment, the RT-DETR deep learning model is used for target detection during the pedestrian loitering detection stage. The model is pre-trained and can identify pedestrians and heads. Then, the DeepSort algorithm is used for tracking. Detection is completed when a certain ID exists within the monitored area for a certain amount of time. Our solution, by optimizing the RT-DETR model structure, the DeepSort algorithm, and dense target optimization, achieves high-accuracy identification of pedestrian loitering behavior. It also improves the model's generalization ability and real-time performance, reduces computational costs, and significantly improves the enforcement efficiency and road safety level of traffic management departments.

[0074] Figure 6 A schematic diagram of a pedestrian loitering and stopping device provided in an embodiment of this application is shown below. Figure 6 As shown: The device includes a processing module 601 and a determination module 602.

[0075] The processing module 601 is used to acquire video streams captured by image acquisition devices in public places, including airports, high-speed rail stations, stadiums, and other places where pedestrians gather. It extracts multiple video frame images from the video stream at preset time intervals. For each of the multiple video frame images, it determines each target in the video frame image based on a pre-trained target detection model. The target is a pedestrian or a preset body part of a pedestrian. Based on a pre-trained tracking model, it tracks the same target in the multiple video frame images and labels the same target in the multiple video frame images using the same identifier, acquiring the labeled multiple video frame images output by the tracking model. The determination module 602 is used to determine that if a target with a certain identifier appears a preset number of times in the multiple video frame images, the target with that identifier is abnormal.

[0076] In one possible implementation, the processing module 601 is further configured to, if the number of targets in a preset number of video frame images all exceed a preset number, then, for the multiple video frame images, use a clustering algorithm to cluster the positions of each target in the video frame images to obtain each target group; for each target group, execute subsequent tracking based on a pre-trained tracking model to track the same target in the multiple video frame images, and use the same identifier to label the same target in the multiple video frame images.

[0077] In one possible implementation, the processing module 601 is specifically used to cluster the position of each target in the video frame image using a preset number of clustering groups and a clustering algorithm to obtain each target group; or, based on the average number of targets in the multiple video frame images, determine the number of clustering groups, and use the number of clustering groups and the clustering algorithm to cluster the position of each target in the video frame image to obtain each target group.

[0078] In one possible implementation, the processing module 601 is specifically used to determine global features in the video frame image based on the pre-trained target detection model FasterNet; and to process the global features according to the network layers of the target detection model to obtain each target in the video frame image.

[0079] In one possible implementation, the processing module 601 is specifically configured to determine the features of each target in the plurality of video frame images based on the Yolov11-cls of the pre-trained tracking model; process the features of each target according to the network layer of the tracking model; track the same target in the plurality of video frame images; and label the same target in the plurality of video frame images using the same identifier.

[0080] Figure 7 This application provides a schematic diagram of an electronic device structure based on an embodiment of the present application. In addition to the above embodiments, this application also provides an electronic device, such as... Figure 7 As shown, it includes: processor 701, communication interface 702, memory 703 and communication bus 704, wherein processor 701, communication interface 702 and memory 703 communicate with each other through communication bus 704. The memory 703 stores a computer program, which, when executed by the processor 701, causes the processor 701 to perform any of the above method steps.

[0081] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0082] The communication interface 702 is used for communication between the above-mentioned electronic device and other devices.

[0083] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0084] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0085] This application also provides a computer-readable storage medium storing a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform any of the above method steps.

[0086] This application provides a computer program product, which includes an executable program that, when executed by a processor, implements the method described herein.

[0087] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0088] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for detecting pedestrian loitering and lingering, characterized in that, The method includes: The method involves acquiring video streams captured by image acquisition devices in public places, including airports, high-speed rail stations, stadiums, and other places where pedestrians gather; extracting multiple video frame images from the video stream at preset time intervals; and determining each target in the video frame image based on a pre-trained target detection model, where the target is a pedestrian or a preset body part of a pedestrian. Based on the pre-trained tracking model, the same target in the multiple video frame images is tracked, and the same identifier is used to label the same target in the multiple video frame images, thereby obtaining the labeled multiple video frame images output by the tracking model; If a target with a certain identifier appears a preset number of times in the multiple video frame images, then it is determined that the target with that identifier is abnormal.

2. The method according to claim 1, characterized in that, After determining each target in the video frame image based on the pre-trained target detection model, and before tracking the same target in the multiple video frame images using the pre-trained tracking model and labeling the same target in the multiple video frame images with the same identifier, the method further includes: If the number of targets in a preset number of video frame images exceeds the preset number, then for multiple video frame images, a clustering algorithm is used to cluster the position of each target in the video frame image to obtain each target group; For each target group, the tracking model is pre-trained to track the same target in the multiple video frames, and the same identifier is used to label the same target in the multiple video frames.

3. The method according to claim 2, characterized in that, The method employs a clustering algorithm to cluster the location of each target in the video frame image, resulting in each target group, including: Using a preset number of cluster groups and clustering algorithm, the location of each target in the video frame image is clustered to obtain each target group; or, Based on the average number of targets in the multiple video frame images, the number of cluster groups is determined. Using the number of cluster groups and the clustering algorithm, the location of each target in the video frame image is clustered to obtain each target group.

4. The method according to claim 1, characterized in that, The target detection model is the RT-DETR model.

5. The method according to claim 1 or 4, characterized in that, The step of determining each target in the video frame image based on the pre-trained target detection model includes: Based on FasterNet, a pre-trained object detection model, determine the global features in the video frame image; The global features are processed by the network layer of the target detection model to obtain each target in the video frame image.

6. The method according to claim 1, characterized in that, The tracking model is the DeepSort model.

7. The method according to claim 1 or 6, characterized in that, The step of tracking the same target in multiple video frames using a pre-trained tracking model and labeling the same target in multiple video frames with the same identifier includes: Based on the Yolov11-cls of the pre-trained tracking model, the features of each target in the multiple video frame images are determined; The features of each target are processed according to the network layer of the tracking model, the same target in the multiple video frame images is tracked, and the same label is used to annotate the same target in the multiple video frame images.

8. A pedestrian loitering and lingering detection device, characterized in that, The device includes: The processing module is used to acquire video streams captured by image acquisition devices in public places, including airports, high-speed rail stations, stadiums, and other places where pedestrians gather. It extracts multiple video frame images from the video stream at preset time intervals. For each of the multiple video frame images, it determines each target in the video frame image based on a pre-trained target detection model; the target is a pedestrian or a preset body part of a pedestrian. Based on a pre-trained tracking model, it tracks the same target in the multiple video frame images and labels the same target in the multiple video frame images using the same identifier, acquiring the labeled multiple video frame images output by the tracking model. The determination module is used to determine that if a target with a certain identifier appears a preset number of times in the multiple video frame images, the target with that identifier is abnormal.

9. An electronic device, characterized in that, The electronic device includes at least a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the steps of the pedestrian loitering and lingering detection method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the steps of the pedestrian loitering and lingering detection method as described in any one of claims 1-7.

Images