Video-based channel-passing crowd line-crossing counting system
By generating stable motion trajectories through multimodal video acquisition and Kalman filtering algorithms, the problem of distinguishing pedestrian features in crowded scenes was solved, and accurate crowd counting and resource scheduling support were achieved in complex lighting environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-05
- Publication Date
- 2026-03-10
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing video crowd counting systems struggle to distinguish overlapping pedestrian features in congested scenarios, resulting in insufficient counting accuracy, missed detections, false detections, and trajectory breaks. Consequently, they are unable to provide accurate congestion warnings and resource scheduling data support.
A multimodal video acquisition and processing module is adopted, which combines RGB and infrared cameras. Multi-scale pedestrian features are extracted through adaptive fusion and perspective correction technology, and continuous trajectories are generated using the Kalman filter algorithm. Accurate counting is performed by combining the counting and resource scheduling module.
Improve image quality in complex lighting environments, accurately distinguish overlapping pedestrian features, generate stable motion trajectories, enhance counting accuracy, and provide accurate cumulative and instantaneous traffic data support.
Smart Images
Figure CN121640377A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of image processing technology, and more specifically, relates to a video-based system for counting crowd crossings through a channel. Background Technology
[0002] In densely populated places such as transportation hubs, large venues, and commercial complexes, passageways are key nodes for the flow of people. The accurate monitoring and counting of people passing through these passageways is of great significance for passenger flow scheduling, safety management, and resource allocation. With the continuous development of video surveillance technology, video-based crowd counting systems have gradually replaced traditional manual counting methods and become the mainstream technical means for passageway crowd monitoring due to their advantages such as non-contact and real-time performance.
[0003] Current pedestrian counting systems typically acquire images of the scene through video capture devices and combine them with algorithms such as object detection and trajectory tracking to achieve pedestrian recognition and counting. The quality of video capture, the accuracy of pedestrian feature extraction, and the accuracy of line-crossing determination are the core factors determining the counting effect. To adapt to the needs of different scenarios, the industry is gradually exploring the application of technologies such as multimodal acquisition and feature fusion in this field to improve the system's adaptability in complex environments. Solving the counting accuracy problem in scenarios such as crowded areas has become a key focus of current technological research and development.
[0004] When encountering overlapping pedestrians in congested scenarios, existing systems cannot fully distinguish the individual characteristics of overlapping pedestrians, resulting in insufficient accuracy in pedestrian feature extraction. This can easily lead to missed detections, false detections, or target sticking. Furthermore, when pedestrians move quickly, turn, or are briefly obscured, issues such as trajectory breaks and misconnections can easily occur, resulting in a large discrepancy between the cumulative count and the actual number of pedestrians. Moreover, these systems cannot provide accurate data support for congestion warnings and resource scheduling. Summary of the Invention
[0005] To address the aforementioned problems and technical deficiencies, this application adopts the following technical solution: a video-based crowd crossing counting system, comprising: The multimodal video acquisition and processing module is deployed and installed at the entrance and exit of the channel to capture multimodal video and preprocess the captured video data. The crowd detection module is used to detect and identify pedestrians in the captured video data and extract multi-scale pedestrian features from the pedestrian targets. The multi-dimensional line crossing determination module is used to track pedestrian features at multiple scales and generate continuous trajectories of pedestrians. The counting and resource scheduling module is used to count and statistically analyze the continuous trajectory of pedestrians, and to count the number of people entering and exiting the passage.
[0006] Preferably, the multimodal video acquisition and processing module includes: a dual-modal acquisition camera, an adaptive fusion submodule, and a perspective correction submodule.
[0007] Furthermore, the dual-modal acquisition camera uses an RGB camera and an infrared camera to simultaneously capture video stream data from the acquisition channel; The adaptive fusion submodule dynamically adjusts the feature fusion weights of RGB video and infrared video based on the illumination intensity threshold, and outputs the fused video frame image. The perspective correction submodule identifies ground markings within the channel using the U-Net semantic segmentation network, generates channel boundary coordinates, and calculates the perspective transformation matrix to complete the correction.
[0008] Furthermore, the extraction of the multi-scale pedestrian features includes: Spatial dimension decomposition is performed on the overlapping regions of pedestrians in video frame images to generate redefined spatial features; Then, local HOG features are extracted from the video frame images and fused with the redefined spatial features to output pedestrian target detection results including bounding boxes and key node coordinates, which are used as multi-scale pedestrian features.
[0009] Furthermore, the Kalman filter algorithm is used to connect pedestrian nodes in consecutive video frames by using multi-scale pedestrian features as pedestrian nodes to generate motion trajectory vectors. Then, based on the channel boundary coordinates, a virtual tripwire is generated, the intersection event between the motion trajectory vector and the virtual tripwire is calculated, the validity of the motion trajectory vector is determined, and the valid motion trajectory vector is output as a continuous trajectory.
[0010] Furthermore, the generation of the motion trajectory vector The bounding box center coordinates (x, y), aspect ratio (w / h), and offsets of a continuous number of key nodes are used as state feature vectors, and the feature covariance matrix of Kalman filtering is generated based on the state feature vectors. The velocity changes of multi-scale pedestrian features in adjacent video frames are used as motion consistency constraints, and the feature similarity of multi-scale pedestrian features is used as similarity constraints, thus applying dual constraints to the feature covariance matrix. The offset of the constrained feature covariance matrix is calculated, and the calculation result is compared with the threshold to filter effective key nodes. The motion trajectory vectors of the effective key nodes are then generated, as shown in the following formula: V=(v x , v y , t) Among them, v x and v y For the effective key nodes, the velocity components in the x and y directions are denoted as t, which is the trajectory duration.
[0011] Furthermore, the counting of the crowd is first based on the direction of the movement trajectory vector to distinguish between the entry and exit directions, then the effective continuous trajectories are counted to generate cumulative counts and instantaneous counts, thereby obtaining the total number of people entering and exiting the channels and the number of people within a preset time period.
[0012] Furthermore, the calculation of the cumulative count involves establishing an inbound counter C1 and an outbound counter C2. For each valid inbound trajectory detected, counter C1 increments by 1, and for each valid outbound trajectory detected, counter C2 increments by 1. The instantaneous counting technique first presets a sliding time window, then uses an inbound counter C1 and an outbound counter C2 to count the number of people entering (C1') and exiting (C2') within the sliding time window, and calculates the instantaneous flow density ρ. ρ=(C1'+C2') / T Where T is the window time length, when the instantaneous flow density ρ is greater than the threshold, it is judged as a congested state, and the number of people entering C1' and exiting C2' is corrected.
[0013] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the content of a video-based channel-passing crowd crossing counting system as described above.
[0014] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the content of a video-based channel-through-crowd-crossing counting system as described above.
[0015] Compared to existing technologies, the beneficial effects of this application are as follows: (1) This application uses dual-modal cameras to synchronously acquire video streams and dynamically adjusts the fusion weight of the two modal features based on the light intensity threshold. In complex lighting environments such as strong light or night, the infrared modal weight is increased to retain the pedestrian contour features, while in normal lighting, the detailed information of the RGB modal is emphasized to improve the image quality under different lighting scenarios. At the same time, the U-Net semantic segmentation network is used to accurately identify the channel ground markings, realize perspective distortion correction, eliminate pedestrian position positioning deviation, and provide a position benchmark for subsequent line crossing judgment. (2) This application redefines the space of the overlapping area of pedestrians by using a spatial dimension splitting and multi-feature fusion strategy, generates distinguishable spatial features, and then fuses local HOG features to form multi-scale pedestrian features, retains the individual feature information of overlapping pedestrians, avoids the problems of pedestrian missed detection, false detection and adhesion in crowded scenes, and improves the accuracy and completeness of pedestrian target detection. (3) This application uses the Kalman filter algorithm, takes multi-scale pedestrian features as pedestrian nodes, and combines motion consistency constraints and feature similarity constraints to impose dual constraints on the feature covariance matrix, so as to avoid trajectory breakage or misconnection caused by pedestrians moving quickly and occlusion, and improve the continuity and stability of the generated motion trajectory vector. Attached Figure Description
[0016] In the attached diagram: Figure 1 This is a schematic diagram of the system structure according to an embodiment of this application. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments. Generally, the components of the embodiments of this application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Example
[0018] like Figure 1 As shown, the video-based channel uses a crowd crossing counting system, which includes: The multimodal video acquisition and processing module is deployed and installed at the entrance and exit of the channel to capture multimodal video and preprocess the captured video data. The multimodal video acquisition and processing module includes: a dual-modal acquisition camera, an adaptive fusion submodule, and a perspective correction submodule.
[0019] The dual-modal acquisition camera uses an RGB camera and an infrared camera to simultaneously capture video stream data from the acquisition channel; The adaptive fusion submodule dynamically adjusts the feature fusion weights of RGB video and infrared video based on the illumination intensity threshold, and outputs the fused video frame image. The perspective correction submodule identifies ground markings within the channel using the U-Net semantic segmentation network, generates channel boundary coordinates, and calculates the perspective transformation matrix to complete the correction.
[0020] The crowd detection module is used to detect and identify pedestrians in the captured video data and extract multi-scale pedestrian features from the pedestrian targets. Extraction of multi-scale pedestrian features includes: Spatial dimension decomposition is performed on the overlapping regions of pedestrians in video frame images to generate redefined spatial features; Then, local HOG features are extracted from the video frame images and fused with the redefined spatial features to output pedestrian target detection results including bounding boxes and key node coordinates, which are used as multi-scale pedestrian features.
[0021] The multi-dimensional line crossing determination module is used to track pedestrian features at multiple scales and generate continuous trajectories of pedestrians. The Kalman filter algorithm is used to take multi-scale pedestrian features as pedestrian nodes, and the pedestrian nodes in consecutive video frames are connected to generate motion trajectory vectors. Then, based on the channel boundary coordinates, a virtual tripwire is generated, the intersection event between the motion trajectory vector and the virtual tripwire is calculated, the validity of the motion trajectory vector is determined, and the valid motion trajectory vector is output as a continuous trajectory.
[0022] Generation of motion trajectory vectors The bounding box center coordinates (x, y), aspect ratio (w / h), and offsets of a continuous number of key nodes are used as state feature vectors, and the feature covariance matrix of Kalman filtering is generated based on the state feature vectors. The velocity changes of multi-scale pedestrian features in adjacent video frames are used as motion consistency constraints, and the feature similarity of multi-scale pedestrian features is used as similarity constraints, thus applying dual constraints to the feature covariance matrix. The offset of the constrained feature covariance matrix is calculated, and the calculation result is compared with the threshold to filter effective key nodes. The motion trajectory vectors of the effective key nodes are then generated, as shown in the following formula: V=(v x , v y , t) Among them, v x and v y For the effective key nodes, the velocity components in the x and y directions are denoted as t, which is the trajectory duration.
[0023] The counting and resource scheduling module is used to count and count pedestrians based on their continuous trajectories, and to count the number of people entering and exiting the passage. The counting of crowds is first based on the direction of the movement trajectory vector to distinguish between the entry and exit directions. Then, the effective continuous trajectories are counted to generate cumulative counts and instantaneous counts, thereby obtaining the total number of people entering and exiting the channels and the number of people within a preset time period.
[0024] The cumulative count is calculated by establishing an inbound counter C1 and an outbound counter C2. For each valid inbound trajectory detected, counter C1 increments by 1, and for each valid outbound trajectory detected, counter C2 increments by 1. The instantaneous counting technique first presets a sliding time window, then uses an inbound counter C1 and an outbound counter C2 to count the number of people entering (C1') and exiting (C2') within the sliding time window, and calculates the instantaneous flow density ρ. ρ=(C1'+C2') / T Where T is the window time length, when the instantaneous flow density ρ is greater than the threshold, it is judged as a congested state, and the number of people entering C1' and exiting C2' is corrected. Example
[0025] From a hardware perspective, this application provides an embodiment of an electronic device comprising all or part of a video-based crowd crossing counting system. The electronic device includes a service processor and a distributed memory. The service processor is connected to the memory, and the distributed memory stores a service self-management program configured to store machine-readable instructions. The service processor executes the service self-management program, and the instructions, when executed by the processor, implement the video-based crowd crossing counting system as described above.
[0026] From a hardware perspective, in order to effectively improve the flexibility, versatility, and efficiency of data acquisition, this application provides an embodiment of an electronic device comprising all or part of a video-based crowd crossing counting system. The electronic device specifically includes the following components: The system includes a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between the core business system, user terminals, and related databases of the video-based channel crowd crossing counting system; the logic controller can be a desktop computer, tablet computer, or mobile terminal, etc., but this embodiment is not limited to these.
[0027] In this embodiment, the logic controller can be implemented with reference to the embodiment of the video-based channel crowd crossing counting system, the content of which is incorporated herein, and repeated details will not be repeated.
[0028] It is understood that the user terminal may include smartphones, tablet electronic devices, network set-top boxes, portable computers, desktop computers, personal digital assistants (PDAs), in-vehicle devices, smart wearable devices, etc., wherein the smart wearable devices may include smart glasses, smartwatches, smart bracelets, etc.
[0029] In practical applications, the video-based channel crowd crossing counting system can be partially executed on the electronic device side as described above, or all operations can be completed in the client device. The choice depends on the processing power of the client device and the limitations of the user's usage scenario, and this application does not impose any limitations on this. If all operations are completed in the client device, the client device may further include a processor.
[0030] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster composed of multiple servers, or a server structure of a distributed device. Example
[0031] Embodiments of this application also provide a computer-readable storage medium capable of implementing a video-based crowd crossing counting system with a server or client as the execution subject in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all the contents of the video-based crowd crossing counting system with a server or client as the execution subject in the above embodiments.
[0032] The embodiments of this application may be provided as methods, apparatus, or computer program products. Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0033] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0034] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.
[0035] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0036] The embodiments described above are merely preferred embodiments of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications, improvements, and substitutions without departing from the concept of this application, and these all fall within the protection scope of this application.
Claims
1. A video-based lane through crowd count system, characterized in that, The application relates to a multi-modal video acquisition and processing system and method. The system comprises a multi-modal video acquisition and processing module, a crowded people detection module, a multi-dimensional overline judgment module and a counting and resource scheduling module. The multi-modal video acquisition and processing module is arranged at the entrance and exit of a channel and is used for multi-modal video shooting and acquisition and pre-processing of the shooting video data. The crowded people detection module is used for detecting and identifying the shooting video data and extracting multi-scale pedestrian features from the pedestrian targets. The multi-dimensional overline judgment module is used for tracking the multi-scale pedestrian features and generating continuous trajectories of the pedestrians.
2. The video-based lane-through crowd count system of claim 1, wherein, The counting and resource scheduling module is used for counting and statistics according to the continuous trajectories of the pedestrians and counting the number of people entering and leaving the channel.
3. The video-based lane-through crowd count system of claim 2, wherein, The multi-modal video acquisition and processing module comprises a dual-modal acquisition camera, an adaptive fusion sub-module and a perspective correction sub-module. The dual-modal acquisition camera adopts an RGB camera and an infrared camera to synchronously shoot and acquire channel video stream data. The adaptive fusion sub-module dynamically adjusts the feature fusion weight of the RGB video and the infrared video based on a light intensity threshold value and outputs the fused video frame image.
4. The video-based lane-through crowd count system of claim 3, wherein, The perspective correction sub-module identifies the ground marking line in the channel through a U-Net semantic segmentation network, generates channel boundary coordinates, calculates a perspective transformation matrix and completes correction. The extraction of the multi-scale pedestrian features comprises the following steps. The overlapping area of the pedestrians in the video frame image is spatially dimensionally split to generate redefined spatial features.
5. The video-based lane-through crowd count system of claim 4, wherein, The local HOG features in the video frame image are extracted, fused with the redefined spatial features and output to generate pedestrian target detection results comprising a bounding box and key node coordinates, which are taken as the multi-scale pedestrian features. The multi-scale pedestrian features are taken as pedestrian nodes, the pedestrian nodes in the continuous video frame image are connected and a motion trajectory vector is generated.
6. The video-based lane-through crowd count system of claim 5, wherein, A virtual stumbling line is generated based on the channel boundary coordinates, the intersection event of the motion trajectory vector and the virtual stumbling line is calculated, whether the motion trajectory vector is valid is judged and the valid motion trajectory vector is output as the continuous trajectory. The generation of the motion trajectory vector The center coordinates (x, y) of the bounding box, the width-height ratio (w / h) and the offset of the continuous number of key nodes are taken as a state feature vector, a feature covariance matrix of Kalman filtering is generated based on the state feature vector. The speed change of the multi-scale pedestrian features in the adjacent video frame image is taken as a motion consistency constraint and the feature similarity of the multi-scale pedestrian features is taken as a similarity constraint, and the feature covariance matrix is doubly constrained. V = (v x , v y , t) where v x and v y are the velocity components in the x, y directions of the effective key node, and t is the trajectory duration.
7. The video-based lane-through crowd count system of claim 5, wherein, The offset of the constrained feature covariance matrix is calculated, the calculation result is compared with a threshold value, the valid key nodes are screened, the motion trajectory vector of the valid key nodes is generated and the formula is as follows.
8. The video-based lane-through crowd count system of claim 7, wherein, The statistics of the number of people is that the entering and leaving directions are distinguished based on the direction of the motion trajectory vector, the valid continuous trajectories are counted, cumulative counting and instantaneous counting are generated and thus the total number of people entering and leaving the channel and the number of people in a preset time period are obtained. The cumulative counting is calculated by establishing an in-counting counter C1 and an out-counting counter C2, the in-counting counter C1 is increased by 1 every time an effective in-direction trajectory is detected and the out-counting counter C2 is increased by 1 every time an effective out-direction trajectory is detected. The technology of instant counting is to preset a sliding time window, use an in-counting counter C1 and an out-counting counter C2 to count the in-count C1' and the out-count C2' in the sliding time window, and calculate the instant flow density ρ: ρ=(C1'+C2') / T Wherein, T is the window time length, when the instant flow density ρ is greater than a threshold value, it is judged as a congestion state, and the in-count C1' and the out-count C2' are corrected.
9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the content of the video-based channel passing crowd overline counting system of claim 1 when executing the program.
10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the content of the video-based channel passing crowd overline counting system of claim 1 when executed by the processor.