A method and system for monitoring public transportation passenger flow based on human body detection

Abstract

The present invention discloses a method and system for monitoring public transit passenger flow based on human body detection. The method comprises: synchronously collecting passengers' skeletal key points, thermal distribution, and contour features using a multi-camera onboard infrared sensor to generate a passenger spatiotemporal feature matrix; outputting a set of passenger motion trajectories based on the passenger spatiotemporal feature matrix; generating a dynamic congestion index based on the length of stay and spatial distribution density of passengers in the set of passenger motion trajectories; calculating the probability of abnormal behavior based on the dynamic congestion index and outputting an abnormal event warning signal; and generating public transit passenger flow monitoring results through spatiotemporal correlation analysis by integrating the dynamic congestion index, abnormal event warning signal, and real-time passenger flow data. The embodiments of the present invention enable high-precision, low-latency, and intelligent passenger flow status monitoring and abnormality warning.

Description

Technical Field

[0001] The present invention belongs to the technical field of passenger flow monitoring, and in particular to a method and system for monitoring public transportation passenger flow based on human body detection. Background Art

[0002] With the rapid development of urban public transportation systems, accurate monitoring of bus passenger flow has become a key technology for improving operational efficiency and service quality. Traditional passenger flow monitoring methods mainly rely on infrared beams, pressure pedals, or single visual sensors. These methods suffer from low detection accuracy, inability to distinguish overlapping passengers, and susceptibility to environmental interference, making it difficult to meet the needs of refined monitoring in complex onboard environments. Especially during peak hours, when passenger density is high and behavior is complex, existing technologies struggle to accurately identify individual motion trajectories, posture changes, and abnormal behavior, resulting in inaccurate congestion assessments and difficulty in timely warning of safety hazards. In addition, insufficient multimodal data fusion and the lack of three-dimensional spatial modeling limit the comprehensiveness and real-time nature of passenger flow analysis. Although some studies have attempted to combine computer vision with sensor data, technical bottlenecks still exist in separating overlapping targets, eliminating shadow interference, and identifying abnormal behavior. Summary of the Invention

[0003] The purpose of the present invention is to provide a bus passenger flow monitoring method and system based on human body detection to address the deficiencies in the existing technology and to achieve high-precision, low-latency, intelligent passenger flow status monitoring and abnormality warning.

[0004] An embodiment of the present application provides a method for monitoring public transportation passenger flow based on human body detection, the method comprising:

[0005] The on-board multi-camera and infrared sensor synchronously collect the passenger's skeletal key points, thermal distribution and contour features, and use the multimodal feature fusion network to extract spatiotemporal correlation features to generate the passenger spatiotemporal feature matrix;

[0006] Based on the passenger spatiotemporal feature matrix, a human detection model optimized for spatiotemporal attention is used to separate and locate overlapping passengers, and a dynamic trajectory tracking algorithm is used to eliminate environmental shadow interference and output a set of passenger motion trajectories;

[0007] Based on the length of stay and spatial distribution density of passengers in the passenger trajectory set, a physical space occupancy mapping model is constructed in combination with a three-dimensional posture estimation model and pressure sensor data to generate a dynamic congestion index;

[0008] Based on the dynamic crowding index, a multi-task learning model is used to analyze the frequency of passengers' body movements and the characteristics of the space occupied by their belongings, calculate the probability of abnormal behavior, and output an abnormal event warning signal;

[0009] The dynamic congestion index, the abnormal event warning signal and the real-time passenger flow data are integrated to generate a public transportation passenger flow monitoring result through spatiotemporal correlation analysis.

[0010] Optionally, the on-board multi-camera and infrared sensor are used to synchronously collect the passenger's skeletal key points, thermal distribution, and contour features, and a multimodal feature fusion network is used to extract spatiotemporal correlation features to generate a passenger spatiotemporal feature matrix, including:

[0011] Based on the RGB video stream captured by multiple cameras, a lightweight skeleton key point detection model is used to extract the coordinates of multiple key points of the passenger and generate a skeleton topology map.

[0012] The skeleton topology map is input into the thermal distribution alignment module, the thermal imaging data of the infrared sensor is processed synchronously, and the passenger thermal distribution mask is generated through the threshold segmentation algorithm;

[0013] Based on the skeleton topology map and heat distribution mask, a contour tracking algorithm is used to extract the passenger contour binary image and generate a set of multimodal feature maps;

[0014] The multimodal feature map is input into the spatiotemporal convolutional fusion network, and the skeletal motion, thermal radiation intensity and contour geometric features are dynamically weighted through the cross-modal attention mechanism to output the passenger spatiotemporal feature matrix.

[0015] Optionally, based on the passenger spatiotemporal feature matrix, a human detection model optimized with spatiotemporal attention is used to separate and locate overlapping passengers, and a dynamic trajectory tracking algorithm is used to eliminate environmental shadow interference, and a set of passenger motion trajectories is output, including:

[0016] The passenger spatiotemporal feature matrix is ​​input into the spatiotemporal attention module to generate an attention weight matrix representing the saliency of passenger movement;

[0017] The pixel features of the overlapping areas are reweighted based on the attention weight matrix, and the YOLOv8 model is used to output the separated passenger location bounding box coordinates;

[0018] The passenger location bounding box coordinates and heat distribution mask are input into the shadow discriminator, and the environmental shadow artifacts are filtered out through the thermal radiation intensity comparison algorithm to generate a purified passenger location sequence;

[0019] The cleaned passenger position sequence is used to initialize the trajectory tracker, and the three-dimensional space trajectory is calculated by combining the multi-camera disparity data to generate a preliminary motion trajectory set.

[0020] The trajectory jump points in the preliminary motion trajectory set are corrected by the acceleration constrained Kalman filter algorithm, and the passenger motion trajectory set with time stamps is output.

[0021] Optionally, the step of constructing a physical space occupancy mapping model based on the length of stay and spatial distribution density of passengers in the passenger trajectory set, in combination with a three-dimensional posture estimation model and pressure sensing data, and generating a dynamic congestion index includes:

[0022] Based on the passenger movement trajectory set, the length of stay and movement path of each passenger in the carriage are counted, and the spatial distribution heat map is generated using kernel density estimation;

[0023] The spatial distribution heat map is input into the 3D posture estimation model to analyze the passenger's sitting and standing postures and luggage carrying status, and output the predicted value of individual space occupancy rate;

[0024] By integrating real-time load data collected by pressure sensors, the predicted space occupancy rate is dynamically calibrated to generate an accurate space occupancy distribution matrix.

[0025] The congestion calculation function is used to integrate the entropy characteristics of the spatial distribution heat map and the weighted summation results of the spatial occupancy distribution matrix to output a standardized dynamic congestion index.

[0026] Optionally, based on the dynamic congestion index, analyzing the frequency of passengers' body movements and the characteristics of the space occupied by their belongings through a multi-task learning model, calculating the probability of abnormal behavior, and outputting an abnormal event warning signal, includes:

[0027] Extract the optical flow features of body movements from the passenger's spatiotemporal feature matrix, and use sliding window Fourier transform to generate the movement frequency spectrogram;

[0028] Based on the motion frequency spectrum and dynamic crowding index, the abnormal frequency detection algorithm is used to screen abnormal limb motion segments and generate a preliminary abnormal candidate set;

[0029] The Mask R-CNN model is used to segment the personal belongings area, and the item type classifier is used to calculate the occupied area ratio and generate the item occupancy feature vector.

[0030] The preliminary anomaly candidate set and item occupancy feature vectors are input into the multi-task learning network. The main network predicts the anomaly probability, and the auxiliary network reconstructs the normal behavior pattern and outputs a comprehensive anomaly score.

[0031] Based on the comparison results of the comprehensive anomaly score and the standard deviation of the historical baseline data, a graded warning signal is triggered, and the type and location coordinates of the abnormal event are output.

[0032] Optionally, the integration of the dynamic congestion index, the abnormal event warning signal, and the real-time passenger flow data to generate a public transport passenger flow monitoring result through spatiotemporal correlation analysis includes:

[0033] The dynamic congestion index is spatiotemporally aligned with real-time boarding and alighting passenger flow data to construct a multi-dimensional spatiotemporal matrix of passenger flow status;

[0034] The location and impact range of abnormal events are marked in the passenger flow state spatiotemporal matrix, and the propagation relationship between events and congestion is modeled using a graph neural network to generate a correlation influence diagram.

[0035] Based on the correlation influence diagram, the monitoring indicator aggregation algorithm is used to integrate the congestion gradient, abnormal event density and passenger flow throughput to generate a monitoring score that reflects the overall health of the real-time passenger flow status.

[0036] The monitoring blind spot data is filled in through the three-dimensional spatial interpolation algorithm, and the monitoring score is mapped to the visualization interface in combination with the carriage layout model. A real-time monitoring report containing heat maps, warning marks and traffic statistics is output.

[0037] Optionally, the method further includes:

[0038] Digital signature technology is used to encrypt electronic evidence containing archived bus passenger flow monitoring results and related passenger flow data to generate a unique data fingerprint. The original electronic evidence and data fingerprint are then synchronized to the distributed evidence storage system through a multi-node storage mechanism in a different location to generate a complete and auditable monitoring report chain.

[0039] Another embodiment of the present application provides a public transportation passenger flow monitoring system based on human body detection, the system comprising:

[0040] The extraction module is used to synchronously collect the skeletal key points, thermal distribution, and contour features of passengers through the on-board multi-camera and infrared sensor, and use the multimodal feature fusion network to extract spatiotemporal correlation features to generate the passenger spatiotemporal feature matrix;

[0041] A positioning module is used to separate and locate overlapping passengers based on the passenger spatiotemporal feature matrix using a human detection model optimized with spatiotemporal attention, eliminate environmental shadow interference through a dynamic trajectory tracking algorithm, and output a set of passenger motion trajectories;

[0042] A construction module is used to construct a physical space occupancy mapping model based on the length of stay and spatial distribution density of passengers in the passenger movement trajectory set, combined with a three-dimensional posture estimation model and pressure sensor data, to generate a dynamic congestion index;

[0043] An output module is configured to analyze the frequency of passengers' body movements and the location characteristics of their belongings using a multi-task learning model based on the dynamic crowding index, calculate the probability of abnormal behavior, and output an abnormal event warning signal;

[0044] The monitoring module is used to integrate the dynamic congestion index, the abnormal event warning signal and the real-time passenger flow data, and generate a public transportation passenger flow monitoring result through spatiotemporal correlation analysis.

[0045] Yet another embodiment of the present application provides a storage medium, wherein the storage medium stores a computer program, wherein the computer program is configured to execute any of the above methods when run.

[0046] Yet another embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute any of the above methods.

[0047] Compared with the existing technology, the present invention provides a bus passenger flow monitoring method based on human body detection. It uses an on-board multi-camera and infrared sensor to synchronously collect passengers' skeletal key points, thermal distribution and contour features to generate a passenger spatiotemporal feature matrix; based on the passenger spatiotemporal feature matrix, it outputs a set of passenger motion trajectories; based on the passenger's stay duration and spatial distribution density in the passenger motion trajectory set, it generates a dynamic congestion index; based on the dynamic congestion index, it calculates the probability of abnormal behavior and outputs an abnormal event warning signal; and by integrating the dynamic congestion index, abnormal event warning signal and real-time passenger flow data, it generates bus passenger flow monitoring results through spatiotemporal correlation analysis, thereby achieving high-precision, low-latency, and intelligent passenger flow status monitoring and abnormal warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A hardware structure block diagram of a computer terminal for a method for monitoring public transportation passenger flow based on human body detection provided by an embodiment of the present invention;

[0049] Figure 2 A schematic flow chart of a method for monitoring public transportation passenger flow based on human body detection provided by an embodiment of the present invention;

[0050] Figure 3 A schematic structural diagram of a public transportation passenger flow monitoring system based on human body detection provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limiting the present invention.

[0052] The embodiment of the present invention first provides a method for monitoring public transportation passenger flow based on human body detection. The method can be applied to electronic devices such as computer terminals, specifically ordinary computers.

[0053] The following describes it in detail by taking running on a computer terminal as an example. Figure 1The hardware structure block diagram of a computer terminal for a bus passenger flow monitoring method based on human body detection provided by an embodiment of the present invention. Figure 1 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0054] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, which, when executed, can cause the processor to execute any one of the bus passenger flow monitoring methods based on human body detection.

[0055] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0056] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any bus passenger flow monitoring method based on human body detection.

[0057] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 1 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0058] It should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0059] See also Figure 2 The embodiment of the present invention provides a method for monitoring public transportation passenger flow based on human body detection, which may include the following steps:

[0060] S201, using the on-board multi-camera and infrared sensor to synchronously collect the passenger's skeletal key points, thermal distribution, and contour features, and using a multimodal feature fusion network to extract spatiotemporal correlation features to generate a passenger spatiotemporal feature matrix;

[0061] Specifically, based on the RGB video stream collected by multiple cameras, a lightweight skeleton key point detection model can be used to extract the coordinates of multiple key points of the passenger and generate a skeleton topology map.

[0062] Multiple cameras (for example, three wide-angle cameras mounted on the roof of a bus, covering a 120° × 90° area) capture RGB video streams at 30 frames per second (FPS). The lightweight skeletal keypoint detection model uses MobileNetV3 as its backbone network, combined with the multi-scale feature fusion capabilities of HRNet (High-Resolution Network), to ensure accuracy while reducing computational latency (single-frame processing time ≤ 50ms).

[0063] Data preprocessing and key point detection:

[0064] ‌Video Stream Frame Processing‌: The video stream is segmented into frames, and the resolution of each frame is adjusted to 1920×1080 (Full HD). Adaptive Histogram Equalization is used to enhance passenger visibility in low-light environments (such as tunnels). For example, the contrast of night scenes is improved by 40%.

[0065] ‌Keypoint detection process‌:

[0066] Feature Extraction: MobileNetV3's Depthwise Separable Convolution extracts spatial features and outputs a 32-channel feature map with only 2.4M parameters.

[0067] Multi-scale Fusion: HRNet's parallel branches process features at different resolutions (original image, 1 / 2 downsampled, and 1 / 4 downsampled), preserving high-resolution details through repeated fusion. For example, the 1 / 4 branch detects the torso, while the original branch precisely locates finger joints.

[0068] Keypoint Regression: Output 17 keypoint coordinates (COCO format) through the fully connected layer, including the head (5 points), upper limbs (6 points), and lower limbs (6 points). The confidence threshold is set to 0.7 (ConfidenceThreshold = 0.7, lower than this value is considered invalid).

[0069] Topology Generation: A skeletal topology graph is constructed based on keypoint coordinates. Nodes are keypoints (e.g., left shoulder, right knee), and edges connect adjacent joints (e.g., shoulder-elbow-wrist). Connections are stored using an adjacency matrix (17×17 in dimension, representing the connection status of 17 keypoints).

[0070] Lightweight optimization strategy:

[0071] ‌Model Pruning‌: Remove redundant convolution kernels in HRNet (such as channels with parameter contribution < 0.1%) and compress the model size to 45% of its original size.

[0072] Quantization: Converts model weights from FP32 (32-bit floating point) to INT8 (8-bit integer), increasing inference speed by 2.3 times (measured on edge devices).

[0073] Real-time performance guarantee: Accelerated by the TensorRT engine, it achieves a processing speed of 25FPS on the NVIDIA Jetson Xavier platform (meeting real-time requirements).

[0074] ‌Anti-interference mechanism‌:

[0075] Motion blur compensation: A Kalman filter is used to predict key point positions (state transition matrix A = [1, 0; 0, 1]) to compensate for coordinate jitter caused by vehicle vibration.

[0076] Occlusion handling: Infer occluded keypoints based on contextual association. For example, if the left elbow is occluded, its coordinates are inferred based on the right elbow and torso positions.

[0077] Output Standardization: The skeleton topology is stored in JSON format, including timestamp, key point coordinates (x, y), and confidence. For example:

[0078] {"frame_001": {"left_shoulder": [320, 150, 0.92], "right_hip": [450,380, 0.88], ...}}.

[0079] The skeleton topology map is input into the thermal distribution alignment module, the thermal imaging data of the infrared sensor is processed synchronously, and the passenger thermal distribution mask is generated through the threshold segmentation algorithm;

[0080] An infrared sensor (such as the FLIR Lepton 3.5, 160×120 resolution) outputs thermal imaging data at 10 frames per second (temperature range -20°C to 150°C). The Thermal Alignment Module aligns the thermal data with the RGB frames using spatio-temporal registration and extracts passenger thermal regions using adaptive threshold segmentation.

[0081] ‌Spatiotemporal registration process‌:

[0082] Time Synchronization: Aligns the timestamps of RGB and thermal imaging frames based on a hardware trigger signal (PPS, Pulse Per Second), with an error of <10ms.

[0083] ‌Space Transformation‌:

[0084] Feature Matching: Extract SIFT (Scale-Invariant Feature Transform) feature points (such as window edges and armrest corners) from the RGB frame and match them with SURF (Speeded-Up Robust Features) feature points from the thermal image.

[0085] Homography Matrix Calculation: Use the RANSAC (Random Sample Consensus) algorithm to find the optimal transformation matrix H (3×3 matrix) to map thermal imaging pixels to the RGB coordinate system. For example, the thermal imaging point (x_ir, y_ir) is aligned to the RGB point (x_rgb, y_rgb) after H transformation.

[0086] Resolution Adaptation: Bicubic interpolation is performed on thermal imaging data to increase the resolution to 1920×1080 (consistent with RGB frames).

[0087] ‌Heat distribution mask generation‌:

[0088] ‌Adaptive Threshold Segmentation‌:

[0089] Temperature background modeling: Calculate the base temperature of the passenger-free area of ​​the vehicle (e.g., 25°C ± 2°C) as the background model.

[0090] Dynamic threshold calculation: The passenger area is located based on the skeletal topology, and a temperature histogram is calculated within the area. The valley of the two peaks is selected as the segmentation threshold T (e.g., body temperature > 32°C).

[0091] Segmentation algorithm: Apply the modified Otsu's method to dynamically adjust T, using the formula T = μ_bg + 2.5 × σ_bg (where μ_bg is the mean background temperature and σ_bg is the standard deviation). For example, if the background temperature is μ_bg = 25°C and σ_bg = 1.5°C, then T = 25 + 3.75 ≈ 28.75°C.

[0092] ‌Mask Optimization‌:

[0093] Morphological operations: Dilation (kernel size 5×5) to fill holes, and Erosion (kernel size 3×3) to smooth edges.

[0094] ‌Bone Constraint‌: Only retains hot areas that overlap with the skeleton topology (IoU>0.6), filtering out interference such as seat heating.

[0095] Output format: The thermal mask is a binary image with the passenger area white (pixel value 255) and the background black (pixel value 0).

[0096] ‌Strategies to combat environmental interference‌:

[0097] Sunlight Reflection Suppression: Identifies spatial continuity in high temperature areas (>50°C), and treats broken areas as reflection noise (such as sunlight reflected from car windows) and removes them directly from the mask.

[0098] Low-temperature passenger detection: In winter scenarios, the silhouette features (step 3) are combined to compensate for the underestimated body temperature caused by clothing obstruction.

[0099] ‌Real-time Calibration‌: Updates the background temperature model every 5 minutes to adapt to changes in cabin air conditioning.

[0100] Based on the skeleton topology map and heat distribution mask, a contour tracking algorithm is used to extract the passenger contour binary image and generate a set of multimodal feature maps;

[0101] The contour tracking algorithm combines the advantages of skeletal position (high semantic information) and thermal distribution (high robustness) to generate an accurate binary contour image (Binary ContourImage) of the passenger contour through the Active Contour Model.

[0102] ‌Contour initialization and optimization‌:

[0103] ‌Initial outline generation‌:

[0104] ‌Skeleton Driven‌: Based on the skeleton topology, a minimum convex hull is constructed. For example, the initial polygon is formed by connecting the key points of the head, shoulders, and hips.

[0105] Heat Distribution Constraint: Use the heat distribution mask as an energy constraint term to force the contour to move closer to the high temperature area.

[0106] ‌Active Contour Evolution‌:

[0107] Energy function design: Define internal energy (contour smoothness) E_int, external energy (heat distribution gradient) E_ext, constraint energy (bone position) E_con, and total energy E_total = α*E_int+β*E_ext+γ*E_con (weights α=0.3, β=0.5, γ=0.2).

[0108] Iterative Optimization: Minimize E_total by gradient descent, moving the contour vertex (Vertex) at each iteration, with a step size of η=0.1 (Learning Rate), and a maximum number of iterations of 100.

[0109] ‌Contour post-processing‌:

[0110] Hole Filling: Scans unclosed areas inside the contour and fills them using the Flood Fill algorithm.

[0111] ‌Edge Thinning‌: Apply the Zhang-Suen Thinning Algorithm to obtain single-pixel width contour lines.

[0112] Multimodal feature map generation:

[0113] ‌Feature Map Type‌:

[0114] ‌Contour binary image‌: pixel values ​​inside the passenger are 1, and outside are 0 (resolution 1920×1080).

[0115] ‌Skeleton heatmap‌: Gaussian distribution centered on key points (standard deviation σ = 15 pixels), representing the importance of joints.

[0116] ‌Heat distribution intensity map‌: Raw thermal imaging data normalized to the [0,1] interval (35°C = 0.5, 45°C = 1.0).

[0117] ‌Space-Time Alignment‌:

[0118] ‌Inter-frame correlation‌: Tracks contour motion based on optical flow to ensure that contour IDs in adjacent frames are consistent (for example, if the contour ID of passenger A is 001 in frame t, it remains 001 in frame t+1).

[0119] Multi-view fusion: Utilizes the disparity of three cameras to calculate depth information and map the outline to a unified 3D coordinate system (the cabin space coordinate system).

[0120] ‌Anti-occlusion and overlap processing‌:

[0121] ‌Overlap Segmentation‌: When the IoU of multiple passenger silhouettes is greater than 0.4, the watershed algorithm is used to separate the overlapping areas (such as two people standing side by side).

[0122] Occlusion reasoning: If the outline missing rate is greater than 30% (such as being blocked by a seat), a complete outline is generated based on the skeleton topology and heat distribution interpolation.

[0123] Feature map collection storage: Stored as HDF5 files in frame sequence, each frame contains three feature maps (contour, skeleton, heat distribution) and timestamp.

[0124] The multimodal feature map is input into the spatiotemporal convolutional fusion network, and the skeletal motion, thermal radiation intensity and contour geometric features are dynamically weighted through the cross-modal attention mechanism to output the passenger spatiotemporal feature matrix.

[0125] The Spatio-Temporal Convolutional Fusion Network (STCFN) adopts a three-channel input structure (contour / skeleton / thermal distribution), extracts spatiotemporal features through 3D convolution, and introduces cross-modal attention to dynamically fuse multi-source information.

[0126] ‌Spatiotemporal feature extraction‌:

[0127] ‌Input tensor construction‌:

[0128] Time window: feature map sequence of 10 consecutive frames (about 0.33 seconds) (3 channels × 10 frames × 1080 × 1920).

[0129] Downsampling: Spatial downsampling to 480×640 (reduces computational effort), while keeping the channel dimension unchanged.

[0130] ‌3D Convolutional Coding‌:

[0131] First layer: 3D convolution kernel (kernel_size=3×7×7), outputting a 64-channel feature map (spatiotemporal dimensions 10×480×640).

[0132] Second layer: 3D convolution kernel (3×5×5), stride (stride=1×2×2), output 128-channel feature map (10×240×320).

[0133] The third layer: 3D convolution kernel (3×3×3), stride (1×2×2), output 256-channel feature map (10×120×160).

[0134] ‌Feature Pyramid (FPN) Enhancement‌:

[0135] Top-Down Path: Fusion of deep semantic features (120×160) and shallow detail features (240×320).

[0136] Lateral Connection: Align the number of channels through 1×1 convolution.

[0137] ‌Cross-modal Attention Fusion‌:

[0138] ‌Attention weight calculation‌:

[0139] Query: Extract motion vectors (such as hip joint displacement velocity) from skeletal motion features (channels 1-64).

[0140] Key and Value: The contour geometry feature (channels 65-128) is used as the key, and the thermal radiation feature (channels 129-192) is used as the value.

[0141] Similarity calculation: Dot product of query and key, and generate attention weight matrix through Softmax.

[0142] Dynamic Weighted Fusion:

[0143] Thermal radiation features are weighted by weight: Weighted_Heat = Attention_Weights × Value

[0144] Concatenate skeleton and contour features: Fused_Feature = Concat(Skeleton, Weighted_Heat)

[0145] Output dimension: 256 channels × 10 frames × 120 × 160 (fusion feature map).

[0146] ‌Feature Matrix Generation and Optimization‌:

[0147] ‌Spatiotemporal Pooling‌:

[0148] Spatial dimensions: Max Pooling up to 15×20 (stride 8×8).

[0149] Temporal dimension: Average Pooling up to 5 frames (step 2).

[0150] ‌Dimensionality reduction output‌:

[0151] The fully connected layer (256×5×15×20→1024 dimensions) compresses features.

[0152] Layer Normalization (LayerNorm) and ReLU activation.

[0153] The final output is the Passenger Spatio-Temporal Feature Matrix, with a dimension of N×1024 (N is the number of passengers, and each passenger has one feature vector).

[0154] Real-time guarantee:

[0155] The amount of network parameters is controlled at 8.7M, and the single inference time on the RTX 3060 GPU is ≤80ms.

[0156] The feature matrix is ​​transmitted to subsequent modules in real time via ZeroMQ.

[0157] This step uses multi-source sensing technology, with visual and infrared sensors working together to obtain passengers' skeletal movement information, body temperature distribution characteristics, and physical contour data. The multimodal fusion network, through a deep learning architecture, aligns these heterogeneous data in time and space and correlates features to construct a spatiotemporal matrix that comprehensively reflects the characteristics of passenger behavior. Skeletal key points are used to analyze posture changes, thermal distribution data helps distinguish overlapping bodies, and contour features enhance the recognition accuracy of passenger edges, achieving high-precision passenger feature extraction and providing a structured data foundation for subsequent behavioral analysis and congestion calculations. Multimodal fusion effectively overcomes the limitations of a single sensor. For example, in low-light conditions, infrared data can compensate for the lack of visual information, while skeletal key points can prevent thermal imaging from misjudging stationary objects.

[0158] S202, based on the passenger spatiotemporal feature matrix, a human detection model optimized for spatiotemporal attention is used to separate and locate overlapping passengers, and a dynamic trajectory tracking algorithm is used to eliminate environmental shadow interference, and a set of passenger motion trajectories is output;

[0159] Specifically, the passenger spatiotemporal feature matrix can be input into the spatiotemporal attention module to generate an attention weight matrix that represents the significance of passenger movement;

[0160] The Spatio-Temporal Attention Module quantifies the dynamic characteristics of passenger movements and identifies areas of significant motion (such as rapid hand waving or sudden standing up), providing priority guidance for separating overlapping passengers. This module employs a dual-path architecture: a temporal path analyzes motion continuity, and a spatial path locates areas of high motion intensity.

[0161] Motion saliency feature extraction:

[0162] Input Data Preprocessing: The passenger spatiotemporal feature matrix (dimension N × 1024, where N is the number of passengers) is first segmented into time series. For example, every 5 frames (approximately 0.17 seconds) is a time segment, and a total of 60 segments (covering 3 seconds) are extracted.

[0163] ‌Time Path Design‌:

[0164] Optical flow feature calculation: Dense optical flow is calculated for the coordinates of skeletal keypoints (such as the hip joint) in consecutive frames, outputting a displacement vector (Δx, Δy). For example, if a passenger's hip joint moves 15 pixels in 0.1 seconds, the velocity vector is (150, 0) pixels / second.

[0165] Motion Energy Accumulation: Construct a Motion Energy Map, where pixel value = square of optical flow amplitude (i.e. Δx 2 +Δy2 ). High energy areas (such as >5000 pixels 2 / Second 2 ) are marked as significant motion areas.

[0166] ‌Spatial path design‌:

[0167] Local Contrast Enhancement: Calculates the local entropy of the feature matrix, which measures spatial complexity. For example, the entropy value is >6.0 (threshold) in dense passenger areas and <3.0 in sparse passenger areas.

[0168] Heatmap generation: The entropy value is superimposed on the motion energy map and a spatial saliency heatmap is generated using Gaussian filtering (standard deviation σ = 8 pixels).

[0169] Dynamic allocation of attention weights:

[0170] Dual-path fusion: The temporal path output (motion energy) and the spatial path output (entropy heat map) are input into the cross-attention layer:

[0171] Query vector: The regional mean of the motion energy map (e.g., a 10×10 pixel block).

[0172] Key vector and value vector: regional features of the spatial entropy map.

[0173] Similarity Calculation: Calculates the cosine similarity between the query and the key, and outputs a similarity score. For example, if a region has high motion energy and high entropy, the score is 0.92 (out of a maximum score of 1.0).

[0174] Weight Normalization: Apply the Softmax function to the similarity scores to generate an attention weight matrix (AttentionWeight Matrix). The dimensions are the same as the input feature map (e.g., 480×640). The weight values ​​range from 0 to 1 and sum to 1. For example, the weight of a waving passenger is 0.35, and the weight of a sitting passenger is 0.05.

[0175] ‌Anti-interference optimization strategy‌:

[0176] Vehicle vibration compensation: Based on the vehicle acceleration data collected by IMU (Inertial Measurement Unit) (e.g. Z-axis acceleration > 2m / s during sudden braking) 2 ), dynamically reduce the weight of the vibration area (weight × 0.5).

[0177] ‌Illumination Sudden Change Suppression‌: When the camera exposure changes by >30% (such as when entering or exiting a tunnel), the attention weight is frozen for 1 second to avoid false triggering.

[0178] Output Normalization: The weight matrix is ​​stored as a floating-point tensor with a resolution of 480×640 and an update frequency of 30Hz per frame. High weight regions (>0.3) are marked in red and displayed in real time in the debug interface.

[0179] The pixel features of the overlapping areas are reweighted based on the attention weight matrix, and the YOLOv8 model is used to output the separated passenger location bounding box coordinates;

[0180] Attention weighting enhances the distinction between overlapping areas, driving the YOLOv8 model to accurately locate adjoining passengers. The reweighting strategy emphasizes significant features, suppresses background noise, and improves small object detection.

[0181] ‌Feature Reweighting Mechanism‌:

[0182] ‌Pixel-level fusion‌: Element-wise multiplication of the original multimodal feature map (contour / skeleton / heat distribution) and the attention weight matrix. For example:

[0183] The eigenvalues ​​of the high-weight region (0.8) are magnified to 1.6 times (original value × 2.0).

[0184] The eigenvalues ​​of the low-weight region (0.1) are compressed to 0.2 times (original value × 0.2).

[0185] Channel Prioritization: Give an additional weight of 1.5 to the skeleton feature channel (channels 1-64) (because skeleton information is most effective for separating overlapping passengers).

[0186] Dynamic Balance: When the weight difference between two passengers is less than 0.2, symmetrical weighting (×1.0 for each) is activated to avoid bias towards one side.

[0187] ‌YOLOv8 Separate Positioning Process‌:

[0188] ‌Model architecture optimization‌:

[0189] The backbone network (Backbone) uses CSPDarknet53 instead (the amount of computation is reduced by 40%).

[0190] A small object layer is added to the detection head to distinguish overlapping passengers with a spacing of less than 20 pixels.

[0191] Overlapping area handling:

[0192] ‌Anchor Adjustment‌: Set special anchor frame sizes for bus scenarios (80×120 pixels for standing passengers, 50×50 pixels for children).

[0193] Non-maximum suppression (NMS) optimization: Soft-NMS algorithm (Soft Non-Maximum Suppression) is used, and the overlap rate (IoU) threshold is relaxed from 0.45 to 0.6 to avoid over-suppression of adherent objects.

[0194] ‌Bounding Box Output‌:

[0195] Coordinate format: normalized bounding box (x_center, y_center, width, height), value range 0~1.

[0196] Confidence threshold: 0.65 (values ​​lower than this are considered false positives).

[0197] Example output: Passenger A’s bounding box is [0.35, 0.42, 0.12, 0.18] with a confidence score of 0.92.

[0198] ‌False detection correction mechanism‌:

[0199] Heat distribution verification: When the average temperature within the bounding box is less than 30°C (the ambient temperature threshold), it is considered a shadow artifact and is forcibly removed.

[0200] ‌Skeleton topology check‌: If the number of key points in the bounding box is less than 5 (e.g., occluded), start contour integrity analysis (see step 3) and fill in the missing boundaries.

[0201] Real-time performance guarantee: The model is quantized to INT8 precision, and the inference speed reaches 45FPS on the embedded platform (NVIDIA Jetson AGX Xavier).

[0202] The passenger location bounding box coordinates and heat distribution mask are input into the shadow discriminator, and the environmental shadow artifacts are filtered out through the thermal radiation intensity comparison algorithm to generate a purified passenger location sequence;

[0203] The Shadow Discriminator combines thermophysical and geometric features to identify false positives caused by shadows cast by trees and handrails outside the window, improving positioning accuracy.

[0204] Thermal radiation comparison algorithm:

[0205] Temperature Gradient Analysis:

[0206] Calculate the standard deviation of the temperature within the bounding box. The standard deviation for the true passenger area is >2.5°C (due to temperature differences between different parts of the human body), and the standard deviation for the shaded area is <1.0°C (uniform).

[0207] Detect the "cold edge effect": Clothing blocks the contours of the human body, creating a temperature gradient (the edge is 3-5°C cooler than the center), while shadows do not have this characteristic.

[0208] Thermal distribution consistency test:

[0209] The bounding box is divided into a 3×3 grid and the temperature difference between the grids is compared. The temperature difference between the real passenger grids is greater than 4°C (for example, head vs. feet), while the temperature difference in the shadow area is less than 1°C.

[0210] Dynamic threshold setting: Summer basic threshold = 4°C, Winter = 2.5°C (taking into account the influence of clothing).

[0211] Geometric feature-assisted discrimination:

[0212] ‌Contour morphological analysis‌:

[0213] True passenger characteristics: Aspect ratio 1.5-4.0 (standing / sitting), area > 800 pixels (minimum valid size).

[0214] Shadow features: abnormal aspect ratio (such as 0.2 or 8.0), too large area (> 5000 pixels) but no skeleton points.

[0215] Projection Direction Verification:

[0216] The theoretical direction of the shadow is predicted by combining the vehicle's GPS position with the sun's altitude angle (calculated using an astronomical algorithm).

[0217] If the angle between the detected object direction and the theoretical projection is greater than 30°, it is determined to be an artifact.

[0218] ‌Cleansed data generation‌:

[0219] Position sequence construction:

[0220] Valid passenger location format: [frame ID, passenger ID, x, y, temperature, confidence].

[0221] Example: [frame 235, ID007, 320.5, 150.2, 36.7, 0.88].

[0222] Sequence Optimization Strategy:

[0223] Short track filtering: Targets that appear continuously for less than 5 frames are considered noise (e.g., a bird flying past a car window).

[0224] Cross-frame correlation: Matches passenger IDs across adjacent frames based on the Hungarian Algorithm (matching cost = position offset + temperature change).

[0225] Output guarantee mechanism:

[0226] The processing delay of each frame is <50ms, and the position accuracy error is ±5 pixels.

[0227] Shadow culling rate >95% (verified by measured data sets).

[0228] The cleaned passenger position sequence is used to initialize the trajectory tracker, and the three-dimensional space trajectory is calculated by combining the multi-camera disparity data to generate a preliminary motion trajectory set.

[0229] The tracker maps the 2D position sequence to the three-dimensional space of the vehicle, reconstructs the depth information through multi-view disparity, and generates a motion trajectory with spatial coordinates.

[0230] Multi-view parallax depth calculation:

[0231] ‌Parallax Baseline Calibration‌:

[0232] The distance between the three cameras (baseline distance) is 12 cm, and the focal length f is 4 mm (wide-angle lens).

[0233] Depth Z calculation formula: Z = (Baseline × f) / Disparity (formula prohibited, only the principle is described).

[0234] Feature point matching:

[0235] Match the corner points of the same passenger’s silhouette in the left / right views and calculate the similarity using the SIFT descriptor (128-dimensional vector).

[0236] Disparity search range: 0~120 pixels (corresponding to depth 0.5m~5m).

[0237] ‌Deep Optimization‌:

[0238] Remove outliers: points whose depth values ​​exceed the physical range of the car (Z < 0.2m or Z > 3m).

[0239] Median filtering: Take the median value of the five feature points of the same passenger to eliminate jitter.

[0240] ‌Trajectory initialization and update‌:

[0241] Tracker selection: Using SORT algorithm (Simple Online and Realtime Tracking):

[0242] ‌State vector‌: [x, y, z, vx, vy, vz] (position + velocity, 6 dimensions).

[0243] Kalman Prediction: Predicts the next frame position based on the Constant Velocity Model.

[0244] ‌Data Association‌:

[0245] Cost Matrix: Calculates the Euclidean distance between the predicted position and the actual position.

[0246] Matching threshold: 3D distance < 0.2m (about 20 pixels) is considered to be the same target.

[0247] ‌Trajectory Generation‌:

[0248] Each trajectory contains: trajectory ID, timestamp sequence (Unix millisecond timestamp), and 3D coordinate sequence.

[0249] Example: trajectory ID = 019, waypoints [(t1,x1,y1,z1), (t2,x2,y2,z2), ...].

[0250] ‌Bus scene adaptation‌:

[0251] Emergency braking: When the vehicle acceleration is greater than 1.5m / s 2 When , it switches to the acceleration model (AccelerationModel), and the state vector is expanded to 9 dimensions (acceleration components ax, ay, az are added).

[0252] ‌Entry and exit event markers‌:

[0253] The door area has a preset 3D geofence with dimensions of 1.5m×0.8m×0.5m (length, width, and height).

[0254] The event type (boarding / disembarking) is marked when the track enters or exits the fence.

[0255] ‌Preliminary trajectory set output‌:

[0256] Data format: JSON array, output once per second.

[0257] Track point density: 30 points / second (synchronized with the frame rate).

[0258] The trajectory jump points in the preliminary motion trajectory set are corrected by the acceleration constrained Kalman filter algorithm, and the passenger motion trajectory set with time stamps is output.

[0259] The Acceleration-Constrained Kalman Filter (ACCF) uses human kinematic limits (such as maximum acceleration) to identify and correct abnormal trajectory jumps (such as sudden coordinate changes caused by occlusion).

[0260] ‌Trip point detection mechanism‌:

[0261] Acceleration threshold setting:

[0262] Normal walking acceleration ≤ 2m / s 2 , running ≤ 5m / s 2 .

[0263] Set dynamic threshold: basic value = 3m / s 2 , relaxed to 4m / s in crowded conditions 2 (May be accelerated due to pushing).

[0264] ‌Mutation Detection Algorithm‌:

[0265] Calculate the acceleration of adjacent trajectory points: a = [v(t)-v(t-1)] / Δt (Δt=1 / 30 second).

[0266] Mark abnormal points: When |a|>threshold and lasts for >3 frames, it is determined as a jump point.

[0267] Kalman Correction Process:

[0268] ‌State Reinitialization‌:

[0269] When a jump point is detected, the Kalman gain coefficient K is frozen to avoid erroneous updates.

[0270] Reset the state vector with the mean position of the trajectory of the previous 3 frames.

[0271] Acceleration Constraint Fusion:

[0272] Add a constraint condition in the prediction step: predicted acceleration ≤ 5m / s 2 (Physical limit).

[0273] If the predicted value exceeds the limit, a constrained optimization algorithm (such as the projected gradient method) is used to force correction.

[0274] ‌Smoothing‌:

[0275] A Savitzky-Golay filter (window length 7, polynomial order 3) is applied to the corrected trajectory to preserve the true motion trend.

[0276] ‌Trajectory set optimization output‌:

[0277] Timestamp Synchronization:

[0278] The vehicle CAN bus clock is aligned (accuracy ±10ms), and the trajectory timestamp is synchronized with the vehicle speed signal.

[0279] Format: Unix timestamp (13-digit milliseconds, such as 1700000000123).

[0280] Track segment storage:

[0281] Segmented storage by passenger ID, with a single trajectory lasting up to 10 minutes (to prevent memory overflow).

[0282] Data fields: ["traj_id", "timestamp", "pos_x_m", "pos_y_m", "pos_z_m", "velocity_mps"].

[0283] ‌Performance Indicators‌:

[0284] Track continuity: >98% (no broken frames).

[0285] Position error: ±0.05m in static scenes, ±0.15m in dynamic scenes.

[0286] Output frequency: 30Hz (the entire vehicle supports tracking 50 people simultaneously).

[0287] This step utilizes a spatiotemporal attention mechanism to focus on areas of significant passenger movement, addressing the challenge of identifying objects occluding each other in densely populated scenes. The dynamic trajectory tracking algorithm combines multi-frame time series analysis with thermal radiation signature comparison to effectively distinguish actual passenger movement from ambient shadow changes, ensuring the accuracy of trajectory data. The model generates a continuous sequence of time-space coordinates for each detected passenger, significantly improving detection accuracy in high-density passenger flow scenarios and providing reliable trajectory data for subsequent behavioral analysis. Eliminating shadow interference prevents the misidentification of objects such as window reflections and luggage shadows as passengers, ensuring the authenticity of statistical data.

[0288] S203, constructing a physical space occupancy mapping model based on the passenger's dwell time and spatial distribution density in the passenger trajectory set, combined with a three-dimensional posture estimation model and pressure sensor data, and generating a dynamic congestion index;

[0289] Specifically, based on the passenger movement trajectory set, the length of time each passenger stays in the carriage and the movement path can be counted, and the spatial distribution heat map can be generated using kernel density estimation;

[0290] The passenger trajectory set (Trajectory Set) contains each passenger's 3D coordinates (unit: meters) and timestamp (Unix millisecond timestamp). Kernel Density Estimation (KDE) uses probability density modeling based on the spatial distribution of trajectory points to intuitively reflect the areas of passenger flow within the train. This process consists of three stages: trajectory preprocessing, dynamic kernel function selection, and heatmap rendering.

[0291] ‌Trajectory data preprocessing‌:

[0292] Time Slicing: Splits the trajectory data into fixed time windows (Window Length). The default window length is W = 30 seconds (i.e., a heat map is generated every 30 seconds). For example, count all trajectory points between 14:00:00 and 14:00:30.

[0293] ‌Effective Area Filter‌: Define the effective space range of the carriage (e.g., 8 meters long x 2.5 meters wide x 2 meters high), and exclude invalid points such as the door area (geofence zone) (such as short stops when passengers get on and off the train).

[0294] ‌Stay duration calculation‌:

[0295] If a passenger stays continuously in a 1 square meter area for more than the threshold T_stay=10 seconds, it is marked as a "stay point".

[0296] The movement path is generated by accumulating the Euclidean distances between adjacent trajectory points, and the path length is accurate to 0.01 meters.

[0297] Adaptive Kernel Density Estimation:

[0298] Kernel function selection: Gaussian kernel is used, and its bandwidth parameter (Bandwidth, symbol h) is dynamically adjusted:

[0299] The basic value h_init = 0.3 meters (about the width of an adult's shoulders).

[0300] Automatically adjust based on passenger density: when the number of people per unit area (1 square meter) is greater than 4, h is reduced to 0.2 meters (to improve resolution); when the density is less than 1 person, h is expanded to 0.5 meters (to avoid sparse noise).

[0301] Density calculation principle:

[0302] The carriage floor is divided into 0.1m x 0.1m grids.

[0303] The density value of each grid point is the sum of the weighted contributions of all trajectory points within a radius h around it. The closer the trajectory point is, the higher the weight (Gaussian weight curve).

[0304] Dynamic Weight Mechanism:

[0305] The weight of the stay point W_stay=1.5 (higher than the moving point W_move=1.0) highlights the static gathering area.

[0306] Trajectory points of elderly people (classified by age of skeletal keypoints) are given an additional weight of ×1.2 (due to their limited range of motion).

[0307] Heatmap Generation and Optimization:

[0308] ‌Pseudocolor Mapping‌: Density values ​​are normalized to the range 0-1, and then rendered using the Jet color scale (Blue-Green-Yellow-Red):

[0309] Low density (0~0.3): blue (sparsely populated area).

[0310] High density (0.7~1): red (congestion danger zone).

[0311] Temporal smoothing: 5 consecutive heat maps (2.5 minutes) are averaged to eliminate instantaneous fluctuations (such as passengers briefly brushing against each other).

[0312] Output Specifications:

[0313] Resolution: 80 × 25 pixels (corresponding to an 8m × 2.5m carriage).

[0314] Data format: JSON array containing grid coordinates (grid_x, grid_y) and density values ​​(density).

[0315] Example: {"time": "14:00:30", "grids": [[0,0,0.12], [0,1,0.35], ..., [79,24,0.87]]}.

[0316] The spatial distribution heat map is input into the 3D posture estimation model to analyze the passenger's sitting and standing postures and luggage carrying status, and output the predicted value of individual space occupancy rate;

[0317] The 3D Pose Estimation Model uses heatmaps to locate passengers and uses multi-camera technology to reconstruct 3D skeletal poses to quantify each passenger's actual space occupation. The model employs a cascade network architecture, sequentially processing pose, baggage, and space calculations.

[0318] ‌3D Skeleton Pose Analysis‌:

[0319] Model Architecture: Improved OpenPose network (ResNet-50 backbone), outputting 3D coordinates of 17 key points (millimeter accuracy).

[0320] Input Data Fusion: After the heatmap locates the passenger ROI (Region of Interest), a multi-view RGB image of the corresponding area is cropped and fed into the network.

[0321] ‌Parallax Depth Optimization‌: Utilizes Multi-View Disparity to correct depth errors (Z-axis jitter < ±5 cm).

[0322] Posture classification logic:

[0323] Standing posture recognition: The height difference between the hip and ankle joints ΔH>0.8 meters (threshold), and the knee joint angle θ_knee∈[165°,195°] (close to upright).

[0324] Sitting posture recognition: ΔH < 0.3 meters, and torso inclination angle θ_torso < 70° (to avoid misjudging squatting).

[0325] Abnormal posture: A fall is marked by a sudden drop in hip-ankle height difference >50% (within 0.5 seconds).

[0326] Baggage detection and occupancy analysis:

[0327] Object segmentation model: Mask R-CNN (MobileNetV3 as the backbone), which recognizes three types of objects:

[0328] Large Luggage: Suitcases and strollers (minimum bounding box area > 0.3 square meters).

[0329] ‌Backpack‌: Backpack / handbag (0.1~0.3 square meters).

[0330] Small Items: Handbags, pet cages (<0.1 square meters).

[0331] ‌Footprint Calculation‌:

[0332] The actual area occupied by the luggage = the projected area of ​​the bounding box × the attitude compensation coefficient.

[0333] The backpack coefficient for a seated passenger is 0 (it takes up no extra space on the legs), and the backpack coefficient for a standing passenger is 1.0 (it takes up the entire space).

[0334] Space occupancy forecast:

[0335] The basic space template is shown in Table 1:

[0336] Table 1

[0337] Passenger type Standing space (square meters) Basic riding space (square meters) aldult 0.35 0.45 (including seat width) child 0.25 0.30

[0338] Dynamic overlay rules:

[0339] Luggage space accumulatively: large luggage + 0.4 square meters, carry-on backpack + 0.15 square meters (when standing only).

[0340] Posture compensation: The waving action temporarily increases the area by 0.1 square meters (safety redundancy).

[0341] Sample output:

[0342] {

[0343] "passenger_id": "019",

[0344] "pose": "standing",

[0345] "occupation": 0.62, / / Adult standing 0.35 + suitcase 0.4 - overlap correction 0.13

[0346] "items": [{"type": "large_luggage", "area": ​​0.4}]

[0347] }.

[0348] By integrating real-time load data collected by pressure sensors, the predicted space occupancy rate is dynamically calibrated to generate an accurate space occupancy distribution matrix.

[0349] Pressure sensors are embedded in an array in the vehicle floor (spacing 0.5m x 0.5m). Each sensor has a measurement range of 0-300kg (accuracy ±1kg). Dynamic Calibration corrects for model errors (such as undetected hidden luggage) by comparing predicted space occupancy with actual pressure distribution.

[0350] Pressure data fusion strategy:

[0351] ‌Space-Time Alignment‌:

[0352] The pressure data timestamps are aligned with the heatmap (error < 100 ms).

[0353] The sensor grid coordinates are mapped to the heat map grid (8×2.5m carriage corresponds to 16×5 sensor array).

[0354] Load distribution algorithm:

[0355] A single sensor load (Load) is distributed to all passengers within a 0.7-meter radius (weighted inversely proportional to distance).

[0356] The theoretical passenger weight (Weight) is predicted based on the height of key skeletal points: the average adult weight is 65 kg (standard deviation σ = 10 kg), and the child weight is 30 kg.

[0357] ‌Calibration mechanism design‌:

[0358] Error detection logic:

[0359] If a passenger predicts a space occupancy of 0.5 square meters, but the assigned load is only 40 kg (lower than the adult standard of 65 kg), it is judged to be an underestimate.

[0360] Hidden luggage speculation:

[0361] When the load is greater than the predicted body weight × 1.25 (e.g. 65kg × 1.25 = 81kg), luggage compensation is activated:

[0362] Overweight up to 20kg: additional space of 0.15 square meters will be allocated for small items.

[0363] Overweight 20-40kg: An additional 0.3 square meters of carry-on bag is required.

[0364] Overweight > 40kg: add 0.4 square meters for large luggage.

[0365] Special scene processing:

[0366] Crowded areas (<0.3m spacing) are calibrated together (considered as an overall pressure unit).

[0367] When a child sits on an adult's lap, the loads are combined and marked as "combined occupancy".

[0368] ‌Space occupancy distribution matrix generation‌:

[0369] ‌Matrix structure‌:

[0370] Dimensions: 80×25 (same as the heatmap grid).

[0371] Each grid value = the sum of the calibrated occupancy rates of all passengers at that location.

[0372] ‌Anti-interference optimization‌:

[0373] Sensor temperature drift compensation: Collect a no-load baseline every 10 minutes.

[0374] Vehicle bump filtering: Perform sliding average on load data (window length 5 seconds).

[0375] Sample output:

[0376] {

[0377] "timestamp": 1700000000123,

[0378] "matrix": [

[0379] [0.0, 0.0, 1.2, 0.0, ...], / / Grid data in row 0

[0380] [0.0, 0.8, 1.7, 0.3, ...], / / Row 1 (1.7 means two passengers + luggage) ...

[0381] ]}.

[0382] The congestion calculation function is used to integrate the entropy characteristics of the spatial distribution heat map and the weighted summation results of the spatial occupancy distribution matrix to output a standardized dynamic congestion index.

[0383] The Dynamic Crowding Index (DCI) is a normalized value ranging from 0 to 100 (100 being extremely crowded), combining two core metrics: spatial disorganization (entropy) and physical occupancy density. The calculation process consists of three stages: feature extraction, weighted aggregation, and normalization.

[0384] ‌Entropy feature calculation‌:

[0385] Information Entropy Definition:

[0386] The entropy value reflects the randomness (disorder) of the passenger distribution. Uniform distribution has high entropy, while clustered distribution has low entropy.

[0387] Formula principle: Entropy value H = -Σ(p_i × log2(p_i)), where p_i is the density ratio of the i-th grid in the heat map.

[0388] ‌Carriage partition entropy calculation‌:

[0389] The carriage is divided into 6 functional areas (front door area, front standing area, seating areas 1 / 2 / 3, and rear door area).

[0390] The entropy value of each partition is calculated independently (to avoid global averaging masking local congestion).

[0391] Examples of the physical meaning of entropy values ​​are shown in Table 2:

[0392] Table 2

[0393] Scenario Entropy range illustrate Stand evenly (ideal) 2.8~3.2 Passengers are evenly spaced Congestion at the front door (abnormal) <1.5 Passengers are highly concentrated in a small area Empty car (abnormal) >4.0 Only a few areas have passengers

[0394] ‌Weighted Sum Strategy‌:

[0395] ‌Input feature normalization‌:

[0396] Entropy value feature H_normalized = (H - H_min) / (H_max - H_min) (historical extreme value record of the entire vehicle).

[0397] Total space occupancy S_total = the sum of all elements in the matrix (unit: square meters per person).

[0398] Health Function

[0399] Basic congestion C_base = S_total / effective area of ​​the carriage (8×2.5=20 square meters) × 50 (linear scaling).

[0400] Entropy correction coefficient K_entropy = 1.5 - 0.5×H_normalized (the coefficient is >1 when the entropy is low, which increases the crowding score).

[0401] Final congestion C_final = C_base × K_entropy.

[0402] Index Standardization and Grading:

[0403] ‌Normalized output‌:

[0404] Dynamic congestion index I = min(100, max(0, C_final × 100 / C_max)).

[0405] Among them, C_max=15 (the maximum recorded value in history, corresponding to 57 people standing in 20 square meters).

[0406] The graded warning thresholds are shown in Table 3:

[0407] Table 3

[0408] Index range grade Response measures 0~30 Comfort No operation 31~60 Mildly crowded Voice prompt guidance 61~80 Moderately crowded Dispatch center alarm 81~100 Severe congestion Forced current limiting and station skipping

[0409] Sample output:

[0410] json

[0411] Copy Code

[0412] {

[0413] "timestamp": 1700000000123,

[0414] "crowding_index": 73,

[0415] "subarea_entropy": {"door_front": 0.8, "stand_zone1": 1.2, ...},

[0416] "alert_level": "moderate"

[0417] }.

[0418] Key Technical Parameters Explanation Table 4

[0419] Table 4

[0420] Parameters / Terms‌ Symbol / Abbreviation‌ ‌Meaning‌ Track point time window‌ W The time length of kernel density statistics (default 30 seconds) Dwell time threshold‌ T_stay Minimum duration for a stop point (default 10 seconds) Gaussian kernel bandwidth‌ h Parameter controlling the smoothness of the density estimate (base value 0.3m) Standing height difference threshold‌ ΔH Vertical distance between hip joint and ankle joint (>0.8m for standing position) Joint Angle Tolerance‌ θ_knee Knee joint angle range (165°~195° is upright) Pressure sensor array‌ Floor-embedded sensor grid (0.5m x 0.5m spacing) Load distribution radius‌ R_load Single sensor influence range (0.7 meters) Baggage weight threshold‌ ΔW Overweight load value that triggers baggage compensation (20kg / 40kg classification) Entropy normalization range‌ H_min / H_max Historical minimum / maximum values ​​of compartment partition entropy (dynamically updated) Health modifier K_entropy Entropy-based crowding weighting factor (1.5 - 0.5 × H_normalized)

[0421] This step combines spatiotemporal behavioral characteristics with physical sensor data, quantifying each passenger's cabin space usage through a machine learning model. Three-dimensional posture estimation identifies different positions, such as sitting, standing, and carrying large luggage. Pressure data verifies the accuracy of visual recognition, ultimately generating an indexed indicator reflecting real-time space utilization efficiency. This breaks through the limitations of traditional methods that rely solely on headcounts and establishes a more scientific congestion assessment system. This index truly reflects the impact of passenger distribution on the ride experience, providing a refined basis for scheduling decisions.

[0422] S204, based on the dynamic congestion index, analyzing the frequency of passengers' body movements and the characteristics of the space occupied by their belongings through a multi-task learning model, calculating the probability of abnormal behavior, and outputting an abnormal event warning signal;

[0423] Specifically, the optical flow features of limb movements can be extracted from the passenger spatiotemporal feature matrix, and the sliding window Fourier transform can be used to generate the movement frequency spectrogram;

[0424] The passenger spatiotemporal feature matrix (dimension N×1024, where N is the number of passengers) stores the motion trajectories and contour change information of skeletal keypoints. Body motion optical flow features quantize the displacement vectors (Δx, Δy) of keypoints between consecutive frames to capture dynamic motion. The specific process consists of three stages: optical flow calculation, sliding window segmentation, and frequency domain conversion.

[0425] Optical flow feature extraction process:

[0426] ‌Key point displacement calculation‌:

[0427] Select the core joints of the torso (such as shoulders and hips) as reference points to avoid interference from hand shaking.

[0428] Calculate the displacement vector between adjacent frames (time interval Δt = 33 milliseconds). For example, the left shoulder coordinate changes from [320, 150] in frame (t) to [325, 148] in frame t+1, with a displacement of Δx = 5 pixels and Δy = -2 pixels.

[0429] ‌Motion direction encoding‌:

[0430] The displacement vector is converted to polar coordinate format: Magnitude = √(Δx 2 + Δy 2 ), direction angle (Angle) = arctan(Δy / Δx).

[0431] Normalize the amplitude by dividing it by the image diagonal length (2200 pixels) to avoid the magnitude being affected by the camera distance.

[0432] ‌Feature vector construction‌:

[0433] Single-frame features: A 12-dimensional vector (4 key points × 3 parameters: Δx, Δy, amplitude) is generated for each passenger.

[0434] Time series: Stores 300 consecutive frames (10 seconds) of data to form a 12×300 matrix (one sequence for each passenger).

[0435] Sliding Window Fourier Transform:

[0436] ‌Window parameter design‌:

[0437] The window length is set to 50 frames (about 1.67 seconds) and the stride is 25 frames (50% overlap) to cover the complete motion cycle.

[0438] Hanning Window Weighting: Reduces spectral leakage and improves low-frequency energy focus.

[0439] Frequency domain conversion logic:

[0440] Perform Fast Fourier Transform (FFT) dimension by dimension (12 dimensions) on the 50 frames of data in the window and output the complex spectrum.

[0441] Extract the amplitude spectrum: Take the complex modulus value and retain the 0-15 Hz frequency band (the effective range of human motion).

[0442] ‌Spectrum Generation‌:

[0443] Stack the 12-dimensional spectrum into a three-dimensional tensor (Dimensions: number of time windows × number of frequency points × feature dimensions).

[0444] Pseudo-color coding: Presented in the form of a heat map, with the horizontal axis representing the time window number and the vertical axis representing the frequency (0-15 Hz). Color intensity represents energy value.

[0445] Example: A waving motion in the 2-4 Hz frequency range (medium-speed motion) is highlighted in yellow (energy value > 0.7).

[0446] ‌Anti-interference optimization strategy‌:

[0447] Vehicle Vibration Compensation:

[0448] The vehicle body acceleration data is collected through the IMU (Inertial Measurement Unit). If the Z-axis vibration is greater than 1.5 m / s 2 , freeze the current window FFT calculation.

[0449] ‌Lighting mutation processing‌:

[0450] When the camera exposure changes by >30% (such as when entering or exiting a tunnel), pause the optical flow calculation for 1 second to avoid distortion of the displacement vector.

[0451] Output Specifications:

[0452] Spectrum resolution: 100 pixels on the time axis (corresponding to 10 seconds), 60 pixels on the frequency axis (0-15 Hz divided into 60 levels), stored as PNG images and JSON data files.

[0453] Based on the motion frequency spectrum and dynamic crowding index, the abnormal frequency detection algorithm is used to screen abnormal limb motion segments and generate a preliminary abnormal candidate set;

[0454] The Abnormal Frequency Detection Algorithm combines motion frequency characteristics with the current congestion level in the carriage to identify motion patterns that exceed reasonable limits (such as high-frequency tremors and prolonged periods of inactivity). The Dynamic Crowding Index (Crowding Index) serves as environmental context to dynamically adjust the judgment threshold.

[0455] ‌Action baseline model construction‌:

[0456] ‌Historical Data Learning‌:

[0457] Collect 100 hours of normal ride videos and calculate the action frequency distribution (BaselineDistribution) under different congestion levels.

[0458] Binning by congestion level: an index of 0-30 is "comfortable", 31-60 is "mild", and 61-100 is "moderate to severe".

[0459] The normal frequency range is defined as shown in Table 5:

[0460] Table 5

[0461] Crowding level Standing Wave (Hertz) Bending (Hz) Sitting position adjustment (Hz) Comfort (0-30) 0.5-3.0 0.1-0.8 0.05-0.5 Moderate to severe (61-100) 0.2-1.5 0.05-0.3 0.02-0.2

[0462] ‌Real-time baseline adjustment‌:

[0463] If the current carriage congestion level = 75 (moderate to severe), a waving frequency > 1.5 Hz will trigger an early warning (the threshold is tightened by 50%).

[0464] ‌Abnormal fragment detection logic‌:

[0465] Frequency Band Energy Analysis:

[0466] Split the spectrum into four behavioral bands:

[0467] Low frequency (0-0.5 Hz): breathing, tremors.

[0468] Medium frequency (0.5-4 Hz): waving, turning around.

[0469] High frequency (4-8 Hz): twitching, seizures.

[0470] Ultra-high frequency (8-15 Hz): Mechanical vibration noise (directly filtered out).

[0471] Calculate the energy ratio of each frequency band. For example, if the high-frequency energy ratio is >30%, it is considered suspicious.

[0472] Temporal continuity judgment:

[0473] The anomaly must appear continuously for ≥3 time windows (approximately 5 seconds) to avoid misjudgment of instantaneous actions.

[0474] In crowded conditions, the time is extended to ≥5 windows (8 seconds) due to limited range of motion.

[0475] ‌Preliminary anomaly candidate set generation‌:

[0476] ‌Candidate entry format‌:

[0477] {

[0478] "passenger_id": "007",

[0479] "start_time": 1700000000123, / / Start timestamp (milliseconds)

[0480] "end_time": 1700000005123, / / end timestamp

[0481] "frequency_band": "high", / / Abnormal frequency band (high frequency)

[0482] "confidence": 0.88 / / Confidence (based on energy deviation)

[0483] }.

[0484] Filtering false positives:

[0485] The confidence level of high-frequency actions (such as jumping) of child passengers (skeletal height < 1.3 meters) is ×0.6 (reduced weight).

[0486] Movements in the door area are not counted (natural behavior of passengers getting on and off the vehicle).

[0487] Output Mechanism:

[0488] Detection is performed once per second, and the single processing delay is less than 100 milliseconds.

[0489] The candidate set is transmitted to the next module in real time through a message queue (such as Kafka).

[0490] The Mask R-CNN model is used to segment the personal belongings area, and the item type classifier is used to calculate the occupied area ratio and generate the item occupancy feature vector.

[0491] The Mask R-CNN (Region-based Convolutional Neural Network) finely segments item regions based on a binary image of passenger silhouettes. The Item Classifier, based on a ResNet-18 network, identifies item categories and quantifies their spatial occupancy.

[0492] Item segmentation process:

[0493] ‌Input Data Collaboration‌:

[0494] Crop the multi-view RGB image (1920×1080) using the passenger’s Purified Bounding Box (output of step 3).

[0495] Thermal Mask: High-temperature objects (such as mobile phones) are enhanced in the mask.

[0496] ‌Mask R-CNN Optimization Design‌:

[0497] Backbone: Lightweight MobileNetV3 is used instead of ResNet-50 (reducing computational complexity by 60%).

[0498] Mask Head: Output resolution 56×56 (balance between accuracy and speed).

[0499] ‌Anchor Configuration‌: Set special sizes for public transportation scenarios (20×20 to 150×150 pixels to cover small items to suitcases).

[0500] ‌Post-segmentation processing‌:

[0501] Morphological optimization: The segmentation mask is first closed (with a kernel size of 5×5) to fill holes, and then opened (with a kernel size of 3×3) to smooth edges.

[0502] ‌Passenger Association‌: Only object regions with an Intersection over Union (IoU) > 0.5 with passenger outlines are retained.

[0503] Item classification and placeholder statistics:

[0504] ‌Classifier Architecture‌:

[0505] Input: Segmented object region (56×56 RGB image).

[0506] Network: ResNet-18 (output layer changed to four categories: large luggage, carry-on backpack, small items, and other).

[0507] Incremental training: Use a bus-specific dataset (including special items such as strollers and wheelchairs).

[0508] ‌Footprint Calculation‌:

[0509] ‌Actual pixel conversion‌: Number of object mask pixels × actual area of ​​a single pixel (e.g. 0.0004 square meters / pixel, based on depth map calibration).

[0510] Dynamic Weight Allocation:

[0511] Item type base weight when crowded

[0512] Large Luggage 1.0 1.5

[0513] Backpack 0.8 1.0

[0514] Small items 0.3 0.5

[0515] ‌Occupancy ratio‌: item area / passenger bounding box area (value range 0~1).

[0516] ‌Item placeholder feature vector generation‌:

[0517] ‌Vector structure‌: 8-dimensional features describing the state of a single passenger item: [

[0519] Number of large luggage, # integer (such as 1)

[0520] Number of backpacks, # integer (such as 2)

[0521] Number of small items, # integer (such as 3)

[0522] Total floor space, # square meters (e.g. 0.42)

[0523] Maximum item size, # square meters (e.g. 0.3)

[0524] Average occupancy ratio, # decimal (such as 0.25)

[0525] High-risk item flag, # 0 / 1 (knife detected = 1)

[0526] Item spatial dispersion # 0~1 (1 means items are scattered around the passenger)

[0527] ].

[0528] The preliminary anomaly candidate set and item occupancy feature vector are input into the multi-task learning network. The main network predicts the anomaly probability, and the auxiliary network reconstructs the normal behavior pattern and outputs a comprehensive anomaly score.

[0529] The Multi-Task Learning Network adopts a dual-branch architecture: the main branch is an anomaly detector, and the auxiliary branch learns to reconstruct normal behavior, improving generalization ability through adversarial training.

[0530] ‌Network architecture design‌:

[0531] ‌Input Data Fusion‌:

[0532] Main branch input: preliminary anomaly candidate set (JSON format) and item vector (8-dimensional) are concatenated into 12-dimensional features.

[0533] Auxiliary branch input: only the item vector (8-dimensional), used to reconstruct normal behavior.

[0534] ‌Master Branch (Anomaly Detector)‌:

[0535] Structure: Three fully connected layers (number of neurons: 64-32-1), output layer Sigmoid activation (value range: 0~1).

[0536] Loss function: Focal Loss (parameter α=0.8) to alleviate sample imbalance (normal behaviors far outnumber abnormalities).

[0537] ‌Auxiliary branch (normal behavior rebuilder)‌:

[0538] Structure: Encoder-Decoder, Latent Layer dimension 16.

[0539] Loss function: Mean Squared Error (MSE), which minimizes the difference between the input and the reconstructed output.

[0540] Adversarial training mechanism:

[0541] Training strategy:

[0542] Normal sample training: The auxiliary branch forces the reconstruction of the input, and the main branch output approaches 0 (low abnormality probability).

[0543] Abnormal sample training: freeze the auxiliary branch weights, and the main branch learning output approaches 1.

[0544] ‌Comprehensive score calculation‌:

[0545] The main branch outputs the original probability P_main (0~1).

[0546] Auxiliary branch reconstruction error E_aux = Euclidean distance between the input vector and the reconstructed vector.

[0547] Comprehensive score Score = P_main × [1 + min(1.0, E_aux / 0.3)] (the larger the reconstruction error, the higher the score, with an upper limit of 2.0).

[0548] Threshold determination and output:

[0549] ‌Rating Calibration Rules‌:

[0550] Scenario Rating Correction

[0551] Crowding > 80 Score × 1.3 (High Sensitivity Mode)

[0552] Night mode (22:00-6:00) Score × 0.7 (reduces false alarms)

[0553] ‌Comprehensive anomaly score output‌:

[0554] The value range is 0~2.0, with two decimal places.

[0555] Example: Snatch suspicion score = 1.72, Fall suspicion score = 1.35.

[0556] Based on the comparison results of the comprehensive anomaly score and the standard deviation of the historical baseline data, a graded warning signal is triggered, and the type and location coordinates of the abnormal event are output.

[0557] The ‌Graded Early Warning System‌ is based on the principles of Statistical Process Control (SPC). It compares real-time scores with historical baselines, dynamically triggering three levels of warning (low, medium, and high), and locating event coordinates through multi-camera intersection.

[0558] ‌Historical Baseline Modeling‌:

[0559] Data source:

[0560] Normal behavior score records for the same route and time period (such as morning rush hour) over the past 90 days.

[0561] Store events grouped by type (falls, altercations, items left behind, etc.).

[0562] Baseline statistics calculation:

[0563] Mean μ and standard deviation σ: Calculated independently for each data set (e.g., for fall events, μ=0.15, σ=0.08).

[0564] Dynamic update cycle: Baseline parameters are recalculated every 24 hours.

[0565] ‌Standard Deviation Comparison Rules‌:

[0566] ‌Grading threshold setting‌:

[0567] Warning level trigger condition response action

[0568] Low level (blue) Score>μ+2σ Local logging

[0569] Intermediate (Yellow) Score>μ+3σ Driver-side vibration reminder

[0570] Advanced (Red) Score>μ+4σ or continuously>3σ Report to the dispatch center and trigger in-vehicle broadcast

[0571] Special event judgment:

[0572] ‌Abandoned Items‌: The rating drops suddenly after the passenger gets off the bus (the item vector is non-zero but the passenger ID disappears).

[0573] Group events: ≥3 people within a 3-meter radius trigger a medium-level warning at the same time.

[0574] Event position calibration and output:

[0575] ‌3D coordinate calculation‌:

[0576] Through multi-camera intersection positioning (Triangulation):

[0577] The center point of the 2D bounding box of the target passenger (e.g., [320, 480] for the left view, [290, 475] for the right view).

[0578] Combine the pre-calibrated internal and external parameters of the camera to calculate the 3D coordinates (X, Y, Z).

[0579] Accuracy optimization: Kalman filter output of the fusion trajectory tracker.

[0580] ‌Warning signal format‌:

[0581] {

[0582] "event_id": "EV20231105007",

[0583] "type": "fall_detected", / / Event type (fall)

[0584] "level": "red", / / High level warning (red)

[0585] "position": [2.15, 1.30, 0.0], / / Carriage coordinate system XYZ (meters)

[0586] "timestamp": 1700000000123,

[0587] "passenger_id": "019"

[0588] }.

[0589] ‌Anti-false alarm mechanism‌:

[0590] Low-level warnings are delayed for 5 seconds to confirm (to avoid instantaneous misjudgments).

[0591] Advanced warnings require review of the item vector (e.g., trigger immediately when the knife flag = 1).

[0592] This step analyzes motion frequency spectra and object occupancy patterns to detect abnormalities such as sudden falls and heated arguments. The model also learns normal behavior patterns as a reference. When significant deviations are detected, a graded alert is triggered, annotating the anomaly type and specific location. This enables proactive safety monitoring and timely identification of potentially dangerous situations. Joint analysis of multiple indicators can reduce false alarm rates, for example by distinguishing between normal turns and abnormal movements associated with sudden illness.

[0593] S205, integrating the dynamic congestion index, the abnormal event warning signal and the real-time passenger flow data, and generating a public transportation passenger flow monitoring result through spatiotemporal correlation analysis.

[0594] Specifically, the dynamic congestion index can be aligned with the real-time boarding and alighting passenger flow data in time and space to construct a multi-dimensional passenger flow state time and space matrix;

[0595] Spatio-temporal alignment is a core preprocessing step for integrating the dynamic crowding index, passenger flow data, and the locations of abnormal events. First, the timestamps of all data sources are unified using the vehicle CAN bus clock synchronization protocol, with an accuracy of ±50 milliseconds. For example, the crowding index is updated every 5 seconds (with timestamps formatted as Unix milliseconds: 17000000005000), while boarding and alighting data is triggered in real time by door pressure sensors (recorded every 0.1 seconds). Linear interpolation is used to resample the passenger flow data to 5-second intervals to ensure timeline alignment.

[0596] Spatial alignment relies on a pre-calibrated Digital Cabin Map. This map divides the 8-meter-long cabin into 80 grids (grid size 0.1m x 0.1m), each assigned a unique ID (e.g., GX_Y, where X is the vertical number 0-79 and Y is the horizontal number 0-24). The dynamic crowding index is mapped to the corresponding grid based on passenger trajectories; boarding and alighting data are linked to the grids using door geofence matching (door geofence ID: DF1 for the front door area and DR1 for the rear door area). The coordinates of an abnormal event (e.g., the fall location [2.3, 1.1, 0] meters) are mapped to the nearest grid using a three-dimensional coordinate transformation matrix (e.g., 2.3 meters vertically corresponds to G23, and 1.1 meters horizontally corresponds to G11, resulting in grid ID G23_11).

[0597] The final constructed Passenger State Matrix is ​​a five-dimensional tensor:

[0598] Dimension 1: Time Slice (one slice every 5 seconds, storing the start timestamp).

[0599] Dimension 2: Spatial Grid (80×25=2000 grids).

[0600] Dimension 3: Base Metrics, including congestion value (integer from 0 to 100), net increase in boarding and alighting (negative number indicates boarding and alighting), and instantaneous passenger flow density (people / square meter, accuracy 0.01).

[0601] Dimension 4: Event Annotation, including event type code (such as 101 fall event), level (1-3), and confidence level (0-1 floating point number).

[0602] Dimension 5: Environmental context, such as vehicle speed (km / h), air conditioning status (0 off / 1 on), and light level (level 1-5).

[0603] Example: Grid G23_11 at time slice 17000000005000 stores the data [Crowding 73, Net Passenger Increase -2, Density 3.1, Event Type 101, Level 2, Confidence 0.85, Vehicle Speed ​​25, Air Conditioning 1, Lighting 3].

[0604] The location and impact range of abnormal events are marked in the passenger flow state spatiotemporal matrix, and the propagation relationship between events and congestion is modeled using a graph neural network to generate a correlation influence diagram.

[0605] The impact radius of abnormal events is marked using the Dynamic Radius Diffusion model. The impact radius is determined by the event level, centered on the event location grid. A Level 1 (low-level) event has a radius of 1 meter (approximately 10 grid cells), a Level 2 (intermediate) event has a radius of 2 meters, and a Level 3 (high-level) event has a radius of 3 meters. For example, a Level 2 fall event has an impact radius of 2 meters, covering a 5×5 grid area (25 grid cells) around the center. Each affected grid cell is labeled with the event's penetration factor, starting at 1.0 at the center and decreasing to 0.2 at the edges (calculated inversely proportional to distance).

[0606] The construction of a Graph Neural Network (GNN) is divided into three steps:

[0607] ‌Graph structure definition‌:

[0608] Node: Each spatial grid is considered a node (2000 nodes in total), and the node feature is the concatenation vector (length 10) of the five-dimensional data in the space-time matrix.

[0609] Edge: Defines connection rules based on passenger movement patterns:

[0610] Create bidirectional edges (distance weight 1.0) between adjacent meshes (up, down, left, and right).

[0611] Add reinforcement edges (weight 1.5) to high-frequency passenger movement paths (such as the path from the door to the seat area).

[0612] Directed edges are established from the center point of the abnormal event to the grid of the affected area (direction: center → edge, weight = permeability coefficient).

[0613] Adjacency Matrix stores 2000×2000 edge weights (sparse matrix storage format COO).

[0614] ‌Message Passing‌:

[0615] The ‌GATv2‌ model (Graph Attention Network v2) is used, and each node aggregates neighbor information through the attention mechanism.

[0616] Key parameters: Number of Attention Heads: 8, Hidden Layer dimension: 128.

[0617] Example of a propagation rule: Node G23_11 (the center of the fall event) sends a message to its eight neighbors. The message content equals node feature × event weight × attention score. After receiving the message, the neighboring nodes update their own states.

[0618] ‌ Association influence diagram output‌:

[0619] After three layers of GNN iteration, the node influence value (0-1 floating point number) and event propagation path (Propagation Path) are generated.

[0620] Impact map visualization: Overlay the carriage map with a heat map. Red highlighted areas (impact value > 0.7) indicate that the event has triggered chain congestion, while blue areas (< 0.3) indicate that they are unaffected.

[0621] Structured data: Stored as an edge list. Each record contains [source node ID, target node ID, influence strength, and propagation delay in milliseconds].

[0622] Based on the correlation influence diagram, the monitoring indicator aggregation algorithm is used to integrate the congestion gradient, abnormal event density and passenger flow throughput to generate a monitoring score that reflects the overall health of the real-time passenger flow status.

[0623] Aggregation Algorithm integrates three core indicators:

[0624] Crowding Gradient

[0625] Calculate the standard deviation of congestion between the current time and the previous 5 minutes (60 time slices), denoted as ΔC.

[0626] Dynamic threshold: ΔC>15 (exponential fluctuation per unit time>15%) is considered a drastic change and is assigned a risk factor (RG) = 1.3; otherwise, RG = 1.0.

[0627] ‌Event Density‌:

[0628] Count the number of events per square meter (Events / m 2 For example, if there are two level 2 events within a 3-meter radius, the area is approximately 28m. 2 , density = 0.071.

[0629] Density classification: low density <0.05 (coefficient ED=0.8), medium density 0.05-0.1 (ED=1.0), high density >0.1 (ED=1.5).

[0630] Passenger Throughput:

[0631] The efficiency ratio calculated based on the door counter is: the actual number of people getting off the bus / the theoretical maximum capacity (e.g., a door width of 1 meter supports 2 people per second).

[0632] When the efficiency ratio is <0.5, it is marked as congested (coefficient TP=1.4), and when it is >0.8, it is marked as smooth (TP=0.7).

[0633] ‌Health score calculation process‌:

[0634] ‌Sub-score generation‌:

[0635] Crowding sub-score (Sub_C) = Current Crowding Index × RG / 100 (normalized to 0-1).

[0636] Event sub-score (Sub_E) = min(1.0, event density × ED × highest event level).

[0637] Throughput sub-score (Sub_T) = (1 - Efficiency Ratio) × TP.

[0638] Weighted Fusion:

[0639] Health Score = 40% × Sub_C + 40% × Sub_E + 20% × Sub_T (weights can be adjusted dynamically).

[0640] ‌Normalized Output‌:

[0641] Final monitoring score = min(100, max(0, (1 - Health Score) × 100)), where higher values ​​indicate better health.

[0642] Example: For the rear door area, Sub_C = 0.6 (current index 60), Sub_E = 0.4 (2 secondary events), and Sub_T = 0.3 (efficiency ratio 0.7). The health score is 0.4 × 0.6 + 0.4 × 0.4 + 0.2 × 0.3 = 0.46, and the monitoring score is 54 points (100 - 46).

[0643] The monitoring blind spot data is filled in through the three-dimensional spatial interpolation algorithm, and the monitoring score is mapped to the visualization interface in combination with the carriage layout model. A real-time monitoring report containing heat maps, warning marks and traffic statistics is output.

[0644] Blind Spot Interpolation: This feature addresses camera-blocked areas (such as behind pillars) and sensor failure grids.

[0645] ‌Interpolation algorithm selection‌:

[0646] ‌Radial Basis Function (RBF)‌ uses valid mesh data as anchor points to generate smooth surfaces based on spatial distance.

[0647] Key parameters: Kernel function selects Gaussian RBF, and influence radius r = 1.2 meters (covering 6 adjacent grids).

[0648] Interpolation weight: The effective grid weight is 1.0, and the blind area grid weight decays based on distance (e.g., a distance of 0.6 meters gives a weight of 0.5). For example, the congestion level of the blind area G40_12 behind the pillar is calculated by averaging the values ​​of the six surrounding grids, including G39_12 and G40_11.

[0649] ‌Carriage Layout Model Integration‌:

[0650] Pre-loaded ‌carriage CAD model‌ (OBJ format), including accurate 3D coordinates of seats, handrails, and doors.

[0651] Monitoring score mapping rules:

[0652] The seat area displays the ride comfort level (scores >70 are green, <40 are red).

[0653] The standing area displays a density heatmap (colored by grid score).

[0654] The door area is overlaid with throughput efficiency arrows (green ↑ indicates smooth flow, red ↓ indicates congestion).

[0655] ‌Visualization interface and report generation‌:

[0656] ‌Real-time 3D Dashboard‌ (3D Dashboard):

[0657] The UE4 engine renders the car model and dynamically updates the heat map (5 frames per second).

[0658] Abnormal events are marked with a flashing icon (at the position of the fall icon [2.3, 1.1, 0.5] meters).

[0659] ‌Monitoring report elements‌:

[0660] Heatmap layer: 80×25 grid rendered in pseudo-color, with support for zooming in to view local density.

[0661] ‌Warning Mark‌: A three-level warning icon (red / yellow / blue triangle) marks the event location. Click to pop up details (e.g., "14:05:23 Backdoor Fall Level 2").

[0662] Traffic Statistics Panel:

[0663] Real-time passenger flow: number of people in the vehicle (e.g., 42 people), full load rate (e.g., 82%).

[0664] Efficiency indicators: average boarding and alighting time (seconds / person), congestion duration (minutes).

[0665] Health Overview: The dashboard displays the monitoring score (e.g. 54 / 100) and the historical trend curve (30 minutes).

[0666] Output Mechanism:

[0667] The report is generated in PDF / HTML format every 30 seconds and uploaded to the cloud dispatch center via the 4G network.

[0668] The LED screen inside the car scrolls to display key indicators (only the health score and the highest level of warning are displayed).

[0669] This step constructs a multi-dimensional data association model, analyzes the impact of abnormal events on the overall passenger flow status, and completes the data in the monitoring blind area through a spatial interpolation algorithm. Finally, a visual report including a heat map, event markers, and flow statistics is generated to intuitively display the real-time status of each area of the carriage, forming a closed-loop monitoring system to provide decision support for operation management. The visual interface helps the driver quickly grasp the overall situation of the vehicle and take timely measures for evacuation or emergency response.

[0670] Furthermore, in practical applications, digital signature technology can also be used to encrypt electronic evidence containing archived bus passenger flow monitoring results and related passenger flow data, generate unique data fingerprints, and synchronize the original electronic evidence and data fingerprints to a distributed evidence storage system through a multi-node storage mechanism in different locations to generate a complete and auditable monitoring report chain.

[0671] ‌1. Digital Signature Encryption and Data Fingerprint Generation‌

[0672] ‌Technical Process:‌

[0673] ‌Archiving and Encapsulation of Electronic Evidence‌

[0674] The system automatically packs the real-time generated bus passenger flow monitoring results (including dynamic congestion index, abnormal event warning signal, passenger flow heat map) and original passenger flow data (skeletal key point trajectories, pressure sensing values) into an electronic evidence package (Evidence Package) according to a time window (such as every 5 minutes).

[0675] Evidence Package Structure:

[0676] Packet Header: Timestamp (precision in milliseconds, such as 2025-03-15 08:23:17.342), vehicle ID (such as BUS-JingA12345), route number (such as Route No. 468).

[0677] Data Body:

[0678] Monitoring Results: Store the dynamic congestion index (value range 0-100), abnormal event type (such as "fall", "crowding conflict"), and location coordinates (such as (carriage X: 2.5m, Y: 1.2m)) in JSON format.

[0679] Original Data: Store the skeletal key point coordinates (25 key points per person, precision ±5 cm) and infrared thermal distribution map (resolution 640×480 pixels) in binary stream.

[0680] ‌Generating Data Fingerprint with Digital Signature‌

[0681] Encryption Algorithm Selection: Use the SHA-256 algorithm (Secure Hash Algorithm 256) to calculate the hash value of the electronic evidence package and generate a data fingerprint. The fingerprint is a 64-bit hexadecimal string (such as 9f86d081...c3b4ea3).

[0682] ‌Digital Signature Process‌:

[0683] The data fingerprint is signed using RSA-2048 asymmetric encryption (Rivest-Shamir-Adleman, a public key encryption algorithm):

[0684] Private key encryption: Use the bus company's private key (Private Key, length 2048 bits) to encrypt the data fingerprint and generate a digital signature.

[0685] Public key decryption: Supervisors can use the public key to verify the authenticity of the signature.

[0686] Binding signatures and evidence: combining digital signatures, data fingerprints, and electronic evidence packages into a Verifiable Evidence Unit.

[0687] Uniqueness guarantee: The probability of data fingerprint collision is less than 10 -77 (That is, theoretically, there will be no duplication in all global data.) Example: The data fingerprint of a sudden braking incident evidence package is a1b2c3...d4e5f6, which is signed and stored in the unit.

[0688] ‌2. Remote multi-node storage and synchronization mechanism‌

[0689] ‌Technical process:‌

[0690] Distributed storage architecture design

[0691] Three types of storage nodes:

[0692] Edge Node: Deployed on an industrial computer (industrial-grade solid-state drive, 1TB capacity) on a bus to temporarily store 72 hours of original evidence.

[0693] Regional Center Node: Located in the bus dispatch center (server cluster, storage array capacity 100TB), it stores 30 days of full evidence.

[0694] Disaster Recovery Node: Set up in a geographically isolated data center (such as a remote city), permanently store data fingerprints and signatures.

[0695] Node Interconnection Protocol: Use the TLS 1.3 encrypted channel (Transport Layer Security, a transport layer security protocol) to transmit data, and use the ECDHE algorithm (Elliptic-Curve Diffie-Hellman Ephemeral, an ephemeral key exchange based on elliptic curves) for key exchange.

[0696] Multi-Node Synchronization Mechanism

[0697] Synchronization Trigger Condition: Automatically upload through the 5G network when the vehicle enters the station (signal strength > -90 dBm).

[0698] Event-Driven Upload: Immediately trigger when an abnormal event is detected (such as crowding index > 85).

[0699] Incremental Synchronization Algorithm: The edge node calculates the data fingerprint of the new evidence package and compares it with the fingerprint library of the central node; only transmit unsynchronized evidence packages (saving more than 70% of bandwidth); the disaster recovery node only receives data fingerprints and signatures (single transmission volume < 1KB).

[0700] Data Consistency Guarantee: Use the RAFT consensus protocol (a distributed consistency algorithm) to ensure data consistency among three nodes and tolerate single-node failures. Example: Vehicle with license plate number Jing A12345 uploads data at 08:30, and the synchronization delay of the three nodes < 500 milliseconds.

[0701] 3. Construction of Monitoring Report Chain and Audit Trail

[0702] Technical Process:

[0703] Design of the Structure of the Evidence Chain

[0704] Blockchain-Type Storage: Each verifiable evidence unit serves as an Evidence Block, containing: the hash of the current block (the data fingerprint of this block); the hash of the previous evidence block (forming a chain structure); timestamp, vehicle location (GPS coordinates). The first block is the Genesis Block, storing the initial registration information of the vehicle (such as Vehicle ID: BUS-Jing A12345, Registration Time: 2024-01-01).

[0705] ‌Proof chain generation‌: Link the proof blocks in chronological order to generate an immutable report chain (AuditChain). Example: The 1024th proof block records the "fall incident at 08:25", and the hash of its previous block points to the 1023rd block (recording passenger flow data at 08:20).

[0706] ‌Audit tracking mechanism‌

[0707] ‌Query interface‌: The regulatory party inputs the time range and vehicle ID, and the system returns the complete report chain (e.g., query 2025-03-15 08:00-09:00 / BUS-Jing A12345 → return 12 proof blocks).

[0708] ‌Tampering detection‌:

[0709] When verifying any proof block:

[0710] Recalculate the fingerprint of the current block data using SHA-256; compare it with the stored fingerprint (if inconsistent, trigger an alarm); check the hash link of adjacent blocks (if broken, mark the tampering location).

[0711] ‌Proof of evidence integrity‌: Generate an RFC3161 timestamp (an international standard timestamp protocol), signed by the National Time Service Center (such as TSA.cn), to prove that the evidence existed at a specific time.

[0712] It can be seen that by synchronously collecting the skeletal key points, thermal distribution, and contour features of passengers through on-vehicle multi-view cameras and infrared sensors, a passenger spatio-temporal feature matrix is generated; based on the passenger spatio-temporal feature matrix, a set of passenger movement trajectories is output; according to the residence duration and spatial distribution density of passengers in the set of passenger movement trajectories, a dynamic crowding index is generated; based on the dynamic crowding index, the probability of abnormal behavior is calculated, and an early warning signal for abnormal events is output; by fusing the dynamic crowding index, the early warning signal for abnormal events, and the real-time passenger flow data, the bus passenger flow monitoring result is generated through spatio-temporal correlation analysis, thereby enabling high-precision, low-latency, and intelligent monitoring of the passenger flow state and abnormal early warning.

[0713] Another embodiment of the present invention provides a bus passenger flow monitoring system based on human body detection, see Figure 3 , the system may include:

[0714] An extraction module 301, configured to synchronously collect the skeletal key points, thermal distribution, and contour features of passengers through on-vehicle multi-view cameras and infrared sensors, extract spatio-temporal correlation features using a multi-modal feature fusion network, and generate a passenger spatio-temporal feature matrix;

[0715] A positioning module 302 is configured to separate and locate overlapping passengers based on the passenger spatiotemporal feature matrix using a spatiotemporal attention-optimized human detection model, eliminate environmental shadow interference through a dynamic trajectory tracking algorithm, and output a set of passenger motion trajectories;

[0716] A construction module 303 is configured to construct a physical space occupancy mapping model based on the dwell time and spatial distribution density of passengers in the passenger trajectory set, combined with a three-dimensional posture estimation model and pressure sensor data, and generate a dynamic congestion index;

[0717] Output module 304 is configured to analyze the frequency of passenger body movements and the location characteristics of personal belongings using a multi-task learning model based on the dynamic crowding index, calculate the probability of abnormal behavior, and output an abnormal event warning signal;

[0718] The monitoring module 305 is used to integrate the dynamic congestion index, the abnormal event warning signal and the real-time passenger flow data, and generate a public transportation passenger flow monitoring result through spatiotemporal correlation analysis.

[0719] It can be seen that the on-board multi-camera and infrared sensor synchronously collect the passengers' skeletal key points, thermal distribution and contour features to generate the passenger spatiotemporal feature matrix; based on the passenger spatiotemporal feature matrix, the passenger motion trajectory set is output; according to the passengers' stay time and spatial distribution density in the passenger motion trajectory set, the dynamic congestion index is generated; based on the dynamic congestion index, the probability of abnormal behavior is calculated, and the abnormal event warning signal is output; the dynamic congestion index, abnormal event warning signal and real-time passenger flow data are integrated to generate bus passenger flow monitoring results through spatiotemporal correlation analysis, thereby realizing high-precision, low-latency and intelligent passenger flow status monitoring and abnormal warning.

[0720] An embodiment of the present invention further provides a storage medium storing a computer program, wherein the computer program is configured to execute the steps of any one of the above method embodiments when running.

[0721] Specifically, in this embodiment, the above-mentioned storage medium may be configured to store a computer program for performing the following steps:

[0722] S201, using the on-board multi-camera and infrared sensor to synchronously collect the passenger's skeletal key points, thermal distribution, and contour features, and using a multimodal feature fusion network to extract spatiotemporal correlation features to generate a passenger spatiotemporal feature matrix;

[0723] S202, based on the passenger spatiotemporal feature matrix, a human detection model optimized for spatiotemporal attention is used to separate and locate overlapping passengers, and a dynamic trajectory tracking algorithm is used to eliminate environmental shadow interference, and a set of passenger motion trajectories is output;

[0724] S203, constructing a physical space occupancy mapping model based on the passenger's dwell time and spatial distribution density in the passenger trajectory set, combined with a three-dimensional posture estimation model and pressure sensor data, and generating a dynamic congestion index;

[0725] S204, based on the dynamic congestion index, analyzing the frequency of passengers' body movements and the characteristics of the space occupied by their belongings through a multi-task learning model, calculating the probability of abnormal behavior, and outputting an abnormal event warning signal;

[0726] S205, integrating the dynamic congestion index, the abnormal event warning signal and the real-time passenger flow data, and generating a public transportation passenger flow monitoring result through spatiotemporal correlation analysis.

[0727] It can be seen that the on-board multi-camera and infrared sensor synchronously collect the passengers' skeletal key points, thermal distribution and contour features to generate the passenger spatiotemporal feature matrix; based on the passenger spatiotemporal feature matrix, the passenger motion trajectory set is output; according to the passengers' stay time and spatial distribution density in the passenger motion trajectory set, the dynamic congestion index is generated; based on the dynamic congestion index, the probability of abnormal behavior is calculated, and the abnormal event warning signal is output; the dynamic congestion index, abnormal event warning signal and real-time passenger flow data are integrated to generate bus passenger flow monitoring results through spatiotemporal correlation analysis, thereby realizing high-precision, low-latency and intelligent passenger flow status monitoring and abnormal warning.

[0728] An embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any one of the above method embodiments.

[0729] Specifically, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0730] Specifically, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0731] S201, using the on-board multi-camera and infrared sensor to synchronously collect the passenger's skeletal key points, thermal distribution, and contour features, and using a multimodal feature fusion network to extract spatiotemporal correlation features to generate a passenger spatiotemporal feature matrix;

[0732] S202, based on the passenger spatiotemporal feature matrix, a human detection model optimized for spatiotemporal attention is used to separate and locate overlapping passengers, and a dynamic trajectory tracking algorithm is used to eliminate environmental shadow interference, and a set of passenger motion trajectories is output;

[0733] S203, constructing a physical space occupancy mapping model based on the passenger's dwell time and spatial distribution density in the passenger trajectory set, combined with a three-dimensional posture estimation model and pressure sensor data, and generating a dynamic congestion index;

[0734] S204, based on the dynamic congestion index, analyzing the frequency of passengers' body movements and the characteristics of the space occupied by their belongings through a multi-task learning model, calculating the probability of abnormal behavior, and outputting an abnormal event warning signal;

[0735] S205, integrating the dynamic congestion index, the abnormal event warning signal and the real-time passenger flow data, and generating a public transportation passenger flow monitoring result through spatiotemporal correlation analysis.

[0736] It can be seen that the on-board multi-camera and infrared sensor synchronously collect the passengers' skeletal key points, thermal distribution and contour features to generate the passenger spatiotemporal feature matrix; based on the passenger spatiotemporal feature matrix, the passenger motion trajectory set is output; according to the passengers' stay time and spatial distribution density in the passenger motion trajectory set, the dynamic congestion index is generated; based on the dynamic congestion index, the probability of abnormal behavior is calculated, and the abnormal event warning signal is output; the dynamic congestion index, abnormal event warning signal and real-time passenger flow data are integrated to generate bus passenger flow monitoring results through spatiotemporal correlation analysis, thereby realizing high-precision, low-latency and intelligent passenger flow status monitoring and abnormal warning.

[0737] The above describes in detail the structure, features and effects of the present invention based on the embodiments shown in the drawings. The above is only a preferred embodiment of the present invention, but the scope of implementation of the present invention is not limited to what is shown in the drawings. Any changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments with equivalent changes, which do not exceed the spirit covered by the description and drawings, should be within the scope of protection of the present invention.

Claims

1. A method for monitoring public transportation passenger flow based on human body detection, characterized in that: The method comprises: The on-board multi-camera and infrared sensor synchronously collect the passenger's skeletal key points, thermal distribution and contour features, and use the multimodal feature fusion network to extract spatiotemporal correlation features to generate the passenger spatiotemporal feature matrix; Based on the passenger spatiotemporal feature matrix, a human detection model optimized for spatiotemporal attention is used to separate and locate overlapping passengers, and a dynamic trajectory tracking algorithm is used to eliminate environmental shadow interference and output a set of passenger motion trajectories; Based on the length of stay and spatial distribution density of passengers in the passenger motion trajectory set, a three-dimensional posture estimation model and pressure sensor data are combined to construct a physical space occupancy mapping model to generate a dynamic congestion index. Specifically, based on the passenger motion trajectory set, the length of stay and movement path of each passenger in the carriage are counted, and kernel density estimation is used to generate a spatial distribution heat map. The spatial distribution heat map is input into the three-dimensional posture estimation model to analyze the passenger's sitting and standing postures and luggage carrying status, and output an individual space occupancy rate prediction value. The space occupancy rate prediction value is dynamically calibrated by integrating real-time load data collected by the pressure sensor to generate an accurate space occupancy distribution matrix. Using a congestion calculation function, the entropy value characteristics of the spatial distribution heat map and the weighted summation result of the space occupancy distribution matrix are integrated to output a standardized dynamic congestion index. Based on the dynamic crowding index, a multi-task learning model is used to analyze the frequency of passengers' body movements and the characteristics of the space occupied by their belongings, calculate the probability of abnormal behavior, and output an abnormal event warning signal; The dynamic congestion index, the abnormal event warning signal and the real-time passenger flow data are integrated to generate a public transportation passenger flow monitoring result through spatiotemporal correlation analysis.

2. The method according to claim 1, characterized in that The method uses a multi-camera and infrared sensor to synchronously collect the key points, thermal distribution and contour features of the passenger's skeleton, and uses a multimodal feature fusion network to extract spatiotemporal correlation features to generate a passenger spatiotemporal feature matrix, including: Based on the RGB video stream captured by multiple cameras, a lightweight skeleton key point detection model is used to extract the coordinates of multiple key points of the passenger and generate a skeleton topology map. The skeleton topology map is input into the thermal distribution alignment module, the thermal imaging data of the infrared sensor is processed synchronously, and the passenger thermal distribution mask is generated through the threshold segmentation algorithm; Based on the skeleton topology map and heat distribution mask, a contour tracking algorithm is used to extract the passenger contour binary image and generate a set of multimodal feature maps; The multimodal feature map is input into the spatiotemporal convolutional fusion network, and the skeletal motion, thermal radiation intensity and contour geometric features are dynamically weighted through the cross-modal attention mechanism to output the passenger spatiotemporal feature matrix.

3. The method according to claim 2, characterized in that Based on the passenger spatiotemporal feature matrix, a human detection model optimized for spatiotemporal attention is used to separate and locate overlapping passengers, and a dynamic trajectory tracking algorithm is used to eliminate environmental shadow interference and output a set of passenger motion trajectories, including: The passenger spatiotemporal feature matrix is ​​input into the spatiotemporal attention module to generate an attention weight matrix representing the saliency of passenger movement; The pixel features of the overlapping areas are reweighted based on the attention weight matrix, and the YOLOv8 model is used to output the separated passenger location bounding box coordinates; The passenger location bounding box coordinates and heat distribution mask are input into the shadow discriminator, and the environmental shadow artifacts are filtered out through the thermal radiation intensity comparison algorithm to generate a purified passenger location sequence; The cleaned passenger position sequence is used to initialize the trajectory tracker, and the three-dimensional space trajectory is calculated by combining the multi-camera disparity data to generate a preliminary motion trajectory set. The trajectory jump points in the preliminary motion trajectory set are corrected by the acceleration constrained Kalman filter algorithm, and the passenger motion trajectory set with time stamps is output.

4. The method according to claim 3, characterized in that Based on the dynamic congestion index, the multi-task learning model is used to analyze the frequency of passengers' body movements and the characteristics of the space occupied by their belongings, calculate the probability of abnormal behavior, and output an abnormal event warning signal, including: Extract the optical flow features of body movements from the passenger's spatiotemporal feature matrix, and use sliding window Fourier transform to generate the movement frequency spectrogram; Based on the motion frequency spectrum and dynamic crowding index, the abnormal frequency detection algorithm is used to screen abnormal limb motion segments and generate a preliminary abnormal candidate set; The Mask R-CNN model is used to segment the personal belongings area, and the item type classifier is used to calculate the occupied area ratio and generate the item occupancy feature vector. The preliminary anomaly candidate set and item occupancy feature vectors are input into the multi-task learning network. The main network predicts the anomaly probability, and the auxiliary network reconstructs the normal behavior pattern and outputs a comprehensive anomaly score. Based on the comparison results of the comprehensive anomaly score and the standard deviation of the historical baseline data, a graded warning signal is triggered, and the type and location coordinates of the abnormal event are output.

5. The method according to claim 4, characterized in that The integration of the dynamic congestion index, the abnormal event warning signal and the real-time passenger flow data to generate a public transport passenger flow monitoring result through spatiotemporal correlation analysis includes: The dynamic congestion index is spatiotemporally aligned with real-time boarding and alighting passenger flow data to construct a multi-dimensional spatiotemporal matrix of passenger flow status; The location and impact range of abnormal events are marked in the passenger flow state spatiotemporal matrix, and the propagation relationship between events and congestion is modeled using a graph neural network to generate a correlation influence diagram. Based on the correlation influence diagram, the monitoring indicator aggregation algorithm is used to integrate the congestion gradient, abnormal event density and passenger flow throughput to generate a monitoring score that reflects the overall health of the real-time passenger flow status. The monitoring blind spot data is filled in through the three-dimensional spatial interpolation algorithm, and the monitoring score is mapped to the visualization interface in combination with the carriage layout model. A real-time monitoring report containing heat maps, warning marks and traffic statistics is output.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Digital signature technology is used to encrypt electronic evidence containing archived bus passenger flow monitoring results and related passenger flow data to generate a unique data fingerprint. The original electronic evidence and data fingerprint are then synchronized to the distributed evidence storage system through a multi-node storage mechanism in a different location to generate a complete and auditable monitoring report chain.

7. A public transportation passenger flow monitoring system based on human body detection, characterized in that: The system comprises: The extraction module is used to synchronously collect the skeletal key points, thermal distribution, and contour features of passengers through the on-board multi-camera and infrared sensor, and use the multimodal feature fusion network to extract spatiotemporal correlation features to generate the passenger spatiotemporal feature matrix; A positioning module is used to separate and locate overlapping passengers based on the passenger spatiotemporal feature matrix using a human detection model optimized with spatiotemporal attention, eliminate environmental shadow interference through a dynamic trajectory tracking algorithm, and output a set of passenger motion trajectories; A construction module is used to construct a physical space occupancy mapping model based on the length of stay and spatial distribution density of passengers in the passenger motion trajectory set, combined with a three-dimensional posture estimation model and pressure sensor data, to generate a dynamic congestion index. Based on the passenger motion trajectory set, the length of stay and movement path of each passenger in the carriage are counted, and a spatial distribution heat map is generated using kernel density estimation. The spatial distribution heat map is input into the three-dimensional posture estimation model to analyze the passenger's sitting and standing postures and luggage carrying status, and output individual space occupancy rate prediction values. The space occupancy rate prediction values ​​are dynamically calibrated by integrating real-time load data collected by the pressure sensor to generate an accurate space occupancy distribution matrix. Using a congestion calculation function, the entropy value characteristics of the spatial distribution heat map and the weighted summation result of the space occupancy distribution matrix are integrated to output a standardized dynamic congestion index. An output module is configured to analyze the frequency of passengers' body movements and the location characteristics of their belongings using a multi-task learning model based on the dynamic crowding index, calculate the probability of abnormal behavior, and output an abnormal event warning signal; The monitoring module is used to integrate the dynamic congestion index, the abnormal event warning signal and the real-time passenger flow data, and generate a public transportation passenger flow monitoring result through spatiotemporal correlation analysis.

8. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 6 when executed.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 6.

Images