Passenger track construction method and system

Abstract

The invention relates to a passenger track construction method and system, and the method comprises the steps: carrying out the time-space synchronization of a high-definition network camera and a 3D millimeter-wave radar, and generating preprocessed RGB video data and 3D millimeter-wave radar point cloud data; extracting a multi-scale feature map and local geometric features of the point cloud, and completing multi-modal feature fusion through transformer; based on the fusion features, multi-target tracking and trajectory management are realized through a Hungary algorithm and Kalman filtering; and cross-camera track continuing is optimized through the Re-ID network, and passenger track data is generated. According to the scheme, the tracking performance is excellent, the MOTA is 85.5%, and the IDF1 after shielding is 91.2%; carrying out ID-free switching across multiple cameras, and generating a three-dimensional track; alarm is advanced by 42 seconds, track backtracking speed is increased by 300 times, low illumination is stable, and transformation cost is low.

Description

Technical Field

[0001] This application relates to the field of security and operation management of public transportation hubs, and in particular to a method and system for constructing passenger tracking information. Background Technology

[0002] In security systems at airports, train stations, and subway stations, existing technologies largely rely on single sensors or multi-sensor fusion solutions that are not adapted to the scenarios, resulting in significant limitations. While pure video analytics is widely used, employing models like YOLO and Faster R-CNN for target detection and Kalman filtering and Hungarian algorithms for tracking, it is susceptible to changes in lighting and occlusion, experiencing a sharp performance drop in low light or densely populated conditions. Furthermore, 2D video lacks depth information, limiting the accuracy of trajectory localization, and cross-camera tracking is prone to ID switching or loss. Pure millimeter-wave radar technology can achieve target localization, but due to sparse point cloud data and low angular resolution, it struggles to provide semantic information, failing to distinguish between similar targets or identify fine movements, thus limiting its application value.

[0003] Existing multi-sensor fusion solutions are primarily geared towards autonomous driving and are not optimized for security inspection scenarios. They suffer from limited fusion levels, often at the data or decision level, failing to fully leverage the advantages of modal complementarity and lacking long-term dependency modeling. Furthermore, they struggle to re-identify passengers after prolonged occlusion, and their high computational complexity makes them unsuitable for the real-time demands of security inspections. To address these shortcomings, there is an urgent need for a lightweight technical solution that enables all-weather, robust tracking, deep fusion of multi-modal data, and modeling of long-term dependencies, thus resolving the core challenge of trajectory construction in security inspection scenarios. Summary of the Invention

[0004] In view of this, this application proposes a method for constructing passenger travel routes, including: The passenger information sensors, including high-definition network cameras and 3D millimeter-wave radar, are synchronized in time and space. Pre-processed passenger RGB video data and pre-processed passenger 3D millimeter-wave radar point cloud data are generated through the passenger information sensors. Based on preprocessed passenger RGB video data and preprocessed passenger 3D millimeter-wave radar point cloud data, multi-scale feature maps and local geometric features of point cloud data are extracted. Based on the multi-scale feature maps and the local geometric features of the point cloud data, multi-modal feature fusion is performed using a transformer to generate fused feature data. Based on the fused feature data, multi-target tracking and trajectory management are performed using the Hungarian algorithm and Kalman filtering to generate tracking and trajectory management data. Based on tracking and trajectory management data, cross-camera trajectory continuity and optimization are performed through the Re-ID network to generate passenger trajectory data.

[0005] In one possible implementation, temporal and spatial synchronization of the passenger information sensor is performed, and preprocessed passenger RGB video data and preprocessed passenger 3D millimeter-wave radar point cloud data are generated through the passenger information sensor, including: Microsecond-level time synchronization of all sensors is achieved through a precision clock synchronization protocol; The Zhang Zhengyou calibration method was used to obtain the camera's intrinsic and extrinsic parameters, and the camera's coordinate system was integrated into the global coordinate system of the security checkpoint. The relative positional relationship between the millimeter-wave radar and the camera is obtained by calibration board, and a 3D-2D mapping relationship is established to transfer the millimeter-wave radar coordinate system to the global coordinate system of the security check channel. Denoising and brightness normalization are performed on passenger RGB video data to generate preprocessed passenger RGB video data; Dynamic target clustering and multipath interference suppression are performed on 3D millimeter-wave point cloud data to generate preprocessed passenger 3D millimeter-wave radar point cloud data.

[0006] In one possible implementation, extracting multi-scale feature maps and local geometric features of the point cloud data based on preprocessed passenger RGB video data and preprocessed passenger 3D millimeter-wave radar point cloud data includes: Multi-scale feature maps are extracted from video frames of preprocessed passenger RGB video data using a CNN backbone network; The feature map is flattened into a sequence, sinusoidal positional encoding is added, and long-range dependencies are modeled through a multi-layer self-attention mechanism of a transformer encoder. The PointNet++ network was used to perform feature encoding on the preprocessed passenger 3D millimeter-wave radar point cloud data to generate encoded point cloud data. 3D sparse convolution is used to extract voxel features from the encoded point cloud data to enhance feature representation and extract the local geometric features of each point in the point cloud.

[0007] In one possible implementation, based on the multi-scale feature map and the local geometric features of the point cloud data, multi-modal feature fusion is performed using a transformer to generate fused feature data, including: The transformer model maps multi-scale feature maps to appearance token sequences and local geometric features of point cloud data to geometric token sequences. Add modal type encoding and position encoding to each appearance token and geometry token; By using the transformer model, we set the appearance token as the query vector and the geometry token as the key vector and value vector to mine complementary information between the two modalities. A learnable weighting factor is introduced to dynamically adjust the contribution of the two modalities; The transformer decoder is used to model the fused features of multiple consecutive frames, capture long temporal dependencies, and generate fused feature data.

[0008] In one possible implementation, based on the fused feature data, multi-target tracking and trajectory management are performed using the Hungarian algorithm and Kalman filtering to generate tracking and trajectory management data, including: Based on the fused feature data, the similarity matrix between the current frame detection result and the historical trajectory is calculated using the Hungarian algorithm; For a successfully matched trajectory, a Kalman filter is used to update the state and predict the position of the next frame.

[0009] If an unmatched detection is detected in multiple consecutive frames, it is initialized as a new trajectory, generating tracking and trajectory management data.

[0010] In one possible implementation, based on tracking and trajectory management data, cross-camera trajectory continuation and optimization are performed via a Re-ID network to generate passenger trajectory data, including: Based on tracking and trajectory management data, a Re-ID network is used to calculate the similarity of passengers under different cameras; When the spatiotemporal constraints are met and the similarity exceeds the threshold, the tracking and trajectory management data are spliced ​​together to generate stitched data. Bézier curves are used to smooth the spliced ​​data of the original track points to generate passenger track data. The accuracy of passenger movement data is improved by fusing observation data from multiple cameras using graph optimization methods.

[0011] In one possible implementation, video feature extraction employs an improved ResNet-IBN network, where the feature extraction formula is: F_v=CNN_v(I)={f_v¹,f_v²,...,f_vᴺ} Where F_v is the original feature set of the video, CNN_v ​​is the CNN feature extraction network specifically for video, I represents the video image data, f_v¹, f_v², ..., f_vᴺ represents the features of a single frame / segment of video, and the features of the i-th layer are calculated as follows: f_v 1 =BN(Conv3×3(ReLU(BN(Conv3×3(f_v 1 ⁻¹))))) Where f_v 1 f_v represents the video features of the current layer. 1 ⁻¹ represents the video features of the previous layer, BN represents batch normalization, Conv3×3 is a 3×3 convolution operation, and ReLU is a linear rectified activation function.

[0012] In one possible implementation, the formula for generating multimodal tokens is: Token_v=Linear(F_v)+PE_pos+ME_v Token_p=Linear(F_p)+PE_pos+ME_p Wherein, Token_v is the video modality token, Token_p is the millimeter-wave point cloud modality token, F_v is the original video feature, F_p is the original millimeter-wave point cloud feature, PE_pos is the position code, ME_v is the video modality type code, and ME_p is the millimeter-wave point cloud modality type code.

[0013] In one possible implementation, the location coding calculation formula is: PE(pos,2i)=sin(pos / 10000^(2i / d)) Where pos is the position index, 2i is the even index of the feature dimension, and d is the total feature dimension of the token.

[0014] The present invention also includes a passenger trajectory construction system for implementing the above method, comprising a data preprocessing and alignment module, a multimodal feature extraction module, a transformer feature fusion module, and a multi-target tracking and trajectory management module; The data preprocessing and alignment module performs time synchronization, spatial coordinate system unification, and data cleaning on multi-source data.

[0015] The multimodal feature extraction module extracts the appearance features of the video data and the geometric features of the millimeter-wave point cloud, respectively.

[0016] The transformer feature fusion module utilizes an improved transformer architecture for cross-modal feature interaction and enhancement.

[0017] The multi-target tracking and trajectory management module performs data association, state estimation, and trajectory generation based on fused features.

[0018] The beneficial effects of this invention are: In terms of tracking performance, multimodal fusion enables the system to maintain high accuracy even under harsh conditions. The multi-target tracking accuracy on the test set reaches 85.5%, which is 17.3% higher than the pure video method and 13.4% higher than the pure millimeter wave method. At the same time, the long-term modeling of the transformer effectively solves the occlusion problem. When the passenger reappears after 3-5 seconds of occlusion, the identity retention rate reaches 91.2%, which far exceeds the 72.8% of the traditional Kalman filter method.

[0019] In terms of trajectory construction, multi-camera collaboration and Re-ID technology enable seamless cross-regional tracking. In a test at an airport, the system completed trajectory tracking across 6 cameras without any identity switching, while traditional methods averaged 2.3 ID switchings. Furthermore, by integrating depth information provided by millimeter-wave point clouds, it generates true 3D trajectories containing x, y, and z coordinates, which can accurately detect actions such as bending over and raising hands, laying a data foundation for abnormal behavior analysis.

[0020] In terms of security check optimization, the system can detect abnormal behaviors such as walking in the wrong direction, loitering, and running in real time and automatically alarm. In a test at a subway station, when a passenger was detected loitering next to the security check machine for more than 3 minutes, the automatic alarm time was 42 seconds earlier than when discovered manually. When a security incident such as leaving items behind occurs, the system can trace the complete movements of the relevant passenger within 5 seconds, which is more than 300 times more efficient than the traditional manual review of multiple video recordings.

[0021] The proposed feature-enhanced transformer fusion module overcomes the shortcomings of traditional decision-level fusion in fully leveraging modal complementarity, achieving deep fusion at both spatial and contextual levels. Furthermore, the entire system employs a unified framework for end-to-end training, avoiding error accumulation in traditional multi-stage methods and improving overall performance.

[0022] The penetrating characteristics of millimeter-wave radar make the system unaffected by light and fog, and it can still work stably under low illumination of 0.01 Lux, which is significantly better than pure vision solutions. At the same time, it supports the upgrading and transformation of existing security inspection and monitoring systems without large-scale hardware replacement. Only the addition of millimeter-wave radar and the deployment of algorithm software are required, which greatly reduces the implementation cost and provides efficient and reliable technical support for intelligent security inspection.

[0023] Other features and aspects of this application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0024] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this application together with the specification and serve to explain the principles of this application.

[0025] Figure 1 A flowchart of a passenger tracking method according to an embodiment of this application is shown; Figure 2 This paper illustrates a flowchart of the spatiotemporal synchronization and multi-source data preprocessing of the passenger information sensor according to an embodiment of this application. Figure 3 A flowchart of multimodal feature extraction according to an embodiment of this application is shown; Figure 4 A flowchart of the transformer multimodal feature fusion process according to an embodiment of this application is shown; Figure 5 A flowchart illustrating the multi-target tracking and trajectory management process according to an embodiment of this application is shown; Figure 6 A flowchart illustrating the passenger tracking optimization process according to an embodiment of this application is shown. Figure 7 A structural diagram of a passenger tracking system according to an embodiment of this application is shown. Detailed Implementation

[0026] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0027] It should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application or to simplify the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0028] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0029] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0030] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0031] The invention of this application is a method and system for constructing passenger travel routes, which is applied in the field of security and operation management of public transportation hubs, and plays a role in security protection, operation optimization, emergency response assistance and service upgrade.

[0032] method specifically refer to Figure 1 , Figure 1 A flowchart of a passenger tracking method according to an embodiment of this application is shown.

[0033] This invention includes a method for constructing passenger tracks, comprising the following steps: Step S100: Synchronizing passenger information sensors in time and space, wherein the passenger information sensors include a high-definition network camera and a 3D millimeter-wave radar; generating preprocessed passenger RGB video data and preprocessed passenger 3D millimeter-wave radar point cloud data through the passenger information sensors; Step S200: Extracting multi-scale feature maps and local geometric features of the point cloud data based on the preprocessed passenger RGB video data and preprocessed passenger 3D millimeter-wave radar point cloud data; Step S300: Performing multi-modal feature fusion using a transformer based on the multi-scale feature maps and local geometric features of the point cloud data to generate fused feature data; Step S400: Performing multi-target tracking and trajectory management using a Hungarian algorithm and Kalman filtering based on the fused feature data to generate tracking and trajectory management data; and Step S500: Performing cross-camera track continuation and optimization using a Re-ID network based on the tracking and trajectory management data to generate passenger track data. This invention utilizes multi-sensor spatiotemporal synchronization, multi-modal feature extraction, transformer fusion, multi-target tracking, and cross-camera optimization to construct passenger tracks in security check scenarios. Its MOTA (Modal-Over-The-Air) rate reaches 85.5%, IDF1 is 91.2%, it eliminates ID switching across six cameras, completes track tracing within 5 seconds, and operates stably in low-light conditions of 0.01 Lux, improving tracking accuracy and robustness, optimizing security check processes, and reducing deployment costs.

[0034] specifically refer to Figure 2 , Figure 2 A flowchart illustrating the spatiotemporal synchronization and multi-source data preprocessing of the passenger information sensor according to an embodiment of this application is shown.

[0035] In one possible implementation, temporal and spatial synchronization of passenger information sensors is performed. This involves generating preprocessed passenger RGB video data and preprocessed passenger 3D millimeter-wave radar point cloud data using the passenger information sensors, including step S101, achieving microsecond-level time synchronization of all sensors through a precision clock synchronization protocol. Step S102, using the Zhang Zhengyou calibration method to obtain the camera's intrinsic and extrinsic parameters, aligning the camera coordinate system to the global coordinate system of the security checkpoint. Step S103, obtaining the relative positional relationship between the millimeter-wave radar and the camera using a calibration board, establishing a 3D-2D mapping relationship, and aligning the millimeter-wave radar coordinate system to the global coordinate system of the security checkpoint. Step S104, performing noise reduction and brightness normalization on the passenger RGB video data to generate preprocessed passenger RGB video data. Finally, step S105, performing dynamic target clustering and multipath interference suppression on the 3D millimeter-wave point cloud data to generate preprocessed passenger 3D millimeter-wave radar point cloud data. Through microsecond-level time synchronization, coordinate unification, and data optimization, the preprocessed RGB video and millimeter-wave point cloud data exhibit high spatiotemporal consistency. Video denoising and normalization improve image quality, while point cloud clustering resists interference and removes false points, laying the foundation for subsequent feature extraction and fusion, and ensuring the reliability of multi-sensor data in security inspection scenarios.

[0036] Specifically, multiple high-definition network cameras with resolutions of 2 megapixels or higher and 3D millimeter-wave radars are evenly deployed at key locations such as the entrance, security scanner, waiting area, and exit of the security checkpoint, forming a comprehensive sensing network. The cameras capture RGB video streams at 25-30fps, while the millimeter-wave radar captures point cloud data at 15-20Hz. Time synchronization employs a precision clock synchronization protocol to achieve microsecond-level time synchronization across all sensors. Spatial alignment is achieved through joint calibration, aligning the camera coordinate system, the millimeter-wave radar coordinate system, and the global coordinate system of the security checkpoint. Specifically, the Zhang Zhengyou calibration method is used to obtain the intrinsic and extrinsic parameters of the cameras, and the relative positional relationship between the millimeter-wave radar and the cameras is obtained through a calibration board, establishing a 3D-2D mapping relationship. Noise reduction and brightness normalization are performed on the video data; dynamic target clustering and multipath interference suppression are implemented on the millimeter-wave point cloud to remove ghost points. The DBSCAN algorithm is used for dynamic target clustering of the millimeter-wave point cloud, and the OS-CFAR algorithm is used for multipath interference suppression.

[0037] In one specific embodiment, the data preprocessing method involves synchronizing the input video frame sequence and point cloud data in time and unifying their spatial coordinates, outputting preprocessed aligned data. The inputs include the video frame sequence, point cloud data, and calibration parameters. First, a time synchronization function is called, inputting the video frame sequence and point cloud data to obtain time-aligned synchronized data. Each item in the synchronized data contains single-frame video data with a corresponding timestamp and a set of point cloud data. Then, a world coordinate system data storage list is initialized. Next, a camera-to-world coordinate system transformation function is called, inputting the video frame data of the current data item and the camera intrinsic parameter matrix from the calibration parameters to obtain video data in the world coordinate system. Then, a radar-to-world coordinate system transformation function is called, inputting the point cloud data of the current data item and the radar transformation parameters from the calibration parameters to obtain point cloud data in the world coordinate system. A world coordinate system data item for the current frame is then constructed, containing video data in the world coordinate system, point cloud data in the world coordinate system, and the timestamp of the current data item. Finally, the world coordinate system data item for the current frame is added to the world coordinate system data storage list, and the preprocessed aligned data list is returned. Finally, the preprocessed aligned data is output in list form, with each element containing video data in world coordinates, point cloud data, and the corresponding timestamp.

[0038] specifically refer to Figure 3 , Figure 3 A flowchart illustrating the multimodal feature extraction process according to an embodiment of this application is shown.

[0039] In one possible implementation, the extraction of multi-scale feature maps and local geometric features of the point cloud data based on preprocessed passenger RGB video data and preprocessed passenger 3D millimeter-wave radar point cloud data includes step S201, extracting multi-scale feature maps from video frames of the preprocessed passenger RGB video data using a CNN backbone network. Step S202, flattening the feature maps into a sequence, adding sinusoidal positional encoding, and modeling long-range dependencies using a multi-layer self-attention mechanism with a transformer encoder. Step S203, using a PointNet++ network to encode features in the preprocessed passenger 3D millimeter-wave radar point cloud data to generate encoded point cloud data. Step S204, using 3D sparse convolution to extract voxel features from the encoded point cloud data to enhance feature representation and extract local geometric features for each point in the point cloud. Multi-scale features are extracted from the video using CNN, and long-range dependencies are modeled using a transformer encoder; the point cloud is encoded with PointNet++, and 3D sparse convolution enhances features and extracts local geometric feature representation. Sufficient dual-modal feature extraction lays the foundation for subsequent fusion and improves feature recognition in security inspection scenarios.

[0040] Preferably, a CNN backbone network such as ResNet50-IBN is used to extract multi-scale feature maps from video frames, and a transformer encoder is introduced to process the feature sequence and capture the spatiotemporal dependencies between frames. Specifically, the feature maps are flattened into sequences, sinusoidal positional encoding is added, and long-range dependencies are modeled through a multi-layer self-attention mechanism. For millimeter-wave point cloud feature extraction, the PointNet++ network encodes the point cloud data and extracts the local geometric features of each point. To address the sparsity problem of point clouds, voxelization is used, and 3D sparse convolution is employed to extract voxel features, enhancing feature representation capabilities.

[0041] specifically refer to Figure 4 , Figure 4 A flowchart of the transformer multimodal feature fusion process according to an embodiment of this application is shown.

[0042] In one possible implementation, multimodal feature fusion is performed using a transformer based on multi-scale feature maps and local geometric features of point cloud data to generate fused feature data. This includes step S301, where the transformer model maps the multi-scale feature maps as appearance token sequences and the local geometric features of the point cloud data as geometric token sequences. Step S302, modality type and position encodings are added to each appearance token and geometric token. Step S303, using the transformer model, appearance tokens are set as query vectors, and geometric tokens are set as key and value vectors to mine complementary information between the two modalities. Step S304, learnable weight factors are introduced to dynamically adjust the contribution of the two modalities. Finally, step S305, a transformer decoder is used to model the fused features across multiple consecutive frames, capturing long-term temporal dependencies and generating fused feature data. The transformer converts bimodal features into tokens and adds encoding to mine complementary information across attention levels; learnable weights dynamically adjust modal contributions; and the decoder models long-term temporal dependencies. The fused features are robust, laying the foundation for subsequent tracking, and maintaining high feature quality even in low-light and other scenarios.

[0043] This invention designs a feature-enhanced Transformer fusion module, in which multimodal token generation maps video features and point cloud features into appearance token sequences and geometric token sequences, respectively. Modality type encoding and location encoding are added to each token to distinguish data source and spatial location. Cross-modal attention fusion employs a multi-head cross-attention mechanism, enabling appearance tokens and geometric tokens to query each other and compute key-value pairs. Specifically, appearance tokens are used as query vectors, and geometric tokens as key and value vectors, mining complementary information between the two modalities. A learnable weight factor is introduced to dynamically adjust the contribution of the two modalities. For example, video features are given higher weight in well-lit conditions, while millimeter-wave features are relied upon in low-light conditions. Temporal feature aggregation utilizes a Transformer decoder to model the fused features of multiple consecutive frames, capturing long-term temporal dependencies, and employs a sliding window mechanism to compute self-attention within the window, balancing computational efficiency and temporal modeling capability.

[0044] In one specific embodiment, the mathematical formula for multi-head cross-attention is: Q_v=Token_v·W_Q,K_p=Token_p·W_K,V_p=Token_p·W_V Where Q_v is the video modal query vector, Token_v is the video modal token sequence, W_Q is the query projection weight matrix, K_p is the millimeter-wave point cloud modal key vector, Token_p is the millimeter-wave point cloud modal token sequence, W_K is the key projection weight matrix, and V_p is the millimeter-wave point cloud modal value vector.

[0045] The complete formula for multimodal fusion is: F_fused=LayerNorm(Token_v+Dropout(CrossAttention(Q_v,K_p,V_p))) F_fused=LayerNorm(F_fused+Dropout(FFN(F_fused))) Where F_fused is the multimodal fusion feature vector, LayerNorm is the layer normalization operation, Token_v represents the video modal token sequence, Dropout is the random deactivation operation, CrossAttention represents the cross-attention mechanism, Q_v represents the video modal query vector, K_p is the millimeter-wave point cloud modal key vector, V_p is the millimeter-wave point cloud modal value vector, and FFN represents the feedforward neural network.

[0046] In one specific embodiment, the dynamic weight adjustment mechanism includes environment-adaptive weight calculation, the formula of which is: w_v=σ(W_env·E+b_env) w_p=1-w_v F_final = w_v·F_v + w_p·F_p Where w_v is the video modal weight coefficient, σ is the Sigmoid activation function, W_env is the environmental feature weight matrix, E is the environmental feature vector, b_env is the environmental feature bias term, w_p is the millimeter-wave point cloud modal weight coefficient, F_final is the final multimodal fusion feature, F_v is the video modal feature, and F_p is the millimeter-wave point cloud modal feature.

[0047] specifically refer to Figure 5 , Figure 5 A flowchart illustrating multi-target tracking and trajectory management according to an embodiment of this application is shown.

[0048] In one possible implementation, multi-target tracking and trajectory management are performed using the Hungarian algorithm and Kalman filtering based on the fused feature data. The generation of tracking and trajectory management data includes step S401, calculating the similarity matrix between the current frame detection result and historical trajectories using the Hungarian algorithm based on the fused feature data; step S402, updating the state of successfully matched trajectories using Kalman filtering to predict the position in the next frame; and step S403, initializing unmatched detections as new trajectories if they are detected in multiple consecutive frames, thus generating tracking and trajectory management data. Based on the fused features, the Hungarian algorithm accurately matches detected and historical trajectories, Kalman filtering updates the trajectory state and predicts the position, and unmatched detections are initialized as new trajectories after multi-frame verification. Multi-target tracking achieves a MOTA (Motion Over Time) of 85.5%, is occlusion resistant, and can establish new vehicle tracks within 5 frames, ensuring efficient and accurate trajectory management in security inspection scenarios.

[0049] This method calculates a similarity matrix between the current frame detection result and historical trajectories based on fused features. The similarity calculation comprehensively considers the cosine distance of appearance features and the Mahalanobis distance of motion features. A Hungarian algorithm is employed for optimal matching to solve the detection-trajectory assignment problem. For successfully matched trajectories, a Kalman filter based on a constant velocity model is used for state updates to predict the position in the next frame. For unmatched detections, if they are detected in multiple consecutive frames, a new trajectory is initialized. A trajectory confidence mechanism is designed to dynamically adjust the confidence level based on trajectory length and matching success rate. When the trajectory confidence level falls below a threshold or exceeds the monitoring area, the trajectory is automatically terminated and the tracking data is output.

[0050] In one specific embodiment, the multi-target tracking method is implemented through two core modules: a multi-target tracker and a 3D Kalman filter. During tracker initialization, an empty trajectory list, initial IDs, and configuration parameters are set. In the update phase, the Kalman filter first predicts the existing trajectory state, then calculates the similarity matrix between the detection results and the trajectories (fusing appearance cosine similarity and motion Mahalanobis distance similarity, dynamically weighted according to detection confidence). Data association is completed using the Hungarian algorithm, resulting in matched pairs, unmatched detections, and trajectories. Subsequently, matched trajectories are updated, and for unmatched detections, new IDs are assigned and the Kalman filter is initialized to create new trajectories. Lost trajectories are deleted, and the updated trajectory is output. The 3D Kalman filter, based on a constant velocity model, initializes state and observation matrices. Through a loop of updating the state and error covariance and correcting the state and covariance using observations, it achieves accurate estimation of the 3D spatial target state, supporting the tracker's motion similarity calculation and trajectory state update.

[0051] specifically refer to Figure 6 , Figure 6 A flowchart illustrating the passenger tracking optimization process according to an embodiment of this application is shown.

[0052] In one possible implementation, passenger trajectory data is generated by performing cross-camera trajectory continuation and optimization using a Re-ID network based on tracking and trajectory management data. This includes step S501: calculating the similarity of passengers across different cameras using the Re-ID network based on the tracking and trajectory management data. Step S502: when the spatiotemporal constraints are met and the similarity exceeds a threshold, trajectory continuation is performed on the tracking and trajectory management data to generate stitched data. Step S503: the stitched data is smoothed using Bézier curves to generate passenger trajectory data. Step S504: multi-camera observation data is fused using graph optimization methods to optimize the accuracy of passenger trajectory data. The Re-ID network calculates passenger similarity across cameras, and continuation of trajectories is performed after meeting spatiotemporal constraints and thresholds; Bézier curves smooth trajectories, and graph optimization improves accuracy by fusing multi-camera data. No ID switching is required across six cameras, resulting in smooth, jitter-free trajectories with accuracy meeting the requirements for 3D trajectories in security checks.

[0053] Specifically, cross-camera re-identification is based on multimodal features (appearance, geometry, motion patterns) and uses a Re-ID network to calculate the similarity of passengers across different cameras. When spatiotemporal constraints are met (e.g., the maximum possible time for a passenger to move from camera A to camera B) and the similarity exceeds a threshold, track continuation is performed. Track smoothing and optimization use Bézier curves to smooth the original track points and eliminate jitter. A graph optimization method (factor graph) is used to fuse multi-camera observation data to optimize track accuracy.

[0054] In one possible implementation, video feature extraction employs an improved ResNet-IBN network, where the feature extraction formula is: F_v=CNN_v(I)={f_v¹,f_v²,...,f_vᴺ} Where F_v is the original feature set of the video, CNN_v ​​is the CNN feature extraction network specifically for video, I represents the video image data, f_v¹, f_v², ..., f_vᴺ represents the features of a single frame / segment of video, and the features of the i-th layer are calculated as follows: f_v 1 =BN(Conv3×3(ReLU(BN(Conv3×3(f_v 1 ⁻¹))))) Where f_v 1 f_v represents the video features of the current layer. 1 ⁻¹ represents the video features of the previous layer, BN represents batch normalization, Conv3×3 is a 3×3 convolution operation, and ReLU is a linear rectified activation function.

[0055] In one possible implementation, the formula for generating multimodal tokens is: Token_v=Linear(F_v)+PE_pos+ME_v Token_p=Linear(F_p)+PE_pos+ME_p Wherein, Token_v is the video modality token, Token_p is the millimeter-wave point cloud modality token, F_v is the original video feature, F_p is the original millimeter-wave point cloud feature, PE_pos is the position code, ME_v is the video modality type code, and ME_p is the millimeter-wave point cloud modality type code.

[0056] In one possible implementation, the location coding calculation formula is: PE(pos,2i)=sin(pos / 10000^(2i / d)) Where pos is the position index, 2i is the even index of the feature dimension, and d is the total feature dimension of the token.

[0057] The computational complexity design of this invention in each core processing stage balances the requirements of feature representation integrity and real-time performance. Specifically, the feature extraction stage has a complexity of O(T·(H·W+N_points)), where T represents the time step, H·W corresponds to the pixel scale of the video frame, and N_points is the amount of millimeter-wave point cloud data. This complexity can efficiently complete the extraction of basic features from both modalities. In the Transformer fusion stage, the original complexity O(T^2·D) is optimized to O(W^2·D) through sliding window optimization, significantly reducing the computational pressure over long time series. In the multi-target tracking stage, the initial O(N_det·N_track) (where N_det is the number of detected targets and N_track is the number of trajectories) is optimized to O(N_det·logN_track) through cascaded matching, improving data association efficiency. The overall complexity is ultimately controlled to O(T·(HW+N)+W^2D+NlogN), fully meeting the real-time processing requirements of security inspection scenarios.

[0058] This invention ensures real-time performance through two key technologies. First, GPU parallel computing, leveraging CUDA technology to accelerate feature extraction and Transformer computation, significantly improving the operational efficiency of core modules. Second, an asynchronous processing pipeline employs an asynchronous task mechanism to process video and radar sensor data in parallel. For example, asynchronous tasks process video streams and point cloud data separately, reducing data waiting time and ensuring efficient system operation.

[0059] system Based on the same concept as the method described above, this application also provides a passenger tracking system. See details below. Figure 7 , Figure 7 A structural diagram of a passenger tracking system according to a specific embodiment of this application is shown.

[0060] The present invention also includes a passenger trajectory construction system 700 for implementing the above method, including a data preprocessing and alignment module 701, a multimodal feature extraction module 702, a transformer feature fusion module 703, and a multi-target tracking and trajectory management module 704; The data preprocessing and alignment module 701 performs time synchronization, spatial coordinate system unification, and data cleaning on multi-source data. It adopts the PTP protocol to achieve microsecond-level time synchronization, obtains camera parameters through the Zhang Zhengyou calibration method, establishes a radar-camera mapping with the calibration board, unifies coordinates to the global system, and also performs video denoising and normalization, and point cloud clustering to remove false points.

[0061] The multimodal feature extraction module 702 extracts the appearance features of the video data and the geometric features of the millimeter-wave point cloud, respectively. ResNet50-IBN is used to extract multi-scale features from the video, and a transformer encoder is used to model inter-frame dependencies; PointNet++ is used to encode the point cloud, and 3D sparse convolution is used to enhance the features and extract local geometric features.

[0062] The transformer feature fusion module 703 utilizes an improved transformer architecture for cross-modal feature interaction and enhancement. It converts bimodal features into tokens and encodes them, uses multi-head cross-attention to mine complementary information, introduces learnable weights to dynamically adjust modal contributions, and the decoder models long-term temporal dependencies.

[0063] The multi-target tracking and trajectory management module 704 performs data association, state estimation, and trajectory generation based on fused features. It calculates the detection and trajectory similarity matrix based on the fused features, uses the Hungarian algorithm for matching, updates the trajectory and makes predictions using Kalman filtering, and initializes a new trajectory after multi-frame verification for unmatched detections.

[0064] In one possible implementation, the system further includes a multi-sensor data acquisition module 705 and a trajectory construction and output module 706. The sensor data acquisition module 705 deploys multiple high-definition cameras and 3D millimeter-wave radar, covering key areas of the security checkpoint. The trajectory construction and output module 706 generates continuous, smooth three-dimensional trajectories and supports abnormal behavior detection.

[0065] Obviously, those skilled in the art should understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the control methods described above. The modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps into a single integrated circuit module. Thus, the present invention is not limited to any specific hardware and software combination.

[0066] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the control methods described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0067] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for constructing passenger travel routes, characterized in that, include: The passenger information sensor is synchronized in time and space to generate preprocessed passenger RGB video data and preprocessed passenger 3D millimeter-wave radar point cloud data; wherein, the passenger information sensor includes a high-definition network camera and a 3D millimeter-wave radar. Based on preprocessed passenger RGB video data and preprocessed passenger 3D millimeter-wave radar point cloud data, multi-scale feature maps and local geometric features of point cloud data are extracted. Based on the multi-scale feature maps and the local geometric features of the point cloud data, multi-modal feature fusion is performed using a transformer to generate fused feature data. Based on the fused feature data, multi-target tracking and trajectory management are performed using the Hungarian algorithm and Kalman filtering to generate tracking and trajectory management data. Based on tracking and trajectory management data, cross-camera trajectory continuity and optimization are performed through the Re-ID network to generate passenger trajectory data.

2. The method according to claim 1, characterized in that, The step of temporal and spatial synchronization of passenger information sensors, and the generation of preprocessed passenger RGB video data and preprocessed passenger 3D millimeter-wave radar point cloud data through passenger information sensors, includes: Microsecond-level time synchronization of all sensors is achieved through a precision clock synchronization protocol; The Zhang Zhengyou calibration method was used to obtain the camera's intrinsic and extrinsic parameters, and the camera's coordinate system was integrated into the global coordinate system of the security checkpoint. The relative positional relationship between the millimeter-wave radar and the camera is obtained by calibration board, and a 3D-2D mapping relationship is established to transfer the millimeter-wave radar coordinate system to the global coordinate system of the security check channel. Denoising and brightness normalization are performed on passenger RGB video data to generate preprocessed passenger RGB video data; Dynamic target clustering and multipath interference suppression are performed on 3D millimeter-wave point cloud data to generate preprocessed passenger 3D millimeter-wave radar point cloud data.

3. The method according to claim 1, characterized in that, The step of extracting multi-scale feature maps and local geometric features of the point cloud data based on preprocessed passenger RGB video data and preprocessed passenger 3D millimeter-wave radar point cloud data includes: Multi-scale feature maps are extracted from video frames of preprocessed passenger RGB video data using a CNN backbone network; The feature map is flattened into a sequence, sinusoidal positional encoding is added, and long-range dependencies are modeled through a multi-layer self-attention mechanism of a transformer encoder. The PointNet++ network was used to perform feature encoding on the preprocessed passenger 3D millimeter-wave radar point cloud data to generate encoded point cloud data. 3D sparse convolution is used to extract voxel features from the encoded point cloud data to enhance feature representation and extract the local geometric features of each point in the point cloud.

4. The method according to claim 1, characterized in that, The step of fusing multimodal features based on multi-scale feature maps and local geometric features of point cloud data using a transformer to generate fused feature data includes: The transformer model maps multi-scale feature maps to appearance token sequences and local geometric features of point cloud data to geometric token sequences. Add modal type encoding and position encoding to each appearance token and geometry token; By using the transformer model, we set the appearance token as the query vector and the geometry token as the key vector and value vector to mine complementary information between the two modalities. A learnable weighting factor is introduced to dynamically adjust the contribution of the two modalities; The transformer decoder is used to model the fused features of multiple consecutive frames, capture long temporal dependencies, and generate fused feature data.

5. The method according to claim 1, characterized in that, The step of generating tracking and trajectory management data based on the fused feature data, using the Hungarian algorithm and Kalman filter, includes: Based on the fused feature data, the similarity matrix between the current frame detection result and the historical trajectory is calculated using the Hungarian algorithm; For a successfully matched trajectory, a Kalman filter is used to update the state and predict the position of the next frame. If an unmatched detection is detected in multiple consecutive frames, it is initialized as a new trajectory, generating tracking and trajectory management data.

6. The method according to claim 1, characterized in that, The process of generating passenger tracking data by performing cross-camera tracking and trajectory management data continuation and optimization through a Re-ID network, based on tracking and trajectory management data, includes: Based on tracking and trajectory management data, a Re-ID network is used to calculate the similarity of passengers under different cameras; When the spatiotemporal constraints are met and the similarity exceeds the threshold, the tracking and trajectory management data are spliced ​​together to generate stitched data. Bézier curves are used to smooth the spliced ​​data of the original track points to generate passenger track data. The accuracy of passenger movement data is improved by fusing observation data from multiple cameras using graph optimization methods.

7. The method according to claim 2, characterized in that, The video feature extraction employs an improved ResNet-IBN network, where the feature extraction formula is as follows: F_v=CNN_v(I)={f_v¹,f_v²,...,f_vᴺ} Where F_v is the original feature set of the video, CNN_v ​​is the CNN feature extraction network specifically for video, I represents the video image data, f_v¹, f_v², ..., f_vᴺ represents the features of a single frame / segment of video, and the features of the i-th layer are calculated as follows: f_v 1 =BN(Conv3×3(ReLU(BN(Conv3×3(f_v 1 ⁻¹))))) Where f_v 1 f_v represents the video features of the current layer. 1 ⁻¹ represents the video features of the previous layer, BN represents batch normalization, Conv3×3 is a 3×3 convolution operation, and ReLU is a linear rectified activation function.

8. The method according to claim 3, characterized in that, The formula for generating the multimodal token is: Token_v=Linear(F_v)+PE_pos+ME_v Token_p=Linear(F_p)+PE_pos+ME_p Wherein, Token_v is the video modality token, Token_p is the millimeter-wave point cloud modality token, F_v is the original video feature, F_p is the original millimeter-wave point cloud feature, PE_pos is the position code, ME_v is the video modality type code, and ME_p is the millimeter-wave point cloud modality type code.

9. The method according to claim 8, characterized in that, The location code calculation formula is as follows: PE(pos,2i)=sin(pos / 10000^(2i / d)) Where pos is the position index, 2i is the even index of the feature dimension, and d is the total feature dimension of the token.

10. A passenger travel history construction system, characterized in that, The method for implementing the method of claims 1-9 includes a data preprocessing and alignment module, a multimodal feature extraction module, a transformer feature fusion module, and a multi-target tracking and trajectory management module; The data preprocessing and alignment module performs time synchronization, spatial coordinate system unification, and data cleaning on multi-source data. The multimodal feature extraction module extracts the appearance features of the video data and the geometric features of the millimeter-wave point cloud, respectively. The transformer feature fusion module utilizes an improved transformer architecture for cross-modal feature interaction and enhancement. The multi-target tracking and trajectory management module performs data association, state estimation, and trajectory generation based on fused features.

Images