A cross-mirror multi-target personnel positioning and tracking method based on asymmetric multi-modal deep fusion
By introducing the SlowFast network and the Transformer asymmetric fusion network, multimodal features are decoupled and fused, solving the robustness and accuracy problems of personnel positioning and tracking in complex industrial scenarios, and realizing high-precision multi-target tracking and trajectory reconstruction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 雅江清洁能源科学技术研究(北京)有限公司
- Filing Date
- 2025-12-15
- Publication Date
- 2026-07-14
Smart Images
Figure CN122391286A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and industrial intelligent monitoring technology, and specifically relates to a personnel location and tracking method based on spatiotemporal geometric constraints and depth metric learning in complex industrial scenarios such as large-scale regional geotechnical engineering construction and intelligent factory operation and maintenance. Background Technology
[0002] During the construction of hydropower projects, the entire construction site involves numerous processes and techniques, and the types, numbers, and density of personnel vary greatly. Traditional appearance re-identification (Re-ID) algorithms struggle to maintain identity consistency. To ensure personnel safety and understand the situation on site, real-time location tracking of personnel is necessary. However, in outdoor environments, traditional appearance re-identification (Re-ID) algorithms are limited by the large area, multiple obstructions, varying lighting, similar appearances, and complex terrain, making it difficult to maintain identity consistency due to highly similar features. Even in indoor environments, methods relying solely on macroscopic motion models such as Kalman filtering are prone to identity swapping in crowded areas or when paths intersect. Furthermore, image distortion caused by wide-angle monitoring and the rigidity of existing linear weighted association mechanisms make it difficult for current technologies to achieve robust, high-precision tracking and trajectory reconstruction in dynamic and complex industrial environments, resulting in inaccurate risk identification and untimely hazard warnings. Summary of the Invention
[0003] The technical problem to be solved by this invention is to provide a cross-camera multi-target personnel localization and tracking method based on asymmetric multimodal deep fusion. This method innovatively introduces a SlowFast network to extract zero-sample micro-motion features to solve the problem of uniform clothing, and designs an asymmetric fusion network based on Transformer to solve the problem of nonlinear correlation of multimodal features.
[0004] The present invention adopts the following technical solution:
[0005] An improved method for cross-camera multi-target personnel localization and tracking based on asymmetric multimodal deep fusion includes the following steps:
[0006] Step 1, Video stream preprocessing and detection based on image quality assessment:
[0007] Perform frame-by-frame quality verification on the input video streams from multiple cameras, filtering out low-quality frames; use a pre-trained target detection model to identify human targets in the verified video frames, and output the two-dimensional bounding boxes of the human targets.
[0008] Step 2, Multimodal Feature Decoupling and Extraction:
[0009] For each person target detected in step 1, three complementary feature modalities are decoupled and extracted:
[0010] Geometric modes are obtained by backprojection of camera parameters. Extracting micro-motion semantic modalities using a SlowFast dual-path network. Predicting macroscopic motion modes using Kalman filters ;
[0011] Step 3, Asymmetric vector construction and deep fusion based on Transformer:
[0012] Based on the feature modes obtained in step 2, detection vectors representing the current observation state are constructed respectively. and trajectory vectors representing historical evolution states This asymmetric vector pair is input into a Transformer-based deep fusion network, and after interaction via a self-attention mechanism, a high-discrimination detection embedding vector is generated. and trajectory embedding vector ;
[0013] Step 4, Global Optimal Identity Association and Trajectory Closed-Loop Update:
[0014] Calculate the detection embedding vector With trajectory embedding vector The distance cost between them is used to perform global optimal matching using the Hungarian algorithm, outputting the continuous motion trajectory of all personnel in the real-world coordinate system, and updating the trajectory status based on the matching results.
[0015] Furthermore, the method for filtering out low-quality frames in step 1 is as follows:
[0016] Calculate the average pixel intensity of each frame of the image and grayscale standard deviation If the average pixel intensity of a certain frame of an image Below the preset brightness threshold or grayscale standard deviation If the change value is below the preset threshold, the frame is determined to be an invalid frame and discarded, and no further detection is performed.
[0017] Furthermore, the geometric modes are obtained in step 2. The method is as follows:
[0018] Construct a unified world coordinate system and establish an imaging model using the camera's intrinsic and extrinsic rotation and translation matrices;
[0019] Using a back-projection algorithm, the midpoint pixel coordinates of the bottom edge of the 2D detection box are mapped to the ground plane in the 3D world coordinate system, resulting in the physical coordinates of the 2D bird's-eye view BEV, which is the geometric modality. .
[0020] Furthermore, step 2 extracts the microscopic motion semantic modalities. The method is as follows:
[0021] Construct a dual-path spatiotemporal convolutional network, SlowFast, and use it as a zero-shot identity representation extractor;
[0022] Maintain a first-in-first-out image buffer of length L frames;
[0023] Slow path: Sparsely sample input frames with low frame rate and large time step, and use deep convolution to capture static body features and semantic portraits of people;
[0024] Fast path: Densely sample input frames with high frame rate and small time step, using fewer channels to capture high-frequency dynamic features and gait fingerprints of people;
[0025] The features of the two paths mentioned above are fused in the deep layers of the network through lateral connections to output a micro-motion semantic modality that is insensitive to clothing color. .
[0026] Furthermore, in step 2, macroscopic motion modes are predicted. The method is as follows:
[0027] The macroscopic motion trajectory of the target in the BEV space is modeled using a linear Kalman filter;
[0028] Define a state vector containing position and velocity. In each frame, using the state transition matrix, predict the target's prior state at the current moment based on the posterior state of the previous frame. This predicted vector is the macroscopic motion mode. .
[0029] Furthermore, in step 3:
[0030] Detection vector The current frame micro-motion semantic modality extracted from step 2 With geometric modes It is pieced together;
[0031] Trajectory Vector Microscopic motion semantic modality updated last in trajectory history With macroscopic motion modes It was pieced together.
[0032] Furthermore, step 3, the processing procedure of the Transformer-based deep fusion network, includes:
[0033] The detection vector is projected using two independent linear projection layers. and trajectory vector Mapped to the same dimension of the latent space;
[0034] The input is fed into a shared-weight Transformer encoder, which uses a multi-head self-attention mechanism to calculate the nonlinear correlation between geometric features, micro features, and macro features, and dynamically adjusts the weights of each feature component according to the scene context.
[0035] After layer normalization, the final detection embedding vector is output. and trajectory embedding vector .
[0036] Furthermore, the deep fusion network employs offline training, using a triplet loss function and combining it with an online hard negative sample mining strategy.
[0037] When constructing triples for training samples, for a given anchor sample, samples that are geometrically close but have different identity labels are specifically selected as difficult negative samples to force the model to learn to distinguish different targets that are similar in appearance and location by using micro-motion semantic features.
[0038] Furthermore, the method for globally optimal identity association in step 4 is as follows:
[0039] Calculate the Euclidean distance between the embedding vectors of all newly detected trajectories in the current frame and the embedding vectors of all existing trajectories, and construct the association cost matrix;
[0040] The Hungarian algorithm is used to perform bipartite graph matching on the association cost matrix to find the allocation scheme with the minimum global cost.
[0041] Furthermore, the method for trajectory closure update in step 4 is as follows:
[0042] For successfully matched trajectories, the currently detected geometric modality is utilized. The posterior state of the Kalman filter is updated using observations, and the target image region of the current frame is pushed into the image buffer of the SlowFast network to update the micro-motion semantic modality. ;
[0043] For unmatched trajectories, initialize new trajectories;
[0044] For multiple consecutive frames of unmatched trajectories, the target is determined to have left and the trajectory is deleted.
[0045] The beneficial effects of this invention are:
[0046] This invention effectively overcomes the challenge of appearance feature failure caused by uniform attire in complex industrial scenarios by introducing a "dual-scale motion" paradigm. Through a SlowFast dual-path network, it innovatively decouples "micro-motion semantics" (i.e., high-frequency gait fingerprints from the Fast path and low-frequency body posture portraits from the Slow path) that are independent of clothing color. This is then combined with high-precision geometric back-projection coordinates and macro-motion trends to construct a complementary "geometric + micro + macro" multimodal feature system. This zero-sample identity representation method allows the system to maintain extremely high identity differentiation even in extreme cases where the Re-ID algorithm completely fails, relying on subtle individual biological movement patterns.
[0047] This invention significantly improves the intelligence and robustness of multi-target tracking through an asymmetric deep fusion architecture based on Transformer. This architecture breaks away from the rigid pattern of traditional linear weighted association, utilizing a self-attention mechanism to achieve nonlinear dynamic arbitration between "instantaneous detection" and "historical trajectory." It can automatically adjust the weights of each modality based on scene contexts such as crowding and occlusion (e.g., automatically amplifying the weights of micro-actions when positions are close), effectively reducing the identity-swapping rate. Simultaneously, the system directly outputs physically meaningful 3D world coordinate trajectories, providing precise data support for industrial safety management and ergonomic analysis. Attached Figure Description
[0048] Figure 1 This is a flowchart illustrating the method of the present invention;
[0049] Figure 2 This is a schematic diagram of the geometric principle of back projection from 2D pixel coordinates to 3D world coordinates;
[0050] Figure 3 This is an internal network architecture diagram of an asymmetric fusion network (FusionModel) based on Transformer. Detailed Implementation
[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0052] Example 1 discloses a cross-camera multi-target personnel localization and tracking method based on asymmetric multimodal deep fusion, such as... Figure 1 As shown, it includes the following steps:
[0053] Step 1, Video stream preprocessing and detection based on image quality assessment:
[0054] To prevent low-quality data from interfering with subsequent feature extraction, the input video streams from multiple cameras are first subjected to frame-by-frame quality verification. Image quality evaluation metrics are defined, including average pixel intensity. and grayscale standard deviation The system automatically filters out invalid frames that are too bright (e.g., no lighting at night, lens obstruction) or lack texture variation (e.g., frozen screen, solid color background).
[0055] By calculating the average pixel intensity of the image To quantify its overall brightness. For a three-channel (BGR) image of size W×H. The average pixel intensity is calculated as follows:
[0056]
[0057] in Indicates coordinates The pixel value of channel c. If If the brightness is below a preset threshold, the frame will be marked as a problem frame.
[0058] Still, monochrome, or extremely monotonous images are also invalid inputs. These images share the common characteristic of having extremely low variance in their pixel values. To efficiently evaluate this, color images are first... Convert to grayscale image Then calculate the standard deviation of its pixel intensity. :
[0059]
[0060]
[0061] if Images with a standard deviation below a preset threshold indicate a lack of sufficient texture and content variation and are thus marked as problematic frames. Images identified as problematic frames are skipped by the system and may be selectively logged as an anomaly. This preprocessing step effectively prevents poor-quality data from interfering with the system's analysis process, thereby significantly improving the robustness and stability of the entire monitoring system in real, complex environments.
[0062] The validated video frames are input into a pre-trained YOLO series object detection model. Forward inference is performed to obtain the 2D bounding boxes and confidence scores of all people targets in the current frame. The final detection result is then obtained through non-maximum suppression (NMS) processing. The image enhancement algorithm introduces a deformable convolutional module into the first layer of the generator in the generative adversarial network and sets a radial distortion correction module in the preprocessing stage, reducing the distortion coefficients. The range is set within [-0.4, -0.2]. The target detection model is trained using a strategy that combines hybrid training with transfer learning.
[0063] Step 2, Multimodal Feature Decoupling and Extraction:
[0064] For each target detected in step 1, this invention innovatively decouples and extracts three complementary feature modes to form a "geometric + micro + macro" feature system:
[0065] (1) Geometric modes ( ): Construct a unified global world coordinate system. Utilize the camera's calibration parameters (including intrinsic and extrinsic rotation and translation matrices), such as... Figure 2 As shown, a back-projection algorithm is used to map the midpoint of the bottom edge of the detection box (i.e., the foot pixel) in the 2D image plane to the ground plane in the 3D world coordinate system. The physical coordinates of the resulting 2D bird's-eye view (BEV) are... This constitutes the geometric modal features This provides absolute spatial constraints for cross-mirror association.
[0066] Using camera intrinsic parameters and extrinsic rotation matrix Translation vector Take the midpoint of the bottom edge of the detection frame. As a foot point.
[0067] Step 1: Convert pixel coordinates to normalized camera coordinates.
[0068]
[0069] Step 2: Use a rotation matrix to transform the world coordinate system direction vector.
[0070]
[0071] Step 3: Ray intersection. Construct the ray. Assuming the ground height Z=0, solve for t to obtain the physical coordinates. .
[0072] Final features .
[0073] (2) Microscopic motion semantic mode ( ): Construct a dual-path spatiotemporal convolutional network (SlowFast) and use it as a zero-shot identity representation extractor.
[0074] The Slow path sparsely samples input frames at a low frame rate (large time step), utilizing a large number of channels and deep convolutions to focus on capturing the static spatial semantics of the target. Even when wearing the same uniform, different individuals still have differences in static biometric features such as body shape (height, weight) and standing posture (such as back tilt and head orientation). The Slow path is responsible for encoding these "semantic portraits".
[0075] Fast path: It samples input frames intensively at a high frame rate (small time steps), using fewer channels to focus on capturing dynamic changes with high temporal resolution. This includes high-frequency motion details such as the target's walking stride frequency, arm swing amplitude, and turning speed. The fast path is responsible for encoding these "motion fingerprints".
[0076] The features of the two paths are fused through lateral connections, outputting a micro-motion vector that is insensitive to clothing color but has high recognizability. .
[0077] Maintain a FIFO image buffer of length L frames. When the buffer is full, perform SlowFast feature extraction.
[0078] Fast Pathway: Input frame count is L, time step is... This path preserves the original high frame rate information, but to reduce computational cost, its convolutional kernel channel count is only a fraction of that of the Slow path. This allows the network to sensitively capture high-frequency body movements, such as walking rhythm and the instantaneous acceleration of arm swing. This is crucial for distinguishing workers walking side-by-side at the same speed but with different gaits.
[0079] Slow Pathway: The input frame count is... Time step This path samples at a low frame rate but has more channels, focusing on extracting spatial details and long-term body features from single-frame images. It can effectively encode information such as the target's contours, body proportions, and habitual poses.
[0080] Feature fusion: After each convolutional block, the high-frequency features of the Fast path are integrated into the Slow path through lateral connections. Finally, after a global average pooling layer, the fused high-dimensional feature vector is output. .
[0081] Macroscopic motion modes ( The macroscopic trajectory of the target in the BEV space is modeled using a linear Kalman filter. A state vector containing position and velocity is defined. In each frame, the prior state of the target at the current moment is predicted using the state transition matrix. This predicted vector is the macroscopic motion mode. This provides constraints on the long-term motion trend of the target.
[0082] For each surviving trajectory ID, maintain a Kalman filter. State vector.
[0083] Assuming a uniform motion model, predict the state of the next frame:
[0084]
[0085] in This is the state transition matrix. The predicted state vector is... .
[0086] This step consists of three parts: 1) semantic feature tracking; 2) spatial coordinate tracking; and 3) velocity direction tracking. This step serves as a bridge between the previous and subsequent steps, receiving real-time detection results from the personnel identification step and constructing a multimodal feature representation for each target ID, involving geometric modalities (…). ), micro-motion semantic modality ( ) and macroscopic motion modes ( The feature representation analysis framework diagram is shown below. This representation includes semantic features (who), spatial coordinates (where), and velocity direction (how to move). These three together constitute the sole decision basis for identity association in the multi-camera reasoning steps.
[0087] Step 3, Asymmetric vector construction and deep fusion based on Transformer:
[0088] A Transformer-based deep fusion network (FusionModel) is constructed to learn a non-linear similarity measure of multimodal features in an end-to-end manner. This invention employs a unique asymmetric input design:
[0089] (1) Constructing an asymmetric input vector:
[0090] Detection vector ( ): splicing together the currently observed microscopic features and geometric position This represents the "currently observed target state";
[0091]
[0092] Physical meaning: The apparent movement characteristics of the target and its current actual location.
[0093] Trajectory vector ( Microscopic features of splicing historical records and predicted macroscopic motion state It represents the "evolutionary state of the historical trajectory".
[0094]
[0095] Physical meaning: The action characteristics and predicted location in the target memory.
[0096] (2) Deep integration of Transformer:
[0097] like Figure 3 As shown, using an independent projection layer to... and Mapping to a unified high-dimensional embedding space; utilizing the multi-head self-attention mechanism in the Transformer encoder to capture long-range dependencies and nonlinear relationships between geometric location, micro-actions, and macro-trends. The model dynamically adjusts the weights of each modality based on the scene context (e.g., automatically increasing the weights of micro-action features when geometric distances are close), generating the final highly discriminative embedding vector. The network's final output is a highly discriminative detection embedding vector that has undergone layer normalization (LayerNorm). and trajectory embedding vector .
[0098] Will and Input the Transformer encoder.
[0099]
[0100] Inside the encoder, the multi-head self-attention mechanism plays a crucial role: it calculates... and Consistency, and and The similarity.
[0101] Case A (Crowded Scenario): When two targets A and B are very close to each other. At this point, the distinguishability between geometric and macroscopic features decreases. The self-attention mechanism detects this and automatically reduces the weight of geometric features, instead focusing heavily on them. The Fast path features (step frequency difference) are used to distinguish between A and B.
[0102] Case B (Occlusion Scenario): When target A is occluded, the detection box drifts. An abnormal transition has occurred. The self-attention mechanism has detected it. and The huge conflict will automatically filter out unreliable ones. Completely dependent on macroscopic motion prediction and micro-semantics To maintain tracking.
[0103] Step 4, Global Optimal Identity Association and Trajectory Closed-Loop Update:
[0104] Calculate the embedding vectors of all new detections Embedded vectors of all surviving trajectories The global association cost matrix is constructed using the L2 Euclidean distance between the trajectories. The Hungarian algorithm is then used for bipartite graph matching of the cost matrix to find the allocation scheme with the minimum global cost. For successfully matched trajectories, the posterior state of the Kalman filter is updated using the current geometric observations, and the target image of the current frame is pushed into the buffer of the SlowFast feature extractor to achieve online updating of micro-features. For unmatched trajectories, new trajectories are initialized; and trajectories that have not been matched for a long time are deleted.
[0105] Through offline training, an intelligent recognition model capable of accurately identifying personnel across cameras is constructed. This trained intelligent fusion model is then deployed to a real-time video stream, enabling real-time location and tracking of all personnel within the covered area of the factory.
Claims
1. A cross-camera multi-target personnel localization and tracking method based on asymmetric multimodal deep fusion, characterized in that, Includes the following steps: Step 1, Video stream preprocessing and detection based on image quality assessment: Data augmentation algorithms are used to process the video stream from the wide-angle camera, and frame-by-frame quality checks are performed on the input video streams from multiple cameras to filter out low-quality frames. A pre-trained target detection model is used to identify human targets in verified video frames and output two-dimensional bounding boxes of the human targets. Step 2, Multimodal Feature Decoupling and Extraction: For each person target detected in step 1, three complementary feature modalities are decoupled and extracted: Geometric modes are obtained by backprojection of camera parameters. ; Extracting micro-motion semantic modalities using a SlowFast dual-path network ; Predicting macroscopic motion modes using Kalman filters ; Step 3, Asymmetric vector construction and deep fusion based on Transformer: Based on the feature modes obtained in step 2, detection vectors representing the current observation state are constructed respectively. and trajectory vectors representing historical evolution states This asymmetric vector pair is input into a Transformer-based deep fusion network, and after interaction via a self-attention mechanism, a high-discrimination detection embedding vector is generated. and trajectory embedding vector ; Step 4, Global Optimal Identity Association and Trajectory Closed-Loop Update: Calculate the detection embedding vector With trajectory embedding vector The distance cost between them is used to perform global optimal matching using the Hungarian algorithm, outputting the continuous motion trajectory of all personnel in the real-world coordinate system, and updating the trajectory status based on the matching results.
2. The cross-camera multi-target personnel localization and tracking method based on asymmetric multimodal deep fusion according to claim 1, characterized in that, The method for filtering out low-quality frames in step 1 is as follows: Calculate the average pixel intensity of each frame of the image and grayscale standard deviation If the average pixel intensity of a certain frame of an image Below the preset brightness threshold or grayscale standard deviation If the change value is below the preset threshold, the frame is determined to be an invalid frame and discarded, and no further detection is performed.
3. The cross-camera multi-target personnel localization and tracking method based on asymmetric multimodal deep fusion according to claim 1, characterized in that, The geometric modes are obtained in step 2. The method is as follows: Construct a unified world coordinate system and establish an imaging model using the camera's intrinsic and extrinsic rotation and translation matrices; Using a back-projection algorithm, the midpoint pixel coordinates of the bottom edge of the 2D detection box are mapped to the ground plane in the 3D world coordinate system, resulting in the physical coordinates of the 2D bird's-eye view BEV, which is the geometric modality. .
4. The cross-camera multi-target personnel localization and tracking method based on asymmetric multimodal deep fusion according to claim 1, characterized in that, Step 2: Extracting microscopic motion semantic modalities The method is as follows: Construct a dual-path spatiotemporal convolutional network, SlowFast, and use it as a zero-shot identity representation extractor; Maintain a first-in-first-out image buffer of length L frames; Slow path: Sparsely sample input frames with low frame rate and large time step, and use deep convolution to capture static body features and semantic portraits of people; Fast path: Densely sample input frames with high frame rate and small time step, using fewer channels to capture high-frequency dynamic features and gait fingerprints of people; The features of the two paths mentioned above are fused in the deep layers of the network through lateral connections to output a micro-motion semantic modality that is insensitive to clothing color. .
5. The cross-camera multi-target personnel localization and tracking method based on asymmetric multimodal deep fusion according to claim 1, characterized in that, Predicting macroscopic motion modes in step 2 The method is as follows: The macroscopic motion trajectory of the target in the BEV space is modeled using a linear Kalman filter; Define a state vector containing position and velocity. In each frame, using the state transition matrix, predict the target's prior state at the current moment based on the posterior state of the previous frame. This predicted vector is the macroscopic motion mode. .
6. The cross-camera multi-target personnel localization and tracking method based on asymmetric multimodal deep fusion according to claim 1, characterized in that, In step 3: Detection vector The current frame micro-motion semantic modality extracted from step 2 With geometric modes It is pieced together; Trajectory Vector Microscopic motion semantic modality updated last in trajectory history With macroscopic motion modes It was pieced together.
7. The cross-camera multi-target personnel localization and tracking method based on asymmetric multimodal deep fusion according to claim 1, characterized in that, Step 3, the processing procedure of the Transformer-based deep fusion network, includes: The detection vector is projected using two independent linear projection layers. and trajectory vector Mapped to the same dimension of the latent space; The input is fed into a shared-weight Transformer encoder, which uses a multi-head self-attention mechanism to calculate the nonlinear correlation between geometric features, micro features, and macro features, and dynamically adjusts the weights of each feature component according to the scene context. After layer normalization, the final detection embedding vector is output. and trajectory embedding vector .
8. The cross-camera multi-target personnel localization and tracking method based on asymmetric multimodal deep fusion according to claim 7, characterized in that: The deep fusion network is trained offline, using a triplet loss function and combined with an online hard negative sample mining strategy. When constructing triples for training samples, for a given anchor sample, samples that are geometrically close but have different identity labels are specifically selected as difficult negative samples.
9. The cross-camera multi-target personnel localization and tracking method based on asymmetric multimodal deep fusion according to claim 1, characterized in that, The method for globally optimal identity association in step 4 is as follows: Calculate the Euclidean distance between the embedding vectors of all newly detected trajectories in the current frame and the embedding vectors of all existing trajectories, and construct the association cost matrix; The Hungarian algorithm is used to perform bipartite graph matching on the association cost matrix to find the allocation scheme with the minimum global cost.
10. The cross-camera multi-target personnel localization and tracking method based on asymmetric multimodal deep fusion according to claim 1, characterized in that, The method for trajectory closure update in step 4 is as follows: For successfully matched trajectories, the currently detected geometric modality is utilized. The posterior state of the Kalman filter is updated using observations, and the target image region of the current frame is pushed into the image buffer of the SlowFast network to update the micro-motion semantic modality. ; For unmatched trajectories, initialize new trajectories; For multiple consecutive frames of unmatched trajectories, the target is determined to have left and the trajectory is deleted.