Self-learning multi-target tracking method and system based on cross-modal perception

Abstract

The application discloses a self-learning multi-target tracking method and system based on cross-modal perception, S1: visible light and infrared video streams are collected, and after time sequence alignment, the streams are input into a cross-modal fusion network based on a transformer, information of two modes is deeply fused through a cross attention mechanism, and a fusion feature map is generated; S2: a key point-based anchor-free detector CenterNet is used to locate a target, and an identity re-identification Re-ID feature is extracted at a target center point; S3: a parallel correlation and prediction process is used, a state adaptive predictor SAP module predicts a motion state of an existing track, and a confidence sequence correlator CSA module dynamically generates a decision confidence interval by learning a statistical distribution of Mahalanobis distance, and decision judgment is performed; S4: a global track corrector GCM module is introduced, track interruption is post-processed, and track splicing and identity ID correction are realized. The application realizes real-time tracking of multiple targets in a complex environment.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a self-learning multi-target tracking method and system based on cross-modal perception. Background Technology

[0002] Multi-object tracking is a core research task in the field of computer vision, aiming to detect and track multiple objects in a video sequence in real time. Due to its broad application prospects in various fields, multi-object tracking technology has become a research hotspot in both academia and industry.

[0003] Despite significant progress in existing multi-target tracking methods, two major challenges remain in complex and dynamic real-world scenarios, impacting tracking accuracy and robustness:

[0004] The first major challenge is the target occlusion problem, especially in crowded or complex environments, where targets frequently occlude each other or are occluded by background objects, causing the detector to temporarily lose the target. When the target reappears, the tracker has difficulty re-associating it with its correct identity, thus causing a large number of identity errors.

[0005] The second major challenge lies in the general lack of online self-learning and environmental adaptation capabilities in existing technologies. The decision-making logic of most current tracking frameworks is static, making them vulnerable to dynamically changing scenes in the real world. When the target's speed, lighting conditions, or scene congestion change, the original fixed thresholds quickly become invalid, leading to tracking errors.

[0006] For example, the paper "Adaptive Keyframe Mining and Spatiotemporal Graph Learning-Guided Multimodal Multi-Target Tracking Method" (publication number CN120070506A) employs a cross-modal feature fusion module to fuse features from visible light and thermal infrared images. It then uses a reinforcement learning-based keyframe extraction module to adaptively segment the entire video, finally feeding the segmented video segments into a graph convolutional network to obtain the final tracking result. However, its data association logic relies on the fixed structure of graph learning and lacks the adaptive capability to dynamically learn online and adjust association decision thresholds based on the scene at the frame level. Summary of the Invention

[0007] The purpose of this invention is to provide a self-learning multi-target tracking method and system based on cross-modal perception, which enables real-time tracking of multiple targets in complex environments and provides an effective and reliable basis for personnel tracking in different scenarios.

[0008] To achieve one of the above objectives, the present invention adopts the following technical solution:

[0009] A self-learning multi-target tracking method based on cross-modal perception, characterized by the following steps:

[0010] S1: Collect visible light and infrared video streams of the monitoring scene, perform time-series alignment processing, and then input them into a Transformer-based cross-modal fusion network to generate information of the two modalities. Use the cross-attention mechanism to deeply fuse the information of the two modalities to generate a fusion feature map.

[0011] S2: The CenterNet keypoint-based anchorless detector is used to locate the object target in the fused feature map, and the Re-ID feature is extracted from the center point of the target.

[0012] S3: A parallel association and prediction process is adopted. The SAP module of the state adaptive predictor is used to predict the motion state of the existing trajectory of the target. Then, the CSA module of the confidence sequence correlater is used to dynamically generate decision confidence intervals by learning the statistical distribution of Mahalanobis distance online, making decision judgments, and generating motion trajectories.

[0013] S4: Introduce the Global Trajectory Corrector (GCM) module to stitch together the interrupted trajectory fragments of the motion trajectory, thereby achieving trajectory stitching and correcting the identity ID.

[0014] S1 specifically includes:

[0015] S101: Employs a binocular camera module integrating visible light RGB and infrared thermal imaging IR sensors to collect data from monitored scenes; specifically including:

[0016] To ensure strict temporal consistency between the two modalities, a high-precision network time protocol (NTP) is used to assign a uniform timestamp to each frame of the visible light and infrared video streams.

[0017] Upon receiving the raw video stream, the system performs preprocessing, including frame decoding, image size normalization, and precise matching and alignment based on timestamps, to construct a time-synchronized sequence of paired visible light images. and infrared images The composed cross-modal video sequence, in which For height, 3 represents the width, and 3 represents the number of channels.

[0018] For the location of the target in the sample, a two-person cross-labeling method is used to randomly divide the constructed total dataset into three independent subsets according to the proportion: training set (70%), validation set (20%) and test set (10%).

[0019] S102: Each visible light image acquired in S101... and infrared images They are respectively converted into serialized data, i.e., feature token sequences, that conform to the Transformer architecture. and S102: The visible light image of each frame... and infrared images These are respectively converted into serialized data conforming to the Transformer architecture, namely, a sequence of feature tokens; specifically including:

[0020] First, each input image Spatially divided into Non-overlapping, fixed-size image blocks , among which For height, For width, For the number of channels, It is the resolution of each image patch;

[0021] Secondly, each flattened image patch is mapped to a D-dimensional feature vector, i.e., a feature token, through a trainable linear projection layer;

[0022] Then, a learnable positional encoding is superimposed on each feature token. This enables the model to understand the original spatial location of each image patch;

[0023] Finally, the visible light and infrared images were converted into two independent feature token sequences containing spatial location information. and .

[0024] S103: The two feature token sequences generated in S102 are... and The features are fed into a cross-modal fusion Transformer network in parallel, so that the features of one modality are used as query Q to extract and weightedly fuse the features of another modality, which is used as key K and value V, and finally output a unified fusion feature map.

[0025] Further, in step S103: the two feature token sequences generated in step S102 are fed in parallel into a cross-modal fusion Transformer network, so that the feature of one modality is used as query Q, and the feature of the other modality, which is used as key K and value V, is extracted and weighted and fused to finally output a unified fused feature map; specifically including:

[0026] The two feature token sequences generated in S102 and The data is fed in parallel into a cross-modal fusion Transformer network composed of multiple stacked encoder layers. The multi-head cross-attention mechanism is used to fully interact and enhance the features in the feature token sequences of the two modalities in each encoder layer.

[0027] For updating the visible light feature stream, its query vector From visible light token sequence And its key Sum From infrared token sequence The infrared characteristic flow, on the other hand, has the following formula:

[0028]

[0029] in, These represent the query, key, and value matrices, respectively. It is the transpose of the key matrix; It is the dimension of the key vector, used for scaling to prevent the gradient from vanishing due to an excessively large dot product result;

[0030] In the multi-head attention mechanism, the process is executed in parallel multiple times, with each head learning a different information association pattern. The outputs of all heads are concatenated and linearly integrated, and then passed through a feedforward network and residual connections to finally generate a high-dimensional fusion feature map containing deep interaction information of the two modalities.

[0031] Furthermore, S2 specifically includes:

[0032] S201: The anchorless detector CenterNet includes a center point heatmap prediction head that operates on the fused feature map output by S1 to generate a center point heatmap of the same size as the feature map. , This represents the heatmap of the center point predicted by the model, specifying the range of values ​​that each pixel can take. Representing a heatmap A single numerical value at a pixel's coordinates represents the probability that that location is the center of an object.

[0033] During the training phase, the true object center point is encoded as a Gaussian kernel peak on the heatmap. The network is optimized by a modified Focal Loss function to reduce the weight of background samples, enabling the model to focus more on learning those difficult-to-identify object center points and solving the problem of imbalance between object center and background samples.

[0034] S202: After locating the center point of an object through a heatmap, the anchorless detector regresses the attributes of the center point of the object through two other parallel prediction heads to form a complete detection box corresponding to the center point of the object.

[0035] S203: An independent identity re-identification (Re-ID) feature extraction head is used to extract corresponding features from the fused feature map output by S1 at each target center point position determined in S201, and map them into a highly discriminative identity embedding vector. This identity re-identification (Re-ID) head is specially trained to ensure that the identity vectors of the same target remain aggregated in the feature space, while the identity vectors of different targets are separated from each other.

[0036] Further, S202 specifically includes:

[0037] The size regression head is responsible for predicting the physical width and height of the target corresponding to the center point of the object. The offset regression head is responsible for predicting a sub-pixel level center point offset. This compensates for the discretization error caused by feature map downsampling;

[0038] Both regression tasks use the L1 loss function for supervised training, effectively calculating the difference between the prediction and the true value. This guides the model to update its parameters, and the formula is as follows:

[0039]

[0040] in It is the model's predicted value for the attribute of the k-th target. It is its corresponding true value.

[0041] Furthermore, S3 specifically includes:

[0042] S301: Motion state prediction based on the state adaptive predictor SAP;

[0043] S302: Measure the positional deviation of the predicted motion state to obtain a normalized statistical distance, and calculate the Mahalanobis distance between the two. As an association cost, it measures the degree of matching between the predicted trajectory in S301 and the detection box in S2, that is, the distance between the predicted trajectory in S301 and the detection box in S2.

[0044] S303: A confidence sequence association (CSA) module is used for self-learning data association. The CSA module makes dynamic decisions by learning its statistical distribution online.

[0045] S304: For the output of S303, for high-confidence matching pairs, the update step of the Gain Kalman Filter (GKF) is used to update the trajectory state and reset its unmatched duration by combining new detection information; for high-confidence mismatched and uncertain detection boxes and trajectories, the supplementary association stage is entered, and the Re-ID features extracted in S203 are used for matching. By calculating the cosine similarity between the unmatched trajectory and the unmatched detection box, the matching failure caused by motion blur or sudden change in motion mode is resolved.

[0046] If both rounds of matching fail, the system confirms the detection as a new target and assigns it a new ID.

[0047] Further, step S301: Motion state prediction based on the state adaptive predictor SAP; specifically including:

[0048] The system uses a Gain Kalman Filter (GKF) to predict the target's state in the current frame, defining a state vector that includes the target's position, size, and corresponding velocity. ,in, The coordinates of the center point, For area, Given the aspect ratio, the dotted terms are their corresponding first derivatives. The core innovation lies in the fact that it does not use fixed observation noise, but rather uses the detection confidence c of a single match on the trajectory. k Adaptively adjust the observation noise covariance matrix:

[0049]

[0050] in, It is the basic noise matrix. It is a preset threshold for distinguishing between high and low quality detection; when the detection quality is poor, the confidence level is... Low noise increases the system's observation noise, reduces its confidence in unreliable observations, and makes it rely more on its own motion predictions to ensure the stability and robustness of trajectory predictions.

[0051] Furthermore, S303: The confidence sequence correlator (CSA) makes dynamic decisions by learning its statistical distribution online; specifically including:

[0052] Treat all successfully matched Mahalanobis distance scores as a data stream, use confidence sequences to estimate the p-quantile of this data stream, and use it as the boundary between normal and abnormal matches;

[0053] This module calculates the radius of uncertainty. Its formula is:

[0054]

[0055] in, This is the number of observed samples. This is the allowed probability of error, where e is an irrational number. Based on this radius, a dynamic confidence interval, i.e., the decision interval, is generated. :

[0056]

[0057] in, It is the empirical quantile of the observed fractions, based on the newly calculated Mahalanobis distance. Regarding this dynamic range, the system will make three types of judgments:

[0058] High confidence matching: ;

[0059] High confidence mismatch: ;

[0060] Uncertainty (Conservative Decision-Making): .

[0061] Furthermore, the S4: Global Trajectory Corrector (GCM) module stitches together the interrupted trajectory fragments of the motion trajectory to achieve trajectory stitching and performs identity ID correction, specifically including:

[0062] S401: Based on the motion trajectory obtained in S3, first standardize all trajectory segments generated in the real-time association and prediction stage, and then standardize each trajectory segment. It is represented as a time sequence of fixed length N. For trajectory segments with a length less than N, zero padding is used to extend them to the standard length, forming a standardized trajectory sequence with a uniform format that can be processed as input by neural networks.

[0063] S402: Employs a parallel dual-branch convolutional architecture to extract temporal and spatial features that characterize the continuity of identity from the normalized trajectory sequence of S401.

[0064] S403: Using an MLP classifier, based on the extracted spatiotemporal features, calculate the probability that any two trajectory segments belong to the same target, and find the globally optimal matching scheme.

[0065] S404: Based on the optimal matching result, the fracture trajectory segments determined to belong to the same target are spliced ​​together in data, and their identity IDs are unified with the IDs originally generated by the system for the target, thus completing the identity correction.

[0066] To achieve the second objective mentioned above, the present invention adopts the following technical solution:

[0067] A self-learning multi-target tracking system based on cross-modal perception, used to execute the self-learning multi-target tracking method based on cross-modal perception described above, includes a data acquisition module, a model training module, an association decision module, and an analysis and processing module.

[0068] The data acquisition module is used to acquire visible light and infrared video streams of the monitoring scene, perform time-series alignment processing, and then input them into a Transformer-based cross-modal fusion network to generate information of the two modalities. The information of the two modalities is then deeply fused using a cross-attention mechanism to generate a fused feature map.

[0069] The model training module is used to locate the object target in the fused feature map using the anchorless frame detector CenterNet based on key points, and extract the Re-ID feature at the center point of the target.

[0070] The associated decision module employs a parallel association and prediction process. It uses the State Adaptive Predictor (SAP) module to predict the motion state of the target's existing trajectory, and then uses the Confidence Sequence Association (CSA) module to dynamically generate decision confidence intervals by learning the statistical distribution of Mahalanobis distance online, making decision judgments, and generating the motion trajectory.

[0071] The analysis and processing module is used to introduce the Global Trajectory Corrector (GCM) module to stitch together the interrupted trajectory fragments of the motion trajectory, thereby achieving trajectory stitching and correcting the identity ID.

[0072] The beneficial effects of this invention are:

[0073] This invention deeply integrates the features of both visible light and infrared modes, significantly improving the robustness of the tracking system in complex lighting conditions and harsh environments such as nighttime, strong light, and shadows. It innovatively introduces a self-learning association mechanism based on confidence sequences, abandoning the reliance on static, fixed thresholds in traditional methods. This mechanism can learn the dynamic distribution of association costs in the scene online and adaptively adjust the decision boundary, enabling the tracker to intelligently respond to changes in target speed, density, and other scene variations, greatly enhancing the system's environmental adaptability and decision reliability. Combined with a state adaptive prediction and global trajectory correction module, this invention can effectively handle short-term and long-term target occlusion, ensuring the continuity and integrity of the output trajectory. It achieves high-precision, highly robust, long-term identity consistency tracking of multiple targets in complex dynamic scenes.

[0074] The method of this invention is also applicable to other scenarios that require multi-target tracking. Attached Figure Description

[0075] Figure 1 This is a flowchart of a specific embodiment 1 of the present invention;

[0076] Figure 2 This is a flowchart of step 1 in a specific embodiment 1 of the present invention;

[0077] Figure 3 This is a flowchart of step 2 in a specific embodiment 1 of the present invention;

[0078] Figure 4 This is a flowchart of step 3 in specific embodiment 1 of the present invention;

[0079] Figure 5 This is a flowchart of step 4 in specific embodiment 1 of the present invention;

[0080] Figure 6 This is a principle block diagram of specific embodiment 2 of the present invention;

[0081] Figure 7 This is a flowchart of a specific embodiment 3 of the present invention. Detailed Implementation

[0082] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Specific Implementation Example 1:

[0084] See Figures 1 to 5 As shown, a self-learning multi-target tracking method based on cross-modal perception includes the following steps:

[0085] S1: Collect visible light and infrared video streams from the monitoring scene, perform time-series alignment, and input them into a Transformer-based cross-modal fusion network. The information from the two modalities is deeply fused through a cross-attention mechanism to generate a fused feature map.

[0086] As the foundation of the entire tracking framework, its focus is on building a comprehensive and robust scene understanding paradigm. By performing deep collaborative processing of heterogeneous visible light and infrared data streams, this invention not only overcomes the information bottleneck of a single sensor in complex environments, but also generates a unified representation containing rich semantic and physical attributes in a higher-dimensional feature space.

[0087] Specifically, it includes:

[0088] Step S101: Use a binocular camera module that integrates visible light RGB and infrared thermal imaging IR sensors to collect data from the monitored scene;

[0089] To ensure strict temporal consistency between the two modal data, a high-precision Network Time Protocol (NTP) is used to assign a unified timestamp to each frame of the visible light and infrared video streams.

[0090] Upon receiving the raw video stream, the system performs preprocessing, including frame decoding, image size normalization, and precise matching and alignment based on timestamps, to construct a time-synchronized sequence of paired visible light images. and infrared images A cross-modal video sequence dataset, in which For height, The width is 3, and the number of channels is 3. The number of channels needs to be 3 because the visible light mode has three channels: red (R), green (G), and blue (B). The infrared mode is a single channel, but in order to adapt to the 3-channel input requirement of the visible light mode, the infrared mode channel is copied into 3 copies and passed to the transformer module for subsequent processing.

[0091] For the location of the target in the sample, a two-person cross-labeling method is used to randomly divide the constructed total dataset into three independent subsets according to the proportion: training set (70%), validation set (20%) and test set (10%).

[0092] Step S102: For each frame of visible light image acquired in S101... and infrared images These are respectively converted into serialized data, i.e., a sequence of tokens, that conform to the Transformer architecture. This includes:

[0093] First, each input image Spatially divided into Non-overlapping, fixed-size image blocks , among which For height, For width, For the number of channels, It is the resolution of each image patch;

[0094] Secondly, each flattened image patch is mapped to a D-dimensional feature vector, i.e., a feature token, through a trainable linear projection layer;

[0095] Then, a learnable positional encoding is superimposed on each feature token. This enables the model to understand the original spatial location of each image patch;

[0096] Finally, the visible light and infrared images were converted into two independent feature token sequences containing spatial location information. and .

[0097] Step S103: The two feature token sequences generated in S102 are fed in parallel into a cross-modal fusion Transformer network, so that the features of one modality are used as query Q, and the features of the other modality, which are used as keys K and values ​​V, are intelligently extracted and weighted and fused, finally outputting a unified fused feature map. This includes:

[0098] The two feature token sequences generated in S102 and A cross-modal fusion Transformer consisting of multiple stacked encoder layers is fed in parallel. The core of this module is a multi-head cross-attention mechanism, in which features in the feature token sequences of the two modalities are fully interacted and enhanced in each encoder layer.

[0099] For updating the visible light feature stream, its query vector From visible light token sequence And its key Sum From infrared token sequence The infrared characteristic flow, on the other hand, has the following formula:

[0100]

[0101] in, These represent the query, key, and value matrices, respectively. It is the transpose of the key matrix; It is the dimension of the key vector, used for scaling to prevent the gradient from vanishing due to an excessively large dot product result;

[0102] In the multi-head attention mechanism, the process is executed in parallel multiple times, with each "head" learning a different information association pattern. The outputs of all heads are concatenated and linearly integrated, and then passed through a feedforward network and residual connections to finally generate a high-dimensional fused feature map containing deep interaction information from both modalities.

[0103] S2: Use CenterNet, a key-point-based anchorless detector, to locate the object target in the fused feature map, and extract the Re-ID feature at the center point of the target;

[0104] The unstructured fusion feature map output by S1 is transformed into a series of structured target-level primitives that can be directly used by tracking algorithms, bridging the gap from raw pixel perception to advanced semantic parsing. By accurately anchoring the spatiotemporal coordinates of each potential target and assigning it a highly separable identity embedding in a high-dimensional manifold space, this invention achieves effective deconstruction and quantization of each independent instance within the scene, providing clear and unambiguous initial objects for subsequent state evolution and identity association.

[0105] Specifically, it includes:

[0106] Step S201: The anchorless detector CenterNet includes a center point heatmap prediction head, which is applied to the fused feature map generated in S1 to generate a center point heatmap with the same size as the feature map. , The heatmap of the center point predicted by the model. Define the range of values ​​that each pixel can take. Representing a heatmap A single numerical value at a pixel's coordinates represents the probability that that location is the center of an object.

[0107] During the training phase, the center point of the real object is encoded as a Gaussian kernel peak on the heatmap. The network is optimized by a modified Focal Loss function to reduce the weight of background samples, so that the model can focus more on learning the center points of people that are difficult to identify, thus solving the problem of imbalance between the center points of people and background samples.

[0108] Step S202: After locating the center point of the object through the heatmap, the anchorless detector CenterNet regresses the attributes of the center point of the object through two other parallel prediction heads to form a complete detection box corresponding to the center point of the object.

[0109] Specifically, it includes:

[0110] The size regression head is responsible for predicting the physical width and height of the target corresponding to the center point of the object. The offset regression head is responsible for predicting a sub-pixel level center point offset. This compensates for the discretization error caused by feature map downsampling;

[0111] Both regression tasks use the L1 loss function for supervised training, effectively calculating the difference between the prediction and the true value. This guides the model to update its parameters, and the formula is as follows:

[0112]

[0113] in It is the model's predicted value for the attribute of the k-th target. It is its corresponding true value.

[0114] Step S203: Using an independent Re-ID feature extraction head, at the center point of each target determined in S201, the corresponding features are extracted from the fused feature map output in S1 and mapped to a highly discriminative identity embedding vector. This Re-ID head is specially trained to ensure that the identity vectors of the same target remain aggregated in the feature space, while the identity vectors of different targets are separated from each other.

[0115] S3: A parallel association and prediction process is adopted. The state adaptive predictor SAP module predicts the motion state of the target's existing trajectory, and a confidence sequence correlator CSA module dynamically generates decision confidence intervals by learning the statistical distribution of Mahalanobis distance online, and makes decision judgments.

[0116] This module elevates the classic data association problem from a static rule-based matching task to a dynamic decision-making process based on statistical inference. By quantitatively modeling the uncertainty between prediction and observation and learning its inherent statistical regularities online, it endows the system with a confidence-based reasoning capability. This technological innovation transforms the entire tracking process from a mechanical execution of instructions into one capable of self-calibration and judgment in complex dynamic environments.

[0117] Specifically, it includes:

[0118] Step S301: Predict motion state based on the state adaptive predictor SAP;

[0119] The system uses a Gain Kalman Filter (GKF) to predict the target's state in the current frame, defining a state vector that includes the target's position, size, and corresponding velocity. ,in, The coordinates of the center point, For area, The aspect ratio is given, and the dotted terms are their corresponding first derivatives (velocities). The core innovation lies in the fact that it does not use fixed observation noise, but rather uses the detection confidence level c of a single match on the trajectory. k Adaptively adjust the observation noise covariance matrix:

[0120]

[0121] in, It is the basic noise matrix. It is a preset threshold for distinguishing between high and low quality detection. When the detection quality is poor, the confidence level is lowered. Low noise increases the system's observation noise, reduces its confidence in unreliable observations, and makes it rely more on its own motion predictions to ensure the stability and robustness of trajectory predictions.

[0122] Step S302: The system measures the positional deviation while considering the uncertainty of trajectory prediction itself, obtains a normalized statistical distance, and calculates the Mahalanobis distance between the two. As the association cost, this distance measures the degree of matching between the predicted trajectory in S301 and the new detection box in S2.

[0123] Step S303: A confidence sequence association (CSA) module is used to perform self-learning data association. The CSA module makes dynamic decisions by learning its statistical distribution online.

[0124] Specifically, it includes:

[0125] Treat all existing successfully matched Mahalanobis distance scores as a data stream, use confidence sequences to estimate the p-quantile of this data stream, take p=0.95, and use it as the boundary between "normal match" and "abnormal match";

[0126] This module calculates the radius of uncertainty. Its formula is:

[0127]

[0128] in, This is the number of observed samples. This is the allowed probability of error, where e is an irrational number, approximately 2.71828. Based on this radius, a dynamic confidence interval, i.e., the decision interval, is generated. :

[0129]

[0130] in, It is the empirical quantile of the observed fractions, based on the newly calculated Mahalanobis distance. Regarding this dynamic range, the system will make three types of judgments:

[0131] High confidence matching: ;

[0132] High confidence mismatch: ;

[0133] Uncertainty and conservative decision-making: .

[0134] Step S304: For the output of S303, the high-confidence matching pairs are updated using the GKF update step, combined with new detection information, to update the trajectory status and reset the unmatched duration; for high-confidence mismatched and uncertain detection boxes and trajectories, the supplementary association stage is entered, and the identity re-identification Re-ID features extracted in S203 are used for matching. By calculating the cosine similarity between the unmatched trajectory and the unmatched detection box, the matching failure caused by motion blur or sudden change in motion mode is resolved.

[0135] If both rounds of matching fail, the system confirms the detection as a new target and assigns it a new ID.

[0136] S4: Introducing the Global Connection Model (GCM) module to post-process trajectory interruptions, achieving trajectory stitching and identity ID correction;

[0137] Specifically, it includes:

[0138] Step S401: First, standardize all trajectory segments generated by the system in the initial tracking phase. Each trajectory segment... It is represented as a time sequence of fixed length N. For trajectory segments with a length less than N, zero padding is used to extend them to the standard length, forming a standardized trajectory sequence with a uniform format that can be processed as input by neural networks.

[0139] Step S402: Employ a parallel dual-branch convolutional architecture to extract temporal and spatial features that characterize the continuity of identity from the normalized trajectory sequence in S401.

[0140] Step S403: Using an MLP classifier, based on the extracted spatiotemporal features, calculate the probability that any two trajectory segments belong to the same target, and find the globally optimal matching scheme.

[0141] Step S404: Based on the optimal matching result, the fracture trajectory segments that are determined to belong to the same target are spliced ​​together in terms of data, and their identity IDs are unified with the IDs originally generated by the system for the target, thus completing the identity correction. Specific Implementation Example 2:

[0143] See Figure 6 As shown, a self-learning multi-target tracking system based on cross-modal perception is used to execute the self-learning multi-target tracking method based on cross-modal perception described in Specific Embodiment 1. It includes a data acquisition module 1, a model training module 2, an association decision module 3, and an analysis and processing module 4.

[0144] The data acquisition module uses a binocular camera module integrating visible light and infrared thermal imaging sensors, employing timestamp synchronization technology, to acquire visible light and infrared video streams of the monitored scene. After time-series alignment processing, it forms a multimodal video stream with precise time-series alignment. The video streams are then input into a Transformer-based cross-modal fusion network to generate information in both modalities. A cross-attention mechanism is used to deeply fuse the information from the two modalities, generating a fused feature map. In this specific embodiment, the acquired synchronized video stream is fed into a Transformer-based cross-modal fusion network. This network utilizes its core cross-attention mechanism to deeply interact and fuse the rich texture details of visible light with the superior all-weather perception capabilities of infrared, ultimately generating a unified fused feature map with an information density far exceeding that of a single modality.

[0145] The model training module is used to locate the object target in the fused feature map using the CenterNet keypoint-based anchorless detector, and extract the Re-ID feature at the center point of the target. In this specific embodiment, the core online tracking logic is executed in each frame of video, including target detection, state prediction and data association, and the operations are performed in parallel on the fused feature map: a CenterNet-based anchorless detector head completes high-precision localization by predicting the center point and size of the object; at the same time, a Re-ID feature extraction head extracts a high-discriminative appearance embedding vector that can uniquely identify the target at the detected target center point position.

[0146] The association decision module employs a parallel association and prediction process. It uses a State Adaptive Predictor (SAP) module to predict the motion state of the target's existing trajectory, and then uses a Confidence Sequence Association (CSA) module to dynamically generate a decision confidence interval by learning the statistical distribution of Mahalanobis distance online, making a decision and generating the motion trajectory. In this specific embodiment: for an existing trajectory, a State Adaptive Predictor (SAP) adaptively predicts its motion position in the current frame based on its historical state and detection confidence. The core of this module is a Confidence Sequence Association (CSA), which dynamically generates a statistically guaranteed decision confidence interval by learning the statistical distribution of the association cost Mahalanobis distance online, and updates or creates the trajectory accordingly.

[0147] The analysis and processing module is used to introduce the Global Trajectory Corrector (GCM) module to stitch together interrupted trajectory fragments of the motion trajectory, achieving trajectory stitching and correcting the identity ID. In this specific embodiment, the initial tracking results are post-processed offline or quasi-online to correct identity errors caused by long-term occlusion or conservative decision-making. The Global Trajectory Corrector (GCM) takes all trajectory segments generated within a certain period as input, analyzes and identifies trajectory fragments that should belong to the same target but are interrupted, performs a stitching operation, merges their data into a complete trajectory, and uniformly corrects their identity ID to the earliest appearing ID.

[0148] The functions of the data acquisition module 1, model training module 2, association decision module 3, and analysis and processing module 4 in this system are the same as those of the self-learning multi-target tracking method based on cross-modal perception in Specific Embodiment 1, so this Specific Embodiment is omitted here. Specific Implementation Example 3:

[0150] See Figure 7 As shown, a self-learning multi-target tracking method based on a cross-modal perception self-learning multi-target tracking system is described in Specific Embodiment 2, which adopts the cross-modal perception self-learning multi-target tracking system in Specific Embodiment 2.

[0151] At a city street corner, a binocular camera module integrating a visible light camera and an infrared thermal imager is deployed, facing a busy subway station entrance. This entrance has a large ventilation opening that continuously exhausts hot air. Using timestamp synchronization technology, the device collects synchronous dual-modal video data of this scene around the clock and transmits it in real time to a self-learning multi-target tracking system based on this invention.

[0152] In a surveillance mission, a regular pedestrian (pedestrian A) and another pedestrian wearing a thick jacket (pedestrian B) enter the monitoring field of view. Pedestrian A walks by normally, while pedestrian B walks towards a subway ventilation opening and stops there. The system's goal is to continuously and accurately track all targets in this complex environment, especially addressing the challenge posed by pedestrian B. The system's workflow is as follows:

[0153] Step S1: The system performs cross-modal perception and feature extraction.

[0154] At the beginning of the task, both pedestrian A and pedestrian B were in the normal area. The system's Transformer fusion module received two video streams: one visible light and one infrared. For well-lit areas, it utilized the rich details of the visible light; for shadowed areas, it relied on the thermal signal of the infrared. Through a cross-attention mechanism, the system generated stable and robust fused features for both pedestrians, which were successfully located and assigned initial identities ID1 (pedestrian A) and ID2 (pedestrian B) by the CenterNet detection head and Re-ID head.

[0155] Step S2: The system continuously detects and predicts the status of pedestrian B walking towards the hot air vent.

[0156] As pedestrian B approaches and stops near the hot air vent, challenges arise: their visible light image becomes blurry due to the darkness; their infrared thermal imaging signal is severely interfered with by the hot airflow from the vent, causing their body outline to become unclear on the thermal image, and the detection box position begins to "jitter." Although the fusion module of S1 tries its best to extract the best features from the two imperfect information sources, the stability of the detection results output by S2 is significantly lower than that of pedestrian A. Meanwhile, the system's state adaptive predictor SAP module still stably predicts the position of pedestrian B in the next frame based on the historical velocity of ID2.

[0157] Step S3: The system's Confidence Sequence Correlator (CSA) module handles the matching challenges caused by environmental noise.

[0158] This is a key step that demonstrates the core advantages of this invention. The Mahalanobis distance between the predicted position of ID2 calculated by the system and its actual "jittering" detection position is consistently in a range that is higher than normal and unstable, such as the matching value of pedestrian A. A traditional tracker that relies on a fixed threshold will frequently determine "match failure" in this situation, causing the trajectory of ID2 to be repeatedly interrupted and erroneously creating new IDs.

[0159] However, the Confidence Sequence Associate (CSA) of this invention, upon observing the persistently high cost fraction flow specific to ID2 online, does not directly determine it as a matching failure. Its self-learning mechanism identifies this as a persistent "new normal" caused by local environmental noise. Therefore, it adaptively adjusts its internal decision confidence interval. Even if in some frames the matching cost falls into the "uncertain" interval, leading to conservative decisions, the system will quickly re-associate in subsequent frames based on stronger evidence. Ultimately, ID2 is firmly and continuously maintained on pedestrian B, without identity switching.

[0160] Step S4: The system's Global Trajectory Corrector (GCM) serves as the final safeguard.

[0161] In this embodiment, since the confidence sequence correlator of S3 successfully maintained the stability of ID 2 online, no trajectory break occurred. If pedestrian A reappears after being briefly occluded, the global trajectory corrector (GCM) module of S4 will be activated in post-processing. By analyzing the consistency of the two trajectory segments in terms of time, space, and appearance, it will intelligently stitch them together and correct them back to the original ID 1.

[0162] This embodiment fully demonstrates that the present invention can achieve continuous tracking of multiple targets in complex urban scenarios through its core cross-modal perception and self-learning decision-making capabilities, effectively solving the core pain point of traditional methods that are difficult to adapt to dynamic environments due to their reliance on static rules.

[0163] The technical solution provided by this invention has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. It should be noted that those skilled in the art can make several improvements and modifications to this invention without departing from the principles of this invention, and these improvements and modifications also fall within the protection scope of the claims of this invention.

Claims

1. A self-learning multi-target tracking method based on cross-modal perception, characterized in that, The method comprises the following steps: S1: collecting visible light and infrared video streams of a monitoring scene, performing timing alignment processing, and inputting information of two modalities generated by a cross-modal fusion network based on a Transformer into the network to generate a fusion feature map by using a cross-attention mechanism to deeply fuse information of the two modalities; S2: positioning an object target in the fusion feature map by using a key point-based anchor-free detector CenterNet and extracting an identity recognition Re-ID feature at a center point of the target; specifically comprising: S201: The anchor-free frame detector CenterNet includes a center point heat map prediction head, which acts on the fused feature map output by S1 to generate a center point heat map with the same size as the feature map , The center point heat map predicted by the representative model specifies the numerical range taken by each pixel point, The single numerical value on the pixel point coordinate of the heat map is represented by ​ S202: after locating the object center point by using a heat map, the anchor-free detector performs regression on the attributes of the object center point by using another two parallel prediction heads to form a complete detection frame of the object corresponding to the center point; S203: an independent identity recognition Re-ID feature extraction head is used to extract a corresponding feature from the fusion feature map output in S1 at each target center point position determined in S201, and map the feature to a high-discrimination identity embedding vector; the identity recognition Re-ID head is trained to ensure that identity vectors of the same target are aggregated in a feature space, and identity vectors of different targets are separated from each other; S3: a parallel correlation and prediction process is used to predict a motion state of the target by using a state adaptive predictor SAP module, and a confidence sequence associator CSA module is used to dynamically generate a decision confidence interval by online learning a statistical distribution of Mahalanobis distance to perform decision judgment and generate a motion trajectory; S4: a global trajectory corrector GCM module is introduced to splice and stitch a motion trajectory and correct an identity ID by splicing and processing a broken trajectory fragment of the motion trajectory. 2.The self-learning multi-target tracking method based on cross-modal perception according to claim 1, characterized in that: The S1 specifically comprises: S101: a binocular camera module integrating a visible light RGB sensor and an infrared thermal imaging IR sensor is used to collect data of a monitoring scene; S102: convert each frame of visible light image collected in S101 and infrared image into serialized data conforming to the sequence of feature tokens processed by the Transformer architecture respectively and respectively and ; S103: input the two feature token sequences generated in S102 into the cross-modal fusion Transformer network in parallel and Parallelly input into the cross-modal fusion Transformer network, taking the features of one modality as the query Q, extracting and weightedly fusing from the features of another modality as the key K and the value V, and finally outputting a unified fusion feature map.

3. The self-learning multi-target tracking method based on cross-modal perception according to claim 2, characterized in that: The S103: two feature token sequences generated in S102 are sent into a cross-modal fusion Transformer network in parallel, features of one modality are used as queries Q to extract and perform weighted fusion from features of another modality as keys K and values V, and finally a unified fusion feature map is output; specifically comprising: The two-way feature token sequence generated in S102 And Parallelly input into a cross-modal fusion Transformer network stacked by multiple encoder layers EncoderLayer, and use a multi-head cross-attention mechanism to fully interact and enhance the features in the two-way feature token sequence in each encoder layer. For updating the visible feature stream, its query vector from the visible token sequence while its key and value from the infrared token sequence The infrared feature stream is the opposite, and is given by where, represent query, key, value matrices respectively; is the transpose of the key matrix; is the dimension of the key vector, used for scaling to prevent gradient vanishing due to too large dot product result; In the multi-head attention mechanism, the process is performed in parallel multiple times, each head learns a different information correlation mode, and after the output results of all heads are spliced and linearly integrated, a high-dimensional fusion feature map containing deep interaction information of the two modalities is finally generated through a feedforward network and a residual connection.

4. The self-learning multi-target tracking method based on cross-modal perception according to claim 1, characterized in that: The S202 specifically comprises: The size regression head is responsible for predicting the physical width and height of the object corresponding to the target of the object center point The offset regression head is responsible for predicting a sub-pixel level center point offset Compensate for the discretization error caused by the feature map downsampling The two regression tasks are supervised by L1 loss function, which effectively calculates the difference between the prediction and the true value , guiding the model to update parameters, whose formula is: wherein is the predicted value of the property of the kth target by the model, is its corresponding true value.

5. The self-learning multi-target tracking method based on cross-modal perception according to claim 1, characterized in that: The S3 specifically comprises: S301: motion state prediction is performed based on a state adaptive predictor SAP; S302: measure the position deviation of the predicted motion state to obtain a normalized statistical distance, and calculate the Mahalanobis distance therebetween , as the association cost, thereby measuring the matching degree of the predicted trajectory in S301 and the detection frame in S2, i.e. the distance between the predicted trajectory in S301 and the detection frame in S2; S303: a confidence sequence associator CSA module is used for self-learning data association, and the CSA module dynamically decides by online learning a statistical distribution thereof; S304: For the output result of S303, for the high confidence matching pair, use the update step of the gain Kalman filter GKF to update the state of the track combined with the new detection information, reset its unmatched duration; for high confidence unmatched and uncertain detection boxes and tracks, enter the supplementary association stage, match using the identity re-identification Re-ID features extracted by S203, solve the matching failure caused by motion blur or motion mode mutation by calculating the cosine similarity between unmatched tracks and unmatched detection boxes; If both rounds of matching fail, the system confirms that the detection is a new target and assigns it a new ID.

6. The self-learning multi-target tracking method based on cross-modal perception according to claim 5, characterized in that: S301: Motion state prediction based on state adaptive predictor SAP; specifically including: The system adopts a gain Kalman filter (GKF) to predict its state in the current frame, and defines a state vector containing the target position, size and corresponding velocity wherein, is the center point coordinate, is the area, is the aspect ratio, and the item with a dot is its corresponding first derivative, the core innovation is that it does not adopt a fixed observation noise, but adjusts the observation noise covariance matrix adaptively according to the detection confidence c k of the last matching on the trajectory: wherein, is the base noise matrix, is a preset threshold to distinguish high and low quality detection, when the detection quality is poor, the confidence is low, the system increases the observation noise, reduces the reliability of the unreliable observation, and relies more on its own motion prediction to ensure the stability and robustness of the trajectory prediction.

7. The self-learning multi-target tracking method based on cross-modal perception according to claim 5, characterized in that: S303: Dynamic decision making by confidence sequence associator CSA by learning the statistical distribution of Mahalanobis distance online; specifically including: All successfully matched Mahalanobis distance scores are treated as a data stream, and the p-quantile of the data stream is estimated using confidence sequence estimation as the boundary between normal matching and abnormal matching; The module calculates the uncertainty radius The formula is: wherein, is the number of observed samples, is the allowed probability of error, e is an irrational number, based on which a dynamic confidence interval, i.e. decision interval, is generated : where, is the empirical quantile of the observed score, according to the newly computed Mahalanobis distance In relation to this dynamic interval, the system makes three types of judgments: High confidence match: ; High confidence mismatch: ; Uncertain conservative decision: .

8. The self-learning multi-target tracking method based on cross-modal perception according to claim 1, characterized in that: S4: Global track corrector GCM module, which performs splicing processing on the interrupted track fragments of the motion track, realizes track stitching, and performs identity ID correction; specifically including: S401: According to the motion trajectory obtained in the S3, first, the real-time association is standardized with all trajectory segments generated in the prediction stage, and each trajectory segment is represented as a fixed length N time sequence. For trajectory segments less than N in length, zero padding is used to expand them to the standard length, forming a standardized trajectory sequence for neural network processing input. S402: A parallel double-branch convolution architecture is used to extract time sequence and spatial features that can represent the identity continuity from the standardized track sequence of S401; S403: Using an MLP classifier, the probability that any two track segments belong to the same target is calculated based on the extracted spatiotemporal features, and the globally optimal matching scheme is found; S404: Based on the optimal matching result, the broken track segments determined to belong to the same target are spliced in data, and their identity ID is unified as the ID originally generated by the system for the target, completing identity correction.

9. A self-learning multi-target tracking system based on cross-modal perception, configured to perform a self-learning multi-target tracking method based on cross-modal perception according to any one of claims 1-8. It includes a data acquisition module, a model training module, an association decision module, and an analysis processing module, The data acquisition module is used to collect visible light and infrared video streams of the monitoring scene, perform time sequence alignment processing, and input the two modal information generated by the cross-modal fusion network based on Transformer, and use the cross-attention mechanism to deeply fuse the information of the two modalities to generate a fusion feature map; The model training module is used to use the key point-based anchor-free detector CenterNet to locate the object target in the fusion feature map and extract the identity re-identification Re-ID feature at the center point of the target; The association decision module uses a parallel association and prediction process, uses the state adaptive predictor SAP module to predict the motion state of the existing track of the target, and then uses the confidence sequence associator CSA module to dynamically generate a decision confidence interval by learning the statistical distribution of Mahalanobis distance online, to make a decision and generate a motion track. The analysis processing module is used to introduce the global track corrector GCM module to perform splicing processing on the interrupted track fragments of the motion track, realize track stitching, and perform identity ID correction.

Images