Cross-camera multi-target tracking method in dynamic scenes based on feature decoupling

Abstract

The present invention discloses a method for cross-camera multi-target tracking in dynamic scenes based on feature decoupling. The present invention allows for synchronous input of multiple perspectives, and decouples the target features in each perspective into local tracking queries and global tracking queries. For target tracking in each perspective, the present invention designs corresponding local tracking queries; local tracking queries can learn inter-frame features for predicting target positions and trajectories. At the same time, the same target in different perspectives has a global tracking query for the interaction of cross-perspective object features; the global tracking query utilizes an attention mechanism to promote information sharing between objects; the present invention designs a global allocation and discriminant loss for the global tracking query to distinguish different objects while associating similar objects between views. The present invention solves the problem of distinguishing different targets within a single perspective while ensuring the consistency of the same target between different perspectives, thereby achieving more stable cross-camera multi-target tracking performance in dynamic scenes.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and pattern recognition, and mainly relates to a cross-camera multi-target tracking method in dynamic scenes based on feature decoupling. Background Art

[0002] Cross-camera multi-object tracking aims to maintain consistent object tracking across different cameras over time, enabling a comprehensive understanding of object motion and interactions in multi-camera environments. Cross-camera multi-object tracking involves correlating and linking object trajectories across different camera perspectives, typically addressing challenges such as occlusion, varying viewpoints, lighting conditions, and scale or appearance changes. Cross-camera multi-object tracking typically involves fusing object trajectory, identity, and behavior information from multiple cameras, aiding in traffic monitoring, athlete behavior analysis, and a variety of other areas.

[0003] Existing cross-camera multi-object tracking methods rely on geometric constraints, feature-based matching, and static camera assumptions. Despite achieving promising results, many methods process each view and tracking step in multiple stages, making it difficult to capture long-term temporal correlations and interactions between objects across views. In single-view multi-object tracking, the Transformer family of multi-object tracking represents a cutting-edge approach that utilizes query-based architectures. These methods use query-based mechanisms to track multiple objects throughout a video sequence, achieving consistent and accurate tracking. However, cross-camera multi-object tracking requires stable single-view tracking and accurate cross-view object association, tasks that cannot be simultaneously addressed by tracking queries in existing Transformer architectures. Therefore, a feature-decoupled cross-camera multi-object tracking method for dynamic scenes is needed to decouple queries in single-view and cross-view dynamic scenes to ensure that they can capture long-term temporal correlations while also achieving cross-view association capabilities. Summary of the Invention

[0004] In order to solve the problems of conflict between single-view and cross-view features and the need for camera calibration for cross-view correlation in existing cross-camera tracking algorithms, this paper proposes an end-to-end cross-view multi-target tracking model based on feature decoupling.

[0005] The technical solution adopted in the present invention is as follows:

[0006] 1. A method for multi-target tracking across cameras in dynamic scenes based on feature decoupling

[0007] The following steps are involved:

[0008] Step S1: The synchronized video data of multiple cameras are simultaneously input into the end-to-end cross-camera multi-object tracking network based on the Transformer architecture;

[0009] Step S2: Decouple the tracking query of the tracking target output by the decoder through MLP, dividing it into local tracking query and global tracking query;

[0010] Step S3: The target detection candidate frame query and the local tracking query after the interaction of the previous frame are input into the decoder and the query interaction module in sequence to realize the tracking of the target in the single view;

[0011] Step S4: Use the attention mechanism to interact with the global tracking query and generate cross-view features;

[0012] Step S5: In the inference phase, the local tracking query and the global tracking query of the target are extracted, and the feature similarity between single view and cross view is calculated for matching, and a global ID is assigned to the same target across cameras.

[0013] The end-to-end cross-camera multi-target tracking network with the Transformer architecture as the core in step S1 includes an image encoder and multiple multi-target tracking modules. Each multi-target tracking module consists of a decoder, a query interaction module, and a global query attention module. The input of each camera corresponds to a multi-target tracking module.

[0014] The step S1 comprises:

[0015] The synchronized multi-camera data is input into the image encoder at the same time, and then input into the corresponding decoder respectively after the image encoder. The input of each camera uses the encoder and decoder with shared parameters;

[0016] The target detection candidate frame query input to the decoder is provided by the target detector to generate the target tracking query; YOLOX is used as the target detector and generates target detection candidate frames for training and reasoning stages; in each camera, YOLOX generates a set of candidate frames containing coordinates and confidence scores; then the generated target candidate frame set is used to learn the single-view and cross-view target association tasks. Cross-view means cross-camera

[0017] The step S2 is specifically as follows:

[0018] In the last layer of the decoder, a learnable linear layer (MLP) is used to map the query dimension from 256 to 512. The 512 dimensions are then divided into two sets of learnable tracking queries: the first 256 dimensions are used as local tracking queries for single-view object tracking, and the second 256 dimensions are used as global tracking queries for cross-camera association learning.

[0019] In the step S3:

[0020] The query interaction module takes as input the hidden states produced by the Transformer decoder and outputs the local tracking query for the next frame after the interaction as a single-view feature. The query interaction module filters the input proposal query and tracks the query based on the score and attention-based temporal interaction, thereby determining the appearance, disappearance, and continuous tracking of the target. This enhances temporal relationship modeling and provides contextual priors for the tracked object.

[0021] In step S4, the global query attention module promotes cross-camera information sharing through the interaction of multi-view global queries, specifically:

[0022] 4.1) Each local tracking query after interaction from a single view has a corresponding global tracking query, which is treated as a token input to the transformer encoder. The global tracking query of each target is used as the input of the global query attention module, and the global tracking query of each view acts as the query, key, and value in the attention mechanism;

[0023] 4.2) The global query attention module consists of multiple layers of multi-head self-attention and MLP blocks. Finally, the global query attention module outputs the global tracking query after interaction as the cross-view feature;

[0024] Global query shares information between different views through multi-layer multi-head self-attention. The advantage of this design is that global query can utilize the temporal modeling of single-view target information through local trajectory query while ensuring consistency between different views.

[0025] 4.3) Assign a unified global ID to the same target in different views at the same moment, and use cross-entropy loss to constrain global queries;

[0026] 4.4) Learning cross-view feature associations via global query:

[0027] First calculate the similarity of all global query embeddings:

[0028]

[0029] Where E is the feature of global tracking query, M is the total number of global queries;

[0030] In order to distinguish different objects while maintaining the consistency of the same object, the global ID is used to establish the similarity between objects with cross-view features, and the global query is constrained by the contrastive loss.

[0031] The step S5 comprises:

[0032] During the inference phase, local tracking queries are used in an online association manner to output the target's location box and local features in a single view. Similarity calculations are then performed using the global tracking query and local tracking query after information sharing, ultimately achieving cross-camera tracking of the same target through cross-view association.

[0033] 2. A terminal

[0034] The terminal includes a memory, a processor, and a cross-camera multi-target tracking program stored in the memory and executable on the processor. When the processor executes the cross-camera multi-target tracking program, the above-mentioned dynamic scene cross-camera multi-target tracking method based on feature decoupling is implemented.

[0035] 3. A computer-readable storage medium

[0036] The computer-readable storage medium stores a cross-camera multi-object tracking program. When the cross-camera multi-object tracking program is executed by the processor, the above-mentioned dynamic scene cross-camera multi-object tracking method based on feature decoupling is implemented.

[0037] This feature-decoupling-based cross-camera multi-object tracking method for dynamic scenes proposes a cross-camera multi-object tracking framework that allows simultaneous multi-view data input into an end-to-end multi-view multi-object tracking network based on the Transformer architecture. For each viewpoint, a corresponding local tracking query is designed to learn inter-frame features to predict the target position and trajectory.

[0038] The proposed method for cross-camera multi-target tracking in dynamic scenes based on feature decoupling proposes a global tracking query for the same target in different viewpoints, which is used for cross-view feature interaction. The global tracking query utilizes a multi-head attention mechanism to promote information sharing between targets.

[0039] The cross-camera multi-target tracking method for dynamic scenes based on feature decoupling of the present invention uses local tracking queries to directly output the detection boxes and features of objects in a single perspective, and uses the global query embedding after information sharing to perform similarity calculation, ultimately achieving cross-perspective association of the same object.

[0040] Beneficial effects of the present invention:

[0041] The present invention allows for simultaneous input from multiple perspectives, and decouples target features in each perspective into local tracking queries and global tracking queries. For target tracking in each perspective, the present invention designs corresponding local tracking queries. Local tracking queries can learn inter-frame features for predicting target positions and trajectories. At the same time, the same target in different perspectives has a global tracking query for the interaction of cross-perspective object features. Global tracking queries use attention mechanisms to promote information sharing between objects. The present invention designs global allocation and discriminant losses for global tracking queries to distinguish different objects while associating similar objects between views. The present invention solves the problem of distinguishing different targets within a single perspective while ensuring the consistency of the same target between different perspectives, achieving more stable cross-camera multi-target tracking performance in dynamic scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Schematic diagram of the end-to-end multi-view multi-target tracking network architecture based on feature decoupling of the present invention;

[0043] Figure 2 Schematic diagram of a single-view tracking query with synchronous input from multiple cameras in the present invention;

[0044] Figure 3 is a schematic diagram of a global query attention module for global tracking query in the present invention;

[0045] Figure 4 This is a flowchart of the cross-camera multi-target tracking association reasoning in the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0047] like Figure 1 As shown in the figure, the present invention simultaneously inputs the synchronized video data of multiple cameras into an end-to-end multi-view multi-target tracking network with the Transformer architecture as the core. Input the video of the same scene from different cameras Where N represents the camera number. The present invention can utilize the interaction between target queries across frames and different perspectives to achieve association with the same target, thereby assigning a unified global ID to the same target.

[0048] The input image passes through the image encoder and decoder, and the MLP decouples the query into local tracking queries and global tracking queries. The query also inputs the detection candidate box query provided by YOLOX. The local tracking query then inputs the query interaction module to output single-view features. The global tracking query outputs cross-view features through the global query attention module. Finally, global tracking IDs are assigned to multiple targets across multiple cameras through cross-camera multi-target tracking association inference.

[0049] For each viewpoint, the proposed method consists of two main components: an object detector and a modified anchor-based tracker. YOLOX is used as the object detector and initially proposes detection candidate boxes for the training and inference phases. In each viewpoint, YOLOX generates a set of detection candidate boxes containing coordinates and confidence scores. Subsequently, the modified anchor-based tracker takes on the task of learning trajectory associations using these generated detection candidate boxes. The input of the query interaction module is the hidden state produced by the Transformer decoder, which outputs the local tracking query for the next frame. Figure 2 As shown, the query interaction module can filter the input proposal query and track the query based on the score, thereby determining the appearance and disappearance of the target. For continuously tracked targets, the query interaction module uses multi-head self-attention to enhance temporal relationship modeling and provide contextual priors for tracking targets.

[0050] Utilizing collective average loss Query q via local proposal det and local tracking query q l To track the trajectory of the object in the entire video, the single-view target is tracked. The loss function of the local tracking query is: in, Contains the loss function for detection and tracking, and Y is the corresponding detection and tracking label. This allows the query features of a single view to accurately distinguish each target.

[0051] like Figure 3 As shown in Fig. 2, the global query attention module is designed to promote cross-camera information sharing by leveraging the interaction of multi-view global queries. Initially, each query interaction module from a single view generates a global tracking query, which is regarded as a token input to the encoder. The global tracking query q for each view is g Acting as query, key, and value. The L-layer global query attention module consists of multi-head self-attention and MLP blocks. The output of the global query attention module is used to learn cross-view representations using query embeddings.

[0052] The global query shares information between different views through an L-layer encoder. Each local tracking query after interaction from a single view has a corresponding global tracking query. The advantage of this design is that the global query can utilize the temporal modeling of the single-view trajectory information through the local trajectory query while ensuring consistency between different views. In order to align the cross-view representations of the same object as much as possible to achieve cross-view alignment. Therefore, a unified global ID is assigned to the same object from different views, and a cross-entropy loss is used to constrain the global query:

[0053]

[0054] Among them, θ(·) is the encoder, f(·) represents the classifier, and Y g Is the label of the global ID.

[0055] Global query can learn a unified representation of the same target in different viewpoints based on cross-entropy loss, but it still lacks effective constraints to distinguish different targets. This paper uses the contrast loss of global query to learn cross-view target association. First, all global query features in different viewpoints are calculated:

[0056]

[0057] Where E is the global query feature, and M is the total number of global queries. To distinguish different targets while maintaining consistency within the same target, we use the global ID to establish similarity between targets for global query embedding. When two global query embeddings belong to the same target, their similarity should be 1, otherwise 0. Finally, we use this as a label to constrain the similarity of global query features using a contrastive loss.

[0058] like Figure 4 As shown, the present invention achieves online association through model inference and association. A detector or model head is used to provide bounding boxes for each view. Single-view features and bounding boxes are then extracted through local tracking queries, while cross-view features are extracted through global tracking queries. Finally, a global ID is assigned to each object based on the similarity of features extracted from single and cross-view perspectives, ultimately achieving cross-view association of the same object.

[0059] As shown in the table below, based on the experimental results on the DIVOTrack dataset, the CVMOTR of the present invention shows significant advantages in both CVMA and CVIDF1, two key performance indicators of cross-camera multi-target tracking, achieving scores of 68.6 and 69.4 respectively, far exceeding other multi-stage methods (such as OSNet, Strong, AGW, MvMHAT, CT, MGN, NFormer) and end-to-end methods (such as CrossMOT). These results demonstrate the efficiency and reliability of CVMOTR in practical applications and demonstrate its leading position in the field of cross-camera multi-target tracking.

[0060]

[0061]

[0062] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention's description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for tracking multiple targets across cameras in dynamic scenes based on feature decoupling, characterized in that: The following steps are involved: Step S1: The synchronized video data of multiple cameras are simultaneously input into an end-to-end cross-camera multi-object tracking network model based on the Transformer architecture; Step S2: Decouple the tracking query of the tracking target output by the decoder through MLP, dividing it into local tracking query and global tracking query; Step S3: The target detection candidate frame query and the local tracking query after the interaction of the previous frame are input into the decoder and the query interaction module in sequence to realize the tracking of the target in the single view; Step S4: Use the attention mechanism to interact with the global tracking query and generate cross-view features; Step S5: In the inference phase, local tracking queries and global tracking queries of the target are extracted, and the feature similarity between single view and cross-view is calculated for matching, and a global ID is assigned to the same target across cameras. In step S4, the global query attention module promotes cross-camera information sharing through the interaction of multi-view global queries, specifically: 4.1) Each local tracking query after interaction from a single view has a corresponding global tracking query. The global tracking query of each target serves as the input of the global query attention module, and the global tracking query of each view acts as the query, key, and value in the attention mechanism; 4.2) The global query attention module consists of multiple layers of multi-head self-attention and MLP blocks. Finally, the global query attention module outputs the global tracking query after interaction as the cross-view feature; 4.3) Assign a unified global ID to the same target in different views at the same moment, and use cross-entropy loss to constrain global queries; 4.4) Learning cross-view feature associations via global query: First calculate the similarity of all global query embeddings: Where E is the feature of global tracking query, M is the total number of global queries; The global ID is used to establish the similarity between objects with cross-view features, and the global query is constrained by the contrastive loss.

2. The method for tracking multiple targets across cameras in dynamic scenes based on feature decoupling according to claim 1, characterized in that: The end-to-end cross-camera multi-target tracking network model with Transformer architecture as the core includes an image encoder and multiple multi-target tracking modules. Each multi-target tracking module consists of a decoder, a query interaction module, and a global query attention module. The input of each camera corresponds to a multi-target tracking module.

3. The method for tracking multiple targets across cameras in dynamic scenes based on feature decoupling according to claim 2, characterized in that: The step S1 comprises: The synchronized multi-camera data is input into the image encoder at the same time, and then input into the corresponding decoder respectively after the image encoder. The input of each camera uses the encoder and decoder with shared parameters; The target detection candidate box query input to the decoder is provided by the target detector to generate the target tracking query; YOLOX is used as the target detector and generates target detection candidate boxes for training and inference stages; in each camera, YOLOX generates a set of candidate boxes containing coordinates and confidence scores; the generated target candidate box set is then used to learn the task of single-view and cross-view target association.

4. The method for tracking multiple targets across cameras in dynamic scenes based on feature decoupling according to claim 1, wherein: The step S2 is specifically as follows: In the last layer of the decoder, a learnable linear layer (MLP) is used to map the query dimension from 256 to 512. The 512 dimensions are then divided into two sets of learnable tracking queries: the first 256 dimensions are used as local tracking queries for single-view object tracking, and the second 256 dimensions are used as global tracking queries for cross-camera association learning.

5. The method for tracking multiple targets across cameras in dynamic scenes based on feature decoupling according to claim 1, wherein: In the step S3: The input of the query interaction module is the hidden state generated by the Transformer decoder, which is used to output the local tracking query after the next frame interaction as a single-view feature.

6. The method for tracking multiple targets across cameras in dynamic scenes based on feature decoupling according to claim 1, characterized in that: The step S5 comprises: During the model inference phase, local tracking queries are used in an online association manner to output the target's location box and local features in a single view. Similarity calculations are then performed using global tracking queries and local tracking queries after information sharing, ultimately achieving cross-camera tracking of the same target through cross-view association.

7. A terminal, characterized in that: The terminal includes a memory, a processor, and a cross-camera multi-target tracking program stored in the memory and executable on the processor. When the processor executes the cross-camera multi-target tracking program, the dynamic scene cross-camera multi-target tracking method based on feature decoupling according to any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a cross-camera multi-object tracking program. When the cross-camera multi-object tracking program is executed by the processor, the dynamic scene cross-camera multi-object tracking method based on feature decoupling according to any one of claims 1 to 6 is implemented.

Images