Multi-target tracking method based on traffic scene

Abstract

The invention discloses a multi-target tracking method based on a traffic scene, and relates to the technical field of target tracking, and the method comprises the steps: constructing a target tracking data set of the traffic scene, the target tracking data set comprising a target category, a target ID, and target two-dimensional detection frame information; in the traffic scene, setting a frame-by-frame ordered bounding box for each tracked target according to a multi-target tracking task; constructing a network structure of an end-to-end multi-target tracking model based on the frame-by-frame ordered bounding box and the target tracking data set; and setting a loss function trained by the end-to-end multi-target tracking model, and performing model optimization and parameter updating. The technical problem that a multi-target tracking method in the prior art is insufficient in accuracy and robustness in a complex traffic scene is solved, and the technical effect of improving tracking precision and robustness is achieved by constructing an end-to-end multi-target tracking model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target tracking, and in particular to a multi-target tracking method based on a traffic scene. BACKGROUND

[0002] The multi-target tracking method has a wide application in traffic monitoring and intelligent traffic system, and realizes continuous tracking of targets such as vehicles and pedestrians through detection and association. However, in a complex traffic scene, due to factors such as light changes, occlusions, target density and scale changes, the traditional detection and association separated tracking mode is prone to cause target loss or frequent ID switching, and it is difficult to maintain the consistency of the track and the stability of the detection result, resulting in reduced tracking accuracy and robustness, which limits the application effect in actual traffic management and intelligent analysis. SUMMARY

[0003] The present application provides a multi-target tracking method based on a traffic scene, which is used to solve the technical problem of insufficient accuracy and robustness of the prior art multi-target tracking method in a complex traffic scene.

[0004] In view of the above problems, the present application provides a multi-target tracking method based on a traffic scene, which comprises:

[0005] A target tracking data set of a traffic scene is constructed, the target tracking data set comprising target categories, target IDs, and target two-dimensional bounding box information; in the traffic scene, frame-by-frame ordered bounding boxes are set for each tracking target in light of a multi-target tracking task; a network structure of an end-to-end multi-target tracking model is constructed based on the frame-by-frame ordered bounding boxes and the target tracking data set; a loss function for training of the end-to-end multi-target tracking model is set, and model optimization and parameter updating are performed.

[0006] One or more technical solutions provided in the present application have at least the following technical effects or advantages:

[0007] In the present application, a target tracking data set of a traffic scene is constructed, the target tracking data set comprising target categories, target IDs, and target two-dimensional bounding box information; in the traffic scene, frame-by-frame ordered bounding boxes are set for each tracking target in light of a multi-target tracking task; a network structure of an end-to-end multi-target tracking model is constructed based on the frame-by-frame ordered bounding boxes and the target tracking data set; a loss function for training of the end-to-end multi-target tracking model is set, and model optimization and parameter updating are performed. The present application solves the technical problem of insufficient accuracy and robustness of the prior art multi-target tracking method in a complex traffic scene, and achieves the technical effect of improving tracking precision and robustness by constructing an end-to-end multi-target tracking model. BRIEF DESCRIPTION OF DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0009] Figure 1 A flowchart of a multi-target tracking method based on a traffic scene provided by an embodiment of the present application is shown in FIG. 1.

[0010] Figure 2 A flowchart of setting a frame-by-frame ordered bounding box in a multi-target tracking method based on a traffic scene provided by an embodiment of the present application is shown in FIG. 2. DETAILED DESCRIPTION

[0011] The present application provides a multi-target tracking method based on a traffic scene, which aims to solve the technical problem of insufficient accuracy and robustness of the prior art multi-target tracking method in a complex traffic scene. By constructing an end-to-end multi-target tracking model, the technical effect of improving tracking accuracy and robustness is achieved.

[0012] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort fall within the scope of protection of the present application.

[0013] It should be noted that any variation of the terms "comprise" and "have" is intended to cover non-exclusive inclusion, for example, a process, method, system, product or server comprising a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices.

[0014] As shown in FIG. 1, the present application provides a multi-target tracking method based on a traffic scene, which comprises: Figure 1

[0015] Step S100: Construct a target tracking dataset of a traffic scene, wherein the target tracking dataset comprises target categories, target IDs and target two-dimensional detection box information.

[0016] ​In the embodiment of the present application, when constructing the target tracking dataset of the traffic scene, first, multi-source video data containing roads, vehicles, pedestrians and non-motor vehicles are obtained based on the high-position installed video acquisition device, and frame-level segmentation is performed on the original video sequence to form continuous image samples. Then, the traffic participant targets in each frame are extracted by using a visual detection algorithm, and the category information of the targets is determined by combining algorithm recognition and manual verification to clearly define the semantic attributes of different traffic objects.

[0017] Meanwhile, in order to realize the continuous identification of the same object in the time sequence, a unique identity ID is assigned according to the target motion trajectory to ensure the consistent representation of the target in multiple frames of images. Then, the two-dimensional bounding box information of the target in the image is labeled in the form of rectangular coordinates, that is, the boundary position of the target on the pixel plane is used as a spatial positioning feature to describe the geometric range of the appearance region of the target. After a series of steps such as video acquisition, target detection, category recognition, identity labeling and bounding box calibration, the target tracking dataset of the traffic scene containing the target category, target ID and target two-dimensional bounding box information is finally obtained.

[0018] Step S200: In the traffic scene, set the frame-by-frame ordered bounding box for each tracking target according to the multi-target tracking task.

[0019] In the embodiment of the present application, when setting the frame-by-frame ordered bounding box for each tracked target in the traffic scene according to the multi-target tracking task, first, target detection is performed on each frame of the video sequence to obtain the detection results containing the target category and two-dimensional position. Then, the K targets detected in the current frame are matched with the historical trajectories of the previous frames to obtain K historical trajectory sets. For the K matched trajectories, when K is greater than or equal to 2, identity prediction is performed on each target in combination with the detection results of the current frame to maintain the identity consistency between consecutive frames. Through the continuous processing process of detection, matching and identity prediction, an ordered bounding box arranged in time sequence is established for each target.

[0020] Further, as shown in Figure 2 In the method provided by the embodiment of the present application, setting the frame-by-frame ordered bounding box for each tracking target in the traffic scene according to the multi-target tracking task further includes:

[0021] For each frame, match the K detected tracking targets with the historical trajectories to obtain K historical trajectory sets, and K is greater than or equal to 2; and perform ID prediction based on the K historical trajectory sets in combination with the detection results of the current frame.

[0022] In the embodiment of the present application, first, K targets are detected in the current frame, and the category, position and appearance feature information thereof are extracted. Then, these detected targets are matched with the historical trajectory set. The historical trajectory set is denoted as Where K represents the historical trajectories of K targets; for the historical trajectory of a target k, it is defined as N represents the time step. This represents the trajectory information of target k at time step N.

[0023] Then, for a given historical trajectory Based on the detection results of the current frame, ID prediction is performed. This process is represented as ID( )= .in, This represents the target detection result in the current frame t. Represents the historical trajectory before frame t, ID( ) represents the target in the current frame t. ID prediction results. This represents an end-to-end multi-target tracking model.

[0024] Step S300: Based on the frame-by-frame ordered bounding boxes and the target tracking dataset, construct the network structure of the end-to-end multi-target tracking model.

[0025] In this embodiment, during the construction of the network structure for an end-to-end multi-target tracking model based on frame-by-frame ordered bounding boxes and a target tracking dataset, a target tracking dataset from a traffic scene is first used as the model input. This dataset contains target categories, target IDs, and target 2D detection box information, providing spatial and identity annotation supervision signals for the network. Subsequently, the overall framework of the end-to-end multi-target tracking model is constructed, consisting of a DETR-based target detector, a learnable ID dictionary, and a Transformer-based ID predictor.

[0026] In the model structure design, the DETR-based object detector uses the input image as the starting point for feature extraction. It employs a convolutional neural network (CNN) as the backbone to extract multi-scale image features and uses a Transformer encoder to model spatial features, capturing the global dependencies between objects in the scene. Then, a Transformer decoder performs object query operations to generate an embedding representation for each detected object. Subsequently, a learnable ID dictionary is constructed, representing the identity information of different objects in high-dimensional vector form. Each identity embedding vector corresponds to a fixed ID, used to maintain object consistency across time series. The detection-derived embedding representation is concatenated with the identity embedding representation to obtain a combined representation containing object features and identity information, used to represent historical trajectories. Finally, a Transformer-based ID predictor processes the combined representation, inputting historical trajectories and the detection results of the current frame, to predict and update object identities, thus achieving integrated learning of detection and association across consecutive frames.

[0027] After the above feature extraction, identity embedding construction and Transformer prediction steps, an end-to-end multi-target tracking network structure that can realize unified optimization of target detection and identity association is finally obtained, improving the tracking accuracy and robustness in complex traffic scenes.

[0028] Further, the method provided by the application embodiment further comprises:

[0029] The end-to-end multi-target tracking model comprises a DETR-based target detector, a learnable ID dictionary, and a Transformer-based ID predictor.

[0030] In the application embodiment, the end-to-end multi-target tracking model is composed of three core components, which are a DETR-based target detector, a learnable ID dictionary, and a Transformer-based ID predictor. Among them, the DETR-based target detector is responsible for extracting target features and spatial position information from the input traffic scene image. After extracting image features using a convolutional neural network, the DETR-based target detector realizes target detection and embedding representation generation through a Transformer encoder and decoder structure; the learnable ID dictionary is used to store and update the identity embedding representation of different targets, and the identity information of each target is expressed in the form of a high-dimensional vector to maintain the consistency of the target in the time sequence; the Transformer-based ID predictor receives the target embedding representation and identity embedding representation output by the detector, and combines historical trajectory information to perform identity prediction on the detected targets in the current frame, thereby realizing end-to-end integrated modeling of target detection and identity association.

[0031] Further, the method provided by the application embodiment further comprises:

[0032] In the DETR-based target detector, a convolutional neural network is used as a backbone network to extract image features according to the input image.

[0033] In the application embodiment, in the DETR-based target detector, a convolutional neural network (CNN) is used as a backbone network to extract image features according to the input image , and multi-level spatial features and semantic information are extracted from the input traffic scene image, providing basic feature input for subsequent Transformer-based feature encoding and target embedding representation.

[0034] Further, the method provided by the application embodiment further comprises:

[0035] The spatial feature processing is performed by using a Transformer-based encoder, and the object query instruction receiving and the embedding representation generation are performed by using a Transformer-based decoder.

[0036] In the embodiment of the present application, first, the spatial feature processing is performed on the features of the input image by using a Transformer-based encoder. The encoder takes the image feature map extracted by a convolutional neural network (CNN) as input, models the relationship between the features through a multi-layer self-attention mechanism, thereby capturing the spatial dependence and global context information between different targets in the traffic scene, and realizing the high-dimensional representation of the global features of the image. Subsequently, the Transformer-based decoder of the DETR-based target detector receives the object query instruction, analyzes and matches the spatial features output by the encoder. The decoder calculates the correspondence between the object query and the encoded features through a multi-head attention mechanism, and gradually generates the embedding representation of the target. The embedding representation output by the decoder is denoted as ={ , ,..., }, where the subscript t represents the current frame, and M represents the number of detected targets.

[0037] Through the feature modeling and embedding generation process completed by the Transformer-based encoder and the Transformer-based decoder, the DETR-based target detector realizes the global spatial feature extraction and object-level embedding representation generation of multiple targets in the traffic scene, and provides basic feature input for the end-to-end multi-target tracking model.

[0038] Further, the method provided in the embodiment of the application further comprises:

[0039] According to the learnable ID dictionary, the embedding identifier and the identity embedding identifier are obtained, and the historical trajectory is obtained by combining the embedding identifier and the identity embedding identifier.

[0040] In the embodiment of the present application, first, the embedding identifier and the identity embedding identifier are obtained according to the learnable ID dictionary. In this process, the learnable ID dictionary is defined as , where I represents the ID dictionary, which contains K+1 learnable words, that is Each word is a C-dimensional embedding representation, which is used to represent the identity feature of the target; and is a special mark, which is used to represent a new target that has not been assigned an identity identifier. The value of K is set according to the composition of the target tracking data set, which is much larger than the average number of targets in a single frame image, so as to ensure the sufficient expression ability of the identity space.

[0041] After obtaining the identity embedding representation, the target embedding representation is extracted from the DETR-based target detector output ={ , ,..., }。Subsequently, the detected target embedding representation is combined with the identity embedding representation selected from the learnable ID dictionary, and feature fusion is realized through a concatenation operation in the channel dimension. The combination process can be represented as . Wherein, represents the tracked target k, the historical trajectory information at time step t, and concat represents concatenation in the channel dimension, represents the identity embedding representation, represents the embedding representation of the target detection output, represents the feedforward neural network, represents feature mapping of the embedding representation of the target detection output using the FFN, so as to be feature spliced with the identity embedding representation.

[0042] Through the above feature extraction and fusion process based on the learnable ID dictionary, a combined representation containing target appearance features and identity features, i.e., the historical trajectory is finally obtained.

[0043] Further, the method provided by the application embodiment further comprises:

[0044] In the ID predictor based on the Transformer, a multi-layer Transformer decoder is used as the ID predictor.

[0045] In the application embodiment, in the ID predictor based on the Transformer, a multi-layer Transformer decoder is used as the ID predictor. The structure takes the historical trajectory and the current frame detection result as input, realizes feature association and identity prediction through the multi-layer stacked Transformer decoder. Each layer of the Transformer decoder is composed of a multi-head attention mechanism and a feedforward neural network, which is used to capture the association relationship of different targets in the time sequence, and to perform feature fusion and semantic aggregation between layers, so as to output the identity prediction result of each target in the continuous frame, and realize identity consistency tracking of multiple targets in the traffic scene.

[0046] Further, the method provided by the application embodiment further comprises:

[0047] Based on the ID predictor, the historical trajectory and the detection result are input, and a predicted ID is output.

[0048] In the embodiment of the present application, in the Transformer-based ID predictor, the input is the historical trajectory and the current frame detection result. The historical trajectory is used to provide the motion information and identity features of the tracked target in the time sequence, and the current frame detection result is used to reflect the target detection state at the latest time. The Transformer-based ID predictor models and fuses the two types of input information through a multi-layer Transformer decoder, captures the dependency relationship of different targets in the time dimension through the multi-head attention mechanism, and completes feature mapping and semantic aggregation through the feedforward neural network. After completing the feature fusion, the Transformer-based ID predictor performs identity discrimination and correlation update on the fused target features, and finally outputs the predicted ID of each detection target.

[0049] Step S400: Set the loss function of the end-to-end multi-target tracking model training, and perform model optimization and parameter update.

[0050] In the embodiment of the present application, in the end-to-end multi-target tracking model, the loss function of the model training is set to realize model optimization and parameter update. The loss function is composed of two parts of classification loss function and target box position regression loss function. Among them, the classification loss function takes the ID of the target as the training label, is used to judge whether two targets belong to the same identity, adopts the cross entropy loss function to calculate, and is expressed as , wherein, represents the predicted output ID category, and L represents the labeled true value.

[0051] The target box position regression loss function adopts the IoU Loss (Intersection over Union loss function), which is used to measure the position difference between the predicted box and the real labeled box. In the specific calculation, the predicted box is denoted as P, and the labeled box is denoted as G. The corresponding IoU is denoted as , and the loss is defined as . Through the joint optimization of the above loss function, the parameter update and performance improvement of the end-to-end multi-target tracking model are realized.

[0052] In the embodiment of the present application, as described above, the embodiment of the present application at least has the following technical effects:

[0053] The application constructs a target tracking data set of a traffic scene, the target tracking data set comprising target categories, target IDs, and target two-dimensional detection box information; in the traffic scene, for each tracking target, a frame-by-frame ordered bounding box is set for a multi-target tracking task; a network structure of an end-to-end multi-target tracking model is constructed based on the frame-by-frame ordered bounding box and the target tracking data set; a loss function for training of the end-to-end multi-target tracking model is set, and model optimization and parameter updating are performed. The application solves the technical problem of insufficient accuracy and robustness of a multi-target tracking method in the prior art in a complex traffic scene, and achieves the technical effect of improving tracking accuracy and robustness by constructing an end-to-end multi-target tracking model.

[0054] It should be noted that the above sequence of the embodiments of the application is only for description, and does not represent the advantages and disadvantages of the embodiments. The above describes specific embodiments of the present application. The processes depicted in the drawings do not necessarily require the specific order and continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are also possible or can be advantageous.

[0055] The above is only a preferred embodiment of the application, and does not limit the application in any form. Although the application has been disclosed as above, it is not intended to limit the application. Any person skilled in the art can make minor changes or modifications to the above disclosed technical content without departing from the scope of the technical solution of the application, and any modification, equivalent change and modification of the above embodiments according to the technical essence of the application are still within the scope of the technical solution of the application.

Claims

1. A multi-target tracking method based on traffic scenarios, characterized in that, The method includes: Construct a target tracking dataset for a traffic scenario, which includes target category, target ID, and target 2D detection box information; In the traffic scenario, in accordance with the multi-target tracking task, a frame-by-frame ordered bounding box is set for each tracked target; Based on the frame-by-frame ordered bounding boxes and the target tracking dataset, the network structure of an end-to-end multi-target tracking model is constructed. Set the loss function for training the end-to-end multi-target tracking model, and perform model optimization and parameter updates.

2. The method as described in claim 1, characterized in that, In the traffic scenario, in accordance with the multi-target tracking task, a frame-by-frame ordered bounding box is set for each tracked target. The method includes: For each frame, the detected K tracking targets are matched with historical trajectories to obtain a set of K historical trajectories, where K is greater than or equal to 2; Based on the K historical trajectory sets and the detection results of the current frame, ID prediction is performed.

3. The method as described in claim 2, characterized in that, Based on the frame-by-frame ordered bounding boxes and the target tracking dataset, a network structure for an end-to-end multi-target tracking model is constructed, the method comprising: The end-to-end multi-target tracking model includes a DETR-based target detector, a learnable ID dictionary, and a Transformer-based ID predictor.

4. The method as described in claim 3, characterized in that, The end-to-end multi-target tracking model includes a DETR-based target detector, and the method further includes: In the DETR object detector, image features are extracted using a convolutional neural network as the backbone network based on the input image.

5. The method as described in claim 4, characterized in that, The method further includes: Spatial feature processing is performed using a Transformer-based encoder; The object query instruction is received and an embedded representation is generated using a Transformer-based decoder.

6. The method as described in claim 3, characterized in that, The method includes: Based on the learnable ID dictionary, obtain the embedding identifier and the identity embedding identifier; The historical trajectory is obtained by combining the embedded identifier and the identity embedded identifier.

7. The method as described in claim 3, characterized in that, The method includes: In the Transformer-based ID predictor, a multi-layer Transformer decoder is used as the ID predictor.

8. The method as described in claim 7, characterized in that, The method includes: Based on the ID predictor, the historical trajectory and the detection result are input, and the predicted ID is output.

Images