Target trajectory tracking method, tracking device, electronic equipment and storage medium

Abstract

The application discloses a target trajectory tracking method, a tracking device, an electronic device and a storage medium. The target trajectory tracking method comprises the following steps: acquiring attribute data of each detection box in each frame of image for an image collected by a vehicle camera device; encoding the attribute data of each detection box in each frame of image to obtain encoded features; inputting the encoded features into a pre-trained converter model for processing to obtain information of a target trajectory; and in the encoder of the model, one query is used to represent one trajectory. The target trajectory tracking method provided by the application embodiment realizes learning-based trajectory tracking, realizes the association and refinement between objects in a learning-based manner, can provide efficiency in a data-driven manner, reduces the labor cost, and has good trajectory tracking effect, high efficiency and good applicability.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a target trajectory tracking method, tracking device, electronic device, and storage medium. Background Technology

[0002] Offline annotation schemes in related technologies use rule-based methods to achieve trajectory tracking. The association between objects is achieved through rule-based methods, which has limited applicability. To solve the problems in the current scenario, modifications are often required, which can easily lead to inapplicability when applied to other scenarios. Therefore, a lot of manpower is needed to make a lot of adjustments to maintain the applicability balance of this method across various scenarios. The scheme design and adjustment process is extremely cumbersome, with poor tracking results, low efficiency, and poor applicability.

[0003] The above statements are for the purpose of providing background information in relation to this application only and do not necessarily constitute prior art. Summary of the Invention

[0004] The purpose of this application is to provide a target trajectory tracking method, tracking device, electronic device, and storage medium. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general description, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0005] According to one aspect of the embodiments of this application, a target trajectory tracking method is provided, comprising:

[0006] For images captured by the vehicle camera, acquire the attribute data of each detection box in each frame of the image;

[0007] The attribute data of each detection box in each frame of the image is encoded to obtain the encoded features;

[0008] The encoded features are input into a pre-trained converter model for processing to obtain information about the target trajectory; in the encoder of the model, a query is used to represent a trajectory.

[0009] In some embodiments of this application, the attribute data of the detection box includes the position, size, orientation angle, velocity, confidence score, and object features of the object in the detection box;

[0010] The acquisition of attribute data for each detection box in each frame of the image includes:

[0011] For each frame of the image, extract the feature map of that image;

[0012] A pre-trained object detection model generates detection boxes on the feature map;

[0013] Identify objects within the detection frame and obtain the object's position, size, orientation angle, speed, confidence score, and object features.

[0014] In some embodiments of this application, encoding the attribute data of each detection box in each frame of the image to obtain encoded features includes:

[0015] Concatenate all attribute data of each detection box into a vector;

[0016] By using a linear feature encoding layer, the concatenated vectors are mapped to a preset feature space through linear transformation to obtain the encoded features.

[0017] In some embodiments of this application, the step of inputting the encoded features into a pre-trained converter model for processing to obtain information about the target trajectory includes:

[0018] The encoded features and their positional encodings are input into the encoder of the converter model for processing to obtain a context vector, which is then input into the decoder.

[0019] The context vector, position encoding, and query of the current decoding state are input into the decoder for processing to obtain information about the target trajectory; the context vector is used as a key and value in the decoder.

[0020] In some embodiments of this application, the step of inputting the context vector, position encoding, and current decoding state query into the decoder for processing to obtain target trajectory information includes:

[0021] The context vector, the positional encoding, and the query of the current decoding state are combined through the multi-head self-attention mechanism of the decoder to obtain a concatenated vector of weighted sum values;

[0022] The feedforward neural network of the decoder processes the concatenated vector of the weighted sum values ​​to generate information about the target trajectory.

[0023] In some embodiments of this application, the decoder further includes a classification task head and a regression task head; the classification task head is used to determine the category to which the target trajectory belongs; the regression task head is used to regress the information of each detection box in the target trajectory.

[0024] In some embodiments of this application, during the pre-training process of the converter model, the prediction results of the converter model are matched with the true values ​​to obtain a matching relationship that meets preset requirements.

[0025] The loss function used in the pre-training process includes at least one of the following: category loss function, localization loss function, and GIoU loss function;

[0026] The category loss function is used to calculate the difference between the predicted category and the true category;

[0027] The positioning loss function is used to calculate the difference between the predicted trajectory location and the actual location.

[0028] The GIoU loss function is used to calculate the overlap between the predicted bounding box value and the true value of the trajectory box.

[0029] According to another aspect of the embodiments of this application, a target trajectory tracking device is provided, comprising:

[0030] The attribute data acquisition module is used to acquire attribute data for each detection box in each frame of the image captured by the vehicle camera device.

[0031] The encoding module is used to encode the attribute data of each detection box in each frame of the image to obtain the encoded features;

[0032] The target trajectory information acquisition module is used to input the encoded features into a pre-trained converter model for processing to obtain the target trajectory information.

[0033] According to another aspect of the embodiments of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the target trajectory tracking method described in any embodiment of this application.

[0034] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, the computer program being executed by a processor to implement the target trajectory tracking method described in any embodiment of this application.

[0035] One aspect of the technical solution provided in this application embodiment may include the following beneficial effects:

[0036] The target trajectory tracking method provided in this application acquires attribute data of each detection box in each frame of the image captured by the vehicle camera device, encodes the attribute data of each detection box in each frame to obtain encoded features, and inputs the encoded features into a pre-trained converter model for processing to obtain target trajectory information. In the encoder of this model, a query is used to represent a trajectory, thus realizing learning-based trajectory tracking. The learning-based approach enables the association and refinement between objects, improves efficiency through data-driven methods, reduces labor costs, and has good trajectory tracking effect, high efficiency, and good applicability.

[0037] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of this application are described below. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A flowchart of a target trajectory tracking method according to an embodiment of this application is shown.

[0040] Figure 2 A flowchart illustrating the acquisition of attribute data for each detection box in each frame of an image is shown in one embodiment of this application.

[0041] Figure 3 A flowchart illustrating a target trajectory tracking method according to an embodiment of this application is shown.

[0042] Figure 4 This document illustrates a flowchart of encoding attribute data for each detection box in each frame of an image, according to one embodiment of this application.

[0043] Figure 5 The diagram illustrates a flowchart in one embodiment of this application, showing how encoded features are input into a pre-trained Transformer model for processing to obtain information about the target trajectory.

[0044] Figure 6 A schematic diagram of a Transformer model in one embodiment of this application is shown.

[0045] Figure 7 A structural block diagram of a target trajectory tracking device according to an embodiment of this application is shown.

[0046] Figure 8 A block diagram of an electronic device structure according to an embodiment of this application is shown.

[0047] Figure 9 A schematic diagram of a computer-readable storage medium according to an embodiment of this application is shown. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0049] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have a meaning consistent with their meaning in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0050] Offline 4D annotation is a type of annotation method where "4D" refers to information in four dimensions. Specifically, 4D annotation typically includes three-dimensional spatial location (X, Y, Z coordinates) and a time dimension (T). That is, the annotation data not only includes the spatial location of the target in an image or video frame, but also the target's position and motion trajectory at different points in time.

[0051] Offline annotation schemes in related technologies use a rule-based approach to track trajectory. For example, DetZero uses a rule-based approach to track trajectory, and then connects to the network to refine the information. This requires fine-tuning of each module and takes a long time to deploy.

[0052] To address the technical problems of cumbersome solution design and adjustment processes, poor tracking performance, low efficiency, and poor applicability in related technologies, this application provides a target trajectory tracking method. For images captured by a vehicle camera device, attribute data of each detection box in each frame is obtained. The attribute data of each detection box in each frame is encoded to obtain encoded features. These encoded features are then input into a pre-trained Transformer model for processing to obtain target trajectory information. In the model's encoder, a query represents a trajectory. This method achieves learning-based trajectory tracking, realizing object association and refinement through a learning-based approach. It improves efficiency through data-driven methods, reduces labor costs, and offers good trajectory tracking performance, high efficiency, and good applicability.

[0053] The following description, in conjunction with the accompanying drawings, describes a target trajectory tracking method, tracking device, electronic device, and storage medium according to embodiments of this application.

[0054] refer to Figure 1 As shown, one embodiment of this application provides a target trajectory tracking method, which may include steps S10-S30:

[0055] S10. For the images captured by the vehicle camera device, obtain the attribute data of each detection box in each frame of the image.

[0056] Vehicle camera devices include, but are not limited to, cameras, dashcams, etc.

[0057] In some implementations, the attribute data of the detection box may include the position, size, orientation angle, velocity, confidence score, and object features of the object in the detection box.

[0058] refer to Figure 2 As shown, obtaining the attribute data of each detection box in each frame of the image can include:

[0059] S101. For each frame of image, extract the feature map of that image.

[0060] Useful feature maps can be extracted from raw images using pre-trained convolutional neural networks (CNNs). For example, common network architectures such as ResNet, VGG, and MobileNet are often used in vehicle monitoring scenarios to extract local and global features. Convolutional and pooling layers can be used to extract low-level features of the image (such as edges, colors, and textures). High-level features can be further captured through fully connected layers and attention mechanisms, aiding in more accurate object detection.

[0061] Before feature map extraction, each frame of the image can be preprocessed. For example, the pixel values ​​of the image can be adjusted to a suitable range by image normalization (e.g., standardizing the pixel values ​​to the range of [0, 1] or [-1, 1]). If the size of the input images is inconsistent, scaling can be performed. The images can be cropped or padded to ensure the consistency of the network input.

[0062] S102. Generate detection boxes on the feature map using a pre-trained target detection model.

[0063] A detection box, also known as a bounding box, is a frame used in object detection tasks to define the location of objects in an image. Specifically, a detection box is a rectangular box used to describe the boundary of an object in an image. It is typically represented by four coordinate values: the coordinates of the top-left and bottom-right corners, or by the coordinates of the center point and its width and height. Detection boxes can be generated on the image's feature map using object detection models (such as YOLO, Faster R-CNN, RetinaNet, etc.) to locate and identify objects. For example, a regression network can predict the bounding box location of each object to determine its position and size.

[0064] S103. Identify the object in the detection box and obtain the object's position, size, orientation angle, speed, confidence score, and object features.

[0065] The category (e.g., vehicle, pedestrian) of a target can be predicted using a classification task head. A classification layer (e.g., softmax) then predicts the confidence score for each detection box; the confidence score characterizes the probability that the detection box contains the target. For vehicles in dynamic scenes, it's necessary not only to obtain the target's position and size but also to estimate its orientation angle, especially in driving environments.

[0066] Orientation angles can be obtained in several ways: They can be predicted using an artificial neural network model via a regression task head (typically the rotation angle around an axis, usually the angle between the vehicle's direction of travel and a reference coordinate axis); or, key points of the vehicle (such as the positions of the front and rear wheels) can be detected, and the orientation angle can be calculated from these key points; the angle can be output via a regression task head; or the angle can be inferred by applying coordinate transformations to the object's bounding box.

[0067] Vehicle speed estimation typically relies on image changes between consecutive frames, inferring object motion based on multi-frame data. For example, optical flow algorithms can be used to estimate the velocity of object pixels in an image, thereby inferring the object's direction and speed. Alternatively, the motion of the same object can be tracked across multiple frames, calculating its displacement and time intervals to infer its velocity. Common target tracking algorithms such as Kalman filters, SORT (Simple Online and Realtime Tracking), or DeepSORT can be used to track vehicles and estimate their speed. In this process, the target is detected in each frame, its positional changes between frames are calculated, and the velocity is estimated by combining these calculations with the time intervals.

[0068] Object features refer to information describing the appearance or other characteristics of a target in an image. For example, for a vehicle, its appearance features (color, shape, brand logo, etc.) might be used for feature description. Feature vectors of target objects can be extracted using artificial neural networks; common methods include using convolutional neural networks to extract deep features. Models such as ResNet, VGG, or Siamese networks can be used to extract feature information related to the target object during training.

[0069] The features of an object can include its external attributes, such as color, shape, and texture. The object features contained in each bounding box are crucial for object tracking and classification.

[0070] In a specific example, such as Figure 3 As shown, each detection box has a set of attributes, with each attribute having a dimension of numChannel. The attribute information includes position (xyz), size (hwl), orientation angle, velocity information, confidence score, and object features extracted from the detection model. All objects detected within time T are labeled as obj1, obj2, obj3, ..., obj n-1 obj n Each frame of an image has Nf detection boxes (if the number of detection boxes in a frame is less than Nf, padding is used). All boxes within time T are aggregated as the model input, with dimensions Nf*T*num channels. Padding is an operation used to adjust the shape or length of data to conform to a predetermined structure. In object detection tasks, padding is mainly used to fill in the insufficient number of detection boxes to ensure that the number of output detection boxes is Nf (i.e., the process of padded to Nf). For example, missing boxes are filled with zero boxes (such as (0, 0, 0, 0)) to ensure that subsequent processing is not affected and to maintain a consistent data format.

[0071] S20. Encode the attribute data of each detection box in each frame of the image to obtain the encoded features.

[0072] The goal of the encoding process is to merge multiple attribute data (such as position, size, velocity, etc.) of each detection box into a unified feature representation. This feature representation, i.e., the encoded feature, can be used as input to the model for further computation or prediction. Using a linear feature encoding layer, these attribute data can be mapped to a high-dimensional feature space through linear transformation.

[0073] In some implementations, reference Figure 4 As shown, the attribute data of each detection box in each frame of the image is encoded to obtain the encoded features, which may include:

[0074] S201. Concatenate all attribute data of each detection box into a vector.

[0075] Before encoding the attributes of each detection box, all attribute data is concatenated into a single vector. For example, for the following six types of attribute data:

[0076] The position is represented by four values ​​(x_min, y_min, x_max, y_max);

[0077] The dimensions are represented by two values ​​(width, height);

[0078] Orientation angle, represented by a single numerical value (theta);

[0079] Speed ​​is represented by two values ​​(v_x, v_y);

[0080] The confidence score is represented by a single numerical value.

[0081] Object features are represented by a fixed-length feature vector (assuming it is 128-dimensional).

[0082] The size of the concatenated vector is 4 (position) + 2 (size) + 1 (orientation angle) + 2 (velocity) + 1 (confidence) + 128 (object features) = 138; that is, the size of the concatenated feature vector corresponding to each detection box is 138.

[0083] S202. Through a linear feature encoding layer, the concatenated vector is mapped to a preset feature space using a linear transformation to obtain the encoded features.

[0084] A linear feature encoding layer is used to map this concatenated feature vector. For example, a fully connected layer can be used to map the input feature vector to another feature space.

[0085] The weight matrix W corresponding to the linear feature encoding layer is a d×138 weight matrix, where d is the dimension of the target feature space. This dimension d can be set to be larger than the original feature dimension to provide richer representation capabilities. A bias term b, with dimension d, can be set.

[0086] The feature vector a (assuming a dimension of 138) of each detection box is input into the linear feature encoding layer, and the operation Wa+b is performed to obtain the encoded vector q, q = Wa+b.

[0087] Through the encoding operation of the linear feature encoding layer, each detection box obtains a new feature representation, i.e., the encoded feature. This feature representation combines all the attribute data of the detection box (position, size, orientation angle, velocity, confidence score, object features, etc.) and maps this information to a new high-dimensional space for subsequent model processing.

[0088] In a specific example, refer to Figure 3 As shown, the model input passes through a linear feature encoding layer (MLP) to encode the attributes of each object, resulting in encoded features with a dimension of Nf*T*d_model. This encoding process converts the attribute information of each object into a feature vector that the model can process.

[0089] S30. Input the encoded features into the pre-trained Transformer model for processing to obtain information about the target trajectory; in the encoder of the Transformer model, a query is used to represent a trajectory.

[0090] The Transformer model, also known as the Transformer Model, has an encoder-decoder architecture. The encoder and decoder of a Transformer model can each include multiple multi-head self-attention layers and feedforward neural network layers.

[0091] The encoder's task is to process the input feature matrix and generate a contextual representation containing information about all bounding boxes. Representing a trajectory with a query not only focuses on the object's current state but also captures the object's motion trajectory through continuous time steps (time series). This helps the model understand the spatial changes of the object at different points in time and predict its future position, allowing for more direct prediction of the object's motion trajectory and improving prediction accuracy.

[0092] In some implementations, reference Figure 5 As shown, the encoded features are input into a pre-trained Transformer model for processing to obtain information about the target trajectory, which may include:

[0093] S301. Input the encoded features and their position encodings into the encoder of the Transformer model for processing to obtain the context vector, and then input the context vector into the decoder.

[0094] S302. Input the context vector, position encoding, and query of the current decoding state into the decoder for processing to obtain information about the target trajectory. The context vector is used as a key and value in the decoder, and in the encoder, a query is used to represent a trajectory.

[0095] The key (Key, K) and value (Value, V) in the decoder are obtained based on the encoder's output.

[0096] For example, inputting the context vector, position encoding, and query of the current decoding state into the encoder for processing to obtain information about the target trajectory may include:

[0097] By combining the context vector, positional encoding, and query of the current decoding state through the multi-head self-attention mechanism of the decoder, a concatenated vector of weighted sum values ​​is obtained.

[0098] The feedforward neural network of the decoder processes the concatenated vector of weighted sum values ​​to generate information about the target trajectory.

[0099] For example, the decoder also includes a classification task head and a regression task head; the classification task head is used to determine the category to which the target trajectory belongs; the regression task head is used to regress information about each detection box in the target trajectory.

[0100] In a specific example, refer to Figure 3 As shown, the encoded features are input into the Transformer model. The structure of the Transformer model is as follows: Figure 6As shown, the Transformer model includes an encoder and a decoder. The encoder extracts global information from the input features, while the decoder generates information about the target trajectory. The encoder includes a normalization layer, a multi-head self-attention layer, and a feedforward neural network layer; the decoder includes a normalization layer, a multi-head self-attention layer, and a feedforward neural network layer.

[0101] The model in this example is a DETR (Detection Transformer) model, an end-to-end object detection model based on the Transformer architecture, where a trajectory is represented by a single query. This model treats object detection as an ensemble prediction problem, directly predicting the set of objects of interest in an image through the global modeling capabilities of the Transformer. By viewing object detection as predicting the set of objects of interest in an image, rather than predicting the bounding box and class of each object individually, this simplifies the detection process and improves detection efficiency.

[0102] The model in this example employs a Transformer encoder-decoder architecture. The encoder extracts image features and performs self-attention, while the decoder generates predictions for the object category and bounding box. By learning a set of learnable object queries, the model in this example can directly and in parallel output the final set of predictions.

[0103] The Transformer model in this example uses bipartite matching to ensure that each predicted target matches only one true target, thus avoiding duplicate predictions. Matching is achieved by calculating the loss between the predicted and true targets.

[0104] The model in this example, based on DETR, uses a query to represent a trajectory in the task, which differs from the DETR approach in related technologies, which uses a query to represent a detected object. In this example, the number of track queries in the task is Nq, i.e., the number of detected trajectories. The output feature dimension after passing through the encoder and decoder of the Transformer model is Nq*d_model.

[0105] The output features of dimension Nq*d_model are input into the task-related task head, which includes a classification task head and a regression task head.

[0106] The classification task head is used to predict the category of the trajectory (the category of the trajectory includes, but is not limited to, the background class). The regression task head is used to regress the information of each bounding box in the trajectory, and the information of the bounding box includes: time T, position (xyz), size (hwl), and object orientation information.

[0107] In some implementations, during the pre-training process of the transformer model, the prediction results of the transformer model are matched with the ground truth to obtain a matching relationship that meets preset requirements. Specifically, the Hungarian algorithm is used to match the prediction results of the Transformer model with the ground truth to obtain a matching relationship that meets preset requirements, thereby increasing the probability of finding the optimal matching relationship between many-to-many prediction results and ground truth.

[0108] The loss function used in the pre-training process includes at least one of the following: the category loss function, the localization loss function, and the GIoU loss function;

[0109] The category loss function is used to calculate the difference between the predicted category and the true category;

[0110] The localization loss function is used to calculate the difference between the predicted trajectory location and the actual location.

[0111] The GIoU loss function is used to calculate the overlap between the predicted bounding box values ​​and the true values ​​of the trajectory boxes.

[0112] Specifically, during model training, the Hungarian algorithm is used to match model predictions with ground truth values. The matching terms are constructed similarly to the loss function, including class loss, bounding box localization (xyz loss), and GIoU (Generalized Intersection over Union) loss.

[0113] During the training of the Transformer model, the model's predictions are matched with the ground truth using the Hungarian algorithm to increase the probability of finding the optimal match in many-to-many prediction-to-ground correspondences. The Hungarian algorithm is a combinatorial optimization algorithm that solves the task assignment problem in polynomial time. It was developed by W.W. Kuhn in 1955 using a theorem by the Hungarian mathematician König, hence its name. The Hungarian algorithm is an efficient algorithm for solving the minimum-cost matching problem.

[0114] The method in this example combines a Transformer model with tracking-by-detection to predict and refine the trajectory of a target, representing each trajectory as a query and optimizing the prediction results through encoding, matching, and loss calculation.

[0115] The method described in this application can be used for offline 4D annotation, enabling object tracking and position / size refinement. It employs a tracking-by-detection approach, where an upstream detection network model (e.g., YOLO, Faster R-CNN, DETR, etc.) outputs bounding boxes, confidence scores, velocity information, and bounding box features extracted from the network model for each frame. This scheme inputs these outputs into the network for learning, thereby obtaining refined trajectory information.

[0116] In a specific example, the target trajectory tracking method may include the following steps:

[0117] 1) such as Figure 3 As shown, each detection box includes a set of attributes, with a dimension of numchannel. The attribute information includes position (xyz), size (hwl), orientation angle, velocity information, confidence score, and object features extracted from the detection model. Each frame has Nf detection boxes (if the number of detection boxes in a frame is less than Nf, padding is used). All detection boxes within time T are collected as the model input, hereinafter referred to as the model input, with a dimension of Nf*T*numchannel.

[0118] 2) The model input passes through a linear feature encoding layer (MLP) to encode the attributes of each object, resulting in encoded features with a dimension of Nf*T*d_model. This encoding process converts the attribute information of each object into a feature vector that the model can process.

[0119] 3) The encoded features are input into the Transformer model, the structure of which is shown in the figure. The Transformer model includes an encoder and a decoder. The encoder extracts image features and performs self-attention operations, while the decoder generates predictions for the object category and bounding box. By learning a set of learnable object queries, the model in this example can directly and in parallel output the final prediction result set.

[0120] The model in this example, based on DETR, represents a trajectory using a query in the task. The number of track queries in the task is Nq, i.e., the number of detected trajectories. The output feature dimension after the encoder and decoder of the Transformer model is Nq*d_model. In this example, each trajectory is represented as a query, and the prediction results are optimized through encoding, matching, and loss calculation.

[0121] 4) Input the output features of dimension Nq*d_model into the task-related task head, which includes a classification task head and a regression task head.

[0122] The classification task head is used to predict the category of a trajectory (the category of the trajectory includes, but is not limited to, the background class).

[0123] The regression task head is used to regress information about each bounding box in the trajectory. The bounding box information includes: time T, position (xyz), size (hwl), and the orientation information of the object.

[0124] This example follows a tracking-by-detection approach. The upstream detection network outputs bounding boxes, confidence scores, velocity information, and bounding box features extracted from the detection model for each frame. These outputs are then fed into a Transformer model for learning, resulting in refined target trajectory information. This learning-based approach achieves object correlation and refinement, improving efficiency through a data-driven method, reducing labor costs, and demonstrating good trajectory tracking performance, high efficiency, and broad applicability.

[0125] The pre-trained Transformer model in this example uses the Hungarian algorithm to match model predictions with ground truth during the pre-training process. The matching terms are constructed similarly to the loss function, including class loss, bounding box localization (xyz loss), and GIoU (Generalized Intersection over Union) loss.

[0126] During the training of the Transformer model, the model's predictions are matched with the ground truth using the Hungarian algorithm to increase the probability of finding the best match in the many-to-many correspondence between predictions and ground truth.

[0127] Loss functions can include:

[0128] Class loss is used to calculate the difference between the predicted class and the true class for each trajectory;

[0129] The localization loss is used to calculate the difference between the predicted (xyz) position and the true position for each trajectory.

[0130] GIoU loss is used to calculate the difference between the predicted bounding box of the trajectory box and the ground truth, measuring the degree of overlap between the two boxes.

[0131] This example of target trajectory tracking combines a Transformer model with a tracking-by-detection approach to predict and refine the target's trajectory, achieving accurate results and high efficiency. The method employs a learning-based approach, which can improve the upper limit of accuracy, and its data-driven approach reduces manual labor costs. This method can be used for offline 4D annotation, enabling object trajectory tracking and positional dimension refinement.

[0132] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0133] refer to Figure 7 As shown, another embodiment of this application provides a target trajectory tracking device, comprising:

[0134] The attribute data acquisition module is used to acquire attribute data for each detection box in each frame of the image captured by the vehicle camera device.

[0135] The encoding module is used to encode the attribute data of each detection box in each frame of the image to obtain the encoded features;

[0136] The target trajectory information acquisition module is used to input the encoded features into a pre-trained Transformer model for processing to obtain the target trajectory information.

[0137] For example, the attribute data of the detection box includes the position, size, orientation angle, velocity, confidence score, and object features of the object in the detection box;

[0138] The attribute data acquisition module may include:

[0139] The feature map extraction unit is used to extract the feature map of each frame of the image.

[0140] The detection box generation unit is used to generate detection boxes on the feature map using a pre-trained target detection model.

[0141] The object recognition unit is used to identify objects in the detection box and obtain the position, size, orientation angle, speed, confidence score and features of the objects.

[0142] For example, the encoding module may include:

[0143] The concatenation unit is used to concatenate all attribute data of each detection box into a vector.

[0144] The mapping unit is used to map the concatenated vector to a preset feature space through a linear feature encoding layer using a linear transformation, so as to obtain the encoded features.

[0145] For example, the target trajectory information acquisition module may include:

[0146] The encoder unit is used to input the encoded features and the position encoding of the encoded features into the encoder of the Transformer model for processing to obtain a context vector, and then input the context vector into the decoder.

[0147] The decoder unit is used to input the context vector, position encoding, and query of the current decoding state into the decoder for processing to obtain information about the target trajectory; the context vector is used as a key and value in the decoder.

[0148] For example, the decoder unit is further specifically used for:

[0149] The context vector, the positional encoding, and the query of the current decoding state are combined through the multi-head self-attention mechanism of the decoder to obtain a concatenated vector of weighted sum values;

[0150] The feedforward neural network of the decoder processes the concatenated vector of the weighted sum values ​​to generate information about the target trajectory.

[0151] For example, the decoder further includes a classification task head and a regression task head; the classification task head is used to determine the category to which the target trajectory belongs; the regression task head is used to regress information of each detection box in the target trajectory.

[0152] For example, training the Transformer model includes:

[0153] The prediction results of the Transformer model are matched with the true values ​​using the Hungarian algorithm to obtain a matching relationship that meets the preset requirements.

[0154] The loss function used during training includes at least one of the following: category loss function, localization loss function, and GIoU loss function;

[0155] The category loss function is used to calculate the difference between the predicted category and the true category;

[0156] The localization loss function is used to calculate the difference between the predicted trajectory location and the actual location.

[0157] The GIoU loss function is used to calculate the overlap between the predicted bounding box values ​​and the true values ​​of the trajectory boxes.

[0158] The target trajectory tracking device of this application embodiment acquires attribute data of each detection box in each frame of the image captured by the vehicle camera device, encodes the attribute data of each detection box in each frame of the image to obtain encoded features, and inputs the encoded features into a pre-trained Transformer model for processing to obtain target trajectory information. In the encoder of the model, a query is used to represent a trajectory. This method realizes learning-based trajectory tracking, realizes the association and refinement between objects through learning, can improve efficiency through data-driven approach, reduce labor costs, and has good trajectory tracking effect, high efficiency, and good applicability.

[0159] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0160] Another embodiment of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the method of any of the above embodiments.

[0161] refer to Figure 8 As shown, the electronic device 10 may include: a processor 100, a memory 101, a bus 102 and a communication interface 103. The processor 100, the communication interface 103 and the memory 101 are connected through the bus 102. The memory 101 stores a computer program that can run on the processor 100. When the processor 100 runs the computer program, it executes the method provided in any of the foregoing embodiments of this application.

[0162] The memory 101 may include high-speed random access memory (RAM), and may also include non-volatile memory, such as at least one disk storage device. Communication between this device network element and at least one other network element is achieved through at least one communication interface 103 (which may be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0163] Bus 102 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. Memory 101 is used to store programs. After receiving an execution instruction, processor 100 executes the program. The methods disclosed in any of the foregoing embodiments of this application can be applied to processor 100, or implemented by processor 100.

[0164] Processor 100 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 100 or by instructions in software form. The processor 100 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), an Off-the-shelf Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 101. The processor 100 reads the information in memory 101 and, in conjunction with its hardware, completes the steps of the above method.

[0165] The electronic devices and methods provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0166] Another embodiment of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the methods of any of the above embodiments. Reference Figure 9 As shown, the computer-readable storage medium is an optical disc 20, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the methods provided in any of the aforementioned embodiments.

[0167] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0168] The computer-readable storage medium provided in the above embodiments of this application and the method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0169] It should be noted that:

[0170] The term "module" is not intended to be limited to a specific physical form. Depending on the application, a module can be implemented as hardware, firmware, software, and / or a combination thereof. Furthermore, different modules may share common components or even be implemented using the same components. Clear boundaries may or may not exist between different modules.

[0171] The algorithms and displays provided herein are not inherently related to any particular computer, virtual device, or other equipment. Various general-purpose devices can also be used with the examples based on this. The required structure for constructing such devices is obvious from the above description. Furthermore, this application is not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0172] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0173] The above embodiments merely illustrate the implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A target trajectory tracking method, characterized in that, include: For images captured by the vehicle camera, acquire the attribute data of each detection box in each frame of the image; The attribute data of each detection box in each frame of the image is encoded to obtain the encoded features; The encoded features are input into a pre-trained converter model for processing to obtain information about the target trajectory; in the encoder of the model, a query is used to represent a trajectory.

2. The method according to claim 1, characterized in that, The attribute data of the detection box includes the position, size, orientation angle, velocity, confidence score, and object features of the object in the detection box; The acquisition of attribute data for each detection box in each frame of the image includes: For each frame of the image, extract the feature map of that image; A pre-trained object detection model generates detection boxes on the feature map; Identify objects within the detection frame and obtain the object's position, size, orientation angle, speed, confidence score, and object features.

3. The method according to claim 1, characterized in that, The step of encoding the attribute data of each detection box in each frame of the image to obtain the encoded features includes: Concatenate all attribute data of each detection box into a vector; By using a linear feature encoding layer, the concatenated vectors are mapped to a preset feature space through linear transformation to obtain the encoded features.

4. The method according to claim 1, characterized in that, The step of inputting the encoded features into a pre-trained converter model for processing to obtain information about the target trajectory includes: The encoded features and their positional encodings are input into the encoder of the converter model for processing to obtain a context vector, which is then input into the decoder. The context vector, position encoding, and query of the current decoding state are input into the decoder for processing to obtain information about the target trajectory; the context vector is used as a key and value in the decoder.

5. The method according to claim 4, characterized in that, The step of inputting the context vector, position encoding, and current decoding state into the decoder for processing to obtain target trajectory information includes: The context vector, the positional encoding, and the query of the current decoding state are combined through the multi-head self-attention mechanism of the decoder to obtain a concatenated vector of weighted sum values; The feedforward neural network of the decoder processes the concatenated vector of the weighted sum values ​​to generate information about the target trajectory.

6. The method according to claim 5, characterized in that, The decoder also includes a classification task head and a regression task head; the classification task head is used to determine the category to which the target trajectory belongs; the regression task head is used to regress the information of each detection box in the target trajectory.

7. The method according to claim 1, characterized in that, During the pre-training process of the converter model, the prediction results of the converter model are matched with the true values ​​to obtain a matching relationship that meets the preset requirements. The loss function used in the pre-training process includes at least one of the following: category loss function, localization loss function, and GIoU loss function; The category loss function is used to calculate the difference between the predicted category and the true category; The positioning loss function is used to calculate the difference between the predicted trajectory location and the actual location. The GIoU loss function is used to calculate the overlap between the predicted bounding box value and the true value of the trajectory box.

8. A target trajectory tracking device, characterized in that, include: The attribute data acquisition module is used to acquire attribute data for each detection box in each frame of the image captured by the vehicle camera device. The encoding module is used to encode the attribute data of each detection box in each frame of the image to obtain the encoded features; The target trajectory information acquisition module is used to input the encoded features into a pre-trained converter model for processing to obtain the target trajectory information.

9. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the target trajectory tracking method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by a processor to implement the target trajectory tracking method as described in any one of claims 1-7.

Images