Obstacle tracking method, apparatus, device, and storage medium

Abstract

The application relates to the field of artificial intelligence and discloses an obstacle tracking method, device and equipment and a storage medium, the method comprising the following steps: acquiring multiple bird's-eye view images under the same time axis; performing obstacle identification on the multiple bird's-eye view images to obtain 3D candidate boxes of obstacles in each bird's-eye view image; inputting each bird's-eye view image into a preset obstacle tracking model according to the time axis; assigning a tracker number to the 3D candidate box of the input bird's-eye view image through the obstacle tracking model; and generating an obstacle trajectory according to the 3D candidate boxes with the same tracker number in each bird's-eye view image. The method fuses multiple steps of a multi-target tracking task into a single model, reduces the use of computing resources, improves system efficiency, improves the speed of obstacle tracking in the process of automatic driving or auxiliary driving, and uses a three-dimensional obstacle box of a bird's-eye view image as model input, thereby improving the accuracy of obstacle detection.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to an obstacle tracking method, apparatus, device, and storage medium. Background Technology

[0002] As a modern means of transportation, automobiles have changed people's lifestyles, driven socio-economic development and cultural progress, and become an indispensable part of society. To prevent traffic accidents and reduce driver workload, assisted driving has become a prevalent trend in driving technology. Vision-based driving systems have become an important component of intelligent transportation systems.

[0003] Multiple Classes Object Tracking (MOT) is a classic problem in computer vision. Without knowing the number of objects beforehand, it's necessary to detect and assign IDs to multiple objects such as pedestrians, cars, and animals in a video for trajectory tracking. Current MOT techniques mainly involve object detection in the image, feature extraction from the detected objects, similarity calculation of obstacles based on the extracted features, matching and associating obstacles using the Hungarian algorithm, and finally tracking the obstacles based on the association results. Because these steps are performed separately, the computational load is high, the response speed is slow, and it cannot react quickly when vehicles are traveling at high speeds. Summary of the Invention

[0004] The main objective of this invention is to solve the technical problem that existing multi-target tracking solutions involve performing each step separately, resulting in slow response times.

[0005] The first aspect of the present invention provides an obstacle tracking method, comprising: acquiring multiple frames of bird's-eye view images on the same time axis; performing obstacle identification on the multiple frames of bird's-eye view images to obtain 3D candidate boxes of obstacles in each frame of bird's-eye view images; inputting each frame of bird's-eye view images into a preset obstacle tracking model according to the time axis, and assigning tracker numbers to the 3D candidate boxes of the input bird's-eye view images through the obstacle tracking model; and generating obstacle trajectories based on the 3D candidate boxes with the same tracker numbers in each frame of bird's-eye view images.

[0006] Optionally, in a first implementation of the first aspect of the present invention, the step of inputting each frame of the bird's-eye view image into a preset obstacle tracking model according to the time axis, and assigning tracker numbers to the 3D candidate boxes of the input bird's-eye view image through the obstacle tracking model includes: inputting each frame of the bird's-eye view image into the preset obstacle tracking model; performing a convolution operation on the 3D candidate boxes of each frame of the input bird's-eye view image through the obstacle tracking model to calculate the appearance features corresponding to each 3D candidate box; processing each 3D candidate box and its corresponding appearance features according to a preset self-attention mechanism and the time axis to generate a historical frame tracker and obtain the matching relationship between the historical frame tracker and each 3D candidate box; and assigning a tracker number corresponding to the historical frame tracker to each 3D candidate box according to the matching relationship.

[0007] Optionally, in a second implementation of the first aspect of the present invention, the step of inputting each frame of the bird's-eye view into a preset obstacle tracking model, and performing a convolution operation on the 3D candidate boxes of each frame of the input bird's-eye view through the obstacle tracking model to calculate the appearance features corresponding to each 3D candidate box includes: inputting each frame of the bird's-eye view into a preset obstacle tracking model, and using the obstacle tracking model to extract candidate box images corresponding to the 3D candidate boxes in the bird's-eye view; encoding the candidate box images through convolutional layers in the obstacle tracking model to obtain the encoded features of the candidate box images; and decoding the encoded features through transposed convolutional layers in the obstacle tracking model to obtain the appearance features corresponding to the 3D candidate boxes.

[0008] Optionally, in a third implementation of the first aspect of the present invention, the step of processing each 3D candidate box and its corresponding appearance features according to a preset self-attention mechanism and the time axis to generate a historical frame tracker and obtain the matching relationship between the historical frame tracker and each 3D candidate box includes: determining the first frame image in each bird's-eye view according to the time axis, and generating a historical frame tracker and tracker history information of the historical frame tracker according to the 3D candidate boxes in the first frame image, wherein the tracker history information records the matching relationship between the corresponding historical frame tracker and the 3D candidate box; matching the 3D candidate boxes of each bird's-eye view frame other than the first frame image with the historical frame tracker of the first frame image frame by frame according to the time axis based on the tracker history information and the appearance features using the self-attention mechanism; updating the tracker history information according to the matching result after the 3D candidate boxes of each bird's-eye view frame are matched with the historical frame tracker of the first frame image; until the 3D candidate boxes of all bird's-eye view frames are matched with the historical frame tracker.

[0009] Optionally, in the fourth implementation of the first aspect of the present invention, the step of updating the tracker history information according to the matching result after the 3D candidate boxes of each bird's-eye view are matched with the historical frame tracker, until the 3D candidate boxes of all bird's-eye views are matched with the historical frame tracker, includes: after the 3D candidate boxes of each bird's-eye view are matched with the historical frame tracker, updating the tracker history information of the historical frame trackers that are successfully matched in the current frame; determining whether there are any 3D candidate boxes in the current frame that are not matched with the historical frame tracker; if so, generating the historical frame tracker of the 3D candidate boxes that are not matched with the historical frame tracker and the corresponding tracker history information; using the current frame bird's-eye view frame tracker and the corresponding tracker history information as the matching input for the next frame bird's-eye view, until the 3D candidate boxes of all bird's-eye views are matched with the historical frame tracker.

[0010] Optionally, in the fifth implementation of the first aspect of the present invention, after the 3D candidate box in each frame of the bird's-eye view is matched with the historical frame tracker, and the tracker history information of the historical frame tracker that was successfully matched in the current frame is updated, the method further includes: determining whether there is a historical frame tracker in the current frame that was not matched with a 3D candidate box; if so, calculating the unmatched duration of the historical frame tracker that was not matched with a 3D candidate box based on the tracker history information; if the unmatched duration is greater than or equal to a preset time threshold, deleting the historical frame tracker that was not matched with a 3D candidate box.

[0011] Optionally, in the sixth implementation of the first aspect of the present invention, before acquiring multiple frames of bird's-eye view images on the same time axis, the method further includes: acquiring multiple frames of sample bird's-eye view images on another time axis, wherein the sample bird's-eye view images carry 3D candidate boxes of obstacles and corresponding number labels for the 3D candidate boxes; inputting the multiple frames of the sample bird's-eye view images into a preset neural network model, assigning tracker numbers based on the 3D candidate boxes of the input sample bird's-eye view images; calculating a loss function based on the number labels of the 3D candidate boxes of the sample bird's-eye view images and the tracker numbers, and obtaining a loss function value; determining whether the loss function value is less than a preset loss threshold; if not, backpropagating the loss function value to update the network parameters of the neural network model until the obtained loss function value is less than the preset threshold; if yes, generating an obstacle tracking model based on the network parameters of the neural network model.

[0012] A second aspect of the present invention provides an obstacle tracking device, comprising: an acquisition module for acquiring multiple frames of bird's-eye view images on the same time axis; an identification module for identifying obstacles in the multiple frames of the bird's-eye view images to obtain 3D candidate boxes of obstacles in each frame of the bird's-eye view images; a numbering and allocation module for inputting each frame of the bird's-eye view images into a preset obstacle tracking model according to the time axis, and assigning tracker numbers to the 3D candidate boxes of the input bird's-eye view images through the obstacle tracking model; and a trajectory generation module for generating obstacle trajectories based on the 3D candidate boxes with the same tracker numbers in each frame of the bird's-eye view images.

[0013] Optionally, in the first implementation of the second aspect of the present invention, the number allocation module specifically includes: a feature calculation unit, used to input each frame of the bird's-eye view into a preset obstacle tracking model, and perform a convolution operation on the 3D candidate boxes of each frame of the input bird's-eye view through the obstacle tracking model to calculate the appearance features corresponding to each 3D candidate box; a matching unit, used to process each 3D candidate box and its corresponding appearance features according to a preset self-attention mechanism and the time axis, generate a historical frame tracker and obtain the matching relationship between the historical frame tracker and each 3D candidate box; and an allocation unit, used to allocate a tracker number corresponding to the historical frame tracker to each 3D candidate box according to the matching relationship.

[0014] Optionally, in a second implementation of the second aspect of the present invention, the feature calculation unit is specifically used to: input each frame of the bird's-eye view into a preset obstacle tracking model; extract candidate box images corresponding to the 3D candidate boxes in the bird's-eye view through the obstacle tracking model; encode the candidate box images through the convolutional layer in the obstacle tracking model to obtain the encoded features of the candidate box images; and decode the encoded features through the transposed convolutional layer in the obstacle tracking model to obtain the appearance features corresponding to the 3D candidate boxes.

[0015] Optionally, in a third implementation of the second aspect of the present invention, the matching unit specifically includes: a tracker generation subunit, configured to determine the first frame image in each frame of the bird's-eye view according to the time axis, and generate a historical frame tracker and tracker history information of the historical frame tracker according to the 3D candidate boxes in the first frame image, wherein the tracker history information records the matching relationship between the corresponding historical frame tracker and the 3D candidate boxes; a self-attention subunit, configured to match the 3D candidate boxes of each frame of the bird's-eye view (excluding the first frame image) with the historical frame tracker of the first frame image frame by frame according to the time axis based on the tracker history information and the appearance features through the self-attention mechanism; and an update subunit, configured to update the tracker history information according to the matching result after the 3D candidate boxes of each frame of the bird's-eye view are matched with the historical frame tracker, until the 3D candidate boxes of all bird's-eye views are matched with the historical frame trackers.

[0016] Optionally, in the fourth implementation of the second aspect of the present invention, the updating subunit is specifically used for: updating the tracker history information of the historical frame tracker that has been successfully matched with the 3D candidate box of each frame of the bird's-eye view after the 3D candidate box is matched with the historical frame tracker; determining whether there is a 3D candidate box in the current frame that has not been matched with the historical frame tracker; if so, generating the historical frame tracker of the 3D candidate box that has not been matched with the historical frame tracker and the corresponding tracker history information; using the current frame bird's-eye view frame tracker and the corresponding tracker history information as the matching input for the next frame bird's-eye view, until the 3D candidate boxes of all bird's-eye views are matched with the historical frame trackers.

[0017] Optionally, in a fifth implementation of the second aspect of the present invention, the updating subunit is further configured to: determine whether there is a historical frame tracker that has not matched a 3D candidate box in the current frame; if so, calculate the unmatched duration of the historical frame tracker that has not matched a 3D candidate box based on the tracker's historical information; if the unmatched duration is greater than or equal to a preset time threshold, delete the historical frame tracker that has not matched a 3D candidate box.

[0018] Optionally, in a sixth implementation of the second aspect of the present invention, the obstacle tracking device further includes a model training module, which is specifically used for: acquiring multiple frames of sample bird's-eye view images on another time axis, wherein the sample bird's-eye view images carry 3D candidate boxes of obstacles and corresponding number labels of the 3D candidate boxes; inputting the multiple frames of the sample bird's-eye view images into a preset neural network model, and assigning tracker numbers based on the 3D candidate boxes of the input sample bird's-eye view images; calculating a loss function based on the number labels of the 3D candidate boxes of the sample bird's-eye view images and the tracker numbers, and obtaining a loss function value; determining whether the loss function value is less than a preset loss threshold; if not, backpropagating the loss function value to update the network parameters of the neural network model until the obtained loss function value is less than the preset threshold; if yes, generating an obstacle tracking model based on the network parameters of the neural network model.

[0019] A third aspect of the present invention provides an obstacle tracking device, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; the at least one processor invokes the instructions in the memory to cause the obstacle tracking device to perform the steps of the obstacle tracking method described above.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the obstacle tracking method described above.

[0021] In the technical solution of this invention, multiple frames of bird's-eye view images on the same time axis are acquired; obstacle recognition is performed on the multiple frames of bird's-eye view images to obtain 3D candidate bounding boxes of obstacles in each frame of the bird's-eye view images; each frame of the bird's-eye view images is input into a preset obstacle tracking model according to the time axis, and the obstacle tracking model assigns tracker numbers to the 3D candidate bounding boxes of the input bird's-eye view images; obstacle trajectories are generated based on the 3D candidate bounding boxes with the same tracker numbers in each frame of the bird's-eye view images. This method integrates multiple steps of a multi-target tracking task into a single model, reducing the use of computing resources, improving system efficiency, increasing the speed of obstacle tracking during autonomous driving or assisted driving, and using the 3D obstacle bounding boxes of the bird's-eye view images as model input improves the accuracy of obstacle detection. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the first embodiment of the obstacle tracking method in this invention; Figure 2 This is a schematic diagram of a second embodiment of the obstacle tracking method in this invention; Figure 3This is a schematic diagram of one embodiment of the obstacle tracking device in this invention; Figure 4 This is a schematic diagram of another embodiment of the obstacle tracking device in this invention; Figure 5 This is a schematic diagram of one embodiment of the obstacle tracking device in this invention. Detailed Implementation

[0023] This application provides an obstacle tracking method, apparatus, device, and storage medium to solve the technical problem that existing multi-target tracking methods involve separate steps, resulting in slow response times.

[0024] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 The first embodiment of the obstacle tracking method in this invention includes: 101. Acquire multiple bird's-eye view images on the same timeline; It is understood that the executing entity of this invention can be an obstacle tracking device, a terminal, or a server; no specific limitation is made here. This embodiment of the invention will be described using a server as an example.

[0026] In this embodiment, the multi-frame bird's-eye view images are mainly point cloud images from a bird's-eye view perspective, i.e., point cloud BEV (bird's eye view). A point cloud BEV view refers to the projection of the point cloud onto a plane perpendicular to the height direction. Typically, before obtaining the BEV view, the space is divided into voxels, and the point cloud is downsampled using voxels. Then, each voxel is projected as a point. A voxel is a cuboid used to divide three-dimensional space according to a fixed-size cuboid. The pixel coordinates of the BEV view can be obtained during voxel projection. The feature values ​​of each pixel can be obtained in various ways, such as through statistical methods called hand-crafted features, including the maximum height value, the intensity value of the point corresponding to the maximum height value, the number of point cloud points in the cuboid, and the average intensity value. Another method is to extract the features of each voxel using a model, such as VoxelNet. This real-time example of the invention does not limit the method of acquiring the bird's-eye view images.

[0027] In this embodiment, the multiple bird's-eye view images are acquired on the same time axis to ensure the continuity of obstacle tracking. If the bird's-eye view images are not acquired on the same time axis, it means that there is no temporal relationship between the multiple bird's-eye view images. After obstacle detection, the obstacles identified in the multiple bird's-eye view images also lack correlation. Only the obstacles can be identified, but obstacle tracking cannot be achieved.

[0028] In this embodiment, the same time axis can be understood as the bird's-eye view images being in the same continuous time interval, such as the time interval from when an autonomous vehicle starts driving to when it stops driving. Within this time interval, bird's-eye view images are generated for each time frame of autonomous driving, and the bird's-eye view images are sorted in chronological order to facilitate subsequent obstacle tracking.

[0029] 102. Perform obstacle recognition on multiple frames of bird's-eye view images to obtain 3D candidate bounding boxes of obstacles in each frame of bird's-eye view image; In this embodiment, the multi-frame bird's-eye view images are mainly point cloud images from a bird's-eye view perspective. Therefore, existing point cloud obstacle recognition methods can be used. For example, in this embodiment, after acquiring multiple frames of bird's-eye view images, obstacle recognition is performed on each frame. The obstacle recognition steps mainly involve voxel mesh filtering, point cloud segmentation, point cloud clustering, and adding bounding boxes. Voxel mesh filtering creates multiple stacked voxel cube meshes to fill the entire point cloud space. The filtering method leaves only one point within each voxel cube; therefore, the larger the length of each side of the cube, the lower the point cloud resolution. However, if the voxel mesh is too large... This process will lose the original features of the object. After voxel mesh filtering, the point cloud map will become a sparser point cloud data that retains the original obstacle features. The purpose of point cloud segmentation is to divide different objects into regions, that is, to distinguish which are road surfaces and which are non-collision objects, so as to facilitate the next step of point cloud clustering. Point cloud clustering refers to combining and aggregating the point clouds of different objects other than the ground to form multiple point sets. Each point set can then represent an obstacle and its size range, thus identifying the obstacle. Adding a bounding box involves using a cube to include the point clouds corresponding to all obstacles, thereby obtaining the 3D candidate boxes of the obstacles.

[0030] 103. Input each frame of bird's-eye view image into the preset obstacle tracking model according to the time axis, and assign tracker numbers to the 3D candidate boxes of the input bird's-eye view image through the obstacle tracking model; In this embodiment, a transformer structure is adopted, and a self-attention mechanism module is used. The obstacle 3D detection box under the current frame bird's-eye view is used as input, and the association between the object and the object box in the historical frame is output. In order to calculate the similarity of 3D candidate boxes in each frame bird's-eye view image in the input model, it is necessary to perform convolution operation on the input bird's-eye view image based on convolutional neural network or transformer to generate the appearance features of the 3D candidate boxes. The appearance features mainly include the size of the 3D candidate boxes and the feature vector generated by the neural network from the image or point cloud in the 3D candidate boxes.

[0031] In this embodiment, after calculating the appearance features of the 3D candidate boxes, the 3D candidate boxes of each frame of the bird's-eye view image are processed by a network based on a transformer self-attention mechanism. The network based on the transformer self-attention mechanism takes the current frame 3D candidate box, the appearance features of the 3D candidate box, and the tracker history information as input to obtain the matching relationship between the current frame 3D candidate box and the historical frame tracker, as well as the updated tracker history information. When the first frame of the bird's-eye view image is input, the tracker history information is zero, and the historical frame tracker corresponding to the 3D candidate box is generated. The matching relationship between the historical frame tracker and the 3D candidate box is used to update the tracker history information until all bird's-eye view images have been calculated.

[0032] In this embodiment, the obstacle tracking model is first obtained through the following steps: acquiring multiple frames of sample bird's-eye view images on another time axis, wherein the sample bird's-eye view images carry 3D candidate boxes of obstacles and corresponding number labels of the 3D candidate boxes; inputting the multiple frames of sample bird's-eye view images into a preset neural network model, and assigning tracker numbers based on the 3D candidate boxes of the input sample bird's-eye view images; calculating a loss function based on the number labels of the 3D candidate boxes of the sample bird's-eye view images and the tracker numbers, and obtaining a loss function value; determining whether the loss function value is less than a preset loss threshold; if not, backpropagating the loss function value to update the network parameters of the neural network model until the obtained loss function value is less than the preset threshold; if yes, generating an obstacle tracking model based on the network parameters of the neural network model.

[0033] 104. Generate obstacle trajectories based on 3D candidate boxes with the same tracker number in each frame of bird's-eye view image.

[0034] In this embodiment, 3D candidate boxes with the same tracker number in each frame of bird's-eye view image are concatenated according to the time axis. Since the obstacles are selected in the 3D candidate boxes, the concatenated 3D candidate boxes with the same tracker number can be used as the obstacle trajectory output.

[0035] In this embodiment, multiple frames of bird's-eye view images along the same timeline are acquired; obstacle recognition is performed on the multiple frames of bird's-eye view images to obtain 3D candidate bounding boxes of obstacles in each frame; each frame of bird's-eye view image is input into a preset obstacle tracking model according to the timeline, and the obstacle tracking model assigns tracker numbers to the 3D candidate bounding boxes of the input bird's-eye view images; obstacle trajectories are generated based on the 3D candidate bounding boxes with the same tracker numbers in each frame of bird's-eye view images. This method integrates multiple steps of a multi-target tracking task into a single model, reducing the use of computing resources, improving system efficiency, increasing the speed of obstacle tracking during autonomous or assisted driving, and using the 3D obstacle bounding boxes of the bird's-eye view images as model input improves the accuracy of obstacle detection.

[0036] Please see Figure 2 The second embodiment of the obstacle tracking method in this invention includes: 201. Obtain multiple bird's-eye view images on the same timeline; 202. Perform obstacle recognition on multiple frames of bird's-eye view images to obtain 3D candidate bounding boxes of obstacles in each frame of bird's-eye view image; 203. Input each frame of bird's-eye view into the preset obstacle tracking model, and perform convolution operation on the 3D candidate boxes of each frame of bird's-eye view through the obstacle tracking model to calculate the appearance features corresponding to each 3D candidate box. In this embodiment, the step of inputting each frame of the bird's-eye view into a preset obstacle tracking model, and performing a convolution operation on the 3D candidate boxes of each frame of the input bird's-eye view using the obstacle tracking model to calculate the appearance features corresponding to each 3D candidate box includes: inputting each frame of the bird's-eye view into the preset obstacle tracking model, and using the obstacle tracking model to extract candidate box images corresponding to the 3D candidate boxes in the bird's-eye view; encoding the candidate box images using the convolutional layers in the obstacle tracking model to obtain the encoded features of the candidate box images; and decoding the encoded features using the transposed convolutional layers in the obstacle tracking model to obtain the appearance features corresponding to the 3D candidate boxes.

[0037] Specifically, the obstacle tracking model includes convolutional layers, transposed convolutional layers, an encoder, and a decoder. The encoder encodes the candidate bounding box image based on the convolutional layers to generate coded features. The decoder decodes the coded features based on the transposed convolutional layers to generate appearance features corresponding to the 3D candidate bounding boxes. The transposed convolutional layer, also known as a deconvolution layer, deconvolutions the coded features to obtain the decoded features. The encoder includes a first two-dimensional convolutional network, a second two-dimensional convolutional network, and a pooling layer connected in sequence. The pooling layer is followed by a first branch, a second branch, and a third branch. The first, second, and third branches are in parallel. The first branch includes one two-dimensional convolutional layer, the second branch includes two two-dimensional convolutional layers, and the third branch includes three two-dimensional convolutional layers. The transposed convolutional layer in the decoder should also be configured accordingly based on the convolutional layers in the encoder to decode the data from the encoder's convolutional layers.

[0038] 204. Determine the first frame image in each frame of the bird's-eye view based on the timeline, and generate a history frame tracker and the tracker history information of the history frame tracker based on the 3D candidate boxes in the first frame image. The tracker history information records the matching relationship between the corresponding history frame tracker and the 3D candidate boxes. 205. Using a self-attention mechanism, based on the tracker's historical information and appearance features, the 3D candidate boxes of each frame of the bird's-eye view (excluding the first frame image) are matched frame by frame with the historical frame tracker of the first frame image according to the timeline. 206. After the 3D candidate boxes of each bird's-eye view are matched with the historical frame tracker, the historical information of the tracker is updated according to the matching results until the 3D candidate boxes of all bird's-eye views are matched with the historical frame tracker. In this embodiment, the first frame image is a bird's-eye view captured by the camera and radar in the order of the timeline. After obstacle recognition is performed on the first frame image in advance, 3D candidate boxes of obstacles are obtained and these 3D candidate boxes are marked for subsequent differentiation of different 3D candidate boxes. A corresponding historical frame tracker is configured for each 3D candidate box to track the marked 3D candidate box. When the first frame is the current frame, the tracker history information of each historical frame tracker only contains the matching relationship between the historical frame tracker and the 3D candidate box of the first frame. As other frame images are tracked in the future, the tracker history information will update the matching relationship between the historical frame tracker and the 3D candidate box of other frames. In practical applications, the goal of obstacle tracking is to find the target that was correctly tracked in all or most of the previous historical frames in the current frame. A self-attention mechanism is used to match the 3D candidate boxes of each frame's bird's-eye view (excluding the first frame) with the historical frame trackers of the first frame image frame by frame along the timeline. Except for the first frame, the matching of 3D candidate boxes and historical frame trackers is mainly based on tracker history information and appearance features. The similarity between the 3D candidate boxes tracked by the historical frame trackers in the previous frame and all 3D candidate boxes in the current frame is calculated using appearance features. The 3D candidate boxes with the highest similarity (greater than or equal to a preset similarity threshold) are successfully matched with the corresponding historical frame trackers. If the highest similarity is less than the preset similarity threshold, it means that the historical frame tracker failed to match the 3D candidate box in the current frame. When a match is successful, the corresponding historical frame tracker identifier is added to the 3D candidate box of the current frame, and the matching relationship between the historical frame trackers and the 3D candidate boxes of the current frame is updated in the tracker history information.

[0039] In this embodiment, the position of the obstacle in the next frame can be predicted by obtaining the position of the 3D candidate box in the frame image and calculating the speed of the obstacle corresponding to the 3D candidate box. The prediction result is matched with the 3D candidate box in the next frame and combined with the appearance features of the 3D candidate box. Specifically, the approximate position of the obstacle can be predicted by the position prediction and the prediction can be fine-tuned by the appearance features to obtain a more accurate prediction.

[0040] In this embodiment, the step of updating the tracker history information based on the matching results after the 3D candidate boxes of each bird's-eye view are matched with the historical frame tracker, until the 3D candidate boxes of all bird's-eye views are matched with the historical frame tracker, includes: updating the tracker history information of the historical frame trackers that are successfully matched in the current frame after the 3D candidate boxes of each bird's-eye view are matched with the historical frame tracker; determining whether there are any 3D candidate boxes in the current frame that are not matched with a historical frame tracker; if so, generating a historical frame tracker for the 3D candidate boxes that are not matched with a historical frame tracker and the corresponding tracker history information; using the current frame bird's-eye view frame tracker and the corresponding tracker history information as the matching input for the next frame bird's-eye view, until the 3D candidate boxes of all bird's-eye views are matched with the historical frame tracker.

[0041] Specifically, the input data is processed primarily through a network based on a transformer-based self-attention mechanism. A transformer is an architecture that transforms one sequence into another using attention-based encoders and decoders. The attention mechanism focuses on the input sequence and determines at each step which other parts of the sequence are important, thus helping to capture global information from the input sequence.

[0042] After the 3D candidate boxes in each frame of the bird's-eye view are matched with the historical frame trackers, and the tracker history information of the historical frame trackers that are successfully matched in the current frame is updated, the method further includes: determining whether there are any historical frame trackers in the current frame that have not matched 3D candidate boxes; if so, calculating the unmatched duration of the historical frame trackers that have not matched 3D candidate boxes based on the tracker history information; if the unmatched duration is greater than or equal to a preset time threshold, deleting the historical frame trackers that have not matched 3D candidate boxes.

[0043] Specifically, the model output updates the historical information of successfully matched trackers based on the 3D candidate box input of the current frame. Trackers that have not matched a 3D candidate box for a certain period of time are deleted, while candidate boxes that have not matched any trackers are used to generate a new tracker and update the historical information. The updated historical information will then be used as one of the model inputs for the next frame.

[0044] 207. Based on the matching relationship, assign the tracker number of the corresponding historical frame tracker to each 3D candidate box; 208. Generate obstacle trajectories based on 3D candidate boxes with the same tracker number in each frame of bird's-eye view image.

[0045] This embodiment, based on the previous embodiment, details the process of inputting each frame of the bird's-eye view image into a preset obstacle tracking model according to the time axis, and assigning tracker numbers to the 3D candidate boxes of the input bird's-eye view image through the obstacle tracking model. The process involves inputting each frame of the bird's-eye view image into the preset obstacle tracking model, performing a convolution operation on the 3D candidate boxes of each frame of the input bird's-eye view image through the obstacle tracking model, calculating the appearance features corresponding to each 3D candidate box, processing each 3D candidate box and its corresponding appearance features according to a preset self-attention mechanism and the time axis, generating a historical frame tracker, and obtaining the matching relationship between the historical frame tracker and each 3D candidate box; and assigning a tracker number corresponding to the historical frame tracker to each 3D candidate box according to the matching relationship. This method integrates multiple steps of a multi-target tracking task into a single model, reducing the use of computational resources, improving system efficiency, increasing the speed of obstacle tracking during autonomous driving or assisted driving, and using the 3D obstacle boxes of the bird's-eye view image as model input to improve the accuracy of obstacle detection.

[0046] The obstacle tracking method in the embodiments of the present invention has been described above. The obstacle tracking device in the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 3 One embodiment of the obstacle tracking device in this invention includes: The acquisition module 301 is used to acquire multiple frames of bird's-eye view images on the same time axis; The recognition module 302 is used to perform obstacle recognition on multiple frames of the bird's-eye view image to obtain 3D candidate boxes of obstacles in each frame of the bird's-eye view image; The numbering and allocation module 303 is used to input each frame of the bird's-eye view image into a preset obstacle tracking model according to the time axis, and to assign tracker numbers to the 3D candidate boxes of the input bird's-eye view image through the obstacle tracking model; The trajectory generation module 304 is used to generate obstacle trajectories based on 3D candidate boxes with the same tracker number in each frame of the bird's-eye view image.

[0047] In this embodiment of the invention, the obstacle tracking device operates the aforementioned obstacle tracking method. The obstacle tracking device acquires multiple frames of bird's-eye view images along the same time axis; performs obstacle identification on the multiple frames of bird's-eye view images to obtain 3D candidate bounding boxes of obstacles in each frame; inputs each frame of bird's-eye view images into a preset obstacle tracking model according to the time axis; assigns tracker numbers to the 3D candidate bounding boxes of the input bird's-eye view images through the obstacle tracking model; and generates obstacle trajectories based on the 3D candidate bounding boxes with the same tracker numbers in each frame of bird's-eye view images. This method integrates multiple steps of a multi-target tracking task into a single model, reducing the use of computational resources, improving system efficiency, and increasing the speed of obstacle tracking during autonomous or assisted driving. Furthermore, by using the 3D obstacle bounding boxes of the bird's-eye view images as model input, it improves the accuracy of obstacle detection.

[0048] Please see Figure 4 A second embodiment of the obstacle tracking device in this invention includes: The acquisition module 301 is used to acquire multiple frames of bird's-eye view images on the same time axis; The recognition module 302 is used to perform obstacle recognition on multiple frames of the bird's-eye view image to obtain 3D candidate boxes of obstacles in each frame of the bird's-eye view image; The numbering and allocation module 303 is used to input each frame of the bird's-eye view image into a preset obstacle tracking model according to the time axis, and to assign tracker numbers to the 3D candidate boxes of the input bird's-eye view image through the obstacle tracking model; The trajectory generation module 304 is used to generate obstacle trajectories based on 3D candidate boxes with the same tracker number in each frame of the bird's-eye view image.

[0049] In this embodiment, the number allocation module 303 specifically includes: a feature calculation unit 3031, used to input each frame of the bird's-eye view into a preset obstacle tracking model, and perform a convolution operation on the 3D candidate boxes of each frame of the input bird's-eye view through the obstacle tracking model to calculate the appearance features corresponding to each 3D candidate box; a matching unit 3032, used to process each 3D candidate box and its corresponding appearance features according to a preset self-attention mechanism and the time axis, generate a historical frame tracker and obtain the matching relationship between the historical frame tracker and each 3D candidate box; and an allocation unit 3033, used to allocate a tracker number corresponding to the historical frame tracker to each 3D candidate box according to the matching relationship.

[0050] In this embodiment, the feature calculation unit 3031 is specifically used to: input the bird's-eye view of each frame into a preset obstacle tracking model, and extract the candidate box image corresponding to the 3D candidate box in the bird's-eye view through the obstacle tracking model; encode the candidate box image through the convolutional layer in the obstacle tracking model to obtain the encoded features of the candidate box image; and decode the encoded features through the transposed convolutional layer in the obstacle tracking model to obtain the appearance features corresponding to the 3D candidate box.

[0051] In this embodiment, the matching unit 3032 specifically includes: a tracker generation subunit 30321, used to determine the first frame image in each frame of the bird's-eye view according to the time axis, and generate a historical frame tracker and tracker history information of the historical frame tracker according to the 3D candidate boxes in the first frame image, wherein the tracker history information records the matching relationship between the corresponding historical frame tracker and the 3D candidate boxes; a self-attention subunit 30322, used to match the 3D candidate boxes of each frame of the bird's-eye view (excluding the first frame image) with the historical frame tracker of the first frame image according to the time axis through the self-attention mechanism based on the tracker history information and the appearance features; and an update subunit 30323, used to update the tracker history information according to the matching result after the 3D candidate boxes of each frame of the bird's-eye view are matched with the historical frame tracker, until the 3D candidate boxes of all bird's-eye views are matched with the historical frame trackers.

[0052] In this embodiment, the update subunit 30323 is specifically used for: updating the tracker history information of the historical frame tracker that has been successfully matched with the 3D candidate box of each frame's bird's-eye view after the 3D candidate box is matched with the historical frame tracker; determining whether there is a 3D candidate box in the current frame that has not been matched with the historical frame tracker; if so, generating the historical frame tracker of the 3D candidate box that has not been matched with the historical frame tracker and the corresponding tracker history information; using the current frame's bird's-eye view frame tracker and the corresponding tracker history information as the matching input for the next frame's bird's-eye view, until the 3D candidate boxes of all bird's-eye views are matched with the historical frame trackers.

[0053] In this embodiment, the update subunit 30323 is further configured to: determine whether there is a historical frame tracker that has not matched a 3D candidate box in the current frame; if so, calculate the unmatched duration of the historical frame tracker that has not matched a 3D candidate box based on the tracker's historical information; if the unmatched duration is greater than or equal to a preset time threshold, delete the historical frame tracker that has not matched a 3D candidate box.

[0054] In this embodiment, the obstacle tracking device further includes a model training module 305, which is specifically used for: acquiring multiple frames of sample bird's-eye view images on another time axis, wherein the sample bird's-eye view images carry 3D candidate boxes of obstacles and corresponding number labels of the 3D candidate boxes; inputting the multiple frames of the sample bird's-eye view images into a preset neural network model, and assigning tracker numbers based on the 3D candidate boxes of the input sample bird's-eye view images; calculating a loss function based on the number labels of the 3D candidate boxes of the sample bird's-eye view images and the tracker numbers, and obtaining a loss function value; determining whether the loss function value is less than a preset loss threshold; if not, backpropagating the loss function value to update the network parameters of the neural network model until the obtained loss function value is less than the preset threshold; if yes, generating an obstacle tracking model based on the network parameters of the neural network model.

[0055] This implementation details the specific functions of each module of the obstacle tracking device and the unit composition of some modules. Through the modules and units of this device, multiple steps of multi-target tracking tasks are integrated into a single model, reducing the use of computing resources, improving system efficiency, increasing the speed of obstacle tracking during autonomous or assisted driving, and using the 3D obstacle box of the bird's-eye view image as the model input to improve the accuracy of obstacle detection.

[0056] above Figure 3 and Figure 4 The obstacle tracking device in the embodiments of the present invention will be described in detail from the perspective of modular functional entities. The obstacle tracking device in the embodiments of the present invention will be described in detail from the perspective of hardware processing.

[0057] Figure 5 This is a schematic diagram of the structure of an obstacle tracking device 500 provided in an embodiment of the present invention. The obstacle tracking device 500 can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 510 (e.g., one or more processors) and a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) for storing application programs 533 or data 532. The memory 520 and storage media 530 can be temporary or persistent storage. The program stored in the storage media 530 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the obstacle tracking device 500. Furthermore, the processor 510 may be configured to communicate with the storage media 530 and execute the series of instruction operations in the storage media 530 on the obstacle tracking device 500 to implement the steps of the obstacle tracking method described above.

[0058] The obstacle tracking device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input / output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 5 The obstacle tracking device structure shown does not constitute a limitation on the obstacle tracking device provided in this application. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0059] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of the obstacle tracking method.

[0060] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0061] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0062] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An obstacle tracking method, characterized in that, The obstacle tracking method includes: Acquire multiple bird's-eye view images on the same timeline; Obstacle identification is performed on multiple frames of the bird's-eye view to obtain 3D candidate bounding boxes of obstacles in each frame of the bird's-eye view; Each frame of the bird's-eye view is input into a preset obstacle tracking model according to the time axis, and tracker numbers are assigned to the 3D candidate boxes of the input bird's-eye view through the obstacle tracking model. Obstacle trajectories are generated based on 3D candidate boxes with the same tracker number in the bird's-eye view of each frame. The step of inputting each frame of the bird's-eye view into a preset obstacle tracking model according to the time axis, and assigning tracker numbers to the 3D candidate boxes of the input bird's-eye view through the obstacle tracking model includes: Each frame of the bird's-eye view is input into a preset obstacle tracking model. The obstacle tracking model performs a convolution operation on the 3D candidate boxes of each frame of the bird's-eye view, and calculates the appearance features corresponding to each 3D candidate box. According to a preset self-attention mechanism and the time axis, each 3D candidate box and its corresponding appearance features are processed to generate a historical frame tracker and obtain the matching relationship between the historical frame tracker and each 3D candidate box. According to the matching relationship, a tracker number corresponding to the historical frame tracker is assigned to each 3D candidate box. The process of processing each 3D candidate box and its corresponding appearance features according to a preset self-attention mechanism and the time axis to generate a historical frame tracker and obtain the matching relationship between the historical frame tracker and each 3D candidate box includes: The first frame image in each bird's-eye view is determined according to the timeline, and a historical frame tracker and the tracker history information of the historical frame tracker are generated based on the 3D candidate boxes in the first frame image. The tracker history information records the matching relationship between the corresponding historical frame tracker and the 3D candidate boxes. The first frame image is the first bird's-eye view captured by the camera and radar in the order of the timeline. The self-attention mechanism matches the 3D candidate bounding boxes of each bird's-eye view (excluding the first frame image) with the historical frame tracker of the first frame image frame by frame according to the time axis, based on the tracker's historical information and appearance features. After the 3D candidate bounding boxes of each bird's-eye view are matched with the historical frame tracker, the historical information of the tracker is updated according to the matching results until the 3D candidate bounding boxes of all bird's-eye views are matched with the historical frame tracker.

2. The obstacle tracking method according to claim 1, characterized in that, The step of inputting each frame of the bird's-eye view into a preset obstacle tracking model, and then performing a convolution operation on the 3D candidate boxes of each frame of the input bird's-eye view through the obstacle tracking model to calculate the appearance features corresponding to each 3D candidate box includes: Each frame of the bird's-eye view is input into a preset obstacle tracking model, and the obstacle tracking model is used to extract the candidate box image corresponding to the 3D candidate box in the bird's-eye view; The candidate bounding box image is encoded by the convolutional layer in the obstacle tracking model to obtain the encoded features of the candidate bounding box image; The encoded features are decoded by the transposed convolutional layer in the obstacle tracking model to obtain the appearance features corresponding to the 3D candidate box.

3. The obstacle tracking method according to claim 1, characterized in that, After the 3D candidate bounding boxes of each bird's-eye view are matched with the historical frame tracker, the historical information of the tracker is updated according to the matching results until the 3D candidate bounding boxes of all bird's-eye views are matched with the historical frame tracker, which includes: After the 3D candidate box in each frame of the bird's-eye view is matched with the historical frame tracker, the tracker history information of the historical frame tracker that was successfully matched in the current frame is updated. Determine if there are any 3D candidate boxes in the current frame that do not match the historical frame tracker; If it exists, generate a historical frame tracker for the 3D candidate box that does not match the historical frame tracker, along with the corresponding tracker history information; The current frame's bird's-eye view tracker and its corresponding historical information are used as the matching input for the next frame's bird's-eye view, until all 3D candidate boxes of the bird's-eye views are matched with the historical frame trackers.

4. The obstacle tracking method according to claim 3, characterized in that, After the 3D candidate bounding boxes in each frame of the bird's-eye view are matched with the historical frame trackers, and the tracker history information of the historical frame trackers that have successfully matched the current frame is updated, the following steps are also included: Determine if the current frame contains a historical frame tracker that did not match a 3D candidate box; If so, calculate the unmatched duration of the historical frames of the tracker that did not match the 3D candidate boxes based on the tracker's historical information; If the duration of the unmatched frame is greater than or equal to a preset time threshold, the historical frame tracker that did not match the 3D candidate box will be deleted.

5. The obstacle tracking method according to any one of claims 1-4, characterized in that, Before acquiring multiple frames of bird's-eye view images on the same timeline, the following steps are also included: Obtain multi-frame sample bird's-eye view images under another time axis, wherein the sample bird's-eye view images carry 3D candidate boxes of obstacles and corresponding number labels of the 3D candidate boxes; The sample bird's-eye view images of multiple frames are input into a preset neural network model, and tracker numbers are assigned based on the 3D candidate boxes of the input sample bird's-eye view images. The loss function is calculated based on the numbering and labeling of the 3D candidate boxes in the sample bird's-eye view and the tracker number, and the loss function value is obtained. Determine whether the value of the loss function is less than a preset loss threshold; If not, the loss function value is backpropagated to update the network parameters of the neural network model until the obtained loss function value is less than the preset loss threshold. If so, an obstacle tracking model is generated based on the network parameters of the neural network model.

6. An obstacle tracking device, characterized in that, The obstacle tracking device includes: The acquisition module is used to acquire multiple frames of bird's-eye view images on the same timeline. The recognition module is used to identify obstacles in multiple frames of the bird's-eye view and obtain 3D candidate boxes of obstacles in each frame of the bird's-eye view. The numbering and allocation module is used to input the bird's-eye view of each frame into a preset obstacle tracking model according to the time axis, and to assign tracker numbers to the 3D candidate boxes of the input bird's-eye view through the obstacle tracking model; The trajectory generation module is used to generate obstacle trajectories based on 3D candidate boxes with the same tracker number in the bird's-eye view of each frame. The numbering and allocation module is used to input the bird's-eye view of each frame into a preset obstacle tracking model, perform convolution operations on the 3D candidate boxes of the input bird's-eye view of each frame through the obstacle tracking model, calculate the appearance features corresponding to each 3D candidate box, process each 3D candidate box and its corresponding appearance features according to a preset self-attention mechanism and the time axis, generate a historical frame tracker and obtain the matching relationship between the historical frame tracker and each 3D candidate box, and assign a tracker number corresponding to the historical frame tracker to each 3D candidate box according to the matching relationship. The numbering and allocation module is used to determine the first frame image in each bird's-eye view according to the timeline, and generate a historical frame tracker and tracker history information of the historical frame tracker based on the 3D candidate boxes in the first frame image. The tracker history information records the matching relationship between the corresponding historical frame tracker and the 3D candidate boxes. The first frame image is the first bird's-eye view captured by the camera and radar in the order of the timeline. Through the self-attention mechanism, based on the tracker history information and the appearance features, the 3D candidate boxes of each bird's-eye view frame other than the first frame image are matched with the historical frame tracker of the first frame image frame by frame according to the timeline. After the 3D candidate boxes of each bird's-eye view frame are matched with the historical frame tracker, the tracker history information is updated according to the matching result until the 3D candidate boxes of all bird's-eye view frames are matched with the historical frame tracker.

7. An obstacle tracking device, characterized in that, The obstacle tracking device includes: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor invokes the instructions in the memory to cause the obstacle tracking device to perform the steps of the obstacle tracking method as described in any one of claims 1-5.

8. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the obstacle tracking method as described in any one of claims 1-5.

Images