Target detection and tracking method and device, computer device and storage medium

Abstract

The application provides a target detection and tracking method and device, computer equipment and a storage medium. The application fuses image data and point cloud data, fuses the features of the target in the continuous frames in a query tracking manner, completes the fusion of multi-modal information and time sequence information, realizes an end-to-end target detection and tracking model, avoids post-processing processes such as NMS, detection frame association and ReID, improves the detection and tracking effect, and reduces the calculation complexity. Meanwhile, the application fuses image features and point cloud BEV features based on the target level, greatly reduces the required calculation amount compared with the current BEVFusion method which fuses features on the whole BEV plane.

Description

Technical Field

[0001] This application relates to the field of information processing technology, and more specifically, to a target detection and tracking method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Object detection and tracking are crucial components of autonomous driving perception algorithms, requiring the detection of targets in every frame and the tracking of their trajectories. Currently, a common approach is to detect first and then track, using deep learning algorithms for target detection, followed by Kalman filtering for prediction and tracking of each target. However, this method heavily relies on the accuracy of the detector's output targets, especially in single-frame detection where historical frame features cannot be utilized, often leading to missed or false detections due to occlusion. In recent years, continuous frame detection methods have been proposed, but most involve feature fusion on BEVs (Battery Electric Vehicles), resulting in high computational costs and difficulty in ensuring feature alignment. Summary of the Invention

[0003] This application provides a target detection and tracking method, apparatus, computer device, and storage medium.

[0004] A first aspect of this application provides a target detection and tracking method, comprising: Acquire the image data and point cloud data to be detected, and perform feature extraction on the image data and point cloud data respectively to obtain image features and point cloud BEV features; The target embedding is obtained by aggregating image features and point cloud BEV features, and the target embedding is filtered to obtain candidate targets for the current frame; Perform feature interaction between the candidate target in the current frame and the tracked target, and output the interaction result; The interaction results are then combined with image features and point cloud BEV features to output the target query. The target query is processed by a linear layer to obtain the target category confidence score and the target bounding box, and the target set is updated based on the target query, the target category confidence score, and the target bounding box.

[0005] In an optional embodiment of this application, the process of acquiring image data and obtaining image features includes: Multiple cameras were used to capture 360° surround view images of the vehicle. The same feature extraction network is used to extract features from images acquired by multiple cameras to obtain image features at multiple different scales for each image.

[0006] In an optional embodiment of this application, the process of acquiring point cloud data and obtaining point cloud BEV features includes: Point cloud data of the vehicle's surroundings are collected by multiple lidar sensors. The point cloud data is preprocessed and voxelized before being input into the point cloud feature extraction network; The point cloud feature extraction network performs sparse convolution processing on the point cloud data, then downsamples and fuses the point cloud data to obtain point cloud BEV features at multiple different scales under BEV.

[0007] In an optional embodiment of this application, the process of aggregating image features and point cloud BEV features to obtain target embeddings, and filtering the target embeddings to obtain candidate targets for the current frame, includes: Obtain the target bounding box and target query used to locate the target; Based on the target bounding box used to locate the target, obtain the target image sampling points and the target point cloud sampling points; based on the target query of the target, obtain the target image sampling point weights and the target point cloud sampling point weights. The first image sampling feature is obtained by linear interpolation sampling calculation of the target image sampling points, the target image sampling point weights, and the high-dimensional features of the image. The first point cloud sampling feature is obtained by linear interpolation sampling calculation of the target point cloud sampling points, the target point cloud sampling point weights, and the point cloud BEV high-dimensional features. The target embedding is obtained by linearly fusing the first image sampling features and the first point cloud sampling features; Target embeddings that meet preset requirements are selected from the target embeddings as candidate targets for the current frame.

[0008] In an optional embodiment of this application, the process of performing feature interaction between the current frame candidate target and the tracked target, and outputting the interaction result, includes: The target embedding of the tracked target and the target embedding of the candidate target in the current frame are concatenated and then input into the self-attention module. The target embedding includes a query, a key, and a value. After performing a linear transformation on each query, key, and value, they are mapped to different subspaces. In each subspace, a weight matrix is ​​assigned to each query, key, and value to obtain queries, keys, and values ​​with multiple attention heads. Calculate the single-head attention for each attention head separately; After concatenating the single-head attention of each attention head, it is mapped to the target dimension through a linear layer to obtain multi-head attention.

[0009] In an optional embodiment of this application, the process of interactively outputting the target query by combining the interaction result with image features and point cloud BEV features includes: Obtain the target bounding box and target query used to locate the target; Based on the target bounding box of the tracked target, obtain the image sampling points and point cloud sampling points of the tracked target; based on the target query of the tracked target, obtain the weights of the image sampling points and point cloud sampling points of the tracked target. The second image sampling feature is obtained by linear interpolation sampling calculation of the tracking target image sampling points, the tracking target image sampling point weights, and image features; The second point cloud sampling feature is obtained by linear interpolation sampling calculation of the tracking target point cloud sampling points, the tracking target point cloud sampling point weights and the point cloud BEV features; The target query is obtained by linearly fusing the second image sampling features, the second point cloud sampling features, and the interaction results.

[0010] In an optional embodiment of this application, after the interaction result is interactively output with image features and point cloud BEV features to generate a target query, the method further includes: If the target query does not contain a tracking target, add the output target query ID as the new target ID; If a target exists in the target query, the ID of the output target query will be retained as the original ID of the target.

[0011] A second aspect of this application provides a target detection and tracking device, comprising: The feature extraction module is used to acquire the image data and point cloud data to be detected, and to perform feature extraction on the image data and point cloud data respectively to obtain image features and point cloud BEV features; The first cross-attention module is used to aggregate image features and point cloud BEV features to obtain target embeddings, and to filter the target embeddings to obtain candidate targets for the current frame; The self-attention module is used to perform feature interaction between the candidate target in the current frame and the tracked target, and output the interaction result; The second cross-attention module is used to interact with the interaction results and image features and point cloud BEV features to output the target query. The target output update module is used to process the target query through a linear layer to obtain the target category confidence and target bounding box, and update the tracking target set according to the target query, target category confidence and target bounding box.

[0012] A third aspect of this application provides a computer device, including: a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above methods.

[0013] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of the method as described in any of the above.

[0014] This application integrates image data and point cloud data, fusing target features from consecutive frames through a tracking query approach. This achieves the fusion of multimodal and temporal information, realizing an end-to-end target detection and tracking model. It avoids post-processing steps such as NMS, bounding box association, and ReID, improving detection and tracking performance while reducing computational complexity. Furthermore, this application fuses image features and point cloud BEV features at the target level, significantly reducing the required computation compared to current methods like BEVFusion, which fuse features across the entire BEV plane. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart of a target detection and tracking method provided in one embodiment of this application; Figure 2 This is a schematic diagram of a target detection and tracking device provided in one embodiment of this application; Figure 3 A schematic diagram illustrating the principle of a tracking fusion module provided in one embodiment of this application; Figure 4 A schematic diagram illustrating the principle of a first cross-attention module / second cross-attention module provided in one embodiment of this application; Figure 5 This is a schematic diagram of a computer device structure provided in one embodiment of this application. Detailed Implementation

[0016] In the process of implementing this application, the inventors discovered that most current continuous frame detection methods perform feature fusion on BEV, which involves a large amount of computation and makes it difficult to guarantee feature alignment.

[0017] Example 1 To address the aforementioned problems, this application provides a target detection and tracking method, such as... Figure 1 As shown, it includes: S101. Obtain the image data and point cloud data to be detected, and perform feature extraction on the image data and point cloud data respectively to obtain image features and point cloud BEV features.

[0018] Specifically, for the image data of the detected target, this embodiment uses surround-view cameras to acquire the data. The image data of the detected target comes from 360° surround-view images of the vehicle captured by multiple cameras. For example, four cameras are placed in the front, rear, left, and right directions of the vehicle, with each camera having a capture range exceeding 120°, thus enabling 360° surround-view image acquisition of the vehicle. Then, the same feature extraction network is used to extract features from the images captured by the multiple cameras to obtain feature maps for each image, ensuring the consistency of the feature maps. Figure 2 As shown. Finally, each image can obtain feature maps at three different scales. The feature extraction network used in this embodiment can be an existing network, such as ResNet50 and FPN network. Of course, other networks that can achieve the feature extraction of this embodiment can also be applied. This embodiment does not make any special limitations.

[0019] For point cloud data, this embodiment uses LiDAR to acquire the data. For example, point cloud data within a range of 100m forward, 60m backward, and 50m to the left and right of the LiDAR can be used. Of course, the above distances and orientations can be adjusted according to the actual situation. After preprocessing and voxelization, the point cloud data is input into the point cloud feature extraction network. Due to the sparsity of point cloud data, there are many empty voxels. Therefore, in this embodiment, a sparse convolution module is introduced into the point cloud feature extraction network. Sparse convolution can cope with this sparsity by focusing only on non-zero points to reduce unnecessary calculations, thereby improving computational efficiency. Finally, the point cloud data is downsampled and fused using, for example, a Second backbone network to obtain point cloud BEV features at multiple different scales under BEV, such as... Figure 2 As shown.

[0020] S102. Aggregate image features and point cloud BEV features to obtain target embedding, and filter the target embedding to obtain candidate targets for the current frame.

[0021] Target embedding refers to a fixed-dimensional feature vector that represents the comprehensive information of a target. It can be represented as a high-dimensional array, such as a list of length 256 or 512, used to represent the target's shape, position, motion state, etc. In this embodiment, the process of obtaining candidate targets for the current frame is implemented based on a cross-attention module, such as... Figure 3 As shown, the image features and point cloud BEV features obtained in S101 are fed into the cross-attention module. The target embedding is obtained by aggregating the image features and point cloud BEV features, and the candidate targets with higher scores in the current frame are selected.

[0022] The cross-attention module in this embodiment uses a sampling point-based method to fuse the features of the target detection object in image data and point cloud data. The specific process of aggregating image features and point cloud BEV features is as follows: Figure 4As shown.

[0023] First, obtain the target bounding box and target query for target localization. The target bounding box provides information on the target's location, size, and velocity. The target query provides feature information for the target. Using the target bounding box, obtain the target image sampling points and target point cloud sampling points. Using the target query, obtain the target image sampling point weights and target point cloud sampling point weights. Then, perform linear interpolation sampling calculations on the target image sampling points, target image sampling point weights, and high-dimensional image features to obtain the first image sampling feature. Perform linear interpolation sampling calculations on the target point cloud sampling points, target point cloud sampling point weights, and point cloud BEV high-dimensional features to obtain the first point cloud sampling feature. High-dimensional image features refer to feature maps with rich semantics but no longer intuitively visual, extracted from planar images (height × width × 3, such as RGB) through neural networks (such as ResNet, ViT) with feature channel dimensions greater than a preset threshold. High-dimensional point cloud BEV features refer to point cloud BEV features with feature channel dimensions greater than a preset threshold used to characterize what objects are present, how tall they are, and whether they are passable at each grid location on the ground in the real world. The calculation formula is as follows:

[0024] in, Target query features, These are the normalized coordinates of the reference point. It is the input multi-scale feature map. Represents the number of feature layers at different scales. C represents the number of feature channels, H and W represent the height and width of the feature map, respectively, and the cross-attention module fuses features from the image and point cloud at different scales; m represents the number of attention heads. Here, k represents the sampling point weights, and k is the sampling point index. Rescale the normalized coordinates to the [number]th [value]. Level input feature map, This is the position offset based on the reference point.

[0025] Next, the first image sampling features and the first point cloud sampling features are linearly fused to obtain the target embedding. Finally, the target embeddings that meet the preset requirements are selected as candidate targets for the current frame, for example, target embeddings with higher scores are selected as candidate targets for the current frame.

[0026] In this embodiment, the sampling points include fixed sampling points and learnable sampling points. Fixed sampling points can be designed based on prior target information, while learnable sampling points are obtained through target lookup. Since image data and point cloud data have data differences and target feature distribution locations differ, different sampling points and sampling point weights are used for sampling. This embodiment fuses image features and point cloud BEV features at the target level. Compared to current methods such as BEVFusion, which fuse features across the entire BEV plane, this significantly reduces the required computation and improves feature alignment by using learnable sampling points.

[0027] S103. Perform feature interaction between the candidate target in the current frame and the tracked target, and output the interaction result.

[0028] In this embodiment, the process of feature interaction between the current frame candidate target and the tracked target is implemented based on a self-attention module. This self-attention module can promote feature interaction between the tracked target and the current frame candidate target, and enhance the fusion of temporal features.

[0029] In this embodiment, the self-attention module employs multi-head attention, enabling feature interaction between the tracked target and candidate targets in the current frame through query embedding. In the multi-head attention mechanism, the query, key, and value are divided into multiple "heads" to process different subspaces in parallel. Each head independently performs standard single-head attention computation, and finally, the outputs of all heads are merged.

[0030] Specifically, firstly, the target embeddings of the tracked target and the current frame's candidate targets are concatenated as input to the self-attention module, where the query, key, and value are all target embeddings. Then, a linear transformation is performed on each query, key, and value, and these transformed queries, keys, and values ​​are mapped to multiple different subspaces. Within each subspace, a weight matrix is ​​assigned to each query, key, and value, resulting in queries, keys, and values ​​for multiple attention heads.

[0031] in, It is the weight matrix corresponding to each head. Different attention heads are represented; in this embodiment, a total of 8 attention heads can be set, with 256 dimensions for query, key, and value, and the input dimensions for each head are... .

[0032] For each head Using single-head attention calculation:

[0033] Each head calculates an attention output. .

[0034] Finally, the outputs of all heads are concatenated, and the concatenated output is mapped to the target dimension through a linear layer (i.e., a fully connected layer) to obtain the final multi-head attention output.

[0035]

[0036] in, It is the weight matrix of the linear layer.

[0037] S104. The interaction result is used to output the target query by interacting with the image features and point cloud BEV features.

[0038] In this embodiment, the principle underlying the interaction between the interaction result and image features and point cloud BEV features to output the target query is similar to the process in S102. The difference lies in the possibility of target input during the interaction between the interaction result and image features and point cloud BEV features to output the target query. When target input exists, the target bounding box and target query input by the cross-attention module become the target bounding box and target query of the target. If no target input exists, both the target bounding box and target query are initialized to learnable parameters.

[0039] Specifically, first, the target bounding box and target query of the tracking target are obtained. Based on the target bounding box, image sampling points and point cloud sampling points of the tracking target are obtained. Based on the target query, the weights of the image sampling points and point cloud sampling points of the tracking target are obtained. Linear interpolation is performed on the image sampling points, their weights, and image features to calculate the second image sampling feature. Linear interpolation is then performed on the point cloud sampling points, their weights, and the point cloud BEV features to calculate the second point cloud sampling feature. Finally, the second image sampling feature, the second point cloud sampling feature, and the interaction result are linearly fused to obtain the target query.

[0040] S105. The target query is processed through a linear layer to obtain the target category confidence and the target bounding box, and the target set is updated according to the target query, the target category confidence and the target bounding box.

[0041] Specifically, after obtaining the target query, the target query is fed into a linear layer to obtain the target category confidence score and the target bounding box, and the final output target is selected based on the target score or other criteria. The target bounding box contains information about the target's position, size, orientation, and velocity.

[0042] Meanwhile, this embodiment can also update the tracked target based on the target query, target category confidence, and target bounding box, enabling the tracked target to remain dynamically updated. During this process, when no tracked target exists, the output target query is for a newly added target, and a new target ID is added to the target query. When a tracked target exists, the output target query is obtained through feature interaction with the tracked target, and the original tracked target ID is retained in the output target query.

[0043] This embodiment utilizes image data from a surround-view camera and point cloud data from a LiDAR system. Image features and point cloud BEV features are obtained through image feature extraction networks and point cloud feature extraction networks, respectively. Information about the target in the image features and point cloud BEV features is sampled using a DETR (Detection Transformer) method, and feature fusion is performed to output the target result for the current frame. Finally, the target set is updated using the final result. This embodiment fuses target features from consecutive frames using a tracking query approach. Each tracking query is responsible for predicting the entire trajectory of a target, and target detection and tracking are achieved through frame-by-frame transmission and updating.

[0044] The solutions in this application embodiment can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0045] It should be understood that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order constraint on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the diagram may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0046] Example 2 like Figures 2 to 4 As shown, one embodiment of this application provides a target detection and tracking device, including: The feature extraction module is used to acquire the image data and point cloud data to be detected, and to perform feature extraction on the image data and point cloud data respectively to obtain image features and point cloud BEV features; The first cross-attention module is used to aggregate image features and point cloud BEV features to obtain target embeddings, and to filter the target embeddings to obtain candidate targets for the current frame; The self-attention module is used to perform feature interaction between the candidate target in the current frame and the tracked target, and output the interaction result; The second cross-attention module is used to interact with the interaction results and image features and point cloud BEV features to output the target query. The target output update module is used to process the target query through a linear layer to obtain the target category confidence and target bounding box, and update the tracking target set according to the target query, target category confidence and target bounding box.

[0047] In this embodiment, the first cross-attention module, the self-attention module, and the second cross-attention module constitute a tracking fusion module. Specific limitations of the above-described target detection and tracking device can be found in the limitations of the target detection and tracking method described above, and will not be repeated here. Each module in the above-described target detection and tracking device can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in a computer device in hardware form, or it can be stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0048] Example 3 In one embodiment, a computer device is provided, the internal structure of which can be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements the target detection and tracking method described above. It includes: memory and a processor; the memory stores a computer program; and the processor executes the computer program to implement any step of the target detection and tracking method described above.

[0049] Example 4 In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can perform any of the steps in the target detection and tracking method described above.

[0050] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0051] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0052] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0053] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0054] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0055] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A target detection and tracking method, characterized in that, include: Acquire the image data and point cloud data to be detected, and perform feature extraction on the image data and point cloud data respectively to obtain image features and point cloud BEV features; The target embedding is obtained by aggregating image features and point cloud BEV features, and the target embedding is filtered to obtain candidate targets for the current frame; Perform feature interaction between the candidate target in the current frame and the tracked target, and output the interaction result; The interaction results are then combined with image features and point cloud BEV features to output the target query. The target query is processed by a linear layer to obtain the target category confidence score and the target bounding box, and the target set is updated based on the target query, the target category confidence score, and the target bounding box.

2. The method according to claim 1, characterized in that, The process of acquiring image data and obtaining image features includes: Multiple cameras were used to capture 360° surround view images of the vehicle. The same feature extraction network is used to extract features from images acquired by multiple cameras to obtain image features at multiple different scales for each image.

3. The method according to claim 1, characterized in that, The process of acquiring point cloud data and obtaining point cloud BEV features includes: Point cloud data of the vehicle's surroundings are collected by multiple lidar sensors. The point cloud data is preprocessed and voxelized before being input into the point cloud feature extraction network; The point cloud feature extraction network performs sparse convolution processing on the point cloud data, then downsamples and fuses the point cloud data to obtain point cloud BEV features at multiple different scales under BEV.

4. The method according to claim 1, characterized in that, The process of aggregating image features and point cloud BEV features to obtain target embeddings, and then filtering the target embeddings to obtain candidate targets for the current frame, includes: Obtain the target bounding box and target query used to locate the target; Based on the target bounding box used to locate the target, obtain the target image sampling points and the target point cloud sampling points; based on the target query of the target, obtain the target image sampling point weights and the target point cloud sampling point weights. The first image sampling feature is obtained by linear interpolation sampling calculation of the target image sampling points, the target image sampling point weights, and the high-dimensional features of the image. The first point cloud sampling feature is obtained by linear interpolation sampling calculation of the target point cloud sampling points, the target point cloud sampling point weights, and the point cloud BEV high-dimensional features. The target embedding is obtained by linearly fusing the first image sampling features and the first point cloud sampling features; Target embeddings that meet preset requirements are selected from the target embeddings as candidate targets for the current frame.

5. The method according to claim 1, characterized in that, The process of performing feature interaction between the current frame candidate target and the tracked target, and outputting the interaction result, includes: The target embedding of the tracked target and the target embedding of the candidate target in the current frame are concatenated and then input into the self-attention module. The target embedding includes a query, a key, and a value. After performing a linear transformation on each query, key, and value, they are mapped to different subspaces. In each subspace, a weight matrix is ​​assigned to each query, key, and value to obtain queries, keys, and values ​​with multiple attention heads. Calculate the single-head attention for each attention head separately; After concatenating the single-head attention of each attention head, it is mapped to the target dimension through a linear layer to obtain multi-head attention.

6. The method according to claim 5, characterized in that, The process of interacting with the interaction results and image features and point cloud BEV features to output the target query includes: Obtain the target bounding box and target query used to locate the target; Based on the target bounding box of the tracked target, obtain the image sampling points and point cloud sampling points of the tracked target; based on the target query of the tracked target, obtain the weights of the image sampling points and point cloud sampling points of the tracked target. The second image sampling feature is obtained by linear interpolation sampling calculation of the tracking target image sampling points, the tracking target image sampling point weights, and image features; The second point cloud sampling feature is obtained by linear interpolation sampling calculation of the tracking target point cloud sampling points, the tracking target point cloud sampling point weights and the point cloud BEV features; The target query is obtained by linearly fusing the second image sampling features, the second point cloud sampling features, and the interaction results.

7. The method according to claim 1, characterized in that, After interacting with the image features and point cloud BEV features to output the target query, the method further includes: If the target query does not contain a tracking target, add the output target query ID as the new target ID; If a target exists in the target query, the ID of the output target query will be retained as the original ID of the target.

8. A target detection and tracking device, characterized in that, include: The feature extraction module is used to acquire the image data and point cloud data to be detected, and to perform feature extraction on the image data and point cloud data respectively to obtain image features and point cloud BEV features; The first cross-attention module is used to aggregate image features and point cloud BEV features to obtain target embeddings, and to filter the target embeddings to obtain candidate targets for the current frame; The self-attention module is used to perform feature interaction between the candidate target in the current frame and the tracked target, and output the interaction result; The second cross-attention module is used to interact with the interaction results and image features and point cloud BEV features to output the target query. The target output update module is used to process the target query through a linear layer to obtain the target category confidence and target bounding box, and update the tracking target set according to the target query, target category confidence and target bounding box.

9. A computer device, comprising: The method includes a memory and a processor, the memory storing a computer program, characterized in that the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images