3D Moving Object Tracking Method and Device Based on Spatiotemporal Domain and Bounding Box Information

Abstract

The present invention provides a method and apparatus for tracking three-dimensional moving targets based on spatiotemporal domain and bounding box information, comprising: for a given point cloud sequence, obtaining mask information of the target in each frame of the point cloud based on the bounding box information of the target in each frame; extracting geometric features from each frame of the point cloud sequence to obtain point features for each frame, wherein the point features include point features and geometric feature description operators of the points; based on an attention mechanism, utilizing feature fusion or a shared similarity matrix, transferring the mask information of the target to the point features of the current frame, optimizing the point features of the current frame using spatiotemporal context information to obtain optimized point features of the current frame; and calculating the bounding box information of the target in the current frame using the optimized point features of the current frame. The present invention fully utilizes the rich spatiotemporal context information in historical frames and the current frame, eliminating the need for cropping the point cloud, and achieving higher-performance, continuous, and accurate tracking of a specified single moving object.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision three-dimensional single target tracking, and in particular to a three-dimensional moving target tracking method and device based on spatiotemporal domain and bounding box information. Background Art

[0002] The ability to track objects in 3D space is crucial for many applications, including robotics, autonomous driving, and surveillance systems. Given an initial state of a specific object, the goal of single object tracking (SOT) is to estimate the pose and position of the tracked object in each frame. Recent methods have focused on using point clouds to solve 3D object tracking because point clouds have unique advantages over image inputs, such as accurate spatial information and robustness to illumination changes.

[0003] Existing methods for 3D single object tracking primarily follow the Siamese neural network paradigm, taking as input a point cloud representing the target cropped from the previous frame and a search region in the current frame. Localization networks such as the Region Proposal Network (RPN) are then used to localize the target in an end-to-end manner. Unlike previous methods, the M2-Track method first crops the target from the point clouds of two adjacent frames and then explicitly models the target's motion between the two consecutive frames. This cropping-based approach only focuses on the point cloud information associated with the target, ignoring the spatiotemporal context of the target. This limitation makes 3D single object tracking a challenging task, especially when the appearance changes significantly or the target disappears due to occlusion. To this end, the TAT method exploits temporally correlated target information by sampling a set of high-quality target templates cropped from historical frames to achieve reliable feature interactions. However, ignoring information in the latest frame can cause the network to fail to capture persistent appearance changes, such as the sparseness of the point cloud as the target moves further away. TAT also ignores the contextual information around the target, which is crucial for 3D single object tracking, resulting in limited tracking performance.

[0004] In addition, the huge differences in size and geometry of tracking targets of different categories also pose challenges to 3D single target tracking, which has been ignored in previous work. The localization networks adopted in existing methods can be divided into two paradigms, namely point feature-based localization networks and voxel feature-based localization networks. For voxel feature-based localization networks, tracking targets with simple shapes and large sizes (such as vehicles) can be well matched to the voxel shape, thereby achieving more accurate localization than point feature-based localization networks (such as RPN). However, for categories with complex geometric shapes and small sizes such as pedestrians, voxelization will lead to a large amount of information loss, thereby reducing tracking performance.

[0005] In summary, the existing technology has the problem of limited single target tracking performance. Summary of the Invention

[0006] The present invention provides a three-dimensional moving target tracking method and device based on spatiotemporal domain and bounding box information, which are used to solve the defects of limited performance in the prior art and achieve higher performance continuous and accurate tracking.

[0007] The present invention provides a three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information, comprising:

[0008] For a given point cloud sequence, mask information of the tracking target in each frame of the point cloud is obtained based on the bounding box information of the tracking target in each frame; geometric features are extracted for each frame of the point cloud sequence to obtain point features of each frame, wherein the point features include points and geometric feature description operators of the points;

[0009] Based on the attention mechanism, the mask information of the tracking target is transferred to the point features of the current frame by means of feature fusion or shared similarity matrix, and the point features of the current frame are optimized using spatiotemporal context information to obtain the optimized point features of the current frame; the spatiotemporal context information includes the spatial context feature relationship within the current frame and the temporal context feature relationship between frames;

[0010] The optimized point features of the current frame are used to calculate the bounding box information of the tracking target in the current frame.

[0011] According to the present invention, a three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information is provided. Based on the attention mechanism, the mask information of the tracking target is transferred to the current frame point feature by using feature fusion, and the current frame point feature is optimized using spatiotemporal context information to obtain the optimized point feature of the current frame. Specifically, the method includes:

[0012] Encoding the mask information of the tracked target in each frame into a mask feature description operator;

[0013] fusing the mask feature description operator with the geometric feature description operator;

[0014] Using a combination of self-attention and cross-attention mechanisms, the feature relationship between the tracking target and the spatiotemporal context information in each frame is modeled;

[0015] Using the cross attention mechanism, based on the feature relationship, the spatiotemporal context information in the historical frame is utilized to optimize the current frame point features to obtain the optimized point features of the current frame.

[0016] According to the present invention, a three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information is provided. Based on the attention mechanism, the mask information of the tracking target is transferred to the current frame point feature by using a shared similarity matrix, and the current frame point feature is optimized using spatiotemporal context information to obtain the optimized point feature of the current frame. Specifically, the method includes:

[0017] Encoding the mask information of the tracked target in each frame into a mask feature description operator;

[0018] For the geometric features and mask features, a self-attention mechanism and a cross-attention mechanism are respectively used to model the feature relationship between the tracking target and the spatiotemporal context information in each frame to obtain a first feature relationship and a second feature relationship;

[0019] Using a cross attention mechanism, based on the first feature relationship and the second feature relationship, and utilizing context information in historical frames to optimize the current frame point feature, a first target point feature and a second target point feature are obtained;

[0020] Based on the first target point feature and the second target point feature, the mask feature description operator is fused with the geometric feature description operator to obtain the optimized point feature of the current frame.

[0021] According to a three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information provided by the present invention, the method of calculating the bounding box information of the tracking target in the current frame by using the optimized point features of the current frame specifically includes:

[0022] Predicting the center position of the tracking target based on each optimized point feature in the current frame to obtain a predicted center position, and predicting whether each point in the current frame belongs to the tracking target;

[0023] The local self-attention mechanism is used to aggregate the point features in the neighborhood of the predicted center position to obtain the bounding box information of the current frame.

[0024] According to a three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information provided by the present invention, the method of calculating the bounding box information of the tracking target in the current frame by using the optimized point features of the current frame specifically includes:

[0025] Predicting the center position of the tracking target based on each optimized point feature in the current frame to obtain a predicted center position, and predicting whether each point in the current frame belongs to the tracking target;

[0026] Using the bounding box size information of the historical frame, a preset number of reference points are sampled near the predicted center position;

[0027] Converting the optimized point features of the current frame into point features defined on the reference points;

[0028] Information aggregation is performed on the point features of the reference point to obtain the bounding box information of the current frame.

[0029] According to a three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information provided by the present invention, for a given point cloud sequence, mask information of the tracking target in each frame of the point cloud is obtained based on the bounding box information of the tracking target in each frame, specifically comprising:

[0030] For an initial frame in the point cloud sequence, based on the bounding box information of the tracking target in the initial frame, calculating point by point whether each point in the initial frame is within the bounding box, and obtaining mask information of the tracking target;

[0031] For a non-initial frame in the point cloud sequence, when predicting bounding box information of a tracking target in the non-initial frame, mask information of the tracking target is predicted.

[0032] The present invention also provides a three-dimensional moving target tracking device based on spatiotemporal domain and bounding box information, comprising:

[0033] A feature unit is configured to obtain, for a given point cloud sequence, mask information of the tracked target in each frame of the point cloud based on the bounding box information of the tracked target in each frame; extract geometric features from each frame of the point cloud sequence to obtain point features of each frame, wherein the point features include points and geometric feature description operators of the points;

[0034] An optimization unit is configured to transfer the mask information of the tracking target to the point features of the current frame by utilizing feature fusion or a shared similarity matrix based on an attention mechanism, and optimize the point features of the current frame using spatiotemporal context information to obtain optimized point features of the current frame; the spatiotemporal context information includes a spatial context feature relationship within the current frame and a temporal context feature relationship between frames;

[0035] A positioning unit is used to calculate the bounding box information of the tracking target in the current frame using the optimized point features in the current frame.

[0036] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, it implements any of the above-described three-dimensional motion target tracking methods based on spatiotemporal domain and bounding box information.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described three-dimensional moving target tracking methods based on spatiotemporal domain and bounding box information.

[0038] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described three-dimensional moving target tracking methods based on spatiotemporal domain and bounding box information.

[0039] The present invention provides a method and apparatus for tracking three-dimensional moving objects based on spatiotemporal domain and bounding box information. For a given point cloud sequence, the method and apparatus obtain mask information of the tracking object in each frame of the point cloud based on the bounding box information of the tracking object in each frame. Geometric feature extraction is performed on each frame of the point cloud sequence to obtain point features for each frame. The point features include points and geometric feature description operators of the points. Based on an attention mechanism, the mask information of the tracking object is transferred to the point features of the current frame by using feature fusion or a shared similarity matrix. The point features of the current frame are optimized using spatiotemporal context information to obtain optimized point features of the current frame. The spatiotemporal context information includes the spatial context feature relationship within the current frame and the temporal context feature relationship between frames. The optimized point features of the current frame are used to calculate the bounding box information of the tracking object in the current frame. Based on the attention mechanism, the present invention uses spatiotemporal domain context and bounding box size information to track three-dimensional moving objects. This method fully utilizes the rich spatiotemporal domain context information in historical frames and the current frame, eliminating the need for cropping the point cloud, and achieving higher-performance continuous and accurate tracking of a specified single moving object. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is one of the flow charts of the three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information provided by the present invention;

[0042] Figure 2 This is the second flow chart of the three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information provided by the present invention;

[0043] Figure 3 A schematic diagram of an embodiment of a three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information provided by the present invention, which uses bounding box size information to adaptively sample and obtain sampling points;

[0044] Figure 4This is a graph showing the output results of several consecutive frames of actual operation of an embodiment of the three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information provided by the present invention;

[0045] Figure 5 It is a structural diagram of a three-dimensional moving target tracking device based on spatiotemporal domain and bounding box information provided by the present invention;

[0046] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention.

[0047] Reference numerals:

[0048] 510: Feature unit; 520: Optimization unit; 530: Positioning unit. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0050] The following combination Figure 1-Figure 4 Describe the three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information of the present invention, Figure 1-Figure 2 : is a flow chart of a three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information provided by the present invention, such as Figure 1 As shown, the method includes the following steps:

[0051] Step 110: For a given point cloud sequence, obtain the mask information of the tracking target in each frame of the point cloud based on the bounding box information of the tracking target in each frame; perform geometric feature extraction on each frame of the point cloud sequence to obtain point features of each frame, wherein the point features include points and geometric feature description operators of the points.

[0052] For a given cloud sequence, mask information calculation and geometric feature extraction are performed. It should be noted that the two steps of mask information calculation and geometric feature extraction do not necessarily have a sequence relationship. They can be performed simultaneously, or mask information calculation can be performed first and then geometric feature extraction.

[0053] Specifically, given a cloud sequence that is directly acquired or predicted, the bounding box information of each frame of the point cloud is used to predict or calculate the mask information of the tracked target in each frame of the point cloud. It should be noted that the bounding box information is the bounding box information of the tracked target and is obtained through calculation or prediction.

[0054] Furthermore, the method for calculating mask information includes: using the bounding box information of each frame of the point cloud to calculate whether each point in the point cloud is within the bounding box, or using point features obtained by geometric feature extraction to predict the mask information. In other words, the mask information includes the determination of whether each point in each frame of the point cloud is within the bounding box.

[0055] When performing geometric feature extraction, geometric feature extraction is performed on each frame of the input point cloud sequence to obtain at least one point feature for each frame. In some embodiments, geometric feature extraction is performed using a feature extraction network to obtain discriminative point features. The feature extraction network can be a hierarchical backbone network.

[0056] Specifically, the hierarchical backbone network can select any hierarchical point cloud feature extraction network based on actual conditions, and the present invention does not impose any restrictions on this. The feature extraction network shares network weights with each frame in the point cloud sequence, decoding each point and its geometric feature description operator. Each point and its geometric feature description operator together constitute a point feature. In general, there is at least one point feature, and all point features constitute a point feature set.

[0057] Step 120: Based on the attention mechanism, the mask information of the tracking target is transferred to the current frame point feature by utilizing feature fusion or sharing of similarity matrix, and the current frame point feature is optimized using spatiotemporal context information to obtain the optimized point feature of the current frame; the spatiotemporal context information includes the spatial context feature relationship within the current frame and the temporal context feature relationship between frames.

[0058] After obtaining the mask information and point features, the attention mechanism is used to transfer the mask information of the tracking target into the point features or mask feature description operator of the current frame by using feature fusion or sharing the similarity matrix, and the context information is used to optimize the feature of the current frame point features.

[0059] In the specific implementation process, the mask of the tracking target in each frame is first encoded into a mask feature description operator, and then the mask information of the tracking target is passed into the point feature or mask feature description operator of the current frame by using feature fusion or shared similarity matrix. At the same time, the self-attention mechanism is combined with the cross-attention mechanism to model the feature relationship between the tracking target and the spatiotemporal domain context, and the feature description operator of the current frame is optimized.

[0060] It should be noted that the spatiotemporal context information includes the spatial context feature relationship within the current frame and the temporal context feature relationship between frames.

[0061] Step 130: Calculate bounding box information of the tracking target in the current frame using the optimized point features in the current frame.

[0062] After obtaining the optimized point features, the bounding box information of the current frame is predicted based on the optimized point feature information. This process can use the bounding box size information of the historical frames for more accurate positioning. Specifically, the optimized point features are used to predict the bounding box information of the current frame using an attention mechanism, or several reference points are sampled based on the bounding boxes of the historical frames, the point features are converted into features at the reference points, and then the bounding box information of the current frame is predicted.

[0063] The optimized point features of each current frame are used to predict the center position of the tracking target and whether it belongs to the tracking target. The latter constitutes the mask information of the tracking target in the current frame.

[0064] After the prediction is completed, in some embodiments, a local self-attention mechanism is used to aggregate information of optimized point features close to the prediction center position, and the bounding box of the tracking target is predicted for the aggregated features to obtain the bounding box information of the tracking target in the current frame.

[0065] In other embodiments, bounding box size information from historical frames is used to uniformly sample a fixed number of reference points near the predicted center position. Point features from the current frame are converted to point feature descriptors defined on the reference points. Information is aggregated from the reference point features to predict the bounding box of the tracking target. Furthermore, information aggregation is performed using a 3D convolutional neural network based on the reference point features.

[0066] Based on the above embodiment, in this method, the mask information of the tracking target is transferred to the point features of the current frame by using the feature fusion method based on the attention mechanism, and the point features of the current frame are optimized by using the spatiotemporal context information to obtain the optimized point features of the current frame, which specifically includes:

[0067] Encoding the mask information of the tracked target in each frame into a mask feature description operator;

[0068] fusing the mask feature description operator with the geometric feature description operator;

[0069] Using a combination of self-attention and cross-attention mechanisms, the feature relationship between the tracking target and the spatiotemporal context information in each frame is modeled;

[0070] Using the cross attention mechanism, based on the feature relationship, the spatiotemporal context information in the historical frame is utilized to optimize the current frame point features to obtain the optimized point features of the current frame.

[0071] Specifically, if the feature fusion method is used, the mask information of the tracking target in each frame is first encoded into a mask feature description operator. iRepresents the tracking target mask information of the i-th frame point cloud, the tracking target mask information M of the i-th frame point cloud i The encoded mask feature description operator is denoted as ME i The geometric feature description operator of the i-th frame point cloud is X i For the current frame t, initialize the value of each mask to 0.5 to get ME t .

[0072] In the specific implementation process, the mask feature description operator ME is first i Fusion with geometric feature description operators:

[0073]

[0074] Then, the self-attention mechanism and the cross-attention mechanism are interleaved to model the relationship between the spatial context features within the current frame and the temporal context features between frames. Assume that the attention mechanism is expressed as:

[0075]

[0076] Among them, d represents the number of dimensions of the feature, Q represents the query feature in the attention mechanism, and K T represents the transpose of the key features in the attention mechanism, V represents the median feature of the attention mechanism, Q, K, and V are all obtained by transforming the input features, and softmax() represents the normalized exponential function.

[0077] The feature relationship of spatiotemporal context information within a frame is modeled through the self-attention mechanism. The feature relationship is expressed as:

[0078]

[0079] Among them, W q , W k , W v is the learnable weight matrix.

[0080] The cross attention mechanism is used to optimize the feature information of the current frame using the context information in the historical frames. In order to save computing resources, it is assumed that only the information of the previous k frames is used, which can be expressed as:

[0081]

[0082] in, Depend on After linear transformation, the above attention mechanism can be repeatedly stacked in multiple layers to obtain the optimized feature representation, that is, the optimized point feature.

[0083] Based on the above embodiment, in this method, the mask information of the tracking target is transferred to the point features of the current frame by sharing the similarity matrix based on the attention mechanism, and the point features of the current frame are optimized by using the spatiotemporal context information to obtain the optimized point features of the current frame, which specifically includes:

[0084] Encoding the mask information of the tracked target in each frame into a mask feature description operator;

[0085] For the geometric features and mask features, a self-attention mechanism and a cross-attention mechanism are respectively used to model the feature relationship between the tracking target and the spatiotemporal context information in each frame to obtain a first feature relationship and a second feature relationship;

[0086] Using a cross attention mechanism, based on the first feature relationship and the second feature relationship, and utilizing context information in historical frames to optimize the current frame point feature, a first target point feature and a second target point feature are obtained;

[0087] Based on the first target point feature and the second target point feature, the mask feature description operator is fused with the geometric feature description operator to obtain the optimized point feature of the current frame.

[0088] Specifically, the mask information of the tracking target in each frame is first encoded into a mask feature description operator. i Represents the tracking target mask information of the i-th frame point cloud, the tracking target mask information M of the i-th frame point cloud i The encoded mask feature description operator is denoted as ME i The geometric feature description operator of the i-th frame point cloud is X i For the current frame t, initialize the value of each mask to 0.5 to get ME t .

[0089] If the shared attention matrix is used, the geometric features and mask features are not fused. The other processing methods are similar to the feature fusion method described above, that is, the self-attention mechanism and the cross-attention mechanism are interleaved for the geometric features and mask features respectively, and the attention matrix in the shared attention mechanism is used to model the relationship between the spatial context features within the current frame and the temporal context features between frames. Assume that the attention mechanism is expressed as:

[0090]

[0091] Among them, d represents the number of dimensions of the feature, Q represents the query feature in the attention mechanism, and K T represents the transpose of the key features in the attention mechanism, V represents the median feature of the attention mechanism, Q, K, and V are all obtained by transforming the input features, and softmax() represents the normalized exponential function.

[0092] For geometric features, the feature relationship of spatiotemporal context information within the frame is modeled through the self-attention mechanism to obtain the first feature relationship:

[0093]

[0094] The cross attention mechanism is used to optimize the geometric feature information of the current frame using the context information in the historical frames. In order to save computing resources, it is assumed that only the information of the first k frames is used to obtain the first target point feature. The first target point feature is expressed as:

[0095]

[0096] For mask features, the attention matrix calculated by reusing geometric features Model the feature interaction and obtain the second feature relationship:

[0097]

[0098] The cross attention mechanism is used to optimize the mask feature information of the current frame using the context information in the historical frames. In order to save computing resources, it is assumed that only the information of the previous k frames is used to obtain the second target point feature. The second target point feature is expressed as:

[0099]

[0100] in, By M i After linear transformation, we get .

[0101] The above attention mechanism can be repeatedly stacked in multiple layers to obtain the optimized feature representation. Finally, the geometric features are fused with the mask features, that is, the first target point features and the second target point features that are finally optimized are fused to obtain the final optimized point features.

[0102]

[0103] In some embodiments, after obtaining the final optimized point features, the bounding box information of the current frame is predicted based on the optimized point feature information. The optimized point features of the current frame are denoted as F, where the i-th point x i The feature is represented as f i , first predict the center position of the tracking target and whether it belongs to the tracking target for each point:

[0104]

[0105]

[0106] in is a parameter-learnable transformation that constitutes the tracking target mask of the current frame, c i is the center position of the tracking target predicted at this point, m i This is a prediction of whether the point belongs to the tracking target. The local self-attention mechanism can be used to aggregate information about point features close to the predicted center, or the bounding box size information of historical frames can be used to sample reference points and aggregate information about the reference points.

[0107] When using the local self-attention mechanism to aggregate information of point features with similar predicted center positions, in some embodiments, calculating the bounding box information of the tracking target in the current frame using the optimized point features of the current frame specifically includes:

[0108] Predicting the center position of the tracking target based on each optimized point feature in the current frame to obtain a predicted center position, and predicting whether each point in the current frame belongs to the tracking target;

[0109] The local self-attention mechanism is used to aggregate the point features in the neighborhood of the predicted center position to obtain the bounding box information of the current frame.

[0110] Specifically, if the local self-attention mechanism is used, point x is defined as i The neighborhood points of are:

[0111] N(x i )={x k |||c i -c k || <r}

[0112] Where r is the specified radius.

[0113] The local self-attention mechanism is expressed as:

[0114]

[0115] Among them, for the matrix Qk T The value of the i,j position, if the point x j Not point x i The neighborhood point of Mask is filled with -inf, which means the point feature f j Not participating point feature f i The optimization process is used to suppress the interaction of point features that do not belong to the same object and obtain the optimized features:

[0116] F′=F+LocalAttn(FW q ,FW k ,FW v )

[0117] Use the optimized feature F′ to predict the bounding box of the tracking target:

[0118] B t =h(F ′ )

[0119] Among them, h represents the parameter learnable transformation function, B t Parameters of the tracking target bounding box.

[0120] Using the bounding box size information of the historical frame, sampling reference points, and aggregating information on the reference points, in some embodiments, using the optimized point features of the current frame to calculate the bounding box information of the tracking target in the current frame specifically includes:

[0121] Predicting the center position of the tracking target based on each optimized point feature in the current frame to obtain a predicted center position, and predicting whether each point in the current frame belongs to the tracking target;

[0122] Using the bounding box size information of the historical frame, a preset number of reference points are sampled near the predicted center position;

[0123] Converting the optimized point features of the current frame into point features defined on the reference points;

[0124] Information aggregation is performed on the point features of the reference point to obtain the bounding box information of the current frame.

[0125] Specifically, if the bounding box size information (w, l, h) of the historical frame is used, the tracking target center c is predicted. v uniformly sample a fixed number n nearby x ×n y ×n z Reference points:

[0126]

[0127] Get as Figure 3 The reference points (i.e., sampling points) shown in the figure use operators such as EdgeConv to transform the point features F into features defined at the sampling points. Since the sampling points are densely distributed along the coordinate axis, the sampling point features can be converted into a three-dimensional feature map. The three-dimensional convolutional neural network is used to aggregate information on the reference point features and predict the bounding box of the tracking target.

[0128] In this embodiment, dynamic reference point sampling is performed using the bounding box size information of the tracking target. Reference points are adaptively sampled using this bounding box size information to assist in target positioning. Reference points can adaptively cover objects of varying sizes and types, enabling robust and precise target tracking. This allows for relatively accurate target positioning even in scenarios where the target point cloud undergoes shape changes, occlusions, or loss. This system demonstrates excellent positioning performance for targets of varying types and sizes, effectively tracking 3D moving objects.

[0129] In one embodiment, the present invention finally outputs the bounding box prediction results of several consecutive frames as follows: Figure 4 As shown, from top to bottom, it represents the point cloud representation of each frame in the point cloud sequence, the B box represents the real position of the tracked target, and the A box represents the prediction result given by the present invention.

[0130] Based on the above embodiment, in the method, for a given point cloud sequence, obtaining the mask information of the tracking target in each frame of the point cloud based on the bounding box information of the tracking target in each frame specifically includes:

[0131] For an initial frame in the point cloud sequence, based on the bounding box information of the tracking target in the initial frame, calculating point by point whether each point in the initial frame is within the bounding box, and obtaining mask information of the tracking target;

[0132] For a non-initial frame in the point cloud sequence, when predicting bounding box information of a tracking target in the non-initial frame, mask information of the tracking target is predicted.

[0133] Specifically, for the bounding box of the tracking target in the initial frame (first frame) in a given point cloud sequence, it is calculated point by point whether each point in the initial frame is within the bounding box to obtain the mask of the tracking target. For other frames in the point cloud sequence, when predicting the bounding box of the tracked object in each frame, the present invention will also predict the mask of the tracking target in that frame, and the prediction result will be reused as input in the subsequent tracking process. It can be understood that the mask prediction is an auxiliary output of the present invention, which is used as the input for subsequent predictions of the algorithm, indicating whether each point belongs to the tracked object, that is, a probability between 0-1, which includes the model's confidence in whether each point falls within the bounding box. Compared with the 01 binary mask of each point calculated using the predicted bounding box, it has richer information.

[0134] The method provided by the present invention fully utilizes the rich spatiotemporal context information in the historical frames and the current frame, and does not require cropping of the point cloud, so that it can better handle problems such as shape changes and occlusion of the tracked target than previous methods. Through the self-attention mechanism and the cross-attention mechanism, the information interaction between the tracked target and the spatiotemporal context in the dynamic scene is modeled. The method has achieved results that are better than the previous methods in terms of accuracy and has achieved performance that exceeds real-time in terms of speed. At the same time, in order to accurately locate moving targets of different categories, sizes, and shapes, the present invention uses the tracking target bounding box size information of the historical frame to perform dynamic reference point sampling, and uses the reference point aggregation point feature information to obtain the final bounding box prediction result. The method has achieved results that are better than the previous methods in terms of tracking accuracy and has achieved a running speed that exceeds real-time.

[0135] The present invention provides a three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information. For a given point cloud sequence, the mask information of the tracking target in each frame of the point cloud is obtained based on the bounding box information of the tracking target in each frame. Geometric features are extracted from each frame of the point cloud sequence to obtain point features for each frame. The point features include points and geometric feature description operators of the points. Based on an attention mechanism, the mask information of the tracking target is transferred to the point features of the current frame by using feature fusion or a shared similarity matrix. The point features of the current frame are optimized using spatiotemporal context information to obtain optimized point features of the current frame. The spatiotemporal context information includes the spatial context feature relationship within the current frame and the temporal context feature relationship between frames. The optimized point features of the current frame are used to calculate the bounding box information of the tracking target in the current frame. Based on the attention mechanism, the present invention uses spatiotemporal domain context and bounding box size information to track three-dimensional moving targets. This method fully utilizes the rich spatiotemporal domain context information in historical frames and the current frame, eliminating the need for cropping the point cloud, and achieving higher-performance continuous and accurate tracking of a specified single moving object.

[0136] The three-dimensional motion target tracking device based on the spatiotemporal domain and bounding box information provided by the present invention is described below. The three-dimensional motion target tracking device based on the spatiotemporal domain and bounding box information described below and the three-dimensional motion target tracking method based on the spatiotemporal domain and bounding box information described above can be referenced to each other. Figure 5 Schematic diagram of the structure of the three-dimensional moving target tracking device based on spatiotemporal domain and bounding box information provided by the present invention. Figure 5 As shown, the device includes:

[0137] The feature unit 510 is configured to obtain, for a given point cloud sequence, mask information of the tracked target in each frame of the point cloud based on the bounding box information of the tracked target in each frame; extract geometric features from each frame of the point cloud sequence to obtain point features of each frame, wherein the point features include points and geometric feature description operators of the points;

[0138] The optimization unit 520 is configured to transfer the mask information of the tracking target to the point features of the current frame by using feature fusion or a shared similarity matrix based on an attention mechanism, and optimize the point features of the current frame using spatiotemporal context information to obtain optimized point features of the current frame; the spatiotemporal context information includes the relationship between spatial context features within the current frame and the relationship between temporal context features between frames;

[0139] The positioning unit 530 is configured to calculate bounding box information of the tracking target in the current frame using the optimized point features in the current frame.

[0140] Based on the above embodiment, in the device, the mask information of the tracking target is transferred to the current frame point feature by using the feature fusion method based on the attention mechanism, and the current frame point feature is optimized by using the spatiotemporal context information to obtain the optimized point feature of the current frame, which specifically includes:

[0141] Encoding the mask information of the tracked target in each frame into a mask feature description operator;

[0142] fusing the mask feature description operator with the geometric feature description operator;

[0143] Using a combination of self-attention and cross-attention mechanisms, the feature relationship between the tracking target and the spatiotemporal context information in each frame is modeled;

[0144] Using the cross attention mechanism, based on the feature relationship, the spatiotemporal context information in the historical frame is utilized to optimize the current frame point features to obtain the optimized point features of the current frame.

[0145] Based on the above embodiment, in the device, the mask information of the tracking target is transferred to the current frame point feature by sharing the similarity matrix based on the attention mechanism, and the current frame point feature is optimized by using the spatiotemporal context information to obtain the optimized point feature of the current frame, which specifically includes:

[0146] Encoding the mask information of the tracked target in each frame into a mask feature description operator;

[0147] For the geometric features and mask features, a self-attention mechanism and a cross-attention mechanism are respectively used to model the feature relationship between the tracking target and the spatiotemporal context information in each frame to obtain a first feature relationship and a second feature relationship;

[0148] Using a cross attention mechanism, based on the first feature relationship and the second feature relationship, and utilizing context information in historical frames to optimize the current frame point feature, a first target point feature and a second target point feature are obtained;

[0149] Based on the first target point feature and the second target point feature, the mask feature description operator is fused with the geometric feature description operator to obtain the optimized point feature of the current frame.

[0150] Based on the above embodiment, in the device, calculating the bounding box information of the tracking target in the current frame using the optimized point features of the current frame specifically includes:

[0151] Predicting the center position of the tracking target based on each optimized point feature in the current frame to obtain a predicted center position, and predicting whether each point in the current frame belongs to the tracking target;

[0152] The local self-attention mechanism is used to aggregate the point features in the neighborhood of the predicted center position to obtain the bounding box information of the current frame.

[0153] Based on the above embodiment, in the device, calculating the bounding box information of the tracking target in the current frame using the optimized point features of the current frame specifically includes:

[0154] Predicting the center position of the tracking target based on each optimized point feature in the current frame to obtain a predicted center position, and predicting whether each point in the current frame belongs to the tracking target;

[0155] Using the bounding box size information of the historical frame, a preset number of reference points are sampled near the predicted center position;

[0156] Converting the optimized point features of the current frame into point features defined on the reference points;

[0157] Information aggregation is performed on the point features of the reference point to obtain the bounding box information of the current frame.

[0158] Based on the above embodiment, in the device, for a given point cloud sequence, obtaining the mask information of the tracking target in each frame of the point cloud based on the bounding box information of the tracking target in each frame specifically includes:

[0159] For an initial frame in the point cloud sequence, based on the bounding box information of the tracking target in the initial frame, calculating point by point whether each point in the initial frame is within the bounding box, and obtaining mask information of the tracking target;

[0160] For a non-initial frame in the point cloud sequence, when predicting bounding box information of a tracking target in the non-initial frame, mask information of the tracking target is predicted.

[0161] The present invention provides a three-dimensional moving target tracking device based on spatiotemporal domain and bounding box information. For a given point cloud sequence, the device obtains the mask information of the tracking target in each frame of the point cloud based on the bounding box information of the tracking target in each frame; extracts geometric features from each frame of the point cloud sequence to obtain point features for each frame, wherein the point features include points and geometric feature description operators of the points; based on an attention mechanism, the mask information of the tracking target is transferred to the point features of the current frame by using feature fusion or a shared similarity matrix; the point features of the current frame are optimized using spatiotemporal context information to obtain optimized point features of the current frame; the spatiotemporal context information includes the spatial context feature relationship within the current frame and the temporal context feature relationship between frames; and the bounding box information of the tracking target in the current frame is calculated using the optimized point features of the current frame. Based on the attention mechanism, the present invention uses spatiotemporal domain context and bounding box size information to track three-dimensional moving targets. This fully utilizes the rich spatiotemporal domain context information in historical frames and the current frame, eliminating the need for cropping the point cloud, and achieving higher performance, continuous, and accurate tracking of a specified single moving object.

[0162] Figure 6 An example of a physical structure diagram of an electronic device is shown below. Figure 6As shown, the electronic device may include: a processor (processor) 610, a communication interface (Communications Interface) 620, a memory (memory) 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the logic instructions in the memory 630 to execute a three-dimensional motion target tracking method based on the spatiotemporal domain and bounding box information, the method including: for a given point cloud sequence, based on the bounding box information of the tracking target in each frame, obtaining the mask information of the tracking target in each frame point cloud; performing geometric feature extraction on each frame of the point cloud sequence to obtain the point features of each frame, the point features including the points and the geometric feature description operators of the points; based on the attention mechanism, using feature fusion or sharing of the similarity matrix, the mask information of the tracking target is transferred to the current frame point features, and the current frame point features are optimized using spatiotemporal context information to obtain the optimized point features of the current frame; the spatiotemporal context information includes the spatial context feature relationship within the current frame and the temporal context feature relationship between frames; and the optimized point features of the current frame are used to calculate the bounding box information of the tracking target in the current frame.

[0163] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0164] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the three-dimensional motion target tracking method based on the spatiotemporal domain and bounding box information provided by the above methods. The method includes: for a given point cloud sequence, based on the bounding box information of the tracking target in each frame, obtaining the mask information of the tracking target in each frame point cloud; performing geometric feature extraction on each frame of the point cloud sequence to obtain point features of each frame, wherein the point features include points and geometric feature description operators of the points; based on the attention mechanism, the mask information of the tracking target is transferred to the current frame point features by using feature fusion or shared similarity matrix, and the current frame point features are optimized by using spatiotemporal context information to obtain the optimized point features of the current frame; the spatiotemporal context information includes the spatial context feature relationship within the current frame and the temporal context feature relationship between frames; and the optimized point features of the current frame are calculated using the optimized point features of the current frame.

[0165] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the three-dimensional motion target tracking method based on the spatiotemporal domain and bounding box information provided by the above-mentioned methods, the method comprising: for a given point cloud sequence, obtaining the mask information of the tracking target in each frame point cloud based on the bounding box information of the tracking target in each frame; performing geometric feature extraction on each frame of the point cloud sequence to obtain point features of each frame, wherein the point features include points and geometric feature description operators of the points; based on the attention mechanism, the mask information of the tracking target is transferred to the point features of the current frame by utilizing feature fusion or sharing of a similarity matrix, and the point features of the current frame are optimized by utilizing spatiotemporal context information to obtain the optimized point features of the current frame; the spatiotemporal context information includes the spatial context feature relationship within the current frame and the temporal context feature relationship between frames; and the optimized point features of the current frame are calculated using the optimized point features of the current frame.

[0166] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0167] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information, characterized in that: include: For a given point cloud sequence, the mask information of the tracking target in each frame of the point cloud is obtained based on the bounding box information of the tracking target in each frame; Extracting geometric features from each frame of the point cloud sequence to obtain point features of each frame, wherein the point features include points and geometric feature description operators of the points; Based on the attention mechanism, the mask information of the tracking target is transferred to the point features of the current frame by means of feature fusion or shared similarity matrix, and the point features of the current frame are optimized using spatiotemporal context information to obtain the optimized point features of the current frame; The spatiotemporal context information includes the spatial context feature relationship within the current frame and the temporal context feature relationship between frames; The optimized point features of the current frame are used to calculate the bounding box information of the tracking target in the current frame.

2. The three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information according to claim 1 is characterized in that: The attention mechanism is based on the feature fusion method to transfer the mask information of the tracking target to the current frame point feature, and the spatiotemporal context information is used to optimize the current frame point feature to obtain the optimized point feature of the current frame, which specifically includes: Encoding the mask information of the tracked target in each frame into a mask feature description operator; fusing the mask feature description operator with the geometric feature description operator; Using a combination of self-attention and cross-attention mechanisms, the feature relationship between the tracking target and the spatiotemporal context information in each frame is modeled; Using the cross attention mechanism, based on the feature relationship, the spatiotemporal context information in the historical frame is utilized to optimize the current frame point features to obtain the optimized point features of the current frame.

3. The three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information according to claim 1, characterized in that: The attention mechanism is based on the shared similarity matrix, which is used to transfer the mask information of the tracking target to the current frame point feature. The spatiotemporal context information is used to optimize the current frame point feature to obtain the optimized point feature of the current frame. Specifically, the following steps are performed: Encoding the mask information of the tracked target in each frame into a mask feature description operator; For the geometric features and mask features, a self-attention mechanism and a cross-attention mechanism are respectively used to model the feature relationship between the tracking target and the spatiotemporal context information in each frame to obtain a first feature relationship and a second feature relationship; Using a cross attention mechanism, based on the first feature relationship and the second feature relationship, and utilizing context information in historical frames to optimize the current frame point feature, a first target point feature and a second target point feature are obtained; Based on the first target point feature and the second target point feature, the mask feature description operator is fused with the geometric feature description operator to obtain the optimized point feature of the current frame.

4. The three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information according to claim 1, characterized in that: The calculating the bounding box information of the tracking target in the current frame by using the optimized point features in the current frame specifically includes: Predicting the center position of the tracking target based on each optimized point feature in the current frame to obtain a predicted center position, and predicting whether each point in the current frame belongs to the tracking target; The local self-attention mechanism is used to aggregate the point features in the neighborhood of the predicted center position to obtain the bounding box information of the current frame.

5. The three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information according to claim 1, characterized in that: The calculating the bounding box information of the tracking target in the current frame by using the optimized point features in the current frame specifically includes: Predicting the center position of the tracking target based on each optimized point feature in the current frame to obtain a predicted center position, and predicting whether each point in the current frame belongs to the tracking target; Using the bounding box size information of the historical frame, a preset number of reference points are sampled near the predicted center position; Converting the optimized point features of the current frame into point features defined on the reference points; Information aggregation is performed on the point features of the reference point to obtain the bounding box information of the current frame.

6. The three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information according to claim 1, characterized in that: For a given point cloud sequence, obtaining the mask information of the tracking target in each frame of the point cloud based on the bounding box information of the tracking target in each frame specifically includes: For an initial frame in the point cloud sequence, based on the bounding box information of the tracking target in the initial frame, calculating point by point whether each point in the initial frame is within the bounding box, and obtaining mask information of the tracking target; For a non-initial frame in the point cloud sequence, when predicting bounding box information of a tracking target in the non-initial frame, mask information of the tracking target is predicted.

7. A three-dimensional moving target tracking device based on spatiotemporal domain and bounding box information, characterized in that: include: A feature unit is configured to obtain, for a given point cloud sequence, mask information of the tracked target in each frame of the point cloud based on the bounding box information of the tracked target in each frame; Extracting geometric features from each frame of the point cloud sequence to obtain point features of each frame, wherein the point features include points and geometric feature description operators of the points; An optimization unit is configured to transfer the mask information of the tracking target to the point features of the current frame by utilizing feature fusion or a shared similarity matrix based on an attention mechanism, and optimize the point features of the current frame by utilizing spatiotemporal context information to obtain optimized point features of the current frame; The spatiotemporal context information includes the spatial context feature relationship within the current frame and the temporal context feature relationship between frames; A positioning unit is used to calculate the bounding box information of the tracking target in the current frame using the optimized point features in the current frame.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information is implemented as described in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the three-dimensional moving target tracking method based on spatiotemporal domain and bounding box information is implemented as described in any one of claims 1 to 6.

Images