Three-dimensional video target tracking method based on multi-level mutual enhancement and correlation pyramid

Abstract

The application discloses a three-dimensional video target tracking method based on multi-level mutual enhancement and a related pyramid, and comprises the following steps: S1, establishing an encoder-decoder paradigm for three-dimensional video single-target tracking; S2, establishing a local self-attention enhancement unit of homologous multi-level target features; S3, establishing a local mutual attention enhancement unit of heterologous multi-scale template and search area features; S4, establishing an encoder with a pyramid structure of lateral full-stage related feature aggregation; S5, establishing a double-branch decoder for motion mode separation; S6, after a single-target tracker containing S1-S5 is constructed by using a server, the network model of the single-target tracker is trained, the network parameters are optimized by reducing a network loss function, and the network converges until the network converges; and S7, for a given target in a video sequence of a three-dimensional video to be processed, the network model is used for tracking. The method can accurately and quickly track a target in a complex environment.

Description

Technical Field

[0001] This invention belongs to the technical field of 3D vision, specifically relating to a 3D video target tracking method based on multi-level mutual enhancement and correlation pyramids. Background Technology

[0002] In recent years, single object tracking (SOT) based on 3D LiDAR has gained increasing attention due to its crucial role in 3D applications. This task, though relatively new, has become a vital part of 3D applications with the development of 3D sensors such as LiDAR, applicable to numerous scenarios including autonomous driving, robotics, and surveillance systems. The goal is to track a specific object in a video by providing the corresponding 3D bounding box in the first frame. Tracking a specific object in a video is challenging, primarily because the object changes at different points in time, often appearing sparse and incomplete due to occlusion by other objects or rapid movement.

[0003] One existing method is as follows Figure 1a As shown, the tracking of a specific target within a 3D target bounding box is first accomplished using an "extractor-matcher-decoder" paradigm. First, the extractor encodes the features of the template and the search region. Then, the matcher is used to establish the template-search relationship and enhance the potential target features, i.e., embedding the template features into the search region features—a process known as correlation. Finally, the decoder uses the features from the matcher to generate the 3D target bounding box. Modern backbone networks such as PointNet and PointNet++ have become the mainstream, even default, choice for extractors. Most trackers in this paradigm are designed with stronger and more refined matchers and more powerful decoders in mind. However, this traditional paradigm still has two problems:

[0004] (1) The relationship between the template and the search features is only modeled in the matcher, which is insufficient for full cross-source interaction and target augmentation, and cannot achieve full cross-source. The traditional paradigm separates the extractor and the matcher, with the matcher responsible for enhancing the template features. Inspired by two-dimensional target tracking methods, previous methods in this paradigm always used a Siamese network in the extractor to independently encode the features of the template and the search region without any intermediate interaction. Then, the matcher was used to fuse the template features into the search features. However, the independent matcher is redundant, requiring additional establishment of the relationship between the template and the search region, bringing additional parameters and computations. Moreover, it is not enough to use only the high semantic layer features extracted by the matcher, and it cannot match the low semantic layer and mid semantic layer features.

[0005] (2) Point downsampling in commonly used backbone networks inevitably exacerbates the sparsity of the point cloud. Most backbone network default configurations have relatively poor matching with 3D target tracking. For example, the most commonly used PointNet++ usually reduces the input points (usually 1024 or 512 points) to 1 / 8 of the input, and the matcher and decoder only have very sparse input information (128 or 64 points). In fact, the feature extraction backbone network plays a core role in target tracking. It needs to provide a discriminative target representation for the sparse input point cloud, distinguishing it from the surrounding background, including interference and noise.

[0006] Another approach, such as motion trackers, proposes a motion-centric paradigm to replace the aforementioned "extractor-matcher-decoder" paradigm. This paradigm constructs a spatial-temporal point cloud to avoid Siamese networks and directly predicts motion. However, this approach still requires a motion transformation module to integrate template information into the search representation and employs a two-stage refinement method to ensure performance, introducing additional network structure and parameter count. Furthermore, it cannot directly achieve template and search region matching in the encoder. Summary of the Invention

[0007] To achieve the above objectives, this invention provides a three-dimensional video target tracking method based on a multi-level mutual enhancement and correlation pyramid, which can accurately and quickly track targets in complex real-world environments.

[0008] This invention adopts the following technical solution: a three-dimensional video target tracking method based on multi-level mutual enhancement and correlation pyramids, comprising the following steps:

[0009] S1. Establish an encoder-decoder paradigm for single-target tracking in 3D video;

[0010] S2. Establish local self-attention enhancement units for multi-level target features of the same origin;

[0011] S3. Establish a local mutual attention enhancement unit between heterogeneous multi-scale templates and search region features;

[0012] S4. Establish an encoder with a pyramid structure that aggregates lateral full-stage relevant features;

[0013] S5. Establish a dual-branch decoder for motion pattern separation;

[0014] S6. Construct a single-target tracker under the paradigm of S1, including a pyramid-structured encoder consisting of a series of point cloud set summarization units, local self-attention enhancement units, and local mutual attention enhancement units, as well as a dual-branch decoder; after constructing the network model of the single-target tracker, train the network model using a server, optimize the network parameters by reducing the network loss function until the network converges.

[0015] S7. For a given target in the video sequence of the 3D video to be processed, use the network model obtained in S6 to track it.

[0016] Furthermore, step S1 specifically includes:

[0017] S11. Establish an encoder that integrates a feature extractor and a matcher to extract the search region P of a single target from a given 3D video. s and a template region P t Simultaneously input into the encoder, perform feature extraction on both and feature interaction between them at one time, and output the target sensitive features after the search region interaction;

[0018] S12. Input the target-sensitive features output from S11 into the decoder to obtain and output the tracking results. The tracking results include the predicted center localization map, coordinate regression value, and rotation angle regression value of the corresponding single target.

[0019] Furthermore, step S2 specifically includes:

[0020] S21. Features X of a single input point cloud i and the corresponding three-dimensional coordinate position P i The features are obtained by performing point-by-point feature transformation using a point-level linear layer f1.

[0021] S22, Features Perform a set of point-level linear transformations f q Get query feature q i According to the three-dimensional coordinate position P i Calculate the Euclidean distance with all source points, select the k nearest neighbors, and then apply two sets of linear transformations f to the corresponding features. k ,f V The key features k were obtained respectively. i Sum value characteristic V i ;

[0022] S23. Calculate the local self-attention of the same source.

[0023]

[0024] Among them, e ij=f e (P i -P j ) is the positional encoding, f e For point-by-point linear transformation, j represents the order of nearest neighbors, k is the number of nearest neighbors, and f a It is a two-layer pointwise linear transformation;

[0025] S24. Based on the attention features obtained in S23, output the template features. and search region features They are respectively:

[0026]

[0027] Where f2 is a point-by-point linear transformation The individual point cloud features input in step S21 are respectively: search region features and template region features. These are the local self-attention features of the search region and the local self-attention features of the template region obtained in step S23, respectively; σ is the softmax function.

[0028] Furthermore, step S3 specifically includes:

[0029] S31. For the input single point cloud feature X i and the corresponding three-dimensional coordinate position P i The features are obtained by performing point-by-point feature transformation using a point-level linear layer f'1.

[0030] S32, Features obtained from S31 Perform a set of point-level linear transformations f' q Get query feature q' i According to position P i Euclidean distances are calculated with all heterogeneous points. After selecting the k nearest neighbors, the corresponding features are subjected to two sets of linear transformations f'. k ,f' v The key features k' were obtained respectively. i Sum feature v' i ;

[0031] S33. Calculate the heterogeneous local self-attention features.

[0032]

[0033] Where e' ij =f' e (P i -P j ) is the positional encoding, f' eFor point-by-point linear transformation, j represents the order of nearest neighbors, k is the number of nearest neighbors, and f' a It is a two-layer pointwise linear transformation;

[0034] S34. The local mutual attention enhancement unit utilizes the heterogeneous local self-attention feature to calculate the output model.

[0035] Plate characteristics and search region features They are respectively:

[0036]

[0037] Where f'2 is a pointwise linear transformation The individual point cloud features input in step S31 are respectively: search region features and template region features. These are the heterogeneous local self-attention features of the search region and the heterogeneous local self-attention features of the template region obtained from step S33, respectively; σ is the softmax function.

[0038] Furthermore, the backbone network of the encoder in step S4 specifically includes three stages: each stage includes a point cloud set summarization unit, a local self-attention unit, and a local mutual attention unit;

[0039] In each stage: the point cloud set summary unit outputs the number of points in the input point cloud in a predetermined ratio;

[0040] The local self-attention unit calculates the feature correlation between each point and its nearest neighbors by Euclidean distance, uses this feature correlation to calculate self-attention and enhance the input features of each point, resulting in the enhanced template features. Search area characteristics Where s and t represent the search region and template region respectively, and i represents the i-th point;

[0041] The local mutual attention unit calculates the Euclidean distance between each point and the source point, and selects the source points with the nearest number of Euclidean distance neighbors for mutual attention calculation. This mutual attention is used to enhance the features of each point, resulting in enhanced template features. Search area characteristics

[0042] Furthermore, the implementation process of the pyramid structure for lateral full-stage relevance feature aggregation of the encoder in step S4 is as follows:

[0043] S41. The multiple heterogeneous related features output by the local mutual attention units in the three stages of the encoder are respectively passed through a set of side one-dimensional convolution modules to unify the feature dimensions to a fixed value. The one-dimensional convolution module includes a one-dimensional convolution layer, a batch regularization layer, a ReLU activation function layer, and a one-dimensional convolution layer.

[0044] S42. The features output in step S41 are spliced ​​together in terms of the number of points in the cloud to form a lateral full-stage related feature aggregation pyramid structure.

[0045] S43. Voxelize the features output in step S42, average the features falling into the same voxel, and finally obtain a target sensitive feature F with a regular size of C×H×L×W. m Where C, H, L, and W represent the feature dimension, height, length, and width, respectively.

[0046] Furthermore, step S5 specifically includes:

[0047] S51, Apply the target sensitive features F obtained in step S43 m A set of motion decomposition convolution modules is fed in, the motion decomposition convolution modules include two parts: two-dimensional convolution that operates in the motion xy plane and one-dimensional convolution that operates in the vertical z direction;

[0048] S52. After three consecutive sets of motion decomposition convolution modules, the obtained features are subjected to maximum pooling in the vertical z-direction and xy-plane respectively to obtain the features F of the bird's-eye view. BEV and the vertical feature F z ;

[0049] S53, the feature F of the bird's-eye view described in S52 BEV Vertical features F z The features F of the bird's-eye view are fed into their respective local feature encoding networks. BEV The local feature encoding network consists of a set of two-dimensional convolutional modules used to predict the target's displacement, discretized offset, and rotation angle in the xy plane; the vertical feature F z The local feature encoding network consists of a set of one-dimensional convolutional modules used to predict the target's vertical displacement and corresponding discretized offset.

[0050] Furthermore, step S6 specifically includes:

[0051] S61. The server generates a template-search region by adding noise in the three coordinate axes of uniform sampling to the labeled target region. After the search region frame is determined, the point cloud of the template region is selected from the previous frame of the search region frame, and each takes a preset number of point clouds.

[0052] S62. The encoder is executed using the server, which involves extracting features from the template region and search region using step S4, analyzing the feature interactions between them, and processing them through a lateral pyramid structure. Finally, the search region features, i.e., the target-sensitive features F, which have undergone multi-level self-enhancement and mutual enhancement, are output. m ;

[0053] S63. Utilize the dual-branch decoder for motion mode separation in step S5 executed by the server to output the positioning results in the xy plane and z direction;

[0054] S64. The network model is trained using a server in an end-to-end manner; the final mathematical expression of the loss function is: in, Represents the focus loss for classification; This represents the discretized offset regression L1 loss. These represent the classification loss based on the bird's-eye view feature map and the center point in the vertical direction, respectively. λ represents the regression loss based on the bird's-eye view feature map and the bounding box offset in the vertical direction, respectively; cls and λ reg These represent the weights of the classification and regression losses, respectively, and are both set to 1;

[0055] S65. Optimize the objective function using the server until the network converges, and obtain the locally optimal network parameters.

[0056] Furthermore, step S7 specifically includes:

[0057] S71. Initialize the given target. In the video sequence of the 3D video to be processed, the first frame will give the 3D target bounding box of the given target to be tracked; tracking starts from the second frame of the video sequence.

[0058] S72. The template region is constructed as a set of point clouds of the first frame and the previous frame. The search frame selects a search region with a preset distance offset from the center of the target box in the previous frame. Together with the template region, the search region features, namely the target sensitive features, are output by the encoder through self-enhancing and mutual-enhancing features.

[0059] S73. Input the target-sensitive features obtained in step S72 into the decoder to obtain the target bounding box prediction result of the current frame.

[0060] Furthermore, both the template area and the search area in step S72 are in the target bounding box coordinate system.

[0061] The beneficial effects of this invention are as follows: This method, through an "encoder-decoder" paradigm, merges the extractor and matcher to jointly learn the required target perception representation; the encoder introduces multi-level self-enhancement and cross-source enhancement in its main branch to enrich template region features and search region features, and respectively achieves their fusion and interaction; it utilizes a pyramid structure to aggregate lateral correlation features, retaining as many points as possible by integrating hierarchically correlated features to address the sparsity problem of point clouds; through a dual-branch decoder with motion pattern separation, the 3D convolution is decomposed into continuous 2D and 1D convolutional blocks, and an additional Z-axis prediction head is attached to the bird's-eye view prediction head to simultaneously and independently learn the motion of the Z-axis and XY planes; this method can accurately track targets in complex environments, and can accurately and quickly track targets in many difficult real-world scenarios. Attached Figure Description

[0062] Figure 1a It is a traditional extractor-matcher-decoder paradigm diagram;

[0063] Figure 1b This is a paradigm diagram of the encoder-decoder of the present invention;

[0064] Figure 2 This is a framework diagram of the three-dimensional video target tracking method of the present invention;

[0065] Figure 3a This is a block diagram of the self-enhancing (SA) / mutual-enhancing (CA) unit of the present invention;

[0066] Figure 3b This is a block diagram of the self-enhancing (SA) module of the present invention;

[0067] Figure 3c This is a block diagram of the mutual enhancement (CA) module of the present invention;

[0068] Figure 4 This is a block diagram of the encoder of the lateral full-stage related feature aggregation pyramid structure of the present invention. Detailed Implementation

[0069] The specific solutions and embodiments of the present invention will be further described in conjunction with the accompanying drawings, making the technical solution clearer and more understandable. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0070] like Figures 1b-4As shown, this embodiment relates to a three-dimensional video target tracking method based on multi-level mutual enhancement and correlation pyramids. It is mainly implemented through seven steps: matching-free paradigm design for single target tracking in three-dimensional video, design of homogeneous multi-level local self-attention enhancement units, design of heterogeneous multi-scale local mutual attention enhancement units, design of lateral full-stage correlation feature aggregation pyramid structure, design of motion mode separation dual-branch decoder, model training, and model inference.

[0071] This method mainly includes the following steps:

[0072] S1. Establish an encoder-decoder paradigm for single-target tracking in 3D video;

[0073] S2. Establish local self-attention enhancement units for multi-level target features of the same origin;

[0074] S3. Establish a local mutual attention enhancement unit between heterogeneous multi-scale templates and search region features;

[0075] S4. Establish an encoder with a pyramid structure that aggregates lateral full-stage relevant features;

[0076] S5. Establish a dual-branch decoder for motion pattern separation;

[0077] S6. Construct a single-target tracker under the paradigm of S1, including a pyramid-structured encoder consisting of a series of point cloud set summarization units, local self-attention enhancement units, and local mutual attention enhancement units, as well as a dual-branch decoder; after construction, use a server to train the network model of the single-target tracker, optimize the network parameters by reducing the network loss function until the network converges.

[0078] S7. For a given target in the video sequence of the 3D video to be processed, use the network model obtained in S6 to track it.

[0079] This method achieved a mean precision of 82.0% on the KITTI validation set, demonstrating excellent tracking performance.

[0080] The following provides a more detailed explanation of each step.

[0081] S1. Establish an encoder-decoder paradigm for single-target tracking in 3D video, such as... Figure 1b As shown, by modifying the traditional "extractor-matcher-decoder" paradigm, the extractor and matcher are merged to jointly learn the required target perception representation, resulting in a new "encoder-decoder" paradigm for 3D target tracking. The specific implementation process is as follows:

[0082] S11. Build an encoder that integrates a feature extractor and a matcher, given a search region P. s and template area P t At the same time, the search area P s Template area P t Both inputs are fed into the encoder, where their respective features are extracted and their features interact at once, outputting the target-sensitive features with good interaction in the search region;

[0083] S12. Input the target sensitive features directly into the decoder to obtain and output the predicted center localization map, coordinate regression value and rotation angle regression value, which is the tracking result.

[0084] S2. Establish a local self-attention and self-enhancing unit for homogeneous multi-level target features, such as Figures 3a-3b As shown, the specific steps for feature enhancement by modifying the template features or search features themselves are as follows:

[0085] S21. For a specific point cloud feature X in the input... i and the corresponding 3D coordinate position P of the point cloud i First, a point-level linear layer f1 is used to perform point-by-point feature transformation to obtain the features.

[0086] S22, to Perform a set of point-level linear transformations f q Get query feature q i According to the three-dimensional coordinate position P i Calculate the Euclidean distance with all source points, select the k nearest neighbors, and then apply two sets of linear transformations f to the corresponding features. k ,f V The key features k were obtained respectively. i Sum value characteristic V i ;

[0087] S23. Calculate the features of local self-attention in the same source area:

[0088]

[0089] Where e ij =f e (P i -P j ) is the positional encoding, f e This is a point-by-point linear transformation, where j represents the order of nearest neighbors, k is the number of nearest neighbors (typically set to 32), and f a It is a two-layer pointwise linear transformation;

[0090] S24. Calculate the output template features using the same source local self-attention feature. and search region features They are respectively:

[0091]

[0092] Where f2 is a pointwise linear transformation.

[0093] S3. Establish a heterogeneous multi-scale template and heterogeneous local mutual enhancement unit of search region features, such as... Figure 3c As shown, full-stage fusion and interaction are performed on template features and search region features to achieve full feature enhancement. The specific steps are as follows:

[0094] S31. For the input single point cloud feature X i and the corresponding three-dimensional coordinate position P i The features are obtained by performing point-by-point feature transformation using a point-level linear layer f'1.

[0095] S32, Features obtained from S31 Perform a set of point-level linear transformations f' q Get query feature q' i According to position P i Euclidean distances are calculated with all heterogeneous points. After selecting the k nearest neighbors, the corresponding features are subjected to two sets of linear transformations f'. k ,f' V The key features k' were obtained respectively. i Sum feature V' i ;

[0096] S33. Calculate the heterogeneous local self-attention features.

[0097]

[0098] Where e' ij =f' e (P i -P j ) is the positional encoding, f' e For point-by-point linear transformation, j represents the order of nearest neighbors, k is the number of nearest neighbors, and f' a It is a two-layer pointwise linear transformation;

[0099] S34. The local mutual attention enhancement unit utilizes the heterogeneous local self-attention feature to calculate the output model.

[0100] Plate characteristics and search region features They are respectively:

[0101]

[0102] Where f'2 is a pointwise linear transformation.

[0103] In this embodiment, skip connections are added to the same-source local self-attention units and different-source local mutual attention units in steps S2 and S3 to accelerate network convergence.

[0104] S4. Construct an encoder with a pyramid structure that aggregates lateral, full-stage relevant features. Its backbone network consists of three stages; each stage includes the local self-attention enhancement unit described in S2 and the local mutual attention enhancement unit described in S3.

[0105] In each stage, the point cloud set summarization unit selects N center points using the farthest point sampling method, divides the point set into several regions using the obtained center points, and then encodes each region to become the feature vector of that center point.

[0106] The local self-attention unit calculates the feature correlation between each point and its nearest neighbors (based on the number of source points) with the Euclidean distance. It then uses this feature correlation to calculate self-attention and enhance the input features of each point, resulting in the enhanced template features. Search area characteristics

[0107] The local mutual attention unit calculates the Euclidean distance between each point and the source point, and selects the source points with the nearest number of Euclidean distance neighbors for mutual attention calculation. This mutual attention is used to enhance the features of each point, resulting in enhanced template features. Search area characteristics

[0108] The point cloud set summarization unit in each of the three stages outputs a number of points in the input point cloud in descending proportions according to a preset ratio, reducing the number of points in the input point cloud to 1 / 2, 1 / 4, and 1 / 8 of the input, respectively. The local self-attention unit calculates the feature correlation between each point and its 32 closest Euclidean distance points, uses this feature correlation to calculate self-attention, and enhances the input features of each point. The local mutual attention unit calculates the Euclidean distance between each point and the source point, and selects the 32 closest Euclidean distance points for mutual attention calculation to enhance the features of each point.

[0109] In addition to the backbone network, the encoder designed in this invention also has a pyramid structure encoder with lateral full-stage relevant feature aggregation. Pyramids are mainly used in 2D detection, but their utilization in 3D target tracking has been underdeveloped, especially in order to compensate for the sparsity caused by downsampling of the backbone network and retain as many points as possible. This method utilizes the pyramid feature hierarchy of the encoder to formulate a lateral full-stage relevant feature aggregation pyramid structure, which has semantics from low to high. Multi-scale relevant features output from local mutual attention enhancement units in all stages are fused together. Note that due to the downsampling of the ensemble abstraction module, the number of points in these features is different. Furthermore, the feature dimensions and semantic levels are also different. This invention proposes a method such as... Figure 4 The lateral full-stage relevant feature aggregation pyramid structure shown retains relevant features from all stages of the backbone network, preserves features from as many points as possible in the backbone network, and performs reasonable fusion. Step S4 specifically includes the following steps:

[0110] S41. The features output by the heterogeneous local self-attention unit obtained from step S34 of the three stages of the encoder are first passed through a set of side one-dimensional convolutional modules, including one layer of one-dimensional convolution, one layer of batch normalization, one layer of ReLU activation function, and one layer of one-dimensional convolution, to unify the feature dimension to 64.

[0111] S42. The features obtained in step S41 are combined with the features of the previous stage in terms of the number of points in the cloud to form a pyramid structure of lateral full-stage related feature aggregation.

[0112] S43. Voxelize the features obtained in step S42, average the features falling into the same voxel, and finally obtain a target sensitive feature F with a regular size of C×H×L×W. m Where C, H, L, and W represent the feature dimension, height, length, and width, respectively.

[0113] This lateral full-stage correlation feature aggregation pyramid structure only requires input of the three-stage features of the search region, and does not require input of the features of the template region.

[0114] S5. Establish a dual-branch decoder for motion pattern separation, such as... Figure 2 The right half of the diagram shows the dual-branch decoder. By decomposing the target's motion into the xy-plane and the vertical direction, the dual-branch decoder is designed with a two-branch structure to learn and predict both types of motion, resulting in a more accurate position prediction. Specifically, it includes the following steps:

[0115] S51, Apply the target sensitive features F obtained in step S43 above. mA set of motion decomposition convolution modules is fed in, which decomposes ordinary 3D convolution into two continuous parts: a 2D convolution that operates in the motion xy plane and a 1D convolution that operates in the vertical z direction.

[0116] S52. After three consecutive sets of motion decomposition convolution modules, the obtained features are subjected to maximum pooling in the vertical z-direction and xy-plane respectively, to obtain the features F of the bird's-eye view. BEV and the vertical feature F z ;

[0117] S53. The features obtained in step S52 are fed into their respective local feature encoding networks, where the features F of the bird's-eye view are... BEV The encoding network consists of a set of two-dimensional convolutional modules, with vertical feature F z The encoding network consists of a set of one-dimensional convolutional modules;

[0118] Among them, the bird's-eye view local feature encoding network predicts the target's displacement, discretized offset, and rotation angle in the xy plane; the vertical local feature encoding network predicts the target's displacement and corresponding discretized offset in the vertical direction.

[0119] S6. Construct a single-object tracker based on the paradigm of S1, including a pyramid-structured encoder consisting of a series of point cloud ensemble summarization units, local self-attention enhancement units, and local mutual attention enhancement units, as well as a two-branch decoder. After construction, train the network model of the single-object tracker using a server, optimize the network parameters by reducing the network loss function until the network converges, and finally obtain the desired result. Figure 2 The network model shown includes the following steps:

[0120] S61. Use the server to generate a pair of template-search regions. After the search region frame is determined, the point cloud of the template region is selected from the previous frame. Each of them takes 1024 points.

[0121] S62. The encoder is executed using the server, which involves extracting features from the template region and search region using step S4, analyzing the feature interactions between them, and processing them through a lateral pyramid structure. Finally, the search region features, i.e., the target-sensitive features F, which have undergone multi-level self-enhancement and mutual enhancement, are output. m ;

[0122] S63. Utilize the server to execute the dual-branch decoder for motion pattern separation described in step S5, and output the positioning results in the xy plane and z direction;

[0123] S64. Network training is performed using a server in an end-to-end manner; the final mathematical expression of the loss function is:

[0124] in, Represents the focus loss for classification; λ represents the discretized offset regression L1 loss; cls and λ reg These represent the weights of the classification and regression losses, respectively, and are both set to 1;

[0125] S65. Optimize the objective function using the server and iteratively update the network parameters using the Adam optimizer to reduce the objective loss function until it converges to a local optimum. At this point, the training is complete, and the network weights of the trained 3D video target tracking method based on multi-level target feature mutual enhancement and correlation feature pyramid are obtained.

[0126] In step S61, both the template region and the search region are regions offset by a certain degree of random perturbation based on the true value of the target box center in the corresponding frame. Both the template region and the search region in step S61 are in the target box coordinate system.

[0127] S7. For a given target in the video sequence of the 3D video to be processed, track it using the network model obtained in S6. This includes the following steps:

[0128] S71. Initialize the given target to be tracked. The 3D target bounding box of the given target to be tracked will be given in the first frame of the new video sequence; tracking starts from the second frame of the video sequence.

[0129] S72. The template region is constructed as a set of point clouds of the first frame and the previous frame. The search frame selects a search region with an offset of two meters from the center of the target box in the previous frame. Together with the template region, the encoder outputs the required self-enhancing and mutual-enhancing search region features, i.e. target sensitive features.

[0130] S73, Input the target sensitive features from step S72 into the decoder ( Figure 2 (As shown on the right), obtain the target bounding box prediction result for the current frame.

[0131] In step S72, both the template area and the search area are in the target bounding box coordinate system.

[0132] The method of this invention can enhance the representation of the appearance features of the target and the search region and the full interaction and relationship between them. At the same time, it greatly reduces the sparsity problem caused by downsampling, making the obtained encoder features more robust. In the decoder, the motion modeling that separates the vertical direction and the motion plane can ensure that the predictions of the two are decoupled. Ultimately, it can achieve accurate target tracking in complex situations where the input point cloud is very sparse.

[0133] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A three-dimensional video object tracking method based on multi-level mutual enhancement and correlation pyramid, characterized in that, The method comprises the following steps: S1, establishing an encoder-decoder paradigm for three-dimensional video single target tracking; S2, establishing a local self-attention enhancement unit of homologous multi-level target features; S3, establishing a local mutual attention enhancement unit of heterologous multi-scale template features and search area features; S4, establishing an encoder with a pyramid structure of lateral full-stage related feature aggregation; The backbone network of the encoder with the pyramid structure of lateral full-stage related feature aggregation in step S4 comprises three stages; each stage comprises a point cloud set abstraction unit, a local self-attention enhancement unit and a local mutual attention enhancement unit; In each stage, the point cloud set abstraction unit reduces the number of points in the input point cloud by a preset ratio and outputs; The local self-attention unit calculates feature correlation for each point and a number of homologous points closest to the point in terms of Euclidean distance, calculates self-attention by using the feature correlation, and enhances the input feature of each point to obtain an enhanced template feature Search region features Wherein s and t respectively represent a search region and a template region, and i represents an i-th point. The local mutual attention unit calculates the Euclidean distance for each point and the foreign point, and selects a number of nearest neighbors of the foreign point with the nearest Euclidean distance to perform mutual attention calculation, and uses mutual attention to enhance the self feature of each point to obtain an enhanced template feature Search area feature S5, establishing a dual-branch decoder for motion mode separation; S6, constructing a single target tracker under the paradigm of S1, comprising an encoder with a pyramid structure composed of a point cloud set abstraction unit, a local self-attention enhancement unit and a local mutual attention enhancement unit connected in series, and a dual-branch decoder; after constructing the network model of the single target tracker, the server is used to train the network model, optimize the network parameters by reducing the network loss function, and the network converges until the network converges; S7, for a given target in the video sequence of the three-dimensional video to be processed, the network model is used for tracking.

2. The method of claim 1, wherein the method is based on a multi-level inter-enhancement and related pyramid for 3D video object tracking. Step S1 specifically comprises: S11、establish an encoder that integrates a feature extractor and a matcher with each other, input a search region P of a single target of a given three-dimensional video and a template region P S to the encoder at the same time, perform feature extraction of each and feature interaction between the two at one time, and output target sensitive features after interaction of the search region t ​ S12, inputting target sensitive features into the decoder to obtain and output tracking results, the tracking results comprising a predicted corresponding single target center positioning map, coordinate regression value and rotation angle regression value.

3. The method of claim 2, wherein the method further comprises: The implementation process of the local self-attention enhancement unit in step S2 is as follows: S21, for the input single point cloud feature X i and the corresponding three-dimensional coordinate position P i , a point-level linear layer f1 is used to perform point-by-point feature transformation to obtain feature S22, on the feature Perform a set of point-level linear transformation f q Get query feature q i , according to three-dimensional coordinate position P i Euclidean distance calculation with all homologous points, select the first k neighbors, and then the corresponding features are subjected to two sets of linear transformation f k ,f V Get key feature k i and value feature V i ; S23, calculating homologous local self-attention features, where e ij = f e (P i - P j ) is a position encoding, f e is a point-wise linear transformation, j represents the order of the neighbors, k is the number of neighbors, f a is a two-layer point-wise linear transformation; S24, compute the output template feature by using the homologous local self-attention feature and search region feature respectively: wherein f2 is a point-wise linear transformation, respectively represent the single point cloud feature input in step S21 as: search region feature, template region feature, respectively are the search region homologous local self-attention feature and the template region homologous local self-attention feature obtained by step S23; and σ is a softmax function.

4. The method of claim 1, wherein the method further comprises: The implementation process of the local mutual attention enhancement unit in step S3 is as follows: S31, for the input single point cloud feature X i and the corresponding three-dimensional coordinate position P i , a point-level linear layer f'1 is used to perform point-by-point feature transformation to obtain the feature S32, the feature obtained in S31 perform a set of point-level linear transformation f' q obtain query feature q' i , according to position P i Calculate the Euclidean distance with all foreign points, select the top k neighbors, and then pass the corresponding features through two sets of linear transformations f' k f' V respectively to obtain key feature k' i and value feature V' i ; S33, calculating heterologous local self-attention features, where e' ij = f' e (P i - P j ) is a position encoding, f' e is a point-wise linear transformation, j represents the order of the neighbors, k is the number of neighbors, and f' a is a two-layer point-wise linear transformation; S34, the local mutual attention enhancement unit utilizes the heterogeneous local self-attention feature to calculate the output template feature and search region features respectively: where f ' 2 is a point-wise linear transformation, respectively represent the single point cloud feature input in step S31 as: search region feature, template region feature, respectively are the search region heterogeneous local self-attention feature and the template region heterogeneous local self-attention feature obtained by step S33; and σ is a softmax function.

5. The method of claim 3, wherein the method further comprises: The implementation process of the encoder with the pyramid structure of lateral full-stage related feature aggregation in step S4 is as follows: S41, the heterologous related features output by the local mutual attention enhancement unit of the three stages of the encoder are respectively unified to a fixed value by a group of side one-dimensional convolution modules, the one-dimensional convolution module comprising one layer of one-dimensional convolution, one layer of batch normalization, one layer of ReLU activation function and one layer of one-dimensional convolution; S42, the features output in step S41 are spliced in the point cloud number dimension to form a lateral full-stage related feature aggregation pyramid structure; S43, voxelizing the features output by the S42 step, averaging the features falling into the same voxel, and finally obtaining a regular target sensitive feature F with a size of CxHxLxW m where C, H, L, and W are the feature dimension, height, length, and width, respectively.

6. The method of claim 5, wherein the method further comprises: Step S5 specifically comprises: S51, Apply the target sensitive features F obtained in step S43 m A set of motion decomposition convolution modules is fed in, the motion decomposition convolution modules include two parts: two-dimensional convolution that operates in the motion xy plane and one-dimensional convolution that operates in the vertical z direction; S52, after the continuous three groups of motion decomposition convolution modules, the obtained features are respectively maximum-pooled in the vertical z direction and the x-y plane to obtain the features F of the bird's eye view BEV and the vertical direction features F z ; S53, the features F of the bird's eye view of S52 BEV , the features F in the vertical direction z are fed into the respective local feature encoding networks, respectively, wherein, Features F of the bird's eye view BEV The local feature encoding network consists of a set of two-dimensional convolutional modules for predicting the target's displacement in the x-y plane, discretized offset, and rotation angle; Features F in the vertical direction z The local feature encoding network consists of a set of one-dimensional convolutional modules for predicting the target's displacement in the vertical direction and the corresponding discretized offset.

7. The method of claim 6, wherein the method further comprises: Step S6 specifically comprises: S61, using the server to generate a template-search area by adding noise in three coordinate axis directions uniformly sampled according to the labeled target area, wherein the search area frame is determined, the point cloud of the template area is selected from the previous frame of the search area frame, and a preset number of point clouds are taken from each of the two; S62, using the server to execute the encoder, i.e. to perform step S4 to extract the features of the template region and the search region, and to perform feature interaction between the two and through the lateral pyramid structure, and finally to output the search region features that have been subjected to multi-level self-enhancement and mutual enhancement, i.e. the target sensitive features F m ; S63, using the server to execute the dual-branch decoder of step S5 to output the positioning results in the x-y plane and the z direction; S64, using the server to train the network model in an end-to-end manner; the mathematical expression of the final loss function is: wherein, represents a focal loss for classification; represents a discretized offset regression L1 loss, respectively represent center point classification loss in vertical direction based on the bird's eye view feature map; respectively represent bounding box offset regression loss in vertical direction based on the bird's eye view feature map; λ cls and λ reg respectively represent weights for classification and regression loss, both set to 1. S65, using the server to optimize the objective function until the network converges to obtain the locally optimal network parameters.

8. The method of claim 7, wherein the method further comprises: The step S7 specifically comprises: S71, initializing a given target, and giving a three-dimensional target frame of the given target to be tracked in a first frame of a video sequence of the three-dimensional video to be processed; and tracking from a second frame of the video sequence; S72, constructing a template region as a point cloud set of the first frame and a previous frame, selecting a search region with a preset distance offset from the center of the target frame of the previous frame, and outputting the search region features required for self-enhancement and mutual enhancement of the target sensitive features through an encoder together with the template region; S73, inputting the target sensitive features obtained through the step S72 into a double-branch decoder to obtain a current frame target frame prediction result.

9. The method of claim 1, wherein the method further comprises: The template region and the search region of the step S72 are both in a target frame coordinate system.

Images