Three-dimensional model target tracking method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a three-dimensional model target tracking method, which comprises the following steps: acquiring current frame three-dimensional model data, previous frame three-dimensional model data and a previous frame three-dimensional target frame, and carrying out data preprocessing to obtain a preprocessing result, the preprocessing result comprising a space-time three-dimensional model and confidence; the preprocessing result is processed through a Transform network, local features are generated, and the local features comprise the features of the previous frame and the features of the current frame; inputting the space-time three-dimensional model into a PointNet network model to generate a global feature; fusing the local features and the global features to obtain fused features; inputting the fusion features into an MLP layer for processing, and generating a target three-dimensional model; and on the basis of the target three-dimensional model, predicting the relative position offset from the previous frame to the current frame through motion prediction, and positioning the target three-dimensional model on the basis of the relative position offset to obtain a three-dimensional target frame of the current frame. And a self-attention mechanism of the Transform network is utilized, so that the method is more adaptive to the three-dimensional model. Global information and motion information are integrated, so that the accuracy of target identification and the stability of tracking are improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the fields of computer technology and related technologies, and in particular, to a method, apparatus, device, and storage medium suitable for three-dimensional model target tracking. Background Art

[0002] Three-dimensional model target tracking is an important research direction in the field of computer vision and machine learning. It involves continuously tracking the target in the three-dimensional model data obtained by sensors such as lidar to obtain its complete motion trajectory.

[0003] 3D model target detection is to obtain the 3D model data of an object through devices such as lidar, and use computer vision and machine learning technology to analyze and process the 3D model data to achieve target recognition and positioning.

[0004] Three-dimensional model target detection has broad application prospects in the fields of autonomous driving, robot navigation, intelligent monitoring, etc. It can improve the perception ability and decision-making level of equipment and enhance the intelligence level of equipment.

[0005] The BAT network integrates the geometric information of the target features and the 3D target box (BoxCloud) in the search area. It has shown improved tracking performance on specific datasets (such as KITTI). However, this method may fail in complex or sparse 3D model scenes, especially ignoring the correlation between 3D target boxes between consecutive frames, which can easily lead to mistracking. Its feature extraction part relies on PointNet++, which is widely used but not designed for tracking and lacks effective feature trade-off capabilities.

[0006] Therefore, there is an urgent need for a 3D model target tracking method to solve the above problems. Summary of the invention

[0007] The embodiments described herein provide a method, apparatus, device and storage medium for tracking a three-dimensional model target, which solve the problems existing in the prior art.

[0008] According to a first aspect of the present disclosure, a method for tracking a three-dimensional model target is provided, comprising:

[0009] Acquire the current frame 3D model data, the previous frame 3D model data and the previous frame 3D target frame and perform data preprocessing to obtain a preprocessing result, wherein the preprocessing result includes: a spatiotemporal 3D model and a confidence level;

[0010] Processing the preprocessing result by using a Transformer network to generate local features, wherein the local features include: previous frame features and current frame features;

[0011] Inputting the spatiotemporal three-dimensional model into the PointNet network model to generate global features;

[0012] Fusing the local feature with the global feature to obtain a fused feature;

[0013] Inputting the fused features into the MLP layer for processing to generate a target three-dimensional model;

[0014] Based on the target three-dimensional model, a relative position offset from a previous frame to a current frame is predicted through motion prediction, and the target three-dimensional model is positioned based on the relative position offset to obtain a three-dimensional target frame of the current frame.

[0015] In some embodiments of the present disclosure, the step of acquiring the current frame 3D model data, the previous frame 3D model data and the previous frame 3D target frame and performing preprocessing to obtain the preprocessing result includes:

[0016] splicing the current frame 3D model data with the previous frame 3D model data, and adding time information to construct a spatiotemporal 3D model;

[0017] Creating a priori target confidence map to represent the position of the target in the spatiotemporal three-dimensional model, so as to judge the target according to the three-dimensional target frame of the previous frame;

[0018] For a point in the three-dimensional model of the previous frame, its prior confidence is set according to its position relative to the three-dimensional target frame of the previous frame. If it is a point of the previous frame and is in the three-dimensional target frame, the prior confidence is 1; if it is not in the three-dimensional target frame, the prior confidence is 0.

[0019] In some embodiments of the present disclosure, the step of acquiring the current frame 3D model data, the previous frame 3D model data and the previous frame 3D target frame and performing preprocessing to obtain the preprocessing result includes:

[0020] The current frame 3D model data P T The previous frame 3D model data P T-1 Spliced ​​together and adding time information, a spatiotemporal three-dimensional model is constructed, which is expressed as the formula:

[0021]

[0022] Where T is time, N is dimension, j is a positive integer, and x, y, and z are coordinates;

[0023] Create a prior target confidence map to represent the position of the target in the spatiotemporal 3D model based on the 3D target box B of the previous frame T-1 To determine the target;

[0024] For a point in the 3D model of the previous frame, according to its relative position to the 3D target frame B of the previous frame, T-1 The location of the prior confidence is set To indicate that the target is in P T-1，T The position in which q T-1，T ∈Q T-1，T , the prior confidence calculation formula is:

[0025]

[0026] Wherein, j is a positive integer, T is time;

[0027] If it is a point of the previous frame and is in the three-dimensional target frame of the previous frame, the prior confidence is 1; if it is not in the three-dimensional target frame of the previous frame, the prior confidence is 0.

[0028] In some embodiments of the present disclosure, the step of using a Transformer network to process the preprocessing result to generate local features, wherein the local features include: features of a previous frame and features of a current frame, includes:

[0029] receiving the preprocessing result;

[0030] Using a 3D model encoder to perform 3D model encoding on the spatiotemporal 3D model data to obtain 3D model encoding features;

[0031] Based on the spatiotemporal three-dimensional model data, extracting the position code in the three-dimensional model coordinates using a K-nearest neighbor algorithm;

[0032] Projecting the 3D model encoding features into the feature space, evaluating the importance of each point feature by using the confidence and calculating the attention weight, and obtaining the 3D model encoding attention weight information;

[0033] Based on the three-dimensional model encoding features, the position encoding and the three-dimensional model encoding attention weight information, a residual connection method is adopted to obtain the current frame features and the previous frame features.

[0034] In some embodiments of the present disclosure, the step of using a Transformer network to process the preprocessing result to generate local features, wherein the local features include: features of a previous frame and features of a current frame, includes:

[0035] receiving the preprocessing result;

[0036] The spatiotemporal 3D model data is encoded using a 3D model encoder to obtain a 3D model encoding feature. The 3D model encoding feature calculation formula is:

[0037] F i-1 =Encoder(P T-1 );

[0038] F i =Encoder(P T );

[0039] Where, T is time, i is a positive integer;

[0040] Based on the spatiotemporal three-dimensional model data, the position code is extracted in the three-dimensional model coordinates using the K nearest neighbor algorithm. The position code calculation formula is:

[0041] P k1 =K(P T ,k1);

[0042] PE1=α(P k1 -P T );

[0043] P k2 =K(P T-1 ,k2);

[0044] PE2=α(P k2 -P T-1 );

[0045] Among them, k1, k2 are the number of neighboring points, P k1 , P k2 is the set of neighboring points;

[0046] The coded features of the three-dimensional model are projected into the feature space, and the importance of each point feature is evaluated by the confidence and the calculated attention weight to obtain the first attention weight information. The calculation formula of the first attention weight information is:

[0047] Q=β(F i ),K=γ(F i ,P k1 ),V=η(F i ,P k1 );

[0048]

[0049] Q=β(F i-1 ),K=γ(F i-1 ,P k2 ),V=η(F i-1 ,P k2 );

[0050]

[0051] Where i is a positive integer, Q, K, V are query, key and value respectively, β, γ, η, δ are mapping functions, d is the length of the feature vector, attn1 is the first attention weight, P k1 , P k2 is a set of neighboring points, PE1 is the position code of the 3D model of the T-th frame, and PE2 is the position code of the 3D model of the T-1th frame;

[0052] Based on the three-dimensional model encoding feature, the position encoding and the three-dimensional model encoding attention weight information, the current frame feature and the previous frame feature are obtained by using the residual connection method. The calculation formula of the current frame feature and the previous frame feature is:

[0053] F T =ε(attn1·(V+PE1))+F i ;

[0054] F T-1 =ε(attn2·(V+PE2))+F i-1 ;

[0055] Among them, i is a positive integer, ε is the MLP layer, attn2 is the first attention weight, V is the value, PE1 is the position code of the 3D model of the Tth frame, and PE2 is the position code of the 3D model of the T-1th frame.

[0056] In some embodiments of the present disclosure, the global-local feature fusion includes:

[0057] According to the previous frame features and the current frame features, a feature attention weight is calculated by using a convolution kernel and a feature product method;

[0058] Then, the local features are processed using the feature attention weights to obtain processed local features;

[0059] The global feature is fused with the processed local feature to obtain a fused feature of the current frame and a fused feature of the previous frame;

[0060] The fusion features of the current frame and the fusion features of the previous frame are concatenated and processed by the MLP layer to obtain the final fusion features.

[0061] In some embodiments of the present disclosure, the previous frame feature F T Split into F T ∈R 128×8×8, and then use a 2×2 convolution kernel (θ(·), ψ(·), v(·)) to map it to three different feature spaces, and then calculate the product between these features to obtain the second attention weight information. This process is represented by f(·). The calculation formula of the second attention weight information is:

[0062] w T =f(θ(F T ),ψ(F T ))v(F T );

[0063] Where, T is time;

[0064] The current frame F T-1 Split into F T-1 ∈R 128×8×8 , and then use a 2×2 convolution kernel (θ(·), ψ(·), v(·)) to map it to three different feature spaces, and then calculate the product between these features to obtain the second attention weight information. This process is represented by f(·). The calculation formula of the second attention weight information is:

[0065] w T-1 =f(θ(F T-1 ),ψ(F T-1 ))v(F T-1 );

[0066] Where, T is time;

[0067] The features converted by v(·) are weighted using the feature attention weights, and F T-1 Divide into 4 groups, perform the above operations on each group of features, and then splice them together to obtain the processed local features F' T-1 ;

[0068] The global feature and the processed local feature F' T-1 Fusion is performed to obtain the fusion feature F of the previous frame (T-1)f =R 128×64 ;

[0069] The features converted by v(·) are weighted using the feature attention weights, and F T Divide into 4 groups, perform the above operations on each group of features, and then splice them together to obtain the processed local features F' T ;

[0070] The global feature and the processed local feature F' T Fusion is performed to obtain the fusion feature F of the current frame Tf =R128×64 ;

[0071] The fusion feature F of the current frame Tf The fusion feature F of the previous frame (T-1)f Processing is performed to obtain the final fusion feature, and the calculation formula of the final fusion feature is as follows:

[0072]

[0073] In some embodiments of the present disclosure, the step of inputting the fused features into the MLP layer for processing to generate a target three-dimensional model includes:

[0074] The fusion features are learned through the MLP layer to generate a classification result, wherein the classification result includes a target three-dimensional model and background points, and the target three-dimensional model includes: the current frame target three-dimensional model, the previous frame target three-dimensional model and the previous frame three-dimensional target frame.

[0075] In some embodiments of the present disclosure, the step of predicting the relative position offset from the previous frame to the current frame through motion prediction based on the target three-dimensional model, positioning the target three-dimensional model based on the relative position offset, and obtaining the three-dimensional target frame of the current frame includes:

[0076] Obtaining a relative displacement offset by calculating the final fusion feature, the current 3D model data, the previous frame 3D model data, and the previous frame 3D target frame through a multi-dimensional relative motion function;

[0077] The relative displacement offset and the previous frame three-dimensional target frame are transformed using a Transform function to obtain the current frame three-dimensional target frame.

[0078] In some embodiments of the present disclosure, the step of predicting the relative position offset from the previous frame to the current frame through motion prediction based on the target three-dimensional model, positioning the target three-dimensional model based on the relative position offset, and obtaining the three-dimensional target frame of the current frame includes:

[0079] A 4-dimensional relative motion is defined between the current frame 3D target frame and the previous frame 3D target frame. T-1,T ∈R 4 , which includes the translation offset (Δx, Δy, Δz) and the yaw offset Δω. The relative motion process is defined as function O, and the calculation formula of function O is:

[0080]

[0081] O(P T , P T-1 ,BT-1 )→(Δx, Δy, Δx, Δω);

[0082] After calculation, the output is the relative motion change of the three-dimensional target frame of the previous frame on the three axes and angles. T-1,T , according to the relative motion change O T-1,T The position of the 3D target frame of the previous frame is converted by using the Transform function to obtain the position of the 3D target frame of the current frame. The calculation formula of the 3D target frame of the current frame is as follows:

[0083] B T =Transform(B T-1 , O T-1,T ).

[0084] In some embodiments of the present disclosure, calculation is performed based on the previous frame 3D model data and the previous frame 3D target frame to obtain the previous frame box information;

[0085] Calculate according to the current frame 3D model data and the current frame 3D target frame to obtain current frame box information;

[0086] The previous frame box information and the current frame box information are input into the BoxCloud comparator for similarity calculation, and the multiple points with the highest similarity are taken out, and the features of the previous frame 3D model are fused to enhance the features of the current frame 3D model, and the final current frame 3D target box is generated from the enhanced features.

[0087] According to a second aspect of the present disclosure, a device for tracking a three-dimensional model target is provided, comprising:

[0088] A preprocessing module is used to obtain the current frame 3D model data, the previous frame 3D model data and the previous frame 3D target frame and perform preprocessing to obtain a preprocessing result, wherein the preprocessing result includes: a spatiotemporal 3D model and a confidence level;

[0089] A feature extraction module, used to process the preprocessing result using a Transformer network to generate local features, wherein the local features include: previous frame features and current frame features;

[0090] A generation module, used for inputting the spatiotemporal three-dimensional model into a PointNet network model to generate global features;

[0091] A fusion module, used for fusing the local features with the global features to obtain fused features;

[0092] A processing module, used for inputting the fused features into the MLP layer for processing to generate a target three-dimensional model;

[0093] The motion prediction module is used to predict the relative position offset from the previous frame to the current frame based on the target three-dimensional model through motion prediction, and locate the target three-dimensional model based on the relative position offset to obtain the three-dimensional target frame of the current frame.

[0094] In some embodiments of the present disclosure, it further includes:

[0095] A first calculation module, used for performing calculation according to the previous frame 3D model data and the previous frame 3D target frame to obtain previous frame box information;

[0096] A second calculation module, used for performing calculation according to the current frame 3D model data and the current frame 3D target frame to obtain current frame box information;

[0097] The third calculation module is used to input the previous frame box information and the current frame box information into the BoxCloud comparator for similarity calculation, and extract multiple points with the highest similarity, fuse the features of the previous frame three-dimensional model to enhance the features of the current frame three-dimensional model, and generate the final three-dimensional target box from the enhanced features.

[0098] In some embodiments of the present disclosure, the preprocessing module 310 is specifically used to:

[0099] splicing the current frame 3D model data with the previous frame 3D model data, and adding time information to construct a spatiotemporal 3D model;

[0100] Creating a priori target confidence map to represent the position of the target in the spatiotemporal three-dimensional model, so as to judge the target according to the three-dimensional target frame of the previous frame;

[0101] For a point in the three-dimensional model of the previous frame, its prior confidence is set according to its position relative to the three-dimensional target frame of the previous frame. If it is a point of the previous frame and is in the three-dimensional target frame, the prior confidence is 1; if it is not in the three-dimensional target frame, the prior confidence is 0.

[0102] In some embodiments of the present disclosure, the feature extraction module is specifically used to:

[0103] receiving the preprocessing result;

[0104] Using a 3D model encoder to perform 3D model encoding on the spatiotemporal 3D model data to obtain 3D model encoding features;

[0105] Based on the spatiotemporal three-dimensional model data, extracting the position code in the three-dimensional model coordinates using a K-nearest neighbor algorithm;

[0106] Projecting the 3D model encoding features into the feature space, evaluating the importance of each point feature by using the confidence and calculating the attention weight, and obtaining the 3D model encoding attention weight information;

[0107] Based on the three-dimensional model encoding features, the position encoding and the three-dimensional model encoding attention weight information, a residual connection method is adopted to obtain the current frame features and the previous frame features.

[0108] In some embodiments of the present disclosure, the fusion module is specifically used to:

[0109] According to the previous frame features and the current frame features, a feature attention weight is calculated by using a convolution kernel and a feature product method;

[0110] Then, the local features are processed using the feature attention weights to obtain processed local features;

[0111] The global feature is fused with the processed local feature to obtain a fused feature of the current frame and a fused feature of the previous frame;

[0112] The fusion features of the current frame and the fusion features of the previous frame are concatenated and processed by the MLP layer to obtain the final fusion features.

[0113] In some embodiments of the present disclosure, the processing module is specifically used to:

[0114] The fusion features are learned through the MLP layer to generate a classification result, wherein the classification result includes a target three-dimensional model and background points, and the target three-dimensional model includes: the current frame target three-dimensional model, the previous frame target three-dimensional model and the previous frame three-dimensional target frame.

[0115] In some embodiments of the present disclosure, the motion prediction module is specifically used to:

[0116] Obtaining a relative displacement offset by calculating the final fusion feature, the current 3D model data, the previous frame 3D model data, and the previous frame 3D target frame through a multi-dimensional relative motion function;

[0117] The relative displacement offset and the previous frame three-dimensional target frame are transformed using a Transform function to obtain the current frame three-dimensional target frame.

[0118] According to a third aspect of the present disclosure, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the method in any one of the above embodiments are implemented.

[0119] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in any one of the above embodiments are implemented.

[0120] The method, device, equipment and storage medium for tracking a three-dimensional model target provided by the embodiments of the present disclosure obtain the three-dimensional model data of the current frame, the three-dimensional model data of the previous frame and the three-dimensional target frame of the previous frame and perform data preprocessing to obtain the preprocessing result, which includes: a spatiotemporal three-dimensional model and a confidence level; the preprocessing result is processed by a Transformer network to generate local features, which include: features of the previous frame and features of the current frame; the spatiotemporal three-dimensional model is input into a PointNet network model to generate global features; the local features and the global features are fused to obtain fused features; the fused features are input into an MLP layer for processing to generate a target three-dimensional model; based on the target three-dimensional model, the relative position offset from the previous frame to the current frame is predicted through motion prediction, and the target three-dimensional model is positioned based on the relative position offset to obtain a three-dimensional target frame of the current frame. The self-attention mechanism of the Transformer network is used to better adapt the three-dimensional model. Global information and motion information are effectively integrated to improve the accuracy of target recognition and the stability of tracking. By utilizing the self-attention mechanism of the Transformer network, the model can automatically capture and integrate global information, improve the recognition ability of targets in complex scenes, and be more suitable for three-dimensional models. Effectively integrating global information and motion information enables the model to comprehensively consider the appearance and motion state of the target during the tracking process, reduce dependence on a single appearance feature, and improve the stability and robustness of tracking. When similar objects appear in the scene, the model can more accurately identify the target and reduce mistracking due to the simultaneous consideration of global information and motion information. Even in scenes with sparse three-dimensional models, the model can effectively track the target and is not easily lost due to the assistance of motion information.

[0121] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0122] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly described below. It should be noted that the drawings described below only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure, wherein:

[0123] Figure 1 It is a flowchart of a method for tracking a three-dimensional model target provided by an embodiment of the present disclosure;

[0124] Figure 2 It is a schematic diagram of the overall process of a method for tracking a three-dimensional model target provided by an embodiment of the present disclosure;

[0125] Figure 3 It is a structural schematic diagram of a three-dimensional model target tracking device provided by an embodiment of the present disclosure;

[0126] Figure 4 It is a structural diagram of a computer device provided in an embodiment of the present disclosure.

[0127] In the drawings, reference numerals with the same last two digits correspond to the same elements. It should be noted that the elements in the drawings are schematic and not drawn to scale. DETAILED DESCRIPTION

[0128] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work also fall within the scope of protection of the present disclosure.

[0129] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase "embodiments" in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0130] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists, A and B exist at the same time, and B exists. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0131] Furthermore, in all embodiments of the present disclosure, terms such as “first” and “second” are used only to distinguish one component (or a part of a component) from another component (or another part of a component).

[0132] In the description of the present application, unless otherwise specified, "plurality" means more than two (including two), and similarly, "multiple groups" means more than two groups (including two).

[0133] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0134] The method for tracking a three-dimensional model target provided by an embodiment of the present disclosure is applied to a terminal device.

[0135] Based on the problems existing in the existing technology, Figure 1 is a flow chart of a method for tracking a three-dimensional model target provided by an embodiment of the present disclosure, such as Figure 1 As shown, the specific process of the method for 3D model target tracking includes:

[0136] Step S110, obtaining the current frame 3D model data, the previous frame 3D model data and the previous frame 3D target frame and performing preprocessing to obtain a preprocessing result, wherein the preprocessing result includes: a spatiotemporal 3D model and a confidence level;

[0137] The purpose of the preprocessing is to enhance the subsequent Transformer network's ability to recognize and parse time series data, accurately identify the identity information of the target to be tracked, and help it understand the role and importance of each frame in the time series, thereby effectively avoiding the overfitting problem that may occur in the network during training.

[0138] In the specific implementation process, the data of the 3D model of two consecutive frames are obtained, namely the 3D model data of the current frame (T frame) and the 3D model data of the previous frame (T-1 frame), as well as the 3D target frame of the previous frame (T-1 frame).

[0139] Among them, the three-dimensional model is a data set composed of a large number of three-dimensional points, which is commonly used in fields such as three-dimensional scanning and lidar.

[0140] The target box, also known as the bounding box (BBox), is a rectangular box used to describe the position and range of the target in the image. It is defined by the coordinates of the upper left and lower right corners of the rectangular box and can be used to mark and locate the target object. In the target detection task, the model achieves target detection and positioning by predicting the target box of the target object. The purpose of the target box is to define the position and size of the object in 2D or 3D space, and to facilitate the operation and analysis of the object. This application takes a three-dimensional target box as an example for explanation.

[0141] In a possible implementation, the preprocessing in step S110 includes:

[0142] Step S1101: pre-process the data using target segmentation based on spatiotemporal features to obtain relative displacement information of the target point.

[0143] S1102, splicing the current frame 3D model data with the previous frame 3D model data, and adding time information to construct a spatiotemporal 3D model;

[0144] S1103, creating a priori target confidence map to represent the position of the target in the spatiotemporal three-dimensional model, so as to judge the target according to the three-dimensional target frame of the previous frame;

[0145] S1104. For a point in the three-dimensional model of the previous frame, set its prior confidence according to its position relative to the three-dimensional target frame of the previous frame. If it is a point of the previous frame and is in the three-dimensional target frame, the prior confidence is 1; if it is not in the three-dimensional target frame, the prior confidence is 0.

[0146] In a possible implementation, the preprocessing is illustrated by way of example, including:

[0147] Two consecutive frames P T and P T-1 And the three-dimensional target frame B T-1 As input, the 3D model data of the T-1th frame is P T-1 , the 3D model data of the Tth frame is P T The two frames of 3D model data can be represented as two sets, each set contains multiple points, and each point is usually composed of 3D space coordinates (x, y, z). First, the 3D models of the previous and next frames are spliced ​​together, and then the time information is added to construct a spatiotemporal 3D model. The spatiotemporal 3D model is expressed as formula (1):

[0148]

[0149] Where T is time, N is dimension, j is a positive integer, and x, y, and z are coordinates;

[0150] Since 3D models are discrete 3D data, it is difficult for Transformer to automatically identify their temporal relationships. Integrating temporal information in the preprocessing stage and clearly identifying the 3D models of the previous and next frames will help Transformer understand and capture the temporal dependencies between 3D models and improve processing performance.

[0151] Since there are multiple objects in the scene, we must T-1 To judge the target. Therefore, first create a priori target confidence map To indicate that the target is in P T-1，T The position in which q T-1，T ∈Q T-1，T Defined as formula (2):

[0152]

[0153] Wherein, j is a positive integer, T is time;

[0154] Q j Considered as p j As the prior confidence of the target point. T-1 The point in the T-1 The confidence of a point is set according to its position, that is, if it is a point in the previous frame and is in the 3D target frame of the previous frame, the prior confidence is 1; if it is not in the 3D target frame of the previous frame, the prior confidence is 0.

[0155] Confidence is used to intelligently identify and weigh the reliability of data when assisting the Transformer network in processing 3D model data, and can more accurately determine which 3D model data are reliable and important, and which are less reliable.

[0156] S120, using the Transformer network to process the preprocessing result to generate local features, the local features include: the previous frame feature F T-1 and the current frame features and the current frame F T ;

[0157] In one possible implementation, a Transformer network is used to process 3D model data, and an appropriate attention mechanism is combined to capture key information in the 3D model data.

[0158] As a specific implementation method, step S120 specifically includes the following steps:

[0159] Step S1201, receiving the preprocessing result;

[0160] Step S1202: Use a 3D model encoder to perform 3D model encoding on the spatiotemporal 3D model data to obtain 3D model encoding features;

[0161] Step S1203: extracting position codes in the three-dimensional model coordinates using a K-nearest neighbor algorithm based on the spatiotemporal three-dimensional model data;

[0162] Step S1204: projecting the coded features of the three-dimensional model into the feature space, evaluating the importance of each point feature by using the confidence and calculating the attention weight, and obtaining first attention weight information;

[0163] Step S1205: Based on the three-dimensional model encoding features, the position encoding and the first attention weight information, the current frame features and the previous frame features are obtained using a residual connection method.

[0164] As a specific implementation method, step S120 is taken as an example to illustrate, which specifically includes the following:

[0165] 1. Input processing

[0166] In a possible implementation, the preprocessing result is input; the three-dimensional model data of the T-1th frame is P T-1 , the 3D model data of the Tth frame is P T The two frames of 3D model data can be represented as two sets, each set contains multiple points, and each point is usually composed of 3D space coordinates (x, y, z).

[0167] 2. 3D model encoding

[0168] In a possible implementation, a 3D model encoder is used to perform feature encoding on the spatiotemporal 3D model data. The encoder maps the input spatiotemporal 3D model data to a high-dimensional feature space and generates a feature vector for each point. Then the 3D model encoding features of the two frames of the 3D model are respectively expressed as formulas (2) and (3):

[0169] F i-1 =Encoder(P T-1 ) (2)

[0170] F i =Encoder(P T ) (3)

[0171] Where i is a positive integer, F i-1 represents the point cloud encoding features of the T-1 frame point cloud, F i Represents the 3D model encoding features of the 3D model of the Tth frame.

[0172] 3. Positional encoding

[0173] Specifically, position encoding is to add position information to each point in the 3D model to make up for the lack of spatial position perception in the Transformer model;

[0174] In a possible implementation, in view of the spatiotemporal three-dimensional model data, a K-nearest neighbor (KNN) algorithm is used to directly extract the position code in the three-dimensional model coordinates;

[0175] In a possible implementation, the specific representation of the position code PE1 of the 3D model of the Tth frame is shown in formula (4) and formula (5):

[0176] P k1 =K(P T ,k1) (4)

[0177] PE1=α(P k1 -P T ) (5)

[0178] Where K(·) represents the KNN algorithm. Given an input point set P T , the KNN algorithm will find its neighboring points and represent them as P k1 , where k1 represents the number of neighboring points. In addition, α(·) represents the MLP layer, i.e., the multi-layer perceptron layer, which is used for nonlinear transformation and feature extraction of deep neural networks, P k1 is the set of neighboring points.

[0179] In a possible implementation, the specific representation of the position code PE2 of the T-1th frame 3D model is as shown in formula (6) and formula (7):

[0180] P k2 =K(P T-1 ,k2) (6)

[0181] PE2=α(P k2 -P T-1 ) (7)

[0182] Where K(·) represents the KNN algorithm. Given an input point set P T-1 , the KNN algorithm will find its neighboring points and represent them as P k2 , where k2 represents the number of neighboring points. In addition, α(·) represents the MLP layer, i.e., the multi-layer perceptron layer, which is used for nonlinear transformation and feature extraction of deep neural networks, P k2 is the set of neighboring points.

[0183] 4. Self-Attention Mechanism

[0184] This step is to encode the 3D model to obtain the 3D model encoding feature F i-1 and F i As input, through the multi-layer attention mechanism, the model can learn the complex relationship between points in the 3D model and automatically focus on the areas that are critical to target recognition. This mechanism not only captures local features, but also effectively integrates global information, making targets easier to identify in complex scenes.

[0185] In a possible implementation, the three-dimensional model encoding feature F i Project it into the feature space, and then evaluate the importance of each point feature by calculating the attention weight to obtain the first attention weight information, as shown in formulas (8) and (9).

[0186] Q=β(F i ),K=γ(F i ,Pk1 ),V=η(F i ,P k1 ) (8)

[0187]

[0188] Where i is a positive integer, Q, K, and V represent query, key, and value, respectively. β, γ, η, and δ represent mapping functions, which effectively transform input data into feature space. d is the feature dimension, which represents the length of the feature vector. attn1 represents the first attention weight. P k1 is the set of neighboring points, P k1 Also included in K and V, PE1 represents the position code of the three-dimensional model of the Tth frame.

[0189] In a possible implementation, in a possible implementation, the three-dimensional model encoding feature F i-1 Project it into the feature space, and then evaluate the importance of each point feature by calculating the attention weight to obtain the three-dimensional model encoding attention weight information, as shown in formulas (10) and (11).

[0190] Q=β(F i-1 ),K=γ(F i-1 ,P k2 ),V=η(F i-1 ,P k2 ) (10)

[0191]

[0192] Where i is a positive integer, Q, K, and V represent query, key, and value, respectively; β, γ, η, and δ represent mapping functions, which effectively transform input data into feature space; d is the feature dimension, which represents the length of the feature vector; and attn2 represents the first attention weight. k2 is the set of neighboring points, P k2 Also included in K and V, PE2 represents the position code of the three-dimensional model of the T-1th frame.

[0193] 5. Output features

[0194] In a possible implementation, based on the three-dimensional model encoding feature, the position encoding and the first attention weight information, a residual connection method is used to obtain the current frame feature F T It is expressed as formula (12):

[0195] F T =ε(attn1·(V+PE1))+F i (12)

[0196] Among them, i is a positive integer, ε represents an MLP layer, attn1 represents the first attention weight, V represents value, and PE1 represents the position encoding of the 3D model of the Tth frame.

[0197] In a possible implementation, based on the three-dimensional model encoding feature, the position encoding and the first attention weight information, a residual connection method is used to obtain the previous frame feature F T-1 It is expressed as formula (13):

[0198] F T-1 =ε(attn2·(V+PE2))+F i-1 (13)

[0199] Among them, i is a positive integer, ε represents an MLP layer, attn2 represents the first attention weight, V represents value, and PE2 represents the position encoding of the three-dimensional model of the T-1th frame.

[0200] Specifically, the features extracted by Transformer are local features, that is, the features of the previous frame F T-1 and the current frame F T All are local features.

[0201] Through the self-attention mechanism of Transformer, attention features are generated by calculating attention weights, which can automatically capture local information and integrate global information, improve the recognition ability of targets in complex fields, and output three-dimensional model features encoded by Transformer, providing strong support for subsequent tracking.

[0202] Step S130: input the spatiotemporal three-dimensional model into the PointNet network model to generate global features.

[0203] The spatiotemporal 3D model data containing time and space information is used as input and sent to the PointNet network model for processing. With its powerful 3D model processing capabilities, PointNet can effectively extract representative global features from complex 3D model data. These global features not only contain the spatial distribution information of the 3D model, but also imply its dynamic characteristics that change over time, providing a solid foundation for subsequent analysis, recognition or classification tasks.

[0204] Step S140: Using the global-local feature fusion mechanism, the local feature and the global feature are fused to obtain a fused feature F f .

[0205] The core of the 3D single target tracking task is to predict the position of the target in the search area. Therefore, embedding the target information into the search area is the key to ensure accurate prediction, and effectively fusing global features and local features is an important step in correctly identifying the target. In this process, with the help of the global-local feature fusion mechanism, by focusing on the global scope and local details, the Transformer network can understand the input features more comprehensively, and then filter out key feature information.

[0206] In a possible implementation, step S140 includes:

[0207] Step S1401: Calculate a second attention weight according to the previous frame feature and the current frame feature by using a convolution kernel and feature product method;

[0208] Step S1402: Process the local feature using the second attention weight to obtain a processed local feature;

[0209] Step S1403: Fusing the global features with the processed local features to obtain fused features of the current frame and fused features of the previous frame;

[0210] Step S1404: concatenate and perform MLP layer processing on the fusion features of the current frame and the fusion features of the previous frame to obtain the final fusion features.

[0211] In a possible implementation, step S140 is illustrated by way of example, including:

[0212] 1. Global-local feature fusion

[0213] In a possible implementation, taking the T-th frame 3D model as an example, F T Split into F T ∈R 128×8×8 , and then use the 2×2 convolution kernel (θ(·), ψ(·), v(·)) to map it to three different feature spaces, and then calculate the product between these features to obtain the second attention weight information. This process is expressed as f(·), formula (14) is as follows:

[0214] w T =f(θ(F T ),ψ(F T ))v(F T ) (14)

[0215] Then, the second attention weight is used to weight the features transformed by v(·), where the correlation of cross-channel positions needs to be considered, and F is converted accordingly. TDivide into 4 groups, perform the above operations on each group of features, and then splice them together, denoted as F' T .

[0216] Then the PointNet network is used to extract the global features of the 3D model and the obtained global features are compared with F' T Splice to obtain the fusion feature F of the Tth frame Tf =R 128×64 .

[0217] In a possible implementation, taking the T-1th frame 3D model as an example, F T-1 Split into F T-1 ∈R 128×8×8 , and then use a 2×2 convolution kernel (θ(·), ψ(·), v(·)) to map it to three different feature spaces, and then calculate the product between these features to obtain the second attention weight information. This process is represented by f(·), and the formula is as follows:

[0218] w T-1 =f(θ(F T-1 ),ψ(F T-1 ))v(F T-1 );

[0219] Then, the second attention weight is used to weight the features transformed by v(·), where the correlation of cross-channel positions needs to be considered, and F is converted accordingly. T-1 Divide into 4 groups, perform the above operations on each group of features, and then splice them together, denoted as F' T-1 .

[0220] The obtained global features are then compared with F' T-1 Fusion is performed to obtain the fusion feature F of the T-1th frame (T-1)f =R 128×64 .

[0221] 2. Feature fusion of two 3D models

[0222] Since the input includes the 3D target frame of the previous frame when extracting the 3D model features of the T-1 frame, in order to determine whether the features of the Tth frame (i.e., the current frame) are reliable, it is necessary to compare and fuse them with the T-1th frame.

[0223] Take the fusion feature F of the T-1th frame (T-1)f And the fusion feature F of the Tth frame Tf As input, fusion feature F f It is expressed as formulas (16), (17) and (18):

[0224] In a possible implementation, the feature F of the T-1th frame is used. T-1and the T-th frame feature F T As input, we get the fusion feature F f It can be expressed as formulas (14), (15) and (16):

[0225]

[0226] Among them, μ, θ, ζ, ∈ constitute the MLP layer, Cat(·) represents the concatenation operation, and Re peat(·) is a repetitive operation used to ensure the consistency of feature dimensions. and It is a fusion feature.

[0227] Step S150: The fusion feature F f Input to the MLP layer for processing to generate the target 3D model;

[0228] Multilayer Perceptron (MLP) is a classic and important neural network model. It consists of different layers of multiple neurons, including an input layer, one or more hidden layers, and an output layer. Information processing and pattern recognition are achieved through forward propagation and back propagation algorithms. In 3D model processing, MLP is usually used to process each point or 3D model block in a 3D model and output new features or labels for each point or block.

[0229] Inputting the fused features into the MLP layer means that the feature vector of each point or 3D model block is used as the input of the MLP. The MLP learns the complex mapping relationship between the input features and the target output through the nonlinear activation function (such as ReLU) and weight matrix in its hidden layer.

[0230] The fused features input to the MLP layer have fully fused the three-dimensional model of the T-1th frame, the three-dimensional model of the Tth frame, and the three-dimensional target frame of the T-1th frame, and the fused features include but are not limited to: three-dimensional model features, spatiotemporal features, and three-dimensional target frame features; the fused features are learned by the MLP layer to generate a classification result, which includes a target three-dimensional model and background points, wherein the background points are the parts to be discarded, and the target three-dimensional model includes the target three-dimensional model of the T-1th frame, the target three-dimensional model of the Tth frame, and the three-dimensional target frame of the T-1th frame.

[0231] Step S160: Based on the target 3D model, predict the relative position offset from the previous frame (T-1 frame) to the current frame (T frame) through motion prediction, and locate the target 3D model based on the relative position offset to obtain the 3D target frame of the current frame.

[0232] Specifically, the target three-dimensional model is used as input to perform motion prediction, wherein the motion prediction uses the three-dimensional model data between consecutive frames to predict the motion state and relative position offset of the target.

[0233] Fusion of motion information is guided by motion information, and it is feasible to track targets in sparse 3D model scenes. The core of motion prediction is relative motion estimation, that is, predicting the relative position change from the T-1 frame to the T frame.

[0234] In a possible implementation, step S160 includes:

[0235] Step S1601: Based on the fusion feature, the current 3D model data, the previous frame 3D model data and the previous frame 3D target frame, a relative displacement offset is obtained by calculating through a multi-dimensional relative motion function.

[0236] Step S1602: The relative displacement offset and the three-dimensional target frame of the previous frame are converted into the three-dimensional target frame of the current frame using a Transform function.

[0237] In a possible implementation, step S160 is illustrated by way of example, including:

[0238] In the target tracking task, it is necessary to locate the target in each frame of the dynamic 3D scene input sequence. Specifically, assume that the 3D model at time T is represented as Among them, N T represents the number of points, 3 represents the number of channels, and the full coordinate system xyz is used. The initial state of the target is represented by its three-dimensional target frame at the first frame P1, and the three-dimensional target frame is represented by B T ∈R 7 , which includes the center coordinates (xyz coordinates), direction (heading angle ω along the upper axis) and size (width, length and height). In this setting, the task of the tracking module is to T , according to the historical frame Output the 3D target box of the target.

[0239] Tracking with the center of motion aims to use explicit motion modeling to drive the positioning target to ensure accurate tracking of the target in consecutive frames. Under this goal, the input of the lidar sequence and the 3D target box of the target in the first frame are known. The task of the tracker is to use the relative target motion between consecutive frames to predict the position of the target. Specifically, under the known conditions, consider at time T (T>1), the 3D target box B of the T-1th frame is known. T-1 , and the consecutive frames P T , P T-1Since this application only considers ground targets, only the relative motion of the target in the front, back, left, and right directions is considered, and the relative motion up and down is not considered. Specifically, a 4-dimensional relative motion is defined between the two 3D target frames of the Tth frame and the T-1th frame. T-1,T ∈R 4 , which contains the translation offset (Δx, Δy, Δz) and the yaw offset Δω. This relative motion process is defined as function O, and formulas (19) and (20) are as follows:

[0240]

[0241] O(P T ,P T-1 ,B T-1 )→(Δx, Δy, Δz, Δω) (20)

[0242] After formulas (16) and (17), the output is the relative motion change O of the three-dimensional target frame on the three axes and angles. T-1,T , using this variation and the position of the 3D target frame in the T-1th frame, the position of the 3D target frame in the Tth frame can be obtained. The Transform function is used here to put the output relative motion variation and the position of the 3D target frame in the T-1th frame together, and then superimpose the time information to obtain the position of the target in the Tth frame, that is, the 3D target frame of the current frame. Formula (21) is as follows:

[0243] B T =Transform(B T-1 , O T-1,T ) (twenty one)

[0244] By incorporating motion information, the model can comprehensively consider the appearance and motion state of the target during tracking, reduce reliance on a single appearance feature, and improve tracking stability. When similar objects appear in the scene, the model can more accurately identify the target and reduce mistracking by considering both global information and motion information. Even in scenes with sparse 3D models, the model can effectively track the target and is not easily lost due to the assistance of motion information.

[0245] In a possible implementation manner, after implementing step S160, the following may also be included:

[0246] Step S170: Calculate the previous frame box information C according to the T-1 frame 3D model data and the T-1 frame 3D target frame. T-1 .

[0247] BoxCloud is a point-to-box relationship representation. Each point can be represented by a 9-dimensional feature, where each feature is the distance from the point to the 8 vertices and 1 center of the box. This will be discussed in detail later. Then use this BoxCloud to map the features of the Template area to the Search area. Assume that the target 3D model is P and its BBox is B.

[0248] In one possible implementation, for any point p in P T-1 , calculate the distance to each vertex (8) and center (1) of the BBox in space, and then generate a 9D vector C according to the pre-set sorting method of these 9 distances T-1 , expressed by formula (22):

[0249]

[0250] Step S180: Calculate the current frame box information C according to the 3D model data of the Tth frame and the 3D target frame of the Tth frame. T .

[0251] In one possible implementation, for any point p in P T , calculate the distance to each vertex (8) and center (1) of the BBox in space, and then generate a 9D vector C according to the pre-set sorting method of these 9 distances T , expressed by formula (23):

[0252]

[0253] Step S190: The previous frame box information C T-1 and the current frame box information C T Input the BoxCloud comparator to calculate the similarity, and take out the k points with the highest similarity (k>0), and fuse the features of the 3D model of the previous frame to calculate the features of the 3D model of the current frame F. T-1 and generate the final 3D target box from the enhanced features.

[0254] In one possible implementation, C is calculated T-1 and C T The similarity between them is expressed as L2 distance, and the k (k>0) points with the highest similarity and their corresponding features are selected, and then the k selected features with the highest scores are integrated into the feature F of the 3D model of the T-1th frame T-1 To remove the 3D model feature F of the current frame T-1 The enhanced features are obtained and the final 3D target box is generated from the enhanced features.

[0255] The method for tracking a three-dimensional model target provided by the embodiment of the present disclosure obtains the three-dimensional model data of the current frame, the three-dimensional model data of the previous frame and the three-dimensional target frame of the previous frame and performs data preprocessing to obtain a preprocessing result, wherein the preprocessing result includes: a spatiotemporal three-dimensional model and a confidence level; based on the Transformer network, the preprocessing result, the three-dimensional model data of the current frame and the three-dimensional model data of the previous frame, the features of the previous frame and the features of the current frame are generated; the local features and the global features are fused to obtain fused features, wherein the local features include: the features of the previous frame and the features of the current frame; the fused features are input into the MLP layer for processing to generate a target three-dimensional model; based on the target three-dimensional model, the relative position offset from the previous frame to the current frame is predicted, and the target three-dimensional model is positioned based on the relative position offset to obtain a three-dimensional target frame. The self-attention mechanism of the Transformer network is used to better adapt the three-dimensional model. Global information and motion information are effectively integrated to improve the accuracy of target recognition and the stability of tracking. By utilizing the self-attention mechanism of the Transformer network, the model can automatically capture and integrate global information, improve the recognition ability of the target in complex fields, and better adapt the three-dimensional model. The effective integration of global information and motion information enables the model to comprehensively consider the appearance and motion state of the target during the tracking process, reduce dependence on a single appearance feature, and improve the stability and robustness of tracking. When similar objects appear in the scene, the model can more accurately identify the target and reduce mistracking due to the simultaneous consideration of global information and motion information. Even in scenes with sparse three-dimensional models, with the assistance of motion information, the model can effectively track the target and is not easily lost.

[0256] Figure 2 It is a schematic diagram of the overall process of a method for tracking a three-dimensional model target provided by an embodiment of the present disclosure.

[0257] Based on the above embodiments, the present disclosure also provides a device for tracking a three-dimensional model target. Figure 3 As shown, the device for tracking a three-dimensional model target includes:

[0258] The preprocessing module 310 is used to obtain the current frame 3D model data, the previous frame 3D model data and the previous frame 3D target frame and perform preprocessing to obtain a preprocessing result, wherein the preprocessing result includes: a spatiotemporal 3D model and a confidence level;

[0259] A feature extraction module 320 is used to process the preprocessing result using a Transformer network to generate local features, wherein the local features include: previous frame features and current frame features;

[0260] A generating module 330, configured to input the spatiotemporal 3D model into a PointNet network model to generate global features;

[0261] A fusion module 340 is used to fuse the local feature with the global feature to obtain a fused feature;

[0262] A processing module 350 is used to input the fused features into the MLP layer for processing to generate a target three-dimensional model;

[0263] The motion prediction module 360 ​​is used to predict the relative position offset from the previous frame to the current frame through motion prediction based on the target 3D model, and locate the target 3D model based on the relative position offset to obtain the 3D target frame of the current frame.

[0264] In a specific implementation, the preprocessing module 310 is specifically used to:

[0265] splicing the current frame 3D model data with the previous frame 3D model data, and adding time information to construct a spatiotemporal 3D model;

[0266] Creating a priori target confidence map to represent the position of the target in the spatiotemporal three-dimensional model, so as to judge the target according to the three-dimensional target frame of the previous frame;

[0267] For a point in the three-dimensional model of the previous frame, its prior confidence is set according to its position relative to the three-dimensional target frame of the previous frame. If it is a point of the previous frame and is in the three-dimensional target frame, the prior confidence is 1; if it is not in the three-dimensional target frame, the prior confidence is 0.

[0268] In a specific implementation, the feature extraction module 320 is specifically used to:

[0269] receiving the preprocessing result;

[0270] Using a 3D model encoder to perform 3D model encoding on the spatiotemporal 3D model data to obtain 3D model encoding features;

[0271] Based on the spatiotemporal three-dimensional model data, extracting the position code in the three-dimensional model coordinates using a K-nearest neighbor algorithm;

[0272] Projecting the 3D model encoding features into the feature space, evaluating the importance of each point feature by using the confidence and calculating the attention weight, and obtaining the 3D model encoding attention weight information;

[0273] Based on the three-dimensional model encoding features, the position encoding and the three-dimensional model encoding attention weight information, a residual connection method is adopted to obtain the current frame features and the previous frame features.

[0274] In a specific implementation, the fusion module 340 is specifically used to:

[0275] According to the previous frame features and the current frame features, a feature attention weight is calculated by using a convolution kernel and a feature product method;

[0276] Then, the local features are processed using the feature attention weights to obtain processed local features;

[0277] The global feature is fused with the processed local feature to obtain a fused feature of the current frame and a fused feature of the previous frame;

[0278] The fusion features of the current frame and the fusion features of the previous frame are concatenated and processed by the MLP layer to obtain the final fusion features.

[0279] In a specific implementation, the processing module 350 is specifically used to:

[0280] The fusion features are learned through the MLP layer to generate a classification result, wherein the classification result includes a target three-dimensional model and background points, and the target three-dimensional model includes: the current frame target three-dimensional model, the previous frame target three-dimensional model and the previous frame three-dimensional target frame.

[0281] In a specific implementation, the motion prediction module 360 ​​is specifically used to:

[0282] Obtaining a relative displacement offset by calculating the final fusion feature, the current 3D model data, the previous frame 3D model data, and the previous frame 3D target frame through a multi-dimensional relative motion function;

[0283] The relative displacement offset and the three-dimensional target frame of the previous frame are transformed using the Transform function to obtain the three-dimensional target frame of the current frame.

[0284] In a possible implementation, the method further includes:

[0285] A first calculation module 370, configured to perform calculation according to the previous frame 3D model data and the previous frame 3D target frame to obtain previous frame box information;

[0286] A second calculation module 380, configured to perform calculation according to the current frame 3D model data and the current frame 3D target frame to obtain current frame box information;

[0287] The third calculation module 390 is used to input the previous frame box information and the current frame box information into the BoxCloud comparator for similarity calculation, and extract multiple points with the highest similarity, fuse the features of the previous frame three-dimensional model, to enhance the features of the current frame three-dimensional model, and generate the final three-dimensional target box from the enhanced features.

[0288] The device for tracking a three-dimensional model target provided by the embodiment of the present disclosure, by utilizing the self-attention mechanism of the Transformer network, the model can automatically capture and integrate global information, improve the recognition ability of the target in complex scenes, and be more suitable for the three-dimensional model. By effectively integrating global information and motion information, the model can comprehensively consider the appearance and motion state of the target during the tracking process, reduce the dependence on a single appearance feature, and improve the stability and robustness of tracking. When similar objects appear in the scene, because global information and motion information are considered at the same time, the model can more accurately identify the target and reduce the situation of mistracking. Even in scenes with sparse three-dimensional models, with the assistance of motion information, the model can effectively track the target and is not easy to lose.

[0289] The present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0290] The computer device includes a memory 410 and a processor 420 that are connected to each other through a system bus. It should be noted that the figure only shows a computer device with components 410-420, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0291] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0292] The memory 410 includes at least one type of readable storage medium, and the readable storage medium includes a non-volatile memory or a volatile memory, for example, a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc., and the RAM may include a static RAM or a dynamic RAM. In some embodiments, the memory 410 may be an internal storage unit of a computer device, for example, a hard disk or a memory of the computer device. In other embodiments, the memory 410 may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card or a flash card (FlashCard) equipped on the computer device. Of course, the memory 410 may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory 410 is generally used to store the operating system and various application software installed on the computer device, such as the program code of the above method. In addition, the memory 410 may also be used to temporarily store various data that have been output or are to be output.

[0293] The processor 420 is generally used to perform the overall operation of the computer device. In this embodiment, the memory 410 is used to store program codes or instructions, the program code includes computer operation instructions, and the processor 420 is used to execute the program codes or instructions stored in the memory 410 or process data, such as running the program code of the above method.

[0294] In this article, the bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus system can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0295] Another embodiment of the present application also provides a computer-readable medium, which may be a computer-readable signal medium or a computer-readable medium. A processor in a computer reads a computer-readable program code stored in the computer-readable medium, so that the processor can execute the functional actions specified in each step or a combination of steps in the above method; and generate a device for implementing the functional actions specified in each block or a combination of blocks in the block diagram.

[0296] Computer-readable media include but are not limited to electronic, magnetic, optical, electromagnetic, infrared memory or semiconductor systems, devices or apparatuses, or any appropriate combination of the foregoing, the memory is used to store program codes or instructions, the program codes include computer operating instructions, and the processor is used to execute the program codes or instructions of the above methods stored in the memory.

[0297] For the definitions of memory and processor, please refer to the description of the aforementioned computer device embodiment and will not be repeated here.

[0298] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0299] Each functional unit or module in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional unit.

[0300] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

[0301] Unless the context clearly indicates otherwise, the singular form of the words used herein and in the appended claims includes the plural and vice versa. Thus, when referring to the singular, the plural form of the corresponding term is generally included. Similarly, the words "comprise" and "include" are to be interpreted as inclusive rather than exclusive. Likewise, the terms "include" and "or" should be interpreted as inclusive unless such interpretation is expressly prohibited herein. Where the term "example" is used herein, particularly when it is located after a group of terms, the "example" is merely exemplary and illustrative and should not be considered exclusive or comprehensive.

[0302] Further aspects and scopes of adaptability become apparent from the description provided herein. It should be understood that various aspects of the present application can be implemented individually or in combination with one or more other aspects. It should also be understood that the description and specific embodiments herein are intended for purposes of illustration only and are not intended to limit the scope of the present application.

[0303] Several embodiments of the present disclosure are described in detail above, but it is obvious that those skilled in the art can make various modifications and variations to the embodiments of the present disclosure without departing from the spirit and scope of the present disclosure. The protection scope of the present disclosure is defined by the attached claims.

Claims

1. A method for tracking a three-dimensional model target, characterized in that: include: Acquire the current frame 3D model data, the previous frame 3D model data and the previous frame 3D target frame and perform data preprocessing to obtain a preprocessing result, wherein the preprocessing result includes: a spatiotemporal 3D model and a confidence level; Processing the preprocessing result by using a Transformer network to generate local features, wherein the local features include: previous frame features and current frame features; Inputting the spatiotemporal three-dimensional model into the PointNet network model to generate global features; Fusing the local feature with the global feature to obtain a fused feature; Inputting the fused features into the MLP layer for processing to generate a target three-dimensional model; Based on the target three-dimensional model, a relative position offset from a previous frame to a current frame is predicted through motion prediction, and the target three-dimensional model is positioned based on the relative position offset to obtain a three-dimensional target frame of the current frame.

2. The method according to claim 1, characterized in that The step of obtaining the current frame 3D model data, the previous frame 3D model data and the previous frame 3D target frame and performing preprocessing to obtain the preprocessing result includes: splicing the current frame 3D model data with the previous frame 3D model data, and adding time information to construct a spatiotemporal 3D model; Creating a priori target confidence map to represent the position of the target in the spatiotemporal three-dimensional model, so as to judge the target according to the three-dimensional target frame of the previous frame; For a point in the three-dimensional model of the previous frame, its prior confidence is set according to its position relative to the three-dimensional target frame of the previous frame. If it is a point of the previous frame and is in the three-dimensional target frame, the prior confidence is 1; if it is not in the three-dimensional target frame, the prior confidence is 0.

3. The method according to claim 1, characterized in that The step of using a Transformer network to process the preprocessing result to generate local features, wherein the local features include features of a previous frame and features of a current frame, comprises: receiving the preprocessing result; Using a 3D model encoder to perform 3D model encoding on the spatiotemporal 3D model data to obtain 3D model encoding features; Based on the spatiotemporal three-dimensional model data, extracting the position code in the three-dimensional model coordinates using a K-nearest neighbor algorithm; Projecting the 3D model encoding features into the feature space, evaluating the importance of each point feature by using the confidence and calculating the attention weight, and obtaining the 3D model encoding attention weight information; Based on the three-dimensional model encoding features, the position encoding and the three-dimensional model encoding attention weight information, a residual connection method is adopted to obtain the current frame features and the previous frame features.

4. The method according to claim 3, characterized in that: The global-local feature fusion includes: According to the previous frame features and the current frame features, a feature attention weight is calculated by using a convolution kernel and a feature product method; Then, the local features are processed using the feature attention weights to obtain processed local features; The global feature is fused with the processed local feature to obtain a fused feature of the current frame and a fused feature of the previous frame; The fusion features of the current frame and the fusion features of the previous frame are concatenated and processed by the MLP layer to obtain the final fusion features.

5. The method according to claim 1, characterized in that: The step of inputting the fused features into the MLP layer for processing to generate a target three-dimensional model includes: The fusion features are learned through the MLP layer to generate a classification result, wherein the classification result includes a target three-dimensional model and background points, and the target three-dimensional model includes: the current frame target three-dimensional model, the previous frame target three-dimensional model and the previous frame three-dimensional target frame.

6. The method according to claim 1, characterized in that The step of predicting the relative position offset from the previous frame to the current frame through motion prediction based on the target three-dimensional model, positioning the target three-dimensional model based on the relative position offset, and obtaining the three-dimensional target frame of the current frame includes: Obtaining a relative displacement offset by calculating the final fusion feature, the current 3D model data, the previous frame 3D model data, and the previous frame 3D target frame through a multi-dimensional relative motion function; The relative displacement offset and the previous frame three-dimensional target frame are transformed using a Transform function to obtain the current frame three-dimensional target frame.

7. The method according to claim 1, characterized in that Also includes: Calculating according to the previous frame 3D model data and the previous frame 3D target frame to obtain previous frame box information; Calculate according to the current frame 3D model data and the current frame 3D target frame to obtain current frame box information; The previous frame box information and the current frame box information are input into the BoxCloud comparator for similarity calculation, and the multiple points with the highest similarity are taken out, and the features of the previous frame 3D model are fused to enhance the features of the current frame 3D model, and the final current frame 3D target box is generated from the enhanced features.

8. A device for tracking a three-dimensional model target, characterized in that: include: A preprocessing module is used to obtain the current frame 3D model data, the previous frame 3D model data and the previous frame 3D target frame and perform preprocessing to obtain a preprocessing result, wherein the preprocessing result includes: a spatiotemporal 3D model and a confidence level; A feature extraction module, used to process the preprocessing result using a Transformer network to generate local features, wherein the local features include: previous frame features and current frame features; A generation module, used for inputting the spatiotemporal three-dimensional model into a PointNet network model to generate global features; A fusion module, used for fusing the local features with the global features to obtain fused features; A processing module, used for inputting the fused features into the MLP layer for processing to generate a target three-dimensional model; The motion prediction module is used to predict the relative position offset from the previous frame to the current frame based on the target three-dimensional model through motion prediction, and locate the target three-dimensional model based on the relative position offset to obtain the three-dimensional target frame of the current frame.

9. A computer device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images