A 3D point cloud single target tracking method based on graph convolution

Abstract

The present invention provides a three-dimensional point cloud single target tracking method based on graph convolution, which belongs to the technical field of three-dimensional point cloud single target tracking. The present invention applies graph convolution to the three-dimensional single target tracking task for the first time. The present invention is based on the existing three-dimensional single target tracking method P2B for improvement. First, the template point cloud and the search point cloud are input into the network, and the point cloud is downsampled to extract the seed point features; secondly, the graph convolution module is used to perform global and local feature fusion, and the template information is embedded into the search area; finally, the seed point with the template information will be sent to the Hough voting module to locate the tracked object in the search area and generate its three-dimensional target box. The present invention improves the tracking quality by performing global and local feature fusion through the graph convolution module; the present invention uses a real point cloud dataset for evaluation and observes the improvement on the advanced baseline.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional point cloud single target tracking, and specifically relates to a three-dimensional point cloud single target tracking method based on graph convolution, and specifically relates to a method for improving and optimizing a P2B target tracking method. Background Art

[0002] Single target tracking of 3D point cloud is an important task in the field of computer vision. The input of single target tracking of 3D point cloud is the tracking object point cloud scanned by LiDAR and the search point cloud sequence of the tracking object. The purpose is to detect the position and size of the target in each frame of the search point cloud sequence. This task is widely used in intelligent robot interaction systems, unmanned driving systems, and can also be used in aviation, military and other fields.

[0003] With the rapid recommendation of deep models, target tracking methods based on deep learning have shown significant improvements and good performance in tracking tasks. These target tracking methods usually implement matching based on the target shape, embedding template information into the search area, and then better judging the search area. However, when considering shape matching, current target tracking methods either consider global feature learning or local feature learning. In addition, current tracking methods consider feature similarity while ignoring spatial matching. Therefore, the present invention designs effective global and local feature learning methods for the matching problem of template point cloud and search area point cloud to improve the performance of target tracking.

[0004] Based on the current target tracking method, there are several problems: (1) Only global matching is performed, that is, the features of all points in the point learning template in each search area are ignored, and the specificity between points is ignored, thereby embedding erroneous information or redundant information. (2) Only local matching is performed, that is, the features of some high-similarity points in the template are learned in each search area, and the template as an important global target information of the tracking target object is ignored, resulting in information loss. (3) When considering the matching of template and search area, only the similarity at the feature level is used, thereby ignoring the spatial correspondence. Therefore, the present invention improves the P2B network structure to achieve better results for the problem of three-dimensional point cloud target tracking. Summary of the invention

[0005] This paper mainly aims to improve the shortcomings of the above three-dimensional point cloud target tracking method and proposes a three-dimensional point cloud single target tracking method based on graph convolution. This method is designed and improved based on the P2B method, and its specific design features are:

[0006] Different from the traditional P2B target tracking method, the present invention fully learns the global and local information of the tracked object. Different from using only the similarity of features, the present invention adds the distance metric, uses two metric scales, and two feature learning methods to obtain information at different levels and fully mix them. This improves the performance of the target tracking method.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A three-dimensional point cloud single target tracking method based on graph convolution includes the following steps:

[0009] Step S1: read point cloud data, determine the tracking object, i.e., the template point cloud, and determine the search area;

[0010] Step S2: Send the template point cloud and the search area point cloud to the shared encoder for point cloud downsampling and feature extraction to obtain the three-dimensional coordinates of the seed point and the feature vector of the seed point;

[0011] Step S3: using the global feature fusion module, sending the template seed points and the search area seed points into the global feature fusion module, performing global feature learning, embedding the template global information into the search area seed point features, and updating the search area seed point features;

[0012] Step S4: using the local feature fusion module, sending the template seed point and the search area seed point into the local feature fusion module, performing local feature learning, embedding the template local information into the search area seed point feature, and updating the search area seed point feature;

[0013] Step S5: Iterate steps S3-S4 to complete feature fusion and obtain seed points rich in template information.

[0014] Step S6: Send the updated search area seed points to Hough voting to find the cluster center and vote to determine the target center position and the deflection angle of the bounding box.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] The present invention uses an improved P2B network to achieve the task of single target tracking in three-dimensional point clouds, and supplements the shortcomings of current target tracking methods: 1) Only global matching is performed, that is, the points in each search area learn the features of all points in the template, ignoring the specificity between points, thereby embedding erroneous information or redundant information. (2) Only local matching is performed, that is, the points in each search area learn the features of some highly similar points in the template, ignoring the template as an important global target information of the tracking target object, resulting in information loss. (3) When considering the matching of template and search area, only the similarity at the feature level is used, thereby ignoring the spatial correspondence. The present invention improves the tracking quality by fusing multiple measurement scales and multiple modules for fusing information; the present invention uses a real point cloud dataset for evaluation and observes the improvement on the advanced baseline. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is the overall structure designed by the present invention.

[0018] Figure 2 This is the global feature learning node connection method designed by the present invention.

[0019] Figure 3 This is the global feature learning node feature connection method designed by the present invention.

[0020] Figure 4 This is the local feature learning node connection method designed by the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention will be further described below in conjunction with specific embodiments and drawings.

[0022] A 3D point cloud single target tracking method based on graph convolution iteration for global and local feature learning includes the following steps:

[0023] Step S1: Read point cloud data, determine the tracking object, i.e., the template point cloud, and determine the search area. The specific process is as follows:

[0024] Step S11: Read the target frame of the object tracked by the previous frame of point cloud as the size of the target, and use the points in the target frame as the template of the tracking object, that is, the template point cloud.

[0025] Step S12: According to the position of the target frame of the tracking object in the previous frame point cloud in the world coordinate system, a search area is selected for the current frame, and the target frame of the tracking object in the previous frame is expanded as the search area. The points in the search area are the search area point cloud.

[0026] Step S2: Send the template point cloud and the search area point cloud to the shared encoder for point cloud downsampling and feature extraction to obtain the template seed point and its 3D coordinates and feature vector as well as the search area seed point and its 3D coordinates and feature vector. The specific process is:

[0027] The template point cloud and the search area point cloud are input into the Point-Net++ network for point cloud downsampling. The sampling method is the farthest point sampling, and their respective seed points are obtained. And perform feature learning; finally get the location of the seed point and the features of the seed points Among them, i = t, s represents the template area point cloud and the search area point cloud respectively, N i represents the number of seed points, and N represents the dimension of the seed point features.

[0028] Step S3: Use the global feature fusion module to send the template seed points and the search area seed points into the global feature fusion module to perform global feature learning to obtain the search area seed points embedded with the template global information. Further, the step S3 is specifically as follows:

[0029] Step S31: Calculate the cosine similarity between the template seed point features obtained in step S2 and the search area seed point features.

[0030] Step S32: Construct a global graph link, such as Figure 2 As shown: The template area and search area seed points are used as graph convolution nodes, and an edge structure from the template area nodes to the search area nodes is established. Each search area node is connected to all nodes in the template area.

[0031] Step S33: Use the global feature learning module to perform feature fusion, such as Figure 3 As shown, first, the template seed point coordinates are connected to the template seed point features, the connected feature tensor is expanded to the (B, N+3, Nt, Ns) dimension and connected with the cosine similarity obtained in step S31 to obtain a tensor of (B, N+3+1, Nt, Ns); the tensor passes through a layer of MLP and a layer of maximum pooling in turn, and the feature tensor after pooling is (B, N, Ns).

[0032] Step S34: Connect the pooled feature tensor with the search area seed point feature in the feature dimension, embed the learned template global information into the search area through a layer of MLP, obtain the search area seed point feature embedded with the template global information, and complete the global update of the search area seed point feature.

[0033] Step S4: Use the local feature fusion module to send the template seed points and the search area seed points to the local feature fusion module to perform local feature learning to obtain the search area seed points embedded with the template local information. Further, the step S4 is specifically as follows:

[0034] Step S41: Calculate the cosine similarity between the template seed point feature obtained in step S2 and the search area seed point feature updated in step S3 to obtain a cosine similarity graph;

[0035] Step S42: Calculate the distance between the template seed point coordinates obtained in step S2 to obtain a distance map;

[0036] Step S43: Construct local graph links, such as Figure 4 As shown: the template area and the search area seed points are used as graph convolution graph nodes. First, the edge structure from the template area node to the search area node is established. According to the cosine similarity graph obtained in step S41, a connection is established if the cosine similarity between the search area node and the template area node is greater than a predetermined threshold. Secondly, the edge structure between the template area nodes is established. According to the distance graph obtained in step S42, a connection is established if the distance between the template area nodes is less than a predetermined threshold. The construction of the graph structure of the graph convolution is completed.

[0037] Step S44: Connect the template seed point features and the search area seed point features updated in S3 in the quantity dimension, use the cosine similarity or distance between the connected nodes as weights and feature tensors, and the graph structure established in S43, input them into the graph convolutional network, embed the local information of the template into the search area seed point features, and complete the local update of the search area seed point features.

[0038] Step S5: Iterate steps S3-S4. Complete feature fusion to obtain seed points rich in template information. Further, step S5 is specifically as follows:

[0039] Step S51: Send the updated search area seed point features to step S3 to learn the template global information again.

[0040] Step S52: After updating the template global information again, the seed point features of the search area are sent to step S4 to learn the template local information again.

[0041] Step S6: Send the search area seed points that have been updated multiple times to Hough voting, find the cluster center and vote, and determine the target center position and target box. Further, the step S6 is specifically as follows:

[0042] Step S61: sending the obtained search area seed point features and three-dimensional coordinates embedded with global and local template information to Hough voting.

[0043] Step S62: Apply MLP regression classification scores S = s1, s2...s to the seed points in the search area j , which is used to determine whether the seed point is a target point or a non-target point; where j represents the index of the seed point.

[0044] Step S63: The voting network uses MLP to regress the offset of each seed point to the target center Each seed point p s,j Corresponding to a potential target center C j ; At the same time, the residual from the seed point feature to the potential target center feature is predicted The location and characteristics of the potential target center are expressed as:

[0045] Step S64: Use ball clustering for all potential target centers to obtain K cluster centers. The cluster is the set of potential target centers in the ball neighborhood. The characteristics of the potential target center are Among them, j represents the index of the seed point, s j , They respectively represent the classification score, three-dimensional coordinates, and feature vector of the potential target center corresponding to the j-th seed point.

[0046] Step S65: Apply MLP-Maxpool-MLP to the K clusters to obtain the target proposal, the offset from the potential target center to the target center, the rotation offset of the target bounding box, and the confidence score of the target.

[0047] Step S66: Select the result with the highest confidence score, obtain the position of the target tracking object center in the current frame, apply the bounding box of the template point cloud tracking object to the current frame against the center point, and use the predicted rotation offset of the target bounding box to offset the bounding box to obtain the position and bounding box of the tracking target in the current frame.

[0048] It is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. A three-dimensional point cloud single target tracking method based on graph convolution, characterized in that: The method comprises the following steps: Step S1: read point cloud data, determine the tracking object, i.e., the template point cloud, and determine the search area; Step S2: Send the template point cloud and the search area point cloud to the shared encoder for point cloud downsampling and feature extraction to obtain the three-dimensional coordinates of the seed point and the feature vector of the seed point; Step S3: using the global feature fusion module, sending the template seed points and the search area seed points into the global feature fusion module, performing global feature learning, embedding the template global information into the search area seed point features, and updating the search area seed point features; Step S4: using the local feature fusion module, sending the template seed point and the search area seed point into the local feature fusion module, performing local feature learning, embedding the template local information into the search area seed point feature, and updating the search area seed point feature; Step S5: Iterate steps S3-S4 to complete feature fusion and obtain seed points rich in template information; Step S6: Send the updated search area seed points to Hough voting, find the cluster center and vote, and determine the target center position and the deflection angle of the bounding box; The step S3 is specifically as follows: Step S31: Calculate the cosine similarity between the template seed point feature obtained in step S2 and the search area seed point feature; Step S32: construct a global graph link: the template area and the search area seed points are used as graph convolution graph nodes, and an edge structure from the template area node to the search area node is established. Each search area node is connected to all nodes in the template area. Step S33: Use the global feature learning module to perform feature fusion: first, connect the template seed point coordinates with the template seed point features, expand the connected feature tensor to the (B, N+3, Nt, Ns) dimension and connect it with the cosine similarity obtained in step S31 to obtain a tensor of (B, N+3+1, Nt, Ns); the tensor passes through a layer of MLP and a layer of maximum pooling in turn, and the feature tensor after pooling is (B, N, Ns); Step S34: Connect the pooled feature tensor with the search area seed point feature in the feature dimension, embed the learned template global information into the search area through a layer of MLP, obtain the search area seed point feature embedded with the template global information, and complete the global update of the search area seed point feature; The step S4 is specifically as follows: Step S41: Calculate the cosine similarity between the template seed point feature obtained in step S2 and the search area seed point feature updated in step S3 to obtain a cosine similarity graph; Step S42: Calculate the distance between the template seed point coordinates obtained in step S2 to obtain a distance map; Step S43: Construct local graph links: The template area and the search area seed points are used as graph convolution graph nodes. First, the edge structure from the template area node to the search area node is established. According to the cosine similarity graph obtained in step S41, a connection is established if the cosine similarity between the search area node and the template area node is greater than a predetermined threshold. Secondly, the edge structure between the template area nodes is established. According to the distance graph obtained in step S42, a connection is established if the distance between the template area nodes is less than a predetermined threshold. The construction of the graph structure of the graph convolution is completed. Step S44: Connect the template seed point features and the search area seed point features updated in S3 in the quantity dimension, use the cosine similarity or distance between the connected nodes as weights and feature tensors, and the graph structure established in S43, input them into the graph convolutional network, embed the local information of the template into the search area seed point features, and complete the local update of the search area seed point features.

2. The three-dimensional point cloud single target tracking method based on graph convolution according to claim 1, characterized in that: The specific process of step S1 is as follows: Step S11: reading the target frame of the object tracked by the point cloud in the previous frame as the size of the object, and using the points in the target frame as a template of the object to be tracked, i.e., a template point cloud; Step S12: According to the position of the target frame of the tracking object in the previous frame point cloud in the world coordinate system, the search area of ​​the current frame is selected, and the target frame of the tracking object in the previous frame is expanded as the search area. The points in the search area are the search area point cloud.

3. A three-dimensional point cloud single target tracking method based on graph convolution according to claim 1 or 2, characterized in that: The step S2 is specifically as follows: The template point cloud and the search area point cloud are input into the Point-Net++ network for point cloud downsampling. The sampling method is the farthest point sampling, and their respective seed points are obtained. And perform feature learning; Finally, the position of the seed point is obtained and the features of the seed points Among them, i = t, s represents the template area point cloud and the search area point cloud respectively, N i represents the number of seed points, and N represents the dimension of the seed point features.

4. The three-dimensional point cloud single target tracking method based on graph convolution according to claim 1 or 2, characterized in that: The step S6 is specifically as follows: Step S61: sending the obtained search area seed point features and three-dimensional coordinates embedded with global and local template information to Hough voting; Step S62: Apply MLP regression classification scores S = s1, s2...s to the seed points in the search area j , used to determine whether the seed point is a target point or a non-target point; where j represents the index of the seed point; Step S63: The voting network uses MLP to regress the offset of each seed point to the target center Each seed point p s,j Corresponding to a potential target center C j ; At the same time, the residual from the seed point feature to the potential target center feature is predicted The location and characteristics of the potential target center are expressed as: Step S64: Use ball clustering for all potential target centers to obtain K cluster centers. The cluster is the set of potential target centers in the ball neighborhood. The characteristics of the potential target center are Among them, j represents the index of the seed point, s j , They represent the classification score, three-dimensional coordinates, and feature vector of the potential target center corresponding to the j-th seed point; Step S65: Apply MLP-Maxpool-MLP to the K clusters to obtain target proposals, offsets from the potential target center to the target center, rotation offsets of the target bounding box, and confidence scores of the targets; Step S66: Select the result with the highest confidence score, obtain the position of the target tracking object center in the current frame, apply the bounding box of the template point cloud tracking object to the current frame against the center point, and use the predicted rotation offset of the target bounding box to offset the bounding box to obtain the position and bounding box of the tracking target in the current frame.

5. The three-dimensional point cloud single target tracking method based on graph convolution according to claim 1 or 2, characterized in that: The step S5 is specifically as follows: Step S51: sending the updated search area seed point features to step S3 to learn the template global information again; Step S52: After updating the template global information again, the seed point features of the search area are sent to step S4 to learn the template local information again.

Images