A cross-modal distillation LiDAR point cloud tracking method

Abstract

The present invention discloses a cross-modal distillation LiDAR point cloud tracking method, which belongs to the field of three-dimensional visual perception technology in artificial intelligence and computer vision. The method includes using a pre-reconstruction task to tune a pre-trained two-dimensional image visual model to a point cloud modality to adapt to the point cloud feature space; using the adjusted model as a teacher network to guide the feature distillation of a point cloud mask encoding module. The present invention makes full use of the twin structure to effectively solve the problems of expensive data collection costs and endless practical scenarios, so that it can train the model based on observed data and can be applied to target tracking with point cloud distribution changes during the test phase; when the template and search area features are fused, the adaptive generation of multiple templates is studied from the perspective of consistency and diversity to obtain information containing changes in perspective and sparse distribution, and with the help of the Transformer self-attention mechanism, the distribution changes of the target in the entire sequence are addressed to complete the accurate estimation of the target state.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional visual perception in artificial intelligence and computer vision, and specifically relates to a cross-modal distillation LiDAR point cloud tracking method. Background Art

[0002] Visual perception is undergoing a technological leap from 2D to 3D. Three-dimensional target tracking by LiDAR is one of the most important tasks in the process of intelligence to understand and perceive complex dynamic scenes, and it plays an important role in the perception of indoor and outdoor scenes. Many domestic companies have rapidly emerged in this field, such as Benewake Photonics, Leishen Intelligent and Tudatong, which are widely used in intelligent terminals or platforms such as rail transit, civil aviation, road traffic and robots. The target tracking and perception technology based on LiDAR is in line with the requirements of national and industry intelligent development. It aims to accurately perceive the target state in continuous frames and is one of the key links in intelligent systems.

[0003] With the development of intelligent terminals such as autonomous driving, drones, and robots, point cloud single target tracking methods based on deep learning have attracted the attention of the industry. Researchers have combined technologies such as sparse convolution, Transformer, graph network, geometric size prior or bird's-eye view to achieve better performance in point cloud tracking. Currently, these tracking methods use the twin network as the main framework, with target templates and search areas as input, and they mainly focus on conventional objects (such as vehicles and pedestrians). However, in actual scenarios, target tracking will fail when the distribution of point clouds changes, such as collision deformation of the target during the test phase, intrusion of new categories of objects, and complex weather interference (see Figure 1 ). The main reasons behind this are: rare categories and scenarios in the application environment emerge in an endless stream, requiring the offline data collection process to cover all possible situations is costly and challenging; and the current visual tracking method is limited to training a specific model on one type of object. In essence, unobserved scenes can be regarded as out-of-domain data relative to the observed objects. Therefore, the present invention regards it as a domain generalization problem, and trains the model based on the observed data, which is of great significance to make it applicable to target tracking with changing point cloud distribution during the testing phase.

[0004] In addition, the target point cloud scanned in the real world is often sparse and perspective-moving, and its perspective and sparsity in the scene will also change over time and space. The target template is an important clue in the tracking process, but it is not fully characterized when modeling the feature relationship with the frame to be tracked. That is, the current tracking method constructs a single target template and does not fully consider the various situations in which the target appears. Different from this, the present invention designs a robust embedding strategy that is not affected by shape disturbances or transformations, thereby paving the way for subsequent accurate state regression.

[0005] In summary, current target tracking methods have the following problems: (1) Current tracking methods train a specific model for a type of target, which is incapable of coping with various environmental changes, and the representation of its twin structure is not fully exploited; (2) When embedding tracking clues (templates) into the search area, only fixed templates are used, ignoring the relationship modeling of temporal changes, in order to fully consider the various sparse changes and perspective movements of the target.

[0006] The information disclosed in this background technology section is only intended to enhance the understanding of the overall background of the invention and should not be regarded as an acknowledgment or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the invention

[0007] The purpose of the present invention is to provide a cross-modal distillation LiDAR point cloud tracking method, which is oriented to the three-dimensional point cloud visual data obtained by the LiDAR sensor, and focuses on the point cloud distribution changes in actual scenes. It starts from the two aspects of target perception feature representation and target state estimation, and aims to solve the problems of insufficient domain generalization and relationship modeling of current point cloud tracking methods, promote reliable perception in actual scenes, and truly meet the requirements of technological innovation and industry development.

[0008] In order to achieve the above object, a technical solution provided by a specific embodiment of the present invention is as follows:

[0009] A cross-modal distillation LiDAR point cloud tracking method includes the following steps:

[0010] Step S1: Use the previous reconstruction task to tune the pre-trained 2D image vision model to the point cloud modality to adapt to the point cloud feature space;

[0011] Step S2: Use the adjusted model as a teacher network to guide the feature distillation of the point cloud mask encoding module;

[0012] Step S3: Obtain template features under different viewing angles and sparseness, and generate an adaptive template;

[0013] Step S4: Concatenate the template features generated in step S3 with the search area features generated in step S2, and send them into the self-attention mechanism to generate the center features of the candidate objects, and then regress their respective confidence scores, three-dimensional bounding box center point coordinates, and yaw angles.

[0014] In one or more embodiments of the present invention, step S1 is an image-to-point cloud tuning module, and the image-to-point cloud tuning module uses the image-to-point cloud tuning module to fine-tune the pre-trained model to adapt to the point cloud feature space.

[0015] In one or more embodiments of the present invention, the specific implementation method of step S1 is:

[0016] S1-1: Noise enhancement is performed on the point cloud, key points are downsampled as tokens, and preliminary token features h0 are extracted, which are then compared with a set of learnable prompts features p l Concatenate them and feed them into the pre-trained image model f;

[0017] S1-2: Before sending the feature into each layer, it is combined with a set of learnable prompts features p l Phase splicing; among them, p l represents the lth layer of the pre-trained image model;

[0018] S1-3: Based on the obtained tuning feature h L , using self-supervision to predict the displacement vector v of the noise point cloud denoise , apply this vector to the noisy point cloud, and then compare it with the corresponding clean point cloud for loss; the loss function is as follows: where d chamfer To characterize the Chamfer distance between point sets, Q and They are clean point cloud and noise enhanced point cloud respectively;

[0019] S1-4: Use the feedback of the loss function to adapt the features to the point cloud and build a teacher network.

[0020] By adopting self-supervision and prompts tuning, the distribution differences between different modalities are eliminated, effectively avoiding potential negative transfer.

[0021] In one or more embodiments of the present invention, step S2 is a point cloud mask encoding module, and the point cloud mask encoding module uses the adjusted model as a teacher network to guide the feature distillation of the point cloud mask encoding module.

[0022] In one or more embodiments of the present invention, the specific implementation method of S2 is:

[0023] S2-1: The input template and the point cloud of the search area are concatenated and recorded as Q, and sent to the teacher network and the student network respectively;

[0024] S2-2: Sample the seed point Q and extract the initial feature F as the token input; then, randomly generate the corresponding mask for all tokens, and send the token not covered by the mask to the encoder of the point cloud Transformer to obtain the feature g encoder (F) The masked tokens are replaced by a set of learnable features w and compared with the feature g encoder (F) After concatenation, it is sent to the decoder. Finally, the output features of the student network are obtained. The whole process can be expressed as:

[0025] S2-3: Calculate the cosine similarity between the features of the teacher network output in step S1 and the features of the student network output, and use it for distillation constraints to guide the feature distillation of the point cloud mask encoding module.

[0026] Through the tuning of step S1, the feature representation of the pre-trained image Transformer is converted to the 3D feature space, and step S2 uses it as a cross-modal teacher network to guide the tracking process; different from the previous two-stream twin network, the present invention splices the input template and the point cloud of the search area, denoted as Q, and uses the subsequent Transform's self-attention to automatically interact and fuse the features of the two.

[0027] In one or more embodiments of the present invention, S3 is adaptive template generation, and the adaptive template generation is intended to design a multi-template learning strategy based on geometric appearance changes to divide and conquer different incomplete and sparse situations.

[0028] In one or more embodiments of the present invention, the specific implementation method of S3 is:

[0029] S3-1: Rotate different anchors according to the bird’s-eye view or sample the farthest point to obtain anchor template features F under different viewing angles and sparseness p , and then densify it to perceive the spatial information and transform it into a bird's-eye view feature

[0030] S3-2: Each template obtained in step S3-1 is fed into MLP to generate embeddings, denoted as

[0031] S3-3: Introducing consistency loss l consistency and diversity loss l diversity , which is used to ensure semantic consistency and diversity during training, constrain the learning process of anchors, and generate adaptive templates.

[0032] Step S3 takes into account that the viewing angle and sparsity of the target in the scene will change with the passage of time and space, so only using a fixed template is not enough to cover the various situations in which the target appears.

[0033] In one or more embodiments of the present invention, S4 is a self-attention mechanism relationship modeling, in which the Transformer can break through the position limitation and establish long-distance relationships in the context. It is a multi-head attention method, and its goal is to find instance individuals of interest from the global perspective.

[0034] In one or more embodiments of the present invention, the specific implementation method of S4 is:

[0035] S4-1: Concatenate the template features generated in step S3 with the search area features, generate Q, K and V through linear transformation, and send them to the self-attention mechanism to generate the center features of the candidate objects.

[0036] S4-2: The target-aware search area features are sent to the region proposal network (RPN) to regress the respective confidence scores, 3D bounding box center point coordinates, and yaw angles.

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] (1) Make full use of the twin structure to effectively solve the problems of expensive data collection costs and the endless emergence of actual scenarios. The model can be trained based on observed data. In the test phase, it can be applied to target tracking with changing point cloud distribution, thus improving the generalization of the model.

[0039] (2) When the template and search area features are fused, the adaptive generation of multiple templates is studied from the perspectives of consistency and diversity to obtain information containing changes in perspective and sparse distribution. With the help of the Transformer self-attention mechanism, the distribution changes of the target in the entire sequence are dealt with, and the discriminative features of the geometric appearance are learned, ultimately completing the accurate estimation of the target state. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A schematic diagram of point cloud distribution changes in an embodiment of the present invention;

[0042] Figure 2 A content summary diagram of a cross-modal distillation LiDAR point cloud tracking method in one embodiment of the present invention;

[0043] Figure 3 This is a flow chart of cross-modal knowledge transfer in one embodiment of the present invention;

[0044] Figure 4 The figure is a flowchart of relationship modeling in one embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0046] like Figure 2 As shown in the figure, a cross-modal distillation LiDAR point cloud tracking method in one embodiment of the present invention addresses the two problems of insufficient domain generalization and insufficient relationship modeling in previous methods, and is respectively aimed at the two subtasks of feature extraction of target perception and target state estimation during the tracking process. On the one hand, in the feature extraction of target perception, a cross-modal knowledge transfer method based on a point cloud mask encoding network is given; on the other hand, in the target state estimation, a relationship modeling method that integrates adaptive template generation and self-attention mechanism is given.

[0047] like Figure 2 As shown in FIG, the cross-modal distillation LiDAR point cloud tracking method includes the following steps:

[0048] Step S1: Use the previous reconstruction task to tune the pre-trained 2D image vision model to the point cloud modality to adapt to the point cloud feature space;

[0049] Step S2: Use the adjusted model as a teacher network to guide the feature distillation of the point cloud mask encoding module;

[0050] Step S3: Obtain template features under different viewing angles and sparseness, and generate an adaptive template;

[0051] Step S4: Concatenate the template features generated in step S3 with the search area features generated in step S2, and send them into the self-attention mechanism to generate the center features of the candidate objects, and then regress their respective confidence scores, three-dimensional bounding box center point coordinates, and yaw angles.

[0052] The image and NLP fields have obtained relatively complete feature representations with the help of large-scale data pre-training. In contrast, ShapeNet used in the 3D field only contains 50,000 point clouds and about 55 categories, which is far less than the 400 million image-text sample pairs of the 2D visual CLIP model. Considering the high cost of 3D data acquisition and annotation, it is challenging to learn a more general feature representation network from scratch based only on unordered point clouds. The present invention makes full use of the richer appearance information representation learned by the image model and promotes the learning of target discriminant features in 3D point cloud tracking through cross-modal knowledge transfer.

[0053] This part is based on the Transformer architecture, and its use of Token input can eliminate the barriers caused by inconsistent data between different modalities. The proposed research route mainly includes two modules: first, for the pre-trained model under a given image modality, the image to point cloud tuning module is used to fine-tune the pre-trained model to adapt to the point cloud feature space; then the adjusted model is used as a teacher network to guide the feature distillation of the point cloud mask encoding module.

[0054] Specifically, step S1 is an image-to-point cloud tuning module, and the image-to-point cloud tuning module uses the image-to-point cloud tuning module to fine-tune the pre-trained model to adapt to the point cloud feature space.

[0055] like Figure 3 As shown, the specific implementation method of step S1 is:

[0056] S1-1: Noise enhancement is performed on the point cloud, key points are downsampled as tokens, and preliminary token features h0 are extracted, which are then compared with a set of learnable prompts features p l Concatenate them and feed them into the pre-trained image model f;

[0057] S1-2: Before sending the feature into each layer, it is combined with a set of learnable prompts features p l Phase splicing; among them, p l represents the lth layer of the pre-trained image model;

[0058] S1-3: Based on the obtained tuning feature h L , using self-supervision to predict the displacement vector v of the noise point cloud denoise , apply this vector to the noisy point cloud, and then compare it with the corresponding clean point cloud for loss; the loss function is as follows: where d chamfer To characterize the Chamfer distance between point sets, Q and They are clean point cloud and noise enhanced point cloud respectively;

[0059] S1-4: Use the feedback of the loss function to adapt the features to the point cloud and build a teacher network.

[0060] By adopting self-supervision and prompts tuning, the distribution differences between different modalities are eliminated, effectively avoiding potential negative transfer.

[0061] Specifically, step S2 is a point cloud mask encoding module, and the point cloud mask encoding module uses the adjusted model as a teacher network to guide the feature distillation of the point cloud mask encoding module.

[0062] like Figure 3 As shown, the specific implementation method of S2 is:

[0063] S2-1: The input template and the point cloud of the search area are concatenated and recorded as Q, and sent to the teacher network and the student network respectively;

[0064] S2-2: Sample the seed point Q and extract the initial feature F as the token input; then, randomly generate the corresponding mask for all tokens, and send the token not covered by the mask to the encoder of the point cloud Transformer to obtain the feature g encoder (F) The masked tokens are replaced by a set of learnable features w and compared with the feature g encoder (F) After concatenation, it is sent to the decoder. Finally, the output features of the student network are obtained. The whole process can be expressed as:

[0065] S2-3: Calculate the cosine similarity between the features of the teacher network output in step S1 and the features of the student network output, and use it for distillation constraints to guide the feature distillation of the point cloud mask encoding module.

[0066] Through the tuning of step S1, the feature representation of the pre-trained image Transformer is converted to the 3D feature space, and step S2 uses it as a cross-modal teacher network to guide the tracking process; different from the previous two-stream twin network, the present invention splices the input template and the point cloud of the search area, denoted as Q, and uses the subsequent Transform's self-attention to automatically interact and fuse the features of the two.

[0067] Specifically, S3 is adaptive template generation, and the adaptive template generation is intended to design a multi-template learning strategy based on geometric appearance changes to divide and conquer different incomplete and sparse situations.

[0068] like Figure 4 As shown, the specific implementation method of S3 is:

[0069] S3-1: Rotate different anchors according to the bird’s-eye view or sample the farthest point to obtain anchor template features F under different viewing angles and sparseness p , and then densify it to perceive the spatial information and transform it into a bird's-eye view feature

[0070] S3-2: Each template obtained in step S3-1 is fed into MLP to generate embeddings, denoted as

[0071] S3-3: Introducing consistency loss l consistency and diversity loss l diversity, which is used to ensure semantic consistency and diversity during training, constrain the learning process of anchors, and generate adaptive templates.

[0072] Step S3 takes into account that the viewing angle and sparsity of the target in the scene will change with the passage of time and space, so only using a fixed template is not enough to cover the various situations in which the target appears.

[0073] Specifically, the S4 is a self-attention mechanism relationship modeling, in which the Transformer can break through the position limitation and establish long-distance relationships in the context. It is a multi-head attention method, and its goal is to find instance individuals of interest from the global perspective.

[0074] like Figure 4 As shown, the specific implementation method of S4 is:

[0075] S4-1: Concatenate the template features generated in step S3 with the search area features, generate Q, K and V through linear transformation, and send them to the self-attention mechanism to generate the center features of the candidate objects.

[0076] S4-2: The target-aware search area features are sent to the region proposal network (RPN) to regress the respective confidence scores, 3D bounding box center point coordinates, and yaw angles.

[0077] In order to solve the problem of point cloud distribution changes in point cloud single target tracking, the present invention transfers 2D knowledge to the 3D field to assist in point cloud representation learning. Pre-training with large-scale data can bring robust feature representation to the model, such as in the natural language and image fields. In contrast, due to the high cost of 3D data acquisition and annotation, complete 3D feature learning is challenging. In view of this, in the feature extraction of target perception, we distill mature image features (such as CLIP / ViT) into the point cloud across modalities. Specifically, by studying the cross-modal knowledge tuning from image to point cloud, the teacher network is constructed to achieve the purpose of guiding point cloud learning with robust image features; by studying the distillation method for point cloud mask encoding networks, efficient knowledge transfer is achieved, making the point cloud feature representation more robust; finally, the tracking model is able to better handle situations such as target collision deformation, intrusion of new categories of objects, and weather interference, thereby enhancing the domain generalization capability.

[0078] In order to solve the problem of relationship modeling between target template and search area, the present invention studies the adaptive generation of multiple templates from the perspectives of consistency and diversity, obtains information containing perspective changes and sparse distribution changes, and achieves the purpose of coping with the distribution changes of the target in the entire sequence; by studying the coupling mode of template features and search area features, and with the help of Transformer self-attention mechanism, the discriminant feature learning of geometric appearance is realized, and finally the accurate estimation of the target state is completed.

[0079] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0080] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A cross-modal distillation LiDAR point cloud tracking method, characterized in that: The following steps are involved: Step S1: Use the previous reconstruction task to tune the pre-trained 2D image vision model to the point cloud modality to adapt to the point cloud feature space; The step S1 is an image-to-point cloud tuning module, which uses the image-to-point cloud tuning module to fine-tune the pre-trained model to adapt to the point cloud feature space; Step S2: Use the adjusted model as a teacher network to guide the feature distillation of the point cloud mask encoding module; The step S2 is a point cloud mask encoding module, and the point cloud mask encoding module uses the adjusted model as a teacher network to guide the feature distillation of the point cloud mask encoding module; Step S3: obtaining template features under different viewing angles and sparsity, and generating an adaptive template; said S3 is adaptive template generation, and said adaptive template generation intends to design a multi-template learning strategy based on geometric appearance changes, and divide and conquer different incomplete and sparse situations; Step S4: splice the template features generated in step S3 with the search area features, and send them into the self-attention mechanism to generate the center features of the candidate objects, and then regress the respective confidence scores, the coordinates of the center points of the three-dimensional bounding boxes, and the yaw angles; S4 is the self-attention mechanism relationship modeling, in which the Transformer can break through the position restrictions and establish long-distance relationships in the context. It is a multi-head attention method, and its goal is to find the instance individuals of interest from the global perspective; Specifically, the specific implementation method of step S1 is: S1-1: Noise enhancement is performed on the point cloud, key points are downsampled as tokens, and preliminary token features h0 are extracted, which are then compared with a set of learnable prompts features p l Concatenate them and feed them into the pre-trained image model f; S1-2: Before sending the feature into each layer, it is combined with a set of learnable prompts features p l Phase splicing; among them, p l represents the lth layer of the pre-trained image model; S1-3: Based on the obtained tuning feature h L , using self-supervision to predict the displacement vector v of the noise point cloud denoise , apply this vector to the noisy point cloud, and then compare it with the corresponding clean point cloud for loss; the loss function is as follows: where d chamfer To characterize the Chamfer distance between point sets, Q and They are clean point cloud and noise enhanced point cloud respectively; S1-4: Use feedback from the loss function to adapt the features to the point cloud and build a teacher network; The specific implementation method of S2 is: S2-1: The input template and the point cloud of the search area are concatenated and recorded as Q, and sent to the teacher network and the student network respectively; S2-2: Sample the seed point Q and extract the initial feature F as the token input; then, randomly generate the corresponding mask for all tokens, and send the token not covered by the mask to the encoder of the point cloud Transformer to obtain the feature g encoder (F), and the tokens covered by the mask are replaced by a set of learnable features w and combined with the feature g encoder (F) After concatenation, it is sent to the decoder; finally, the output features of the student network are obtained. The whole process can be expressed as: S2-3: Calculate the cosine similarity between the features of the teacher network output in step S1 and the features of the student network output, and use it to perform distillation constraints to guide the feature distillation of the point cloud mask encoding module; The specific implementation method of S3 is: S3-1: Rotate different anchors according to the bird’s-eye view or sample the farthest point to obtain anchor template features F under different viewing angles and sparseness p , and then densify it to perceive the spatial information and transform it into a bird's-eye view feature S3-2: Each template obtained in step S3-1 is fed into MLP to generate embeddings, denoted as S3-3: Introducing consistency loss l consistency and diversity loss l diversity , used to ensure semantic consistency and diversity during training, constrain the learning process of anchors, and generate adaptive templates; The specific implementation method of S4 is: S4-1: concatenate the template features generated in step S3 with the search area features, generate Q, K and V through linear transformation, and send them to the self-attention mechanism to generate the center features of the candidate objects; S4-2: Send the target-perceived search area features to the region candidate network RPN to regress the respective confidence scores, 3D bounding box center point coordinates and yaw angles.

Images