Model training method, point cloud representation method and device

Abstract

The invention provides a model training method and device and a point cloud representation method and device in the field of computer vision, which are used for carrying out variation on features output by a coding model, carrying out comparative learning based on a variation result, and replacing existing fixed features with features obtained through variation, namely features obeying a certain distribution rule. Therefore, overfitting in a single data field can be avoided, and the migration ability of the model is enhanced. The method comprises the steps that firstly, point cloud data are acquired, and the point cloud data comprise data of a plurality of points; then, enhancing the point cloud data for at least two times to obtain different first enhanced point cloud data and second enhanced point cloud data; then, the first enhanced point cloud data and the second enhanced point cloud data serve as input of a coding model, a first point cloud feature and a second point cloud feature are obtained through features output by the coding model, and the first point cloud feature and the second point cloud feature are obtained through variation; and then updating the coding model.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a method and device for representing point clouds. Background Art

[0002] 3D vision has a wide range of application scenarios in many fields such as autonomous driving, robotics, and augmented reality. Point cloud is a commonly used data structure that provides a concise way to represent 3D objects, bringing convenience to the development of 3D vision. Point cloud data can be collected through radar or depth cameras, so algorithms for processing point cloud data have become an important field of 3D vision research.

[0003] Generally, deep learning can be used to represent point cloud data. However, deep learning-based point cloud representation usually relies on a large amount of labeled data, so it takes a lot of cost to label samples. Therefore, how to achieve a lower-cost unsupervised learning method to represent point cloud data has become an urgent problem to be solved. Summary of the invention

[0004] The present application provides a model training method, a point cloud representation method and a device in the field of computer vision, which are used to vary the features output by the encoding model and perform comparative learning based on the variational results. The features obtained by variation, that is, features that obey a certain distribution law, replace the existing fixed features, thereby avoiding overfitting in a single data field and enhancing the migration ability of the model.

[0005] In a first aspect, the present application provides a model training method, comprising: first, obtaining point cloud data, i.e., training data, wherein the point cloud data includes data of multiple points; then, enhancing the point cloud data at least twice, such as flipping, translating, or a combination thereof, to obtain first enhanced point cloud data and second enhanced point cloud data; then, using the first enhanced point cloud data and the second enhanced point cloud data as inputs of a coding model, respectively, and obtaining first point cloud features and second point cloud features through features output by the coding model. The coding model can be used to encode the input point cloud data, wherein the first point cloud features are obtained by performing variations on features output by the coding model for the first enhanced point cloud data, and the second point cloud features are obtained by performing variations on features output by the coding model for the second enhanced point cloud data; then, updating the coding model according to the first point cloud features, the second point cloud features, and the first similarity between the first point cloud features and the second point cloud features to obtain an updated coding model.

[0006] In the implementation mode of the present application, the features output by the encoding model can be subjected to variational operations to obtain features that conform to certain distribution laws, or to make the final features obey certain distribution laws. Even if there are large differences between the training data and the real data, the potential features extracted from the point cloud data can be represented by obeying certain distribution characteristics, so that the model has strong migration capabilities in different data domains and strong generalization capabilities. Compared with using fixed values ​​to represent features, the features obtained in the present application are features that obey certain distribution laws, so that the model that outputs the features has stronger migration capabilities.

[0007] In a possible implementation, the aforementioned updating of the coding model according to the first point cloud feature, the second point cloud feature, and the first similarity between the first point cloud feature and the second point cloud feature to obtain an updated coding model may include: acquiring first generated point cloud data corresponding to the first point cloud feature and second generated point cloud data corresponding to the second point cloud feature through a decoding model; updating the coding model according to the first generated point cloud data, the second generated point cloud data, and the first similarity to obtain an updated coding model.

[0008] Therefore, in the implementation manner of the present application, a decoding model is used to generate point cloud data, and self-supervised learning is performed through the point cloud data output by the decoding model to obtain a coding model with stronger migration capabilities.

[0009] In a possible implementation, the aforementioned updating of the coding model according to the first generated point cloud data, the second generated point cloud data and the first similarity to obtain an updated coding model may include: calculating a first loss value according to the first generated point cloud data, the first enhanced point cloud data, the second generated point cloud data and the second enhanced point cloud data; obtaining a second loss value according to the first similarity; and updating the coding model according to the first loss value and the second loss value to obtain an updated coding model.

[0010] In an implementation manner of the present application, the encoding model can be updated by combining the loss value calculated by the encoding model output value and the loss value calculated by the decoding model output value, so that the output of the decoding model can be used to implement self-supervised learning, thereby improving the output accuracy of the encoding model after training and improving the model convergence speed.

[0011] In a possible implementation, the aforementioned calculation of the first loss value based on the first generated point cloud data, the first enhanced point cloud data, the second generated point cloud data, and the second enhanced point cloud data may include: calculating the first loss value based on at least one of the second similarity, the third similarity, the fourth similarity, or the fifth similarity, wherein the second similarity includes the similarity between the first generated point cloud data and the second enhanced point cloud data, the third similarity includes the similarity between the second generated point cloud data and the first enhanced point cloud data, and the second similarity and the third similarity can be understood as cross-supervision between the point cloud data, the fourth similarity includes the similarity between the first generated point cloud data and the first enhanced point cloud data, and the fifth similarity includes the similarity between the second generated point cloud data and the second enhanced point cloud data, and the fourth similarity and the fifth similarity can be understood as self-supervision of the point cloud data. Therefore, in the implementation of the present application, in the process of updating the model, the update can be performed in a cross-supervision and / or self-supervision manner, so that the encoder retains the invariant features by cross-sampling and reconstructing the features, while promoting the consistency of the feature distribution between the positive samples.

[0012] In a possible implementation, the method further includes: updating the decoding model to obtain an updated decoding model. In the process of updating the encoding model, the decoding model is also updated to improve the output accuracy of the decoding model, thereby performing self-supervised learning based on a more accurate decoding model output and improving the model convergence speed.

[0013] In a possible implementation, the aforementioned method of using the first enhanced point cloud data and the second enhanced point cloud data as inputs of the encoding model, respectively, and obtaining the first point cloud feature and the second point cloud feature through the features output by the encoding model may include: using the first enhanced point cloud data and the second enhanced point cloud data as inputs of the encoding model, respectively, and outputting the first feature corresponding to the first enhanced point cloud data and the second feature corresponding to the second enhanced point cloud data; constraining the first feature and the second feature to obey the corresponding Gaussian distribution, respectively, to obtain the first point cloud feature and the second point cloud feature, that is, the first point cloud feature and the second point cloud feature obey the Gaussian distribution.

[0014] In the implementation manner of the present application, the features output by the encoder are constrained to obey a Gaussian distribution, thereby utilizing the features that obey a Gaussian distribution to represent potential features in the point cloud data. Compared with features represented by fixed values, the extracted features have stronger expressive power and stronger migration capabilities.

[0015] In a possible implementation, the aforementioned constraining the first feature and the second feature to obey the corresponding Gaussian distribution to obtain the first point cloud feature and the second point cloud feature may include: obtaining the distance between the distribution obeyed by the first feature and the distribution obeyed by the second feature; and mapping the first feature and the second feature to the space respectively according to the distance between the distribution obeyed by the first feature and the distribution obeyed by the second feature, such as mapping the features to the space of [0, 1], thereby obtaining the first point cloud feature and the second point cloud feature obeying the Gaussian distribution. In the implementation of the present application, the distance between the distributions obeyed by point clouds of different categories can be calculated, and the features can be projected into the space based on the distance. By adjusting the distance between the distributions, the output features can be made to obey the Gaussian distribution, thereby obtaining the features that characterize the point cloud.

[0016] In a possible implementation, the features output by the updated coding model are used to construct maps, classification tasks, segmentation tasks, or point cloud registration. Therefore, the coding model obtained by the method provided by the present application can be applied to various scenarios that require processing of point cloud data, and has a very strong generalization ability.

[0017] In a second aspect, the present application provides a point cloud representation method, comprising:

[0018] First, input point cloud data is obtained. The input point cloud data may include point cloud data collected by sensors such as radar or depth cameras, and the point cloud data may include one or more points. Then, the input point cloud data is used as input to the encoding model, and point cloud features corresponding to the input point cloud data are obtained according to features output by the encoding model.

[0019] Therefore, in the point cloud representation method provided by the present application, the point cloud feature can be the feature output by the encoding model, or it can be the feature obtained by constraining the feature output by the encoding model to obey the Gaussian distribution. The point cloud representation method can retain the implicit features in the point cloud data, and the point cloud features it represents are more accurate, so that downstream tasks can be performed based on more accurate point cloud features, thereby improving the output accuracy of downstream tasks.

[0020] In one possible implementation, the features output by the encoding model are used to construct a map, a classification task, a segmentation task, or a point cloud registration.

[0021] In addition, the training process of the encoding model can refer to the description of the aforementioned first aspect or any optional implementation manner of the first aspect, and will not be repeated here.

[0022] In a third aspect, the present application provides a model training device, comprising:

[0023] An acquisition module is used to acquire point cloud data, where the point cloud data includes data of multiple points;

[0024] An enhancement module, used to enhance the point cloud data at least twice to obtain first enhanced point cloud data and second enhanced point cloud data;

[0025] The encoding module is used to use the first enhanced point cloud data and the second enhanced point cloud data as inputs of the encoding model respectively, and obtain the first point cloud feature and the second point cloud feature through the features output by the encoding model. The encoding model is used to encode the input point cloud, and the first point cloud feature is obtained by performing variation on the features output by the encoding model for the first enhanced point cloud data, and the second point cloud feature is obtained by performing variation on the features output by the encoding model for the second enhanced point cloud data;

[0026] The training module is used to update the encoding model according to the first point cloud feature, the second point cloud feature and the first similarity between the first point cloud feature and the second point cloud feature to obtain an updated encoding model.

[0027] In a possible implementation manner, the above device may further include:

[0028] A decoding module, used for obtaining first generated point cloud data corresponding to the first point cloud feature and second generated point cloud data corresponding to the second point cloud feature through a decoding model;

[0029] The training module is also used to update the encoding model according to the first generated point cloud data, the second generated point cloud data and the first similarity to obtain an updated encoding model.

[0030] In a possible implementation, the above-mentioned training module can be specifically used to: calculate a first loss value based on the first generated point cloud data, the first enhanced point cloud data, the second generated point cloud data and the second enhanced point cloud data; obtain a second loss value based on the first similarity; update the encoding model based on the first loss value and the second loss value to obtain an updated encoding model.

[0031] In a possible implementation, the above-mentioned training module is specifically used to calculate the first loss value based on at least one of the second similarity, the third similarity, the fourth similarity or the fifth similarity, the second similarity includes the similarity between the first generated point cloud data and the second enhanced point cloud data, the third similarity includes the similarity between the second generated point cloud data and the first enhanced point cloud data, the fourth similarity includes the similarity between the first generated point cloud data and the first enhanced point cloud data, and the fifth similarity includes the similarity between the second generated point cloud data and the second enhanced point cloud data.

[0032] In a possible implementation, the training module is further used to update the decoding model to obtain an updated decoding model.

[0033] In a possible implementation, the above-mentioned encoding module is specifically used to: use the first enhanced point cloud data and the second enhanced point cloud data as inputs of the encoding model respectively, and output a first feature corresponding to the first enhanced point cloud data and a second feature corresponding to the second enhanced point cloud data; constrain the first feature and the second feature to obey the corresponding Gaussian distribution respectively, to obtain the first point cloud feature and the second point cloud feature.

[0034] In a possible implementation, the encoding module is specifically used to: obtain the distance between the distribution obeyed by the first feature and the distribution obeyed by the second feature; and map the first feature and the second feature into the space respectively according to the distance between the distribution obeyed by the first feature and the distribution obeyed by the second feature to obtain a first point cloud feature and a second point cloud feature.

[0035] In a possible implementation, the features output by the updated encoding model are used to construct a map, a classification task, a segmentation task, or a point cloud registration.

[0036] In a fourth aspect, the present application provides a point cloud representation device, comprising:

[0037] Input module, used to obtain input point cloud data;

[0038] The encoding module is used to take the input point cloud data as the input of the encoding model, and obtain the point cloud features corresponding to the input point cloud data according to the features output by the encoding model.

[0039] In a possible implementation, the features output by the encoding model are used to construct a map, a classification task, a segmentation task, or point cloud registration.

[0040] The training process of the encoding model can refer to the description of the first aspect or any optional implementation manner of the first aspect, and will not be repeated here.

[0041] The effects achieved by any optional implementation of the third and fourth aspects can be found in the description of any optional implementation of the first or second aspects, and will not be repeated here.

[0042] In a fifth aspect, an embodiment of the present application provides a model training device, comprising: a processor and a memory, wherein the processor and the memory are interconnected via a line, and the processor calls the program code in the memory to execute the processing-related functions in any of the methods shown in the first aspect above. Optionally, the model training device can be a chip.

[0043] In the sixth aspect, an embodiment of the present application provides a training device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform functions related to processing in the above-mentioned first aspect or any optional embodiment of the first aspect.

[0044] In a seventh aspect, an embodiment of the present application provides a point cloud representation device, comprising: a processor and a memory, wherein the processor and the memory are interconnected via a line, and the processor calls a program code in the memory to execute the processing-related functions in the second aspect or any optional implementation of the second aspect. Optionally, the point cloud representation device may be a chip.

[0045] In the eighth aspect, an embodiment of the present application provides a point cloud representation device, which may also be referred to as a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform functions related to processing as described in the second aspect or any optional embodiment of the second aspect.

[0046] In a ninth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enables the computer to execute a method in any optional implementation of the first aspect or the second aspect above.

[0047] In the tenth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute a method in any optional implementation of the first aspect or the second aspect above.

[0048] In the eleventh aspect, an embodiment of the present application provides a vehicle, which includes a point cloud representation device as described in the seventh aspect or the eighth aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of a system architecture provided for this application;

[0050] Figure 2 Another system architecture diagram provided for this application;

[0051] Figure 3 A flowchart of a model training method provided in this application;

[0052] Figure 4 A flowchart of another model training method provided for this application;

[0053] Figure 5 A flowchart of another model training method provided for this application;

[0054] Figure 6 A schematic diagram of a point cloud representation method provided in this application;

[0055] Figure 7 A schematic diagram of the structure of a model training device provided in this application;

[0056] Figure 8 A schematic diagram of the structure of a point cloud representation device provided in this application;

[0057] Fig. 9 A schematic diagram of the structure of another model training device provided in this application;

[0058] Fig.10 A schematic diagram of the structure of another point cloud representation device provided by the present application;

[0059] Fig.11 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0061] The method provided in this application can be applied to artificial intelligence (AI) scenarios. AI is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robots, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

[0062] First, the overall workflow of the artificial intelligence system is described. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensed process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecological process of the system.

[0063] (1) Infrastructure

[0064] The infrastructure provides computing power support for the AI ​​system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0065] (2) Data

[0066] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0067] (3) Data processing

[0068] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0069] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0070] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0071] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0072] (4) General ability

[0073] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0074] (5) Smart products and industry applications

[0075] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0076] The embodiments of the present application involve related applications of neural networks. In order to better understand the solutions of the embodiments of the present application, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0077] (1) Neural Network

[0078] A neural network may be composed of neural units. A neural unit may refer to an operation unit with xs and intercept 1 as input. The output of the operation unit may be as shown in formula (1-1):

[0079]

[0080] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For x s , b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0081] (2) Deep Neural Networks

[0082] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple intermediate layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, intermediate layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all intermediate layers, or hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

[0083] Although DNN looks complicated, each layer can be expressed as a linear relationship expression: in, is the input vector, is the output vector, is the offset vector or bias parameter, w is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Taking the coefficient w as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0084] In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as

[0085] It should be noted that the input layer does not have a W parameter. In a deep neural network, more intermediate layers allow the network to better characterize complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (a weight matrix formed by many layers of vectors W).

[0086] (3) Convolutional Neural Network

[0087] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract features is independent of position. Convolution kernels can be formalized as matrices of random size, and convolution kernels can obtain reasonable weights through learning during the training process of convolutional neural networks. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0088] (4) Graph Convolutional Network (GCN)

[0089] Graph neural network is a deep learning model that models and processes non-Euclidean spatial data (such as graph data). Its principle is to use pairwise message passing so that graph nodes iteratively update their corresponding representations by exchanging information with their neighbors.

[0090] GCN is similar to CNN, but the difference is that the input of CNN is usually two-dimensional structured data, while the input of GCN is usually graph structured data. GCN has cleverly designed a method to extract features from graph data, so that these features can be used to perform node classification, graph classification, link prediction, and graph embedding.

[0091] (5) Loss Function

[0092] In the process of training deep neural networks, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually a process of optimization before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict lower, and continue to adjust until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible. The loss function can usually include loss functions such as squared error, cross entropy, logarithm, exponential, etc. For example, the squared error can be used as a loss function, defined as The specific loss function can be selected according to the actual application scenario.

[0093] (6) Back propagation algorithm

[0094] An algorithm that calculates the gradient of model parameters based on the loss function and updates the model parameters. The neural network can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0095] (7) Supervised Learning

[0096] It can also be called supervised learning. It is a training method / learning method in machine learning. It refers to the process of training a model by letting the machine learn a large amount of labeled sample data, so that the model can get the corresponding output based on the input.

[0097] Usually, supervised learning requires a large amount of labeled data, that is, data with labels. The more labeled data there is, the better the machine learning algorithm will be, and the more capable it will be of selecting the correct answer from the input data. It can be understood that supervised learning is about finding the relationship between input and output. For example, input an image or text, and output classification features or segmentation features.

[0098] (8) Unsupervised Learning

[0099] Unsupervised learning is a training method / learning method in machine learning. It can be understood as letting the model learn by itself, with its input being samples without labels, and learning the relationship between input data based on clustering or other methods.

[0100] Unsupervised learning is often used in data mining to learn the relationship between data in a large amount of unlabeled data. The training data is unlabeled, and the training goal is to classify or distinguish the observed values. For example, unsupervised learning should be able to distinguish "cat" pictures from a large number of various pictures based on the characteristics of all "cat" pictures without any additional prompts.

[0101] Compared with supervised learning, the input data in the training process of unsupervised learning does not need to carry labels, can be applied to more complex tasks, and can be used to learn deeper relationships between data.

[0102] (9) Self-supervised learning

[0103] It can also be called contrastive learning, which constructs a proxy task (pretext task). Self-supervised learning uses these pretext tasks to enable the feature extractor (encoder) based on deep learning to have the ability to encode the deep features of the point cloud without label information. This makes the output of the encoder effective in downstream tasks such as classification and segmentation.

[0104] Contrastive learning is a special self-supervised learning scheme. It is different from the encoding-decoding scheme of generative learning. It focuses on learning the common features between instances of the same category and distinguishing the differences between instances of different categories. The core of contrastive learning is differentiation. Its encoder is better at learning holistic and discriminative features.

[0105] Unsupervised learning or self-supervised learning has achieved remarkable results in the fields of text, images, etc. The method provided in this application can apply unsupervised learning to 3D vision. In the process of processing point cloud data, contrastive learning is used to train the encoding used to extract point cloud features.

[0106] The method provided in the embodiment of the present application can be executed on a server or on a terminal device. The terminal device can be a mobile phone with image processing function, a tablet personal computer (TPC), a media player, a smart TV, a laptop computer (LC), a personal digital assistant (PDA), a personal computer (PC), a camera, a video camera, a smart watch, a wearable device (WD) or an autonomous driving vehicle, etc., and the embodiment of the present application does not limit this.

[0107] The following introduces the system architecture provided by the embodiments of the present application.

[0108] See also Figure 1 , the embodiment of the present application provides a system architecture 200. As shown in the system architecture 200, the data acquisition device 260 can be used to collect training data. After the data acquisition device 260 collects the training data, the training data is stored in the database 230, and the training device 220 trains the target model / rule 201 based on the training data maintained in the database 230. The target model can be the encoding model mentioned in the present application.

[0109] The following describes how the training device 220 obtains the target model / rule 201 based on the training data. Exemplarily, the training device 220 implements self-supervised learning based on the data, thereby completing the training of the target model / rule 201. For a detailed description, please refer to the training method below.

[0110] The target model / rule 201 in the embodiment of the present application can specifically be a neural network. It should be noted that in actual applications, the training data maintained in the database 230 does not necessarily all come from the collection of the data acquisition device 260, and may also be received from other devices. It should also be noted that the training device 220 does not necessarily train the target model / rule 201 entirely based on the training data maintained by the database 230, and may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0111] The target model / rule 201 trained by the training device 220 can be applied to different systems or devices, such as Figure 2The execution device 210 shown in the figure may be a terminal, such as a smart car, a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR), a vehicle terminal, a television, etc., or a server or a cloud. Figure 2 In the embodiment, the execution device 210 is configured with a transceiver 212, which may include an input / output (I / O) interface or other wireless or wired communication interface, etc., for interacting with external devices. Taking the I / O interface as an example, a user may input data to the I / O interface through the client device 240.

[0112] When the execution device 210 preprocesses the input data, or when the computing module 212 of the execution device 210 performs calculations and other related processing, the execution device 210 can call the data, code, etc. in the data storage system 250 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 250.

[0113] Finally, the I / O interface 212 returns the processing result to the client device 240 so as to provide it to the user.

[0114] It is worth noting that the training device 220 can generate corresponding target models / rules 201 based on different training data for different goals or different tasks. The corresponding target models / rules 201 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0115] In the attached Figure 1 In the case shown in the figure, the user can manually give the input data, and the manual giving can be operated through the interface provided by the transceiver 212. In another case, the client device 240 can automatically send the input data to the transceiver 212. If the client device 240 is required to automatically send the input data, it is necessary to obtain the user's authorization, and the user can set the corresponding authority in the client device 240. The user can view the results output by the execution device 210 on the client device 240, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 240 can also be used as a data acquisition terminal to collect the input data of the input transceiver 212 and the output result of the output transceiver 212 as shown in the figure as new sample data, and store them in the database 230. Of course, it is also possible not to collect through the client device 240, but the transceiver 212 directly stores the input data of the input transceiver 212 and the output result of the output transceiver 212 as new sample data in the database 230.

[0116] It is worth noting that Figure 2This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 2 In the embodiment, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 can also be placed in the execution device 210.

[0117] like Figure 1 As shown, the target model / rule 201 is obtained through training by the training device 220. The target model / rule 201 in the embodiment of the present application may be a trained or updated coding model in the present application.

[0118] For example, for a scenario where the method provided by the present application is applied to a cloud or server, the system architecture of the application of the method provided by the present application can be as follows: Figure 2 In the system architecture 300, the server cluster 310 is implemented by one or more servers, and optionally cooperates with other computing devices, such as data storage, routers, etc. The server cluster 310 can use the data in the data storage system 250, or call the program code in the data storage system 250 to implement the steps of the method provided in the present application.

[0119] Users can operate their respective user devices (e.g., local device 101 and local device 102) to interact with server cluster 310. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0120] The local device of each user can interact with the server cluster 310 through a communication network of any communication mechanism / communication standard, and the communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof. Specifically, the communication network may include a wireless network, a wired network, or a combination of a wireless network and a wired network. The wireless network includes, but is not limited to: a fifth generation mobile communication technology (5th-Generation, 5G) system, a long term evolution (long termevolution, LTE) system, a global system for mobile communication (global system for mobile communication, GSM) or a code division multiple access (code division multiple access, CDMA) network, a wideband code division multiple access (wideband code division multiple access, WCDMA) network, wireless fidelity (wireless fidelity, WiFi), Bluetooth (bluetooth), Zigbee protocol (Zigbee), radio frequency identification technology (radio frequency identification, RFID), long range (Lora) wireless communication, near field communication (near field communication, NFC) in any one or more combinations. The wired network may include a fiber optic communication network or a network composed of a coaxial cable, etc.

[0121] In another implementation, one aspect or multiple aspects of the execution device 210 may be implemented by each local device. For example, the local device 101 may provide local data or feedback calculation results to the execution device 210 .

[0122] It should be noted that all functions of the execution device 210 may also be implemented by the local device. For example, the local device 101 implements the functions of the execution device 210 and provides services to its own user, or provides services to the user of the local device 102.

[0123] The method provided in this application can be applied to the field of 3D vision, which has a wide range of application scenarios in multiple fields such as autonomous driving, robotics, and augmented reality. Point cloud is a commonly used data structure that provides a concise way to represent 3D objects, which facilitates the development of 3D vision. Nowadays, point cloud data can be easily obtained through radar or depth cameras, so algorithms for point cloud data processing have become an important field of 3D vision research. In the past, a variety of artificial descriptors were designed using some geometric features to achieve the encoding of point cloud data. However, these methods require the introduction of a large amount of prior knowledge and can only be applied to some specific environments, with limited generalization and flexibility. In order to solve these problems, deep learning is introduced into the representation of point cloud data, and good results have been achieved in classification, segmentation, target detection and other fields. However, these deep learning-based methods often rely on a large amount of labeled data, which requires a lot of manpower costs to mark each sample.

[0124] For example, there are some existing point cloud self-supervised learning schemes based on autoencoders. Its basic principle is to occlude points that are not visible in the camera view, train an autoencoder to reconstruct the occluded points, and pre-train the encoder to have more powerful 3D encoding capabilities. Specifically, all occluded points can be masked in the camera view, such as taking any point cloud dataset, and generating occluded objects for each input in the following way: (a) randomly sampling the camera viewpoint, (b) deleting the points hidden in the viewpoint. Train an encoder-decoder model to complete the covered point cloud (the encoder can be any model that learns point cloud representation, and the decoder can be any model). Use the learned encoder weights as the initialization of any downstream task. However, the autoencoder method is easy to learn the common features of 3D objects, but it is difficult to learn the difference features between different categories; and the fixed-value feature vector is used to express 3D samples, and the migration ability between virtual data and real data is limited.

[0125] For another example, there is an existing image-point cloud cross-modal self-supervision scheme based on contrastive learning. Its basic principle is: contrastive learning constructs consistency between features of samples of the same category in 3D point clouds, while making features of samples of different categories tend to be far away. The introduction of 2D images constructs cross-modal contrast, making point cloud samples and their corresponding image samples tend to be consistent. It includes two branches, namely: the point cloud branch, which establishes intra-modal correspondence by applying invariance to point cloud enhancement; the image branch, which simply formulates cross-modal correspondence by introducing contrast loss between rendered 2D image features and point cloud prototype features. The model is jointly trained by combining the learning objectives of the two branches. During the reasoning process, the image branch is discarded, and only the point cloud feature extractor is used as the backbone of the downstream task.

[0126] However, the data distribution between different modalities varies greatly, which results in limited help for pre-training feature extractors for image-based point clouds. Using fixed-value feature vectors to represent 3D samples has limited ability to migrate between virtual data and real data.

[0127] Therefore, at present, the self-supervised learning scheme applied to 2D images relies on a large amount of real sample data, and the cost required for the collection and processing of 3D samples is much greater than that of 2D images. Therefore, the current mainstream solution is to train on virtual data and migrate the model to real data for use. However, there are large differences between virtual data and real data, and the traditional scheme has limited migration capabilities between virtual data and real data. Therefore, the present application is mainly aimed at improving the generalization ability of 3D self-supervised learning, so that the model trained on virtual data still maintains good results on real data. The method and application scenarios provided by the present application are introduced below in conjunction with the accompanying drawings.

[0128] First, the method provided in this application can be applied to a variety of systems that use point cloud feature extractors, including but not limited to point cloud target classification, point cloud segmentation, and point cloud registration. The point cloud feature extractors of these systems can be pre-trained to improve the accuracy of the above tasks. Based on this, the method provided in this application can be applied to scenarios such as intelligent driving, robotics, gesture operations, etc. that require map construction, classification tasks, segmentation tasks, positioning, or action recognition.

[0129] Scenario 1: Intelligent Driving

[0130] Intelligent driving can include driving scenarios such as automatic driving and assisted driving. Taking automatic driving as an example, automatic driving is to make decisions for the driving operation of the vehicle, such as moving forward, stopping or changing lanes, by collecting environmental information around the vehicle. In the automatic driving scenario, it is usually necessary to collect environmental information around the vehicle and realize very accurate recognition of the environmental information, so as to make decisions on vehicle operations based on more accurate recognition results, and realize safer automatic driving. For example, sensors such as laser radar, millimeter wave radar or depth camera are usually provided in the vehicle, and the data collected by them are usually point cloud data. Through the method provided in this application, a more accurate coding model can be learned to output, and through a variation-based self-supervised learning method, the coding model can learn the characteristic distribution of the point cloud and the deep information of the point cloud, so as to output more accurate point cloud target classification, point cloud segmentation or point cloud registration results, etc., which can be used for downstream tasks such as mapping and positioning, thereby improving the accuracy of downstream tasks and improving the safety of vehicle automatic driving.

[0131] Scene 2: Robot

[0132] With the advancement of technology, robots are used in more commercial or household scenarios. Such as sweeping robots, delivery robots, etc. Usually, during the operation of the robot, it is necessary to combine the environmental information of the robot to decide the next action of the robot, such as moving forward, back legs, turning the camera, etc. In some scenarios, point cloud data can be collected by radar or depth camera. In this scenario, the encoding model can be trained by the method provided in this application to extract features from the point cloud data, so that further downstream tasks such as mapping and positioning can be performed based on the improved features, thereby improving the accuracy of downstream tasks and making the robot's action decisions more adapted to the environment.

[0133] The following is an introduction to the method flow provided in this application in combination with the aforementioned system architecture and application scenarios.

[0134] The present application provides a model training method and a point cloud method. It can be understood that the model training method and the point cloud method provided by the present application can be respectively referred to as a training phase and an inference phase. The training phase is the model training method provided by the present application, and the inference phase is the process of the point cloud representation method provided by the present application.

[0135] 1. Training Phase

[0136] See also Figure 3 , a flow chart of a model training method provided in this application is as follows.

[0137] 301. Obtain point cloud data.

[0138] The point cloud data may include data formed by a plurality of points. The point cloud data may be collected by a camera, may be collected by a laser radar, or may be data read from stored data.

[0139] For example, the point cloud data may include a plurality of points distributed in space, each point having corresponding coordinates and corresponding data. If the point cloud data is data collected by a depth camera, each point has corresponding coordinates and a depth value.

[0140] 302. Enhance the point cloud data at least twice to obtain first enhanced point cloud data and second enhanced point cloud data.

[0141] After obtaining the point cloud data, the points in the point cloud data can be enhanced at least twice. This application takes two different enhancement methods as an example to obtain first enhanced point cloud data and second enhanced point cloud data, and the first enhanced point cloud data is different from the second enhanced point cloud data. Of course, in some scenarios, if more enhancements are performed, third enhanced point cloud data or fourth enhanced point cloud data can also be obtained. This application only takes two different enhancement methods as an example for exemplary introduction, and is not intended to be limiting.

[0142] Specific data enhancement methods may include but are not limited to translation, rotation, masking or a combination of enhancement methods. The aforementioned different enhancement methods may include different enhancement categories, such as different enhancement categories such as translation and rotation, etc., or the same category but with different enhancement values, such as different translation distances, different rotation angles, or different mask areas; of course, different combinations of enhancement methods, or combinations of the aforementioned multiple different categories or different enhancement values, etc., may be used. The appropriate enhancement method may be selected based on the actual application scenario.

[0143] 303. Use the first enhanced point cloud data and the second enhanced point cloud data as inputs of the encoding model respectively, and obtain the first point cloud feature and the second point cloud feature through the features output by the encoding model.

[0144] After obtaining the first enhanced point cloud data and the second enhanced point cloud data, the first enhanced point cloud data and the second enhanced point cloud data are respectively used as inputs of the encoding model, and the first point cloud features and the second point cloud features are obtained according to the features output by the encoding model. For example, the features output by the encoding model can be varied to obtain the first point cloud features and the second point cloud features. Specifically, the features output by the encoding model for the first enhanced point cloud data can be varied to obtain the first point cloud features, and the features output by the encoding model for the second enhanced point cloud data can be varied to obtain the second point cloud features.

[0145] Usually, the data obtained by the variational operation obeys a certain regular distribution, such as Gaussian distribution. Therefore, the features obtained by performing variational operations on the features output by the encoding model usually also obey Gaussian distribution, that is, the first point cloud features and the second point cloud features obtained usually obey Gaussian distribution. Of course, in addition to Gaussian distribution, other data distribution laws can also be replaced, such as t-distribution, chi-square distribution, etc. This application takes Gaussian distribution as an example for illustrative introduction, and it is not limited to it.

[0146] In a possible implementation, the first enhanced point cloud data and the second enhanced point cloud data can be used as inputs of the encoding model, respectively, to output a first feature corresponding to the first enhanced point cloud data and a second feature corresponding to the second enhanced point cloud data; the first feature and the second feature are respectively constrained to obey the corresponding Gaussian distribution to obtain the first point cloud feature and the second point cloud feature. Therefore, in the implementation of the present application, the features output by the encoding model can be constrained so that the obtained features conform to the Gaussian distribution, thereby using features that obey the Gaussian distribution to represent the potential features of the point cloud data, avoiding overfitting on a single data domain and improving the migration ability of the model.

[0147] Optionally, the distance between the distribution obeyed by the first feature and the distribution obeyed by the second feature can be obtained, and the distance can be used to represent the similarity between the distribution obeyed by the first feature and the distribution obeyed by the second feature; then, according to the distance between the distribution obeyed by the first feature and the distribution obeyed by the second feature, the first feature and the second feature are respectively mapped to the space, such as being mapped to a space represented by a distance of [0, 1], to obtain a first point cloud feature and a second point cloud feature, and the first point cloud feature and the second point cloud feature respectively obey a Gaussian distribution with a mean of μ and a standard deviation of σ. Accordingly, when setting the optimization target for comparative learning later, it can be set to make the distance between the features obeying the same distribution of the features output by the encoding model as close as possible, and the distance between the features obeying different distributions is farther, thereby achieving more accurate classification of positive samples (samples corresponding to features of the same category) and negative samples (samples corresponding to features of different categories).

[0148] 304. Update the encoding model according to the first point cloud feature, the second point cloud feature, and the first similarity between the first point cloud feature and the second point cloud feature to obtain an updated encoding model.

[0149] After obtaining the first point cloud feature and the second point cloud feature, the similarity between the first point cloud feature and the second point cloud feature may be calculated, which is referred to herein as the first similarity for ease of distinction.

[0150] That is, the encoding model can be updated based on the first point cloud feature, the second point cloud feature and the first similarity to obtain an updated encoding model.

[0151] Therefore, in the implementation mode of the present application, a variational operation can be performed on the features output by the encoding model to obtain features that conform to a certain distribution law. Even if there are large differences between virtual data and real data, the potential features extracted from the point cloud data can be represented by making the features obey a certain distribution, thereby enabling the model to have a strong migration ability in different data domains.

[0152] In a possible implementation, the decoding model may be combined to perform comparative learning to implement the training of the encoding model.

[0153] Optionally, the first point cloud feature and the second point cloud feature can be used as inputs of the decoding model, and the first generated point cloud data corresponding to the first point cloud feature and the second generated point cloud data corresponding to the second point cloud feature can be obtained through the decoding model. The encoding model is updated according to the first generated point cloud data, the second generated point cloud data and the first similarity to obtain an updated encoding model. Therefore, in the implementation of the present application, comparative learning can be performed in combination with the output results of the decoding model, so that the model can converge faster.

[0154] Optionally, a first loss value may be calculated based on the first generated point cloud data, the first enhanced point cloud data, the second generated point cloud data, and the second enhanced point cloud data; a second loss value may be obtained based on the first similarity; and the encoding model may be updated based on the first loss value and the second loss value to obtain an updated encoding model. Therefore, in the implementation of the present application, the point cloud output by the encoding model may be combined to implement self-supervised learning to obtain an encoding model with stronger migration capability.

[0155] Optionally, after the first generated point cloud data and the second generated point cloud data are output through the decoding model, the first generated point cloud data and the second generated point cloud data and the corresponding first enhanced point cloud data and the second enhanced point cloud data can be used as supervision data to calculate the loss value, and the encoding model can be updated based on the loss value, thereby realizing self-supervised learning and obtaining an updated encoding model.

[0156] Optionally, the loss value can be calculated based on the similarity between the generated point cloud data and the enhanced point cloud data. When calculating the similarity, cross-supervision can be performed between the generated point cloud data and the enhanced point cloud data, so that the encoder retains the unchanged features by cross-sampling and reconstructing the features, while promoting the consistency of feature distribution between positive samples. Self-supervision can also be performed to achieve model updates from multiple dimensions.

[0157] For example, the first loss value can be calculated based on at least one of the second similarity, the third similarity, the fourth similarity, or the fifth similarity, wherein the second similarity is the similarity between the first generated point cloud data and the second enhanced point cloud data, the third similarity is the similarity between the second generated point cloud data and the first enhanced point cloud data, the fourth similarity includes the similarity between the first generated point cloud data and the first enhanced point cloud data, and the fifth similarity includes the similarity between the second generated point cloud data and the second enhanced point cloud data. The second similarity and the third similarity can be understood as being used for cross-supervision, and the fourth similarity and the fifth similarity can be understood as being used for self-supervision.

[0158] In addition, it should be noted that the similarity between the features mentioned in this application, such as the first similarity, the second similarity, the third similarity, the fourth similarity or the fifth similarity, etc., can be specifically represented or measured by cosine similarity, KL divergence or JS divergence, etc., and can also be represented and measured in other ways. In addition, during the iterative update process of the encoding model, the decoding model can also be updated synchronously to obtain an updated decoding model. Thereby, during the output process of the encoding model, the output of the decoding model is also more accurate. Thereby, based on the more accurate output of the decoding model updated each time, a more accurate loss value can be calculated, thereby improving the model convergence speed.

[0159] The foregoing describes the method flow of the training phase provided by the present application. The following describes the method flow of the training phase provided by the present application in more detail in combination with specific application scenarios.

[0160] For example, the architecture of the training phase in the method provided in this application can be as follows: Figure 4 shown.

[0161] After obtaining the point cloud data, two data enhancement methods are used to enhance the points in the point cloud data as an example. Figure 4 As shown in , the point cloud data is enhanced twice in different ways to obtain enhanced point cloud data 1 (ie, first enhanced point cloud data) and enhanced point cloud data 2 (ie, second enhanced point cloud data).

[0162] The enhanced point cloud data 1 and the enhanced point cloud data 2 are respectively used as the input of the variational contrast module, and the variational contrast module includes a coding model. The variational contrast module can constrain the distribution of the features of the enhanced point cloud data 1 and the enhanced point cloud data 2, and output point cloud features 1 (i.e., the first point cloud features) and point cloud features 2 (i.e., the second point cloud features) that obey the Gaussian distribution. The point cloud features 1 and point cloud features 2 that obey the Gaussian distribution are used as the input of the generation model, and the generated point cloud data 1 (i.e., the first generated point cloud data) and the generated point cloud data 2 (i.e., the second generated point cloud data) are output. The generation model can also be called a decoding model or a decoder, etc.

[0163] Subsequently, cross-supervision is performed based on enhanced point cloud data 1, enhanced point cloud data 2, generated point cloud data 1 and generated point cloud data 2 to achieve the update of the encoding model.

[0164] Combine the following Figure 4 The architecture shown provides a more detailed introduction to the training phase of the method provided in this application.

[0165] See also Figure 5 , a flow chart of another model training method provided in this application is described as follows.

[0166] The training phase provided in this application can be divided into multiple modules, such as Figure 5 The input module, variational contrast module and generative cross-supervision module shown in are introduced below respectively.

[0167] 1. Input module

[0168] First, the input module obtains the point cloud data to be represented and enhances the points in the point cloud data.

[0169] For example, point cloud data can be expressed as: P∈RN*3, and two different data enhancements are applied to it, such as rotation, translation, masking or a combination thereof, to obtain enhanced point cloud data, which are expressed as p and q respectively.

[0170] 2. Variational Contrast Module

[0171] For the point cloud data p and q obtained after data enhancement, they can be used as the input of a shared weight encoder E(.), and encoded by the encoder E(.), that is, the encoding model, to obtain the features fp and fq.

[0172] Next, the features generated by E(.) are constrained to obey Gaussian distribution. Specifically, the features fp and fq can be projected into space, such as projecting the features fp and fq into a high-dimensional latent space. Among them, each input feature can obtain two vectors μp, μq, σp and σq respectively. For example, fp and fq can be projected into a high-dimensional latent space through a fully connected layer. For example, two single-layer fully connected layers fcμ(.) and fcσ(.) can be used to project the features fp and fq into a high-dimensional latent space, and each sample can obtain two vectors μp, μq, σp and σq respectively.

[0173] Specifically, in order to constrain samples of the same category (also called positive samples) to similar Gaussian distributions, and samples of different categories (also called negative samples) to distributions as far as possible, the distance between two point cloud features can be calculated, and when mapping features, the distance between the two features can be mapped to between 0 and 1.

[0174] For example, KL divergence or JS divergence can be used to measure the similarity between samples, and the distance between point cloud features can be represented by similarity.

[0175] For example, KL divergence can be expressed as:

[0176]

[0177] JS divergence is an algorithm for calculating similarity based on KL divergence. Usually, KL divergence is not actually a symmetric function. When the input order of the two distributions p(x) and q(x) is different, the calculation results are also different. In the implementation of the present application, it is necessary to calculate the similarity of the distribution between the positive samples generated by two random data enhancements. This similarity should be order-independent. Therefore, JS divergence can also be used to calculate the similarity between point cloud data, which can be expressed as:

[0178]

[0179] Therefore, in order to make the two point cloud features of the final output obey Gaussian distribution, the distance between samples can be mapped to between 0 and 1. Generally, the smaller the distance value, the higher the similarity between the two samples. For example, the two point cloud features can be made to obey Gaussian distribution with mean μ and standard deviation σ, as expressed as:

[0180]

[0181] Therefore, for the variational contrast module, the optimization objective can be expressed as:

[0182]

[0183] That is, the distance between the distributions obeyed by positive samples is as small as possible, while the distance between the distributions obeyed by negative samples is as large as possible.

[0184] Accordingly, when setting the loss function for the variational contrast module, it can be set based on the optimization objective.

[0185] As can be seen from the definition of KL divergence:

[0186]

[0187] Where p(x) and q(x) obey multidimensional Gaussian distribution, so the first term of the above formula can be expressed as:

[0188] ∫p(x)lnp(x)dx=∫ln N(x;μ p ,σ p 2 )ln N(x;μ p ,σ p 2 )dx

[0189] Among them, μ p and σ p are the mean and standard deviation of the distribution p(x) respectively. From the variational approximation of the Gaussian distribution, we can get:

[0190]

[0191] Where N is the dimension of the vectors μ and σ. For two independent Gaussian distributions p(x) and q(x), it can be proved that the linear combination of p(x) and q(x) still obeys the Gaussian distribution:

[0192]

[0193] Therefore, we can get:

[0194]

[0195] The JS divergence of two multidimensional Gaussian distributions can be described by their mean and standard deviation:

[0196]

[0197] Since the distance is inversely proportional to the similarity, -DJS can be used to describe the similarity between Gaussian distributions. Therefore, the loss function set for the variational contrast module can be expressed as:

[0198]

[0199] Where BS represents the batch size and τ is the temperature hyperparameter.

[0200] 3. Generate cross-supervision module

[0201] Based on variational contrastive learning, this application introduces a generative supervision signal to enable the encoder to focus on fine-grained features and avoid losing too many details under the contrast loss constraint.

[0202] First, sample the generated vector g from p(x) and q(x) p With g q In the sampling process, in order to avoid the problem of non-differentiability caused by direct sampling, the present application can adopt a re-parameterization solution to solve this sampling problem, such as:

[0203] g=μ+σ∈

[0204] Among them, ε~N(0,1), it can be proved that g~N(μ,σ 2 ).

[0205] g p With g q is fed into a decoder D(.) with shared parameters. Exemplarily, the decoder D(.) can also be called a generative model or a decoding model, which can include a three-layer fully connected network and a batch normalization layer. The sampled features are decoded to obtain new point cloud data p gen With q gen ∈R N*3 .

[0206] For the generated cross-supervision module, when setting the optimization target, you can use your own signal to supervise the generated point cloud, or you can use cross-supervision to supervise the generated point cloud.

[0207] For example, we can use the traditional self-supervision method to supervise p gen and q supervision q gen That is, when calculating the loss function, we can use p and p gen The similarity between q and q genThe similarity between them is used to calculate the loss function value, thereby achieving self-supervised learning.

[0208] For example, we can use cross-supervision to use p to supervise q gen and q supervises p gen That is, when calculating the loss function, we can use p and q gen The similarity between q and p gen The loss function value is calculated by the similarity between them, thereby realizing cross-supervised learning. This application can increase the difficulty of training through cross-supervised learning, so that the model focuses on the local features of the point cloud rather than the global features; and because g p With g q It is sampled from two different distributions p(x) and q(x), so the cross-supervision approach is also conducive to the two distributions becoming closer, which is the optimization goal of contrastive learning.

[0209] Accordingly, for generating the cross-supervision module, the optimization objective set can be expressed as:

[0210] max(sim(p gen ,q)+sim(q gen ,p))

[0211] Even if the two distributions are close.

[0212] Therefore, for updating the encoding model and the decoding model, cross chamfer loss can be used for training.

[0213] If it can be expressed as:

[0214]

[0215] Loss = Chamfer(p gen ,q)+Chamfer(q gen ,p)+DC(p,q)

[0216] The model training method provided in this application provides a variational contrastive learning method for self-supervised point cloud representation. Different from traditional contrastive learning, variational contrastive learning uses features that obey Gaussian distribution instead of fixed feature embedding to express potential features, avoiding overfitting in a single data domain, improving the migration ability of the model, improving the performance of the model in actual scenarios, and improving the feasibility and feasibility of the solution.

[0217] And the idea of ​​generative cross-supervision is integrated into contrastive learning, and the method of sampling and reconstructing features and performing cross-supervision is used to make the coding model learn to match the retained invariant features, while promoting the consistency of feature distribution between positive samples. By incorporating the idea of ​​generative cross-supervision, the deep features of the network are extracted and the generalization performance of the network is improved; it has been tested for a variety of point cloud tasks and has been proven to have strong migration capabilities and generalization. And for the provided variational contrastive learning, DCLoss (that is, the aforementioned loss function) is set to constrain the distance between the positive and negative sample distributions. The coding model obtained by the method provided in this application can extract high-level semantic features of the point cloud, design corresponding loss functions, constrain the distance between the positive and negative sample distributions, improve the representation capability of the deep features of the point cloud data, and thereby improve the performance accuracy on a variety of tasks.

[0218] Taking the data collected from actual application scenarios as an example, the accuracy of point cloud configuration in the actual collected data was improved from 98.5% to 98.75%. And it was tested on multiple public data and different tasks, and excellent results were achieved. For example, in the linear classification task of the ScanObjectNN dataset, the accuracy reached 87.4%, which is 5.7% higher than that of CrossPoint (SOTA); in the small sample linear classification task, the accuracy of the ModelNet40 dataset was 92.4%, which was 2.7% higher than that of CrossPoint (SOTA); the accuracy of the ScanObjectNN dataset was 83.6%, which was 10.9% higher than that of CrossPoint (SOTA). It can be seen that the encoding model trained by the method provided in this application has a strong data domain migration capability and has good effects on a variety of tasks.

[0219] 2. Reasoning Stage

[0220] In the aforementioned training phase, the encoding model and the decoding model are trained. The trained encoding model or variational contrast module can be deployed in application scenarios, such as in vehicles, terminals, servers, or cloud platforms.

[0221] See also Figure 6 , a flow chart of a point cloud representation method provided in this application is described as follows.

[0222] 601. Obtain input point cloud data.

[0223] The input point cloud data may be point cloud data collected by the device or data read from stored data. The device deployed by the method provided in this application may be equipped with sensors such as radar or depth camera, and the data structure collected is a point cloud structure, that is, it includes data of multiple points.

[0224] 602. Use the input point cloud data as input of the encoding model, and obtain point cloud features corresponding to the input point cloud data according to features output by the encoding model.

[0225] After the input point cloud data is obtained, the input point cloud data can be used as the input of the encoding model, and the point cloud features corresponding to the input point cloud are obtained according to the features of the output of the encoding model.

[0226] The training process of the coding model can refer to the aforementioned training stage, which will not be described in detail here. For example, the coding model deployed in the device can be as follows: Figure 5 The encoder E(.) shown in the figure and the fully connected layers fcμ(.) and fcσ(.) for projection, and the features output by the fully connected layers fcμ(.) and fcσ(.) can be constrained. Alternatively, it can also be understood as deploying the aforementioned variational contrast module in the device so that the final output features obey the Gaussian distribution. The specific process can be referred to in the aforementioned Figure 5 The detailed introduction of the variational contrast module is omitted here. The features output by the trained encoding model or the variational contrast module can be used for tasks such as map construction, classification tasks, segmentation tasks, or point cloud registration.

[0227] The point cloud feature may be a feature output by the encoding model, or may be obtained by performing variation on the feature output by the encoding model.

[0228] Therefore, in the implementation mode of the present application, a variational approach is used to describe point cloud features, that is, features that obey a certain distribution rule (such as Gaussian distribution) are used instead of fixed feature embedding to express potential features, thereby avoiding overfitting of the model on a single data and improving the performance of the model in actual scenarios.

[0229] The above is an introduction to the method flow provided by the present application. The following is an introduction to the device for executing the method flow in combination with the above method flow.

[0230] See also Figure 7 , a schematic diagram of the structure of a model training device provided in the present application, the model training device may include:

[0231] An acquisition module 701 is used to acquire point cloud data;

[0232] An enhancement module 702 is used to enhance the point cloud data at least twice to obtain first enhanced point cloud data and second enhanced point cloud data;

[0233] The encoding module 703 is used to use the first enhanced point cloud data and the second enhanced point cloud data as inputs of the encoding model respectively, and obtain the first point cloud feature and the second point cloud feature through the features output by the encoding model. The encoding model is used to encode the input point cloud, and the first point cloud feature is obtained by performing variation on the features output by the encoding model for the first enhanced point cloud data, and the second point cloud feature is obtained by performing variation on the features output by the encoding model for the second enhanced point cloud data;

[0234] The training module 704 is used to update the encoding model according to the first point cloud feature, the second point cloud feature and the first similarity between the first point cloud feature and the second point cloud feature to obtain an updated encoding model.

[0235] In a possible implementation manner, the above device may further include:

[0236] A decoding module 705, configured to obtain first generated point cloud data corresponding to the first point cloud feature and second generated point cloud data corresponding to the second point cloud feature through a decoding model;

[0237] The training module 704 is further used to update the encoding model according to the first generated point cloud data, the second generated point cloud data and the first similarity to obtain an updated encoding model.

[0238] In a possible implementation, the above-mentioned training module 704 can be specifically used to: calculate a first loss value based on the first generated point cloud data, the first enhanced point cloud data, the second generated point cloud data and the second enhanced point cloud data; obtain a second loss value based on the first similarity; update the encoding model based on the first loss value and the second loss value to obtain an updated encoding model.

[0239] In a possible implementation, the training module 704 is specifically used to calculate the first loss value based on at least one of the second similarity, the third similarity, the fourth similarity or the fifth similarity, where the second similarity includes the similarity between the first generated point cloud data and the second enhanced point cloud data, the third similarity includes the similarity between the second generated point cloud data and the first enhanced point cloud data, the fourth similarity includes the similarity between the first generated point cloud data and the first enhanced point cloud data, and the fifth similarity includes the similarity between the second generated point cloud data and the second enhanced point cloud data.

[0240] In a possible implementation, the training module 704 is further used to update the decoding model to obtain an updated decoding model.

[0241] In a possible implementation, the encoding module 703 is specifically used to: use the first enhanced point cloud data and the second enhanced point cloud data as inputs of the encoding model, respectively, and output a first feature corresponding to the first enhanced point cloud data and a second feature corresponding to the second enhanced point cloud data; respectively constrain the first feature and the second feature to obey the corresponding Gaussian distribution, to obtain a first point cloud feature and a second point cloud feature.

[0242] In a possible implementation, the encoding module 703 is specifically used to: obtain the distance between the distribution obeyed by the first feature and the distribution obeyed by the second feature; and map the first feature and the second feature into the space respectively according to the distance between the distribution obeyed by the first feature and the distribution obeyed by the second feature to obtain the first point cloud feature and the second point cloud feature.

[0243] In a possible implementation, the features output by the updated encoding model are used to construct a map, a classification task, a segmentation task, or a point cloud registration.

[0244] See also Figure 8 , the present application provides a schematic diagram of the structure of a point cloud representation device, the point cloud representation device comprising:

[0245] An input module 801 is used to obtain input point cloud data;

[0246] The encoding module 802 is used to use the input point cloud data as the input of the encoding model, and obtain the point cloud features corresponding to the input point cloud data according to the features output by the encoding model.

[0247] The update process of the coding model can be specifically referred to in the above Figure 3-Figure 5 The relevant description will not be repeated here.

[0248] In a possible implementation, the features output by the encoding model are used to construct a map, a classification task, a segmentation task, or point cloud registration.

[0249] See also Fig. 9 , a structural schematic diagram of another model training device provided in this application is described as follows.

[0250] The model training device may include a processor 901 and a memory 902. The processor 901 and the memory 902 are interconnected via a line. The memory 902 stores program instructions and data.

[0251] The memory 902 stores the aforementioned Figure 3-Figure 5 The program instructions and data corresponding to the steps in .

[0252] Processor 901 is used to execute the above Figure 3-Figure 5 The method steps performed by the model training device shown in any embodiment.

[0253] Optionally, the model training device may further include a transceiver 903 for receiving or sending data.

[0254] The present application also provides a computer-readable storage medium in which a program is stored. When the program is run on a computer, the computer executes the above-mentioned Figure 3-Figure 5 The illustrated embodiments describe steps in a method.

[0255] Optionally, the aforementioned Fig. 9 The model training device shown in is a chip.

[0256] The embodiment of the present application also provides a model training device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface. The program instructions are executed by the processing unit. The processing unit is used to execute the aforementioned Figure 3-Figure 5 The method steps performed by the model training device shown in any embodiment.

[0257] See also Fig.10 , a structural schematic diagram of another point cloud representation device provided by the present application is described as follows.

[0258] The point cloud representation device may include a processor 1001 and a memory 1002. The processor 1001 and the memory 1002 are interconnected via a line. The memory 1002 stores program instructions and data.

[0259] The memory 1002 stores the aforementioned Figure 6 Program instructions and data of the point cloud representation method.

[0260] Processor 1001 is used to execute the above Figure 6 Method steps for point cloud representation.

[0261] Optionally, the point cloud representation device may further include a transceiver 1003 for receiving or sending data.

[0262] The present application also provides a computer-readable storage medium in which a program is stored. When the program is run on a computer, the computer executes the above-mentioned Figure 6 The illustrated embodiments describe steps in a method.

[0263] Optionally, the aforementioned Fig.10 The point cloud shown in indicates that the device is a chip.

[0264] The present application also provides a point cloud representation device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface. The program instructions are executed by the processing unit. The processing unit is used to execute the aforementioned Figure 6 The method steps shown in any embodiment.

[0265] The embodiment of the present application also provides a digital processing chip. The digital processing chip integrates a circuit and one or more interfaces for implementing the functions of the above-mentioned processor 901, processor 1001, or processor 901, processor 1001. When a memory is integrated in the digital processing chip, the digital processing chip can complete the method steps of any one or more of the aforementioned embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory through a communication interface. The digital processing chip implements the actions performed by the model training device or the point cloud representation device in the above-mentioned embodiment according to the program code stored in the external memory.

[0266] The present application also provides a computer program product which, when executed on a computer, enables the computer to execute the above Figure 3-Figure 5 or Figure 6 The illustrated embodiments describe steps in a method.

[0267] The model training device or point cloud representation device provided in the embodiment of the present application may be a chip, and the chip includes: a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit. The processing unit may execute the computer execution instructions stored in the storage unit, and the chip may execute the above-mentioned Figure 3-Figure 5 or Figure 6 Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0268] Specifically, the aforementioned processing unit or processor may be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0269] For example, see Fig.11 , Fig.11 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 110, NPU 110 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 1103, which is controlled by the controller 1104 to extract matrix data from the memory and perform multiplication operations.

[0270] In some implementations, the operation circuit 1103 includes multiple processing units (PEs) inside. In some implementations, the operation circuit 1103 is a two-dimensional systolic array. The operation circuit 1103 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1103 is a general-purpose matrix processor.

[0271] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 1102 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1101 and performs matrix operations with matrix B, and the partial results or final results of the matrix are stored in the accumulator 1108.

[0272] The unified memory 1106 is used to store input data and output data. The weight data is directly transferred to the weight memory 1102 through the direct memory access controller (DMAC) 1105. The input data is also transferred to the unified memory 1106 through the DMAC.

[0273] The bus interface unit (BIU) 1110 is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 1109 .

[0274] The bus interface unit 1110 (BIU) is used for the instruction fetch memory 1109 to obtain instructions from the external memory, and is also used for the storage unit access controller 1105 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0275] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1106 or to transfer weight data to the weight memory 1102 or to transfer input data to the input memory 1101.

[0276] The vector calculation unit 1107 includes multiple operation processing units, and further processes the output of the operation circuit when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, upsampling of feature planes, etc.

[0277] In some implementations, the vector calculation unit 1107 can store the processed output vector to the unified memory 1106. For example, the vector calculation unit 1107 can apply a linear function and / or a nonlinear function to the output of the operation circuit 1103, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 1107 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1103, for example, for use in a subsequent layer in a neural network.

[0278] An instruction fetch buffer 1109 connected to the controller 1104 is used to store instructions used by the controller 1104;

[0279] Unified memory 1106, input memory 1101, weight memory 1102 and instruction fetch memory 1109 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0280] The operations of each layer in the recurrent neural network may be performed by the operation circuit 1103 or the vector calculation unit 1107 .

[0281] The processor mentioned in any of the above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the above Figure 3-Figure 5 or Figure 6 The method of procedure.

[0282] The present application also provides a vehicle, which includes the above Figure 8 or Fig.10 Point cloud representation of the device.

[0283] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0284] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0285] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0286] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0287] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0288] Finally, it should be noted that the above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A model training method, characterized in that: include: Acquire point cloud data, wherein the point cloud data includes data of a plurality of points; Enhance the point cloud data at least twice to obtain first enhanced point cloud data and second enhanced point cloud data; The first enhanced point cloud data and the second enhanced point cloud data are respectively used as inputs of a coding model, and a first point cloud feature and a second point cloud feature are obtained through features output by the coding model, wherein the first point cloud feature is obtained by performing variation on features output by the coding model for the first enhanced point cloud data, and the second point cloud feature is obtained by performing variation on features output by the coding model for the second enhanced point cloud data; The encoding model is updated according to the first point cloud features, the second point cloud features, and a first similarity between the first point cloud features and the second point cloud features to obtain an updated encoding model.

2. The method according to claim 1, characterized in that The updating of the encoding model according to the first point cloud feature, the second point cloud feature, and the first similarity between the first point cloud feature and the second point cloud feature to obtain an updated encoding model includes: Acquire first generated point cloud data corresponding to the first point cloud feature and second generated point cloud data corresponding to the second point cloud feature through a decoding model; The encoding model is updated according to the first generated point cloud data, the second generated point cloud data and the first similarity to obtain the updated encoding model.

3. The method according to claim 2, characterized in that The updating the encoding model according to the first generated point cloud data, the second generated point cloud data and the first similarity to obtain the updated encoding model includes: Calculate a first loss value according to the first generated point cloud data, the first enhanced point cloud data, the second generated point cloud data, and the second enhanced point cloud data; Obtaining a second loss value according to the first similarity; The encoding model is updated according to the first loss value and the second loss value to obtain the updated encoding model.

4. The method according to claim 3, characterized in that The calculating a first loss value according to the first generated point cloud data, the first enhanced point cloud data, the second generated point cloud data and the second enhanced point cloud data comprises: The first loss value is calculated based on at least one of a second similarity, a third similarity, a fourth similarity or a fifth similarity, the second similarity including the similarity between the first generated point cloud data and the second enhanced point cloud data, the third similarity including the similarity between the second generated point cloud data and the first enhanced point cloud data, the fourth similarity including the similarity between the first generated point cloud data and the first enhanced point cloud data, and the fifth similarity including the similarity between the second generated point cloud data and the second enhanced point cloud data.

5. The method according to any one of claims 2 to 4, characterized in that: The method further comprises: The decoding model is also updated to obtain an updated decoding model.

6. The method according to any one of claims 1 to 5, characterized in that The step of taking the first enhanced point cloud data and the second enhanced point cloud data as inputs of a coding model respectively, and obtaining first point cloud features and second point cloud features through features output by the coding model, comprises: Using the first enhanced point cloud data and the second enhanced point cloud data as inputs of a coding model respectively, and outputting a first feature corresponding to the first enhanced point cloud data and a second feature corresponding to the second enhanced point cloud data; The first feature and the second feature are respectively constrained to obey corresponding Gaussian distributions to obtain the first point cloud feature and the second point cloud feature.

7. The method according to any one of claims 1 to 6, characterized in that The features output by the updated encoding model are used to construct maps, classification tasks, segmentation tasks or point cloud registration.

8. A point cloud representation method, characterized in that: include: Get input point cloud data; The input point cloud data is used as an input of a coding model, and point cloud features corresponding to the input point cloud data are obtained according to features output by the coding model. The coding model includes a coding model trained by the method described in any one of claims 1-7.

9. A model training device, characterized in that: include: An acquisition module, used to acquire point cloud data, wherein the point cloud data includes data of a plurality of points; An enhancement module, used to enhance the point cloud data at least twice to obtain first enhanced point cloud data and second enhanced point cloud data; an encoding module, configured to use the first enhanced point cloud data and the second enhanced point cloud data as inputs of an encoding model respectively, and obtain a first point cloud feature and a second point cloud feature through features output by the encoding model, wherein the first point cloud feature is obtained by performing variation on features output by the encoding model for the first enhanced point cloud data, and the second point cloud feature is obtained by performing variation on features output by the encoding model for the second enhanced point cloud data; A training module is used to update the encoding model according to the first point cloud features, the second point cloud features and the first similarity between the first point cloud features and the second point cloud features to obtain an updated encoding model.

10. The device according to claim 9, characterized in that The device also includes: A decoding module, used for obtaining first generated point cloud data corresponding to the first point cloud feature and second generated point cloud data corresponding to the second point cloud feature through a decoding model; The training module is further used to update the encoding model according to the first generated point cloud data, the second generated point cloud data and the first similarity to obtain the updated encoding model.

11. The device according to claim 10, characterized in that The training module is specifically used for: Calculate a first loss value according to the first generated point cloud data, the first enhanced point cloud data, the second generated point cloud data, and the second enhanced point cloud data; Obtaining a second loss value according to the first similarity; The encoding model is updated according to the first loss value and the second loss value to obtain the updated encoding model.

12. The device according to claim 11, characterized in that The training module is specifically used to calculate the first loss value based on at least one of a second similarity, a third similarity, a fourth similarity or a fifth similarity, wherein the second similarity includes the similarity between the first generated point cloud data and the second enhanced point cloud data, the third similarity includes the similarity between the second generated point cloud data and the first enhanced point cloud data, the fourth similarity includes the similarity between the first generated point cloud data and the first enhanced point cloud data, and the fifth similarity includes the similarity between the second generated point cloud data and the second enhanced point cloud data.

13. The device according to any one of claims 10 to 12, characterized in that The training module is also used to update the decoding model to obtain an updated decoding model.

14. The device according to any one of claims 9 to 13, characterized in that The encoding module is specifically used for: Using the first enhanced point cloud data and the second enhanced point cloud data as inputs of a coding model respectively, and outputting a first feature corresponding to the first enhanced point cloud data and a second feature corresponding to the second enhanced point cloud data; The first feature and the second feature are respectively constrained to obey corresponding Gaussian distributions to obtain the first point cloud feature and the second point cloud feature.

15. The device according to any one of claims 9 to 14, characterized in that The features output by the updated encoding model are used to construct maps, classification tasks, segmentation tasks or point cloud registration.

16. A point cloud representation device, characterized in that: include: Input module, used to obtain input point cloud data; An encoding module is used to use the input point cloud data as an input of an encoding model, and obtain point cloud features corresponding to the input point cloud data according to features output by the encoding model, wherein the encoding model includes an encoding model trained by the method described in any one of claims 1 to 7.

17. A model training device, characterized in that: The method comprises a processor, wherein the processor is coupled to a memory, wherein the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

18. A point cloud representation device, characterized in that: The method comprises a processor, the processor is coupled to a memory, the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the method according to claim 8 is implemented.

19. A computer-readable storage medium comprising a program, which, when executed by a processing unit, performs the method according to any one of claims 1 to 7 or 8.

20. A vehicle, characterized in that: Comprising a point cloud representation device as described in claim 16 or 18.

Images