Training of deep learning model, point cloud data processing method and related equipment

Abstract

The application discloses a training method of a deep learning model. The deep learning model is used for simulating an ICP algorithm. The method comprises the following steps: obtaining a training label sample set of a target object, wherein each training label sample in the training label sample set is determined by using the ICP algorithm; and using the training label sample set for training of the deep learning model. The application also discloses a point cloud data processing method and related equipment. The deep learning model is used for simulating the ICP algorithm, so that the calculation speed of the point cloud posture is accelerated, and the output is more robust.

Description

Technical Field

[0001] The disclosed embodiments of this application relate to the field of point cloud processing, and more specifically, to a method and related equipment for training a deep learning model, processing point cloud data. Background Technology

[0002] In the field of autonomous driving, the main task of the perception system is to perceive information such as the target's position, speed, category, and behavior prediction, with the detection and tracking of moving targets being the most crucial. ICP (Iterative Closest Point) plays a significant role in moving target analysis because it can calculate the pose of two input point clouds. Based on the pose, it can calculate not only the magnitude of the target's velocity but also its direction. However, the ICP algorithm is an iterative optimization method; achieving high accuracy requires multiple iterations, making it relatively time-consuming overall.

[0003] Furthermore, in autonomous driving scenarios, the number of point clouds is relatively large, and the number of targets in most scenarios is also large. Therefore, whether based on the entire point cloud or on batch processing, it is very challenging to use the ICP algorithm for real-time calculation online. Summary of the Invention

[0004] According to embodiments of this application, this application proposes a method and related equipment for training deep learning models and processing point cloud data, in order to solve the above-mentioned problems and accelerate the calculation speed and output more robustly.

[0005] The first aspect of this application discloses a training method for a deep learning model, the deep learning model being used to simulate the ICP algorithm, the method comprising: obtaining a training label sample set of a target object, wherein each training label sample in the training label sample set is determined using the ICP algorithm; and using the training label sample set for training the deep learning model.

[0006] In some embodiments, obtaining training label samples of the target object includes: obtaining two consecutive frame point clouds of the target object; processing the two frame point clouds using the ICP algorithm to obtain initial rotation and translation values; and using the initial rotation and translation values ​​as the training label samples.

[0007] In some embodiments, obtaining training label samples of the target object includes: obtaining two consecutive frame point clouds of the target object; processing the two frame point clouds using the ICP algorithm to obtain initial rotation and translation values; labeling the two frame point clouds using the initial rotation and translation values ​​to obtain displacement information of each point in the two frame point clouds; and using the initial rotation and translation values ​​and the displacement information of each point as training label samples.

[0008] In some embodiments, using the training label samples for training the deep learning model includes: inputting the training label samples into the deep learning model to obtain predicted rotation and translation values; and adjusting the network parameters of the deep learning model based on the difference between the predicted rotation and translation values ​​and the initial rotation and translation values.

[0009] In some embodiments, the deep learning model includes an input layer, an encoding network, a decoding network, and an output layer; inputting the training label samples into the deep learning model to obtain predicted rotation and translation values ​​includes: inputting the training label samples into the input layer to obtain first point cloud data of the corresponding dimension; inputting the first point cloud data into the encoding network for downsampling to obtain second point cloud data; inputting the second point cloud data into the decoding network for upsampling to obtain third point cloud data; and inputting the third point cloud data into the output layer to obtain the predicted rotation and translation values.

[0010] In some embodiments, the encoding network includes: an encoding layer, a pooling layer, and a concatenation layer; the step of inputting the first point cloud data into the encoding network for downsampling to obtain the second point cloud data includes: inputting the first point cloud data into the encoding layer for downsampling to output a first point cloud feature; inputting the first point cloud feature into the pooling layer to output a second point cloud feature; inputting the first point cloud feature and the second point cloud feature into the concatenation layer for concatenation to output a third point cloud feature; the step of inputting the second point cloud data into the decoding network for upsampling to obtain the third point cloud data includes: inputting the third point cloud feature into the decoding network for upsampling to obtain a fourth point cloud feature.

[0011] The second aspect of this application discloses a method for processing point cloud data, comprising: acquiring two consecutive frames of point cloud data of a target object; using a deep learning model to simulate the ICP algorithm to process the two frames of point cloud data to obtain the pose data of the target object, wherein the deep learning model is trained using a sample label set according to the training method described in the first aspect, and at least one sample label in the sample label set is calculated based on the ICP algorithm; and determining the motion information of the target object based on the pose data.

[0012] In some embodiments, obtaining two consecutive frames of point cloud data of a target object includes: obtaining two consecutive frames of raw point cloud data of the target object; obtaining position information corresponding to the two frames of raw point cloud data; and using the position information corresponding to the two frames of raw point cloud data, transforming one frame of raw point cloud data to the coordinate system of the other frame of raw point cloud data, thereby obtaining the two frames of point cloud data.

[0013] In some embodiments, the step of using a deep learning model to simulate the ICP algorithm to process the two frames of point cloud data includes: obtaining the original dimensional features corresponding to each point in the two frames of point cloud data; and using the deep learning model to perform feature compilation operations on the original dimensional features corresponding to each point in the two frames of point cloud data, thereby obtaining the target dimensional features corresponding to each point in the two frames of point cloud data.

[0014] In some embodiments, the step of using the deep learning model to perform feature compilation on the original dimensional features corresponding to each point in the two frames of point cloud data includes: using an encoding network to encode the original dimensional features corresponding to each point in the two frames of point cloud data to obtain the intermediate dimensional features of each point; and using a decoding network to decode the intermediate dimensional features of each point based on the intermediate dimensional features of each point to obtain the target dimensional features corresponding to each point in the two frames of point cloud data, thereby obtaining the pose data in the two frames of point cloud data.

[0015] A third aspect of this application discloses an electronic device including a memory and a processor coupled to each other, the processor being configured to execute program instructions stored in the memory to implement the deep learning model training method described in the first aspect, or to implement the point cloud data processing method described in the second aspect.

[0016] The fourth aspect of this application discloses a non-volatile computer-readable storage medium storing program instructions thereon, which, when executed by a processor, implement the training method for the deep learning model described in the first aspect, or implement the point cloud data processing method described in the second aspect.

[0017] The beneficial effects of this application are: obtaining a training label sample set of the target object, wherein each training label sample in the training label sample set is determined using the ICP algorithm, training the training label sample set to obtain a deep learning model, and using the deep learning model to simulate the ICP algorithm to accelerate the calculation speed of point cloud pose and achieve more robust output. Attached Figure Description

[0018] The present application will be further described below with reference to the accompanying drawings and embodiments. In the drawings:

[0019] Figure 1 This is a flowchart illustrating the training method of a deep learning model according to an embodiment of this application;

[0020] Figure 2 This is a schematic diagram illustrating the effect of training a deep learning model according to an embodiment of this application;

[0021] Figure 3 This is a flowchart illustrating a point cloud data processing method according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of the structure of a non-volatile computer-readable storage medium according to an embodiment of this application. Detailed Implementation

[0024] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0025] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, "many" in this application means two or more. Moreover, the term "at least one" in this application means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. Furthermore, the terms "first," "second," and "third" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0026] To enable those skilled in the art to better understand the technical solution of this application, the technical solution of this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] Please see Figure 1 , Figure 1 This is a flowchart illustrating the training method of a deep learning model according to an embodiment of this application. The deep learning model is used to simulate the ICP algorithm. The deep learning model is used for inference, that is, inputting pre-coded input features into the deep learning model, such as the result of running the ICP algorithm offline. The model is obtained after calculation through the defined deep learning model structure / computation graph. The obtained deep learning model is then used to simulate the ICP algorithm for application to the motion state analysis of the target object.

[0028] The method can be implemented by an electronic device with computing capabilities, such as a microcomputer, server, and mobile devices such as laptops and tablets. It should be noted that if substantially the same result is achieved, the method of this application does not necessarily require further explanation. Figure 1 The sequence of processes shown is limited.

[0029] In some possible implementations, this method can be implemented by the processor calling computer-readable instructions stored in memory, such as... Figure 1 As shown, the method may include the following steps:

[0030] S11: Obtain the training label sample set of the target object, where each training label sample in the training label sample set is determined using the ICP algorithm.

[0031] A training label sample set for the target object is obtained. The target object can be a human body, tree, lane, vehicle, building, etc. The training label sample set includes multiple training label samples. Each training label sample in the training label sample set is determined using the ICP algorithm. The ICP algorithm is an algorithm based on data registration and uses the nearest point search method to solve problems based on freeform surfaces. For example, the ICP algorithm can be used to calculate a training label sample from two consecutive frames of point cloud data of the target object, thus generating the training label sample set.

[0032] S12: Use the training label sample set for training the deep learning model.

[0033] The training label sample set for the target object, which can be a human body, tree, lane, vehicle, building, etc., includes multiple training label samples. Each training label sample in the training label sample set is determined using the ICP algorithm. Taking a vehicle as an example, the training label sample set for the target object can be a set of pose information of the target vehicle calculated using the ICP algorithm based on the point cloud data of the target vehicle. The training label sample set is used for training the deep learning model, that is, training the deep learning model based on the training label sample set of the target object, and optimizing the deep learning model to obtain a deep learning model for simulating the ICP algorithm.

[0034] In this embodiment, a training label sample set of the target object is obtained, wherein each training label sample in the training label sample set is determined using the ICP algorithm. The training label sample set is used to train a deep learning model. By using the deep learning model to simulate the ICP algorithm, the calculation speed of point cloud pose is accelerated and the output is more robust.

[0035] In some embodiments, obtaining training label samples of the target object includes: obtaining two consecutive frame point clouds of the target object; processing the two frame point clouds using the ICP algorithm to obtain initial rotation and translation values; and using the initial rotation and translation values ​​as training label samples.

[0036] Point cloud data is acquired using radar sensors, such as lidar (mechanical or solid-state lidar). Radar sensors can acquire point cloud data at different times; for example, the first frame of point cloud data might be acquired at time t-1, and the second frame at time t. When the radar sensor's acquisition frequency is high, the intervals between each time step are small. Therefore, point cloud data acquired at adjacent time steps can be used to determine whether the objects corresponding to these point cloud data are in motion. Objects in motion are then identified as moving targets.

[0037] Two consecutive frames of point cloud data of the target object are acquired. For example, the first frame of point cloud data of the target object is acquired at the first moment, and the second frame of point cloud data is acquired at the second moment. The two frames of point cloud data are processed using the ICP algorithm, that is, the pose information of the two frames of point cloud data is calculated using the ICP algorithm to obtain initial rotation and translation values, such as the translation distance and rotation angle of the target object. The initial rotation and translation values ​​are used as training label samples. That is, the initial rotation and translation values ​​are obtained by processing the two frames of point cloud data using the ICP algorithm. For example, it can be the translation x, y, z and rotation angles (pitch, yaw, roll) of each point in the two frames of point cloud data. Then, the initial rotation and translation values ​​of the two frames of point cloud data can be used as training label samples of the target object and input into the deep learning model for training.

[0038] In some embodiments, obtaining training label samples of the target object includes: obtaining two consecutive frame point clouds of the target object; processing the two frame point clouds using the ICP algorithm to obtain initial rotation and translation values; labeling the two frame point clouds using the initial rotation and translation values ​​to obtain displacement information of each point in the two frame point clouds; and using the initial rotation and translation values ​​and the displacement information of each point as training label samples.

[0039] Two consecutive frames of point cloud data of the target object are acquired. For example, the first frame of point cloud data of the target object is acquired at the first moment, and the second frame of point cloud data is acquired at the second moment. The two frames of point cloud data are processed using the ICP algorithm, that is, the pose information of the two frames of point cloud data is calculated using the ICP algorithm to obtain initial rotation and translation values, such as the translation distance and rotation angle of the target object. The initial rotation and translation values ​​are used as training label samples. That is, the initial rotation and translation values ​​are obtained by processing the two frames of point cloud data using the ICP algorithm. For example, it can be the translation x, y, z and rotation angles (pitch, yaw, roll) of each point in the two frames of point cloud data. The initial rotation and translation values ​​are used to label two frames of point clouds to obtain the displacement information of each point in the two frames of point clouds. That is, the displacement information of each point in the two frames of point clouds is calculated using the initial rotation and translation values. At this time, the initial rotation and translation values ​​and the displacement information of each point are used as training label samples. For example, the information of each point and the overall rotation and translation values ​​are saved offline to serve as labels for deep model training. Then, the initial rotation and translation values ​​of several consecutive frames of point clouds can be used as training label samples of the target object and input into the deep learning model for training.

[0040] In some embodiments, using training labeled samples for training a deep learning model includes: inputting training labeled samples into a deep learning model to obtain predicted rotation and translation values; and adjusting the network parameters of the deep learning model based on the difference between the predicted rotation and translation values ​​and the initial rotation and translation values.

[0041] The ICP algorithm is used to process the point cloud to obtain initial rotation and translation values. These values ​​could be the translations (x, y, z) and rotation angles (Pitch, Yaw, Roll) of each point in the point cloud. Training labeled samples are then input into a deep learning model to obtain predicted rotation and translation values. For example, the translations (x, y, z) and rotation angles (Pitch, Yaw, Roll) of each point in two frames of the point cloud can be used as training labeled samples to train the deep learning model and obtain predicted rotation and translation values. These predicted values ​​could be the sin and cosine values ​​corresponding to the translations (x, y, z) and rotation angles (yaw) of each point in two frames of the point cloud. Based on the difference between the predicted and initial rotation and translation values, the network parameters of the deep learning model are adjusted. The model is then optimized using a loss function, such as L1 regression loss, and the general SGD algorithm is employed for multiple iterative optimizations.

[0042] In some embodiments, the deep learning model includes an input layer, an encoding network, a decoding network, and an output layer; inputting training label samples into the deep learning model to obtain predicted rotation and translation values ​​includes: inputting training label samples into the input layer to obtain first point cloud data of the corresponding dimension; inputting the first point cloud data into the encoding network for downsampling to obtain second point cloud data; inputting the second point cloud data into the decoding network for upsampling to obtain third point cloud data; and inputting the third point cloud data into the output layer to obtain predicted rotation and translation values.

[0043] Deep learning models can be built based on CNN (Convolutional Neural Network), DBN (Deep Belief Network), RNN (Recurrent Neural Network), RNTN (Recursive Neural Tensor Network), autoencoders, or GAN (Generative Adversarial Networks). A deep learning model includes an input layer, an encoding network, a decoding network, and an output layer.

[0044] The training label samples are input into the input layer to obtain the first point cloud data of the corresponding dimension. That is, the input layer is used to calculate the dimension of the training label samples to obtain the first point cloud data of the corresponding dimension. For example, if the training label samples are three-dimensional data, that is, they correspond to xyz three-dimensional coordinates, then the data is extracted according to the x-axis, y-axis, and z-axis to obtain the first point cloud data of the corresponding x-axis, y-axis, and z-axis.

[0045] At this point, the first point cloud data is input into the encoding network for downsampling to obtain the second point cloud data. For example, the second point cloud data can be 512-dimensional point cloud features. That is, the encoding network is used to downsample the first point cloud data, gradually mapping to obtain point cloud data with larger dimensions. The encoding layers can include a first encoding layer, a second encoding layer, a third encoding layer, a fourth encoding layer, a fifth encoding layer, a sixth encoding layer, a seventh encoding layer, and an eighth encoding layer. For example, the first encoding layer has 32 neural network nodes, the second encoding layer has 64 neural network nodes, the third encoding layer has 64 neural network nodes, the fourth encoding layer has 128 neural network nodes, the fifth encoding layer has 128 neural network nodes, the sixth encoding layer has 256 neural network nodes, the seventh encoding layer has 256 neural network nodes, and the eighth encoding layer has 512 neural network nodes.

[0046] Furthermore, the second point cloud data is input into the decoding network for upsampling to obtain the third point cloud data. For example, the third point cloud data can be 5-dimensional point cloud features. The decoding network can include a first decoding layer, a second decoding layer, a third decoding layer, a fourth decoding layer, a fifth decoding layer, a sixth decoding layer, and a seventh decoding layer. For example, the first decoding layer has 512 neural network nodes, the second decoding layer has 256 neural network nodes, the third decoding layer has 128 neural network nodes, the fourth decoding layer has 64 neural network nodes, the fifth decoding layer has 32 neural network nodes, the sixth decoding layer has 16 neural network nodes, and the seventh decoding layer has 5 neural network nodes.

[0047] The third point cloud data is input into the output layer to obtain the predicted rotation and translation values. That is, the output layer is used to calculate the dimensions of the third point cloud data to obtain the predicted rotation and translation values ​​of the corresponding dimensions. For example, it can be the sin and cos values ​​corresponding to the translation x, y, z and rotation yaw angles of each point.

[0048] In some embodiments, the encoding network includes: an encoding layer, a pooling layer, and a concatenation layer; inputting first point cloud data into the encoding network for downsampling to obtain second point cloud data includes: inputting the first point cloud data into the encoding layer for downsampling to output first point cloud features; inputting the first point cloud features into the pooling layer to output second point cloud features; inputting the first point cloud features and the second point cloud features into the concatenation layer for concatenation to output third point cloud features; inputting the second point cloud data into the decoding network for upsampling to obtain third point cloud data includes: inputting the third point cloud features into the decoding network for upsampling to obtain fourth point cloud features.

[0049] A deep learning model consists of an input layer, an encoder network, a decoder network, and an output layer. The encoder network includes an encoder layer, a pooling layer, and a concatenation layer. The first point cloud data is input into the encoder layer for downsampling, outputting the first point cloud feature. Specifically, the first point cloud data is input into the first encoder layer, outputting a 32-dimensional point cloud feature. This 32-dimensional point cloud feature is then input into the second encoder layer, outputting a 64-dimensional point cloud feature. The 64-dimensional point cloud feature is then input into the third encoder layer, outputting another 64-dimensional point cloud feature. This 64-dimensional point cloud feature is then input into the fourth encoder layer, outputting a 128-dimensional point cloud feature. This 128-dimensional point cloud feature is then input into the fifth encoder layer, outputting another 128-dimensional point cloud feature. This 128-dimensional point cloud feature is then input into the sixth encoder layer, outputting a 256-dimensional point cloud feature. This 256-dimensional point cloud feature is then input into the seventh encoder layer, outputting another 256-dimensional point cloud feature. Finally, this 512-dimensional point cloud feature is input into the eighth encoder layer, outputting a 512-dimensional point cloud feature. This 512-dimensional point cloud feature is then used as the first point cloud feature described above.

[0050] The first point cloud feature is input into a pooling layer, which outputs a second point cloud feature. This pooling layer can use average pooling or max pooling, i.e., pooling the 512-dimensional point cloud feature to obtain the second point cloud feature. The first and second point cloud features are then input into a stitching layer for concatenation, outputting a third point cloud feature. For example, concatenating the 512-dimensional first and second point cloud features in the stitching layer outputs a 1024-dimensional third point cloud feature, thus obtaining the second point cloud data. The third point cloud feature is input into the decoding network for upsampling to obtain the fourth point cloud feature. Specifically, the second point cloud data is input into the first decoding layer, outputting a 512-dimensional point cloud feature. This 512-dimensional feature is then input into the second decoding layer, outputting a 256-dimensional point cloud feature. The 256-dimensional feature is input into the third decoding layer, outputting a 128-dimensional point cloud feature. This 128-dimensional feature is input into the fourth decoding layer, outputting a 64-dimensional point cloud feature. The 64-dimensional feature is input into the fifth decoding layer, outputting a 32-dimensional point cloud feature. The 32-dimensional feature is input into the sixth decoding layer, outputting a 16-dimensional point cloud feature. Finally, the 16-dimensional feature is input into the seventh decoding layer, outputting a 5-dimensional point cloud feature. This 5-dimensional point cloud feature is then used as the third and fourth point cloud data as described above.

[0051] To facilitate understanding, the training process of a deep learning model will be described with an example, such as... Figure 2 As shown, Figure 2 This is a schematic diagram illustrating the training effect of a deep learning model according to an embodiment of this application. The input N×3 represents N points in the input point cloud, each with three coordinates: x, y, and z. Mapping is performed in the first MLP layer, for example, MLP(32, 64, 64, 128, 128, 256, 256, 512). The value after MLP represents multiple fully connected layers, and the number represents the number of neural network nodes in each layer. MLP is performed individually on each of the N points in the point cloud, gradually mapping the 3-dimensional input to 512 dimensions, i.e., N×512. Then, the 512-dimensional features of all points are maximized in their respective dimensions (max-pooling) to obtain a 512-dimensional overall feature of the input point cloud, i.e., the overall 512. The 512-dimensional features of each point and the overall 512-dimensional feature are directly concatenated to form a 1024-dimensional feature for each point, i.e., N×1024. This is achieved through a second MLP layer, for example, MLP(512,256,128,64,32,16,5). The values ​​after MLP represent multiple fully connected layers, and the numbers represent the number of neural network nodes in each layer. MLP is performed on N points in the point cloud above, thus gradually mapping 1024 dimensions to 5 dimensions, i.e., N×5. For example, it can be the sin and cos values ​​corresponding to the translation x, y, z and rotation yaw angles of each point in the point cloud.

[0052] Please see Figure 3 , Figure 3 This is a flowchart illustrating a point cloud data processing method according to an embodiment of this application. This method can be applied to vehicle-mounted devices with computing and other functions. It should be noted that if substantially the same result is obtained, the method of this application is not necessarily identical. Figure 3 The sequence of processes shown is limited.

[0053] In some possible implementations, this method can be implemented by the processor calling computer-readable instructions stored in memory, such as... Figure 3 As shown, the method may include the following steps:

[0054] S31: Obtain two consecutive frames of point cloud data of the target object.

[0055] Two consecutive frames of point data for a target object can be collected at different times for the same target object. The two consecutive frames of point data can be point cloud data at time t and point cloud data at time t-1. The point cloud data includes point cloud information and the original pose information of the target object at the corresponding time. The target object can be a human body, tree, lane, vehicle, building, etc.

[0056] Furthermore, point cloud data of the target object can be constructed by receiving information collected by sensor devices, including radar sensors. These radar sensors can be radar devices used for autonomous driving and meeting accuracy requirements, providing point cloud perception. Point cloud data is collected using radar sensors, such as millimeter-wave radar and lidar. Radar sensors can be mounted on a mobile device, such as an autonomous vehicle. LiDAR can include mechanical lidar, semi-solid-state lidar, or solid-state lidar.

[0057] S32: Using a deep learning model, the ICP algorithm is simulated to process two frames of point cloud data to obtain the pose data of the target object.

[0058] The deep learning model is trained using the sample label set according to the training method of the deep learning model described above. At least one sample label in the sample label set is calculated based on the ICP algorithm, that is, the training label sample set of the target object is obtained. Each training label sample in the training label sample set is determined using the ICP algorithm, and then the deep learning model is trained using the training label sample set.

[0059] Two consecutive frames of point cloud data of the target object are acquired. A deep learning model is used to simulate the ICP algorithm to process the two frames of point cloud data. That is, the two frames of point cloud data are input into the deep learning model to calculate the pose data of the target object. For example, if the target object is a vehicle, the pose data of the target object can be the pose of each point cloud pair of the vehicle, such as the sin and cos values ​​corresponding to the translation x, y, z and rotation yaw angles of each point in the point cloud data.

[0060] S33: Determine the motion information of the target object based on the pose data.

[0061] Furthermore, based on the pose data, the motion information of the target object is determined. For example, the sin and cos values ​​corresponding to the translation x, y, z and rotation yaw angles of each point in the point cloud data are used to calculate the moving speed and direction of the target object.

[0062] In some embodiments, obtaining two consecutive frames of point cloud data of a target object includes: obtaining two consecutive frames of raw point cloud data of the target object; obtaining position information corresponding to the two frames of raw point cloud data; and using the position information corresponding to the two frames of raw point cloud data, transforming one frame of raw point cloud data into the coordinate system of the other frame of raw point cloud data, thereby obtaining two frames of point cloud data.

[0063] Two consecutive frames of raw point cloud data of the target object are acquired. These two consecutive frames can be the raw point cloud data at time t-1 and time t. The corresponding positional information of the two frames is then obtained, for example, the positional information of the raw point cloud data at time t and time t-1. Using the positional information of the two frames, for example, the origin of the point cloud at time t-1 is O1 and the origin of the point cloud at time t is O2, one frame of raw point cloud data is further transformed to the coordinate system of the other frame. This results in two frames of point cloud data, essentially projecting the point cloud cluster at time t-1 into the coordinate system of the point cloud cluster at time t. The pose information of the point cloud cluster at time t-1 also uses O2 as the origin, thus eliminating errors caused by the object's own motion.

[0064] In some embodiments, a deep learning model is used to simulate the ICP algorithm to process two frames of point cloud data, including: obtaining the original dimensional features corresponding to each point in the two frames of point cloud data; and using a deep learning model to perform feature compilation operations on the original dimensional features corresponding to each point in the two frames of point cloud data, thereby obtaining the target dimensional features corresponding to each point in the two frames of point cloud data.

[0065] The process involves obtaining the raw dimensional features corresponding to each point in two frames of point cloud data. For example, each point in the two frames has 3D data, corresponding to x, y, and z coordinates. These raw dimensional features are then extracted along the x, y, and z axes. A deep learning model is then used to compile these raw dimensional features. This model consists of an input layer, an encoder network, a decoder network, and an output layer. The input layer takes the x, y, and z coordinates of each point in the two frames as input, and the compilation process yields the target dimensional features for each point. These could be the sin and cosine values ​​corresponding to the x, y, z translations and yaw rotations of each point.

[0066] In some embodiments, a deep learning model is used to perform feature compilation operations on the original dimensional features corresponding to each point in two frames of point cloud data. This includes: using an encoding network to encode the original dimensional features corresponding to each point in the two frames of point cloud data to obtain the intermediate dimensional features of each point; and using a decoding network to decode the intermediate dimensional features of each point based on the intermediate dimensional features of each point to obtain the target dimensional features corresponding to each point in the two frames of point cloud data, thereby obtaining the pose data in the two frames of point cloud data.

[0067] The original dimensional features corresponding to each point in two frames of point cloud data are encoded using an encoding network to obtain the intermediate dimensional features of each point. The encoding network includes an encoding layer, a pooling layer, and a concatenation layer. The encoding network can include a first encoding layer, a second encoding layer, a third encoding layer, a fourth encoding layer, a fifth encoding layer, a sixth encoding layer, a seventh encoding layer, and an eighth encoding layer. For example, the first encoding layer has 32 neural network nodes, the second encoding layer has 64 neural network nodes, the third encoding layer has 64 neural network nodes, the fourth encoding layer has 128 neural network nodes, the fifth encoding layer has 128 neural network nodes, the sixth encoding layer has 256 neural network nodes, the seventh encoding layer has 256 neural network nodes, and the eighth encoding layer has 512 neural network nodes.

[0068] At this point, the original dimensional features corresponding to each point in the two frames of point cloud data are input into the first encoding layer, outputting 32-dimensional point cloud features. These 32-dimensional features are then input into the second encoding layer, outputting 64-dimensional point cloud features. The 64-dimensional features are then input into the third encoding layer, outputting another 64-dimensional point cloud feature. This 64-dimensional feature is then input into the fourth encoding layer, outputting 128-dimensional point cloud features. This 128-dimensional feature is then input into the fifth encoding layer, outputting yet another 128-dimensional point cloud feature. This 128-dimensional feature is then input into the sixth encoding layer, outputting 256-dimensional point cloud features. This 256-dimensional feature is then input into the seventh encoding layer, outputting another 256-dimensional point cloud feature. Finally, this 512-dimensional point cloud feature is input into the eighth encoding layer, outputting 512-dimensional point cloud features. This 512-dimensional point cloud feature is then used as the intermediate dimensional feature corresponding to each point.

[0069] Based on the intermediate dimension features of each point, a decoding network is used to decode the intermediate dimension features of each point to obtain the target dimension features corresponding to each point in the two frames of point cloud data. That is, the 512-dimensional point cloud features are pooled to obtain the second point cloud features. The first and second point cloud features are then input into the stitching layer for stitching to output the third point cloud features. For example, the 512-dimensional first and second point cloud features are stitched together in the stitching layer to output the 1024-dimensional third point cloud features, thus obtaining the second point cloud data. The third point cloud feature is input into the decoding network for upsampling to obtain the fourth point cloud feature. Specifically, the second point cloud data is input into the first decoding layer, outputting a 512-dimensional point cloud feature. This 512-dimensional feature is then input into the second decoding layer, outputting a 256-dimensional point cloud feature. The 256-dimensional feature is input into the third decoding layer, outputting a 128-dimensional point cloud feature. This 128-dimensional feature is input into the fourth decoding layer, outputting a 64-dimensional point cloud feature. This 64-dimensional feature is input into the fifth decoding layer, outputting a 32-dimensional point cloud feature. This 32-dimensional feature is input into the sixth decoding layer, outputting a 16-dimensional point cloud feature. This 16-dimensional feature is input into the seventh decoding layer, outputting a 5-dimensional point cloud feature. This 5-dimensional point cloud feature is then used as the target dimension feature for each point.

[0070] Furthermore, the 5D point cloud features are transformed to obtain the pose data in the two frames of point cloud data, which can be the sin and cos values ​​corresponding to the translation x, y, z and rotation yaw angles of each point in the two frames of point cloud data.

[0071] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0072] Please see Figure 4 , Figure 4This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. The electronic device 40 includes a memory 41 and a processor 42 coupled to each other. The processor 42 is used to execute program instructions stored in the memory 41 to implement the steps of the above-described deep learning model training method embodiment, or to implement the steps of the above-described point cloud data processing method embodiment. In a specific implementation scenario, the electronic device 40 may include, but is not limited to, a microcomputer or a server.

[0073] Specifically, processor 42 controls itself and memory 41 to implement the steps of the training method embodiment of the deep learning model described above, or to implement the steps of the point cloud data processing method embodiment described above. Processor 42 can also be called a CPU (Central Processing Unit), and may be an integrated circuit chip with signal processing capabilities. Processor 42 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 42 can be implemented using integrated circuit chips.

[0074] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a non-volatile computer-readable storage medium according to an embodiment of this application. The non-volatile computer-readable storage medium 50 is used to store a computer program 501. When the computer program 501 is executed by a processor, for example by the aforementioned... Figure 4 When the processor 42 in the embodiment is executed, it is used to implement the steps of the above-described training method embodiment for deep learning models, or to implement the steps of the above-described point cloud data processing method embodiment.

[0075] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0076] In the several embodiments provided in this application, it should be understood that the disclosed methods and related devices can be implemented in other ways. For example, the related device implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication disconnection shown or discussed may be indirect coupling or communication disconnection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0077] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0078] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0079] Those skilled in the art will readily recognize that numerous modifications and variations can be made to the apparatus and method while maintaining the teachings of this application. Therefore, the above disclosure should be considered limited only by the scope of the appended claims.

Claims

1. A method for training a deep learning model, characterized in that, The deep learning model is used to simulate the ICP algorithm, and the method includes: Obtain a training label sample set for the target object, wherein each training label sample in the training label sample set is determined using the ICP algorithm; The training label sample set is used for training the deep learning model; The step of obtaining the training label samples of the target object includes: Two consecutive frame point clouds of the target object at different times are obtained; the pose information of the two frame point clouds is calculated using the ICP algorithm to obtain initial rotation and translation values; the two frame point clouds are labeled using the initial rotation and translation values ​​to obtain displacement information of each point in the two frame point clouds; the initial rotation and translation values ​​and the displacement information of each point are used as training label samples.

2. The method according to claim 1, characterized in that, Obtaining training label samples of the target object includes: Obtain two consecutive frames of point cloud data of the target object; The two point clouds are processed using the ICP algorithm to obtain initial rotation and translation values; The initial rotation and translation values ​​are used as the training label samples.

3. The method according to claim 2, characterized in that, Using the training labeled samples for training the deep learning model includes: The training label samples are input into the deep learning model to obtain the predicted rotation and translation values; The network parameters of the deep learning model are adjusted based on the difference between the predicted rotation and translation values ​​and the initial rotation and translation values.

4. The method according to claim 3, characterized in that, The deep learning model includes an input layer, an encoder network, a decoder network, and an output layer; The training labeled samples are input into the deep learning model to obtain predicted rotation and translation values, including: The training label samples are input into the input layer to obtain the first point cloud data of the corresponding dimension; The first point cloud data is input into the encoding network for downsampling to obtain the second point cloud data; The second point cloud data is input into the decoding network for upsampling to obtain the third point cloud data; The third point cloud data is input into the output layer to obtain the predicted rotation and translation values.

5. The method according to claim 4, characterized in that, The coding network includes: a coding layer, a pooling layer, and a splicing layer; The step of inputting the first point cloud data into the encoding network for downsampling to obtain the second point cloud data includes: The first point cloud data is input into the coding layer for downsampling, and the first point cloud feature is output. The first point cloud feature is input into the pooling layer, and the second point cloud feature is output. The first point cloud feature and the second point cloud feature are input into the stitching layer for stitching, and the third point cloud feature is output. The step of inputting the second point cloud data into the decoding network for upsampling to obtain the third point cloud data includes: The third point cloud feature is input into the decoding network for upsampling to obtain the fourth point cloud feature.

6. A method for processing point cloud data, characterized in that, include: Acquire two consecutive frames of point cloud data of the target object; Using a deep learning model, the ICP algorithm is simulated to process the two frames of point cloud data to obtain the pose data of the target object. The deep learning model is trained using a sample label set according to the training method of any one of claims 1-5. At least one sample label in the sample label set is calculated based on the ICP algorithm. Based on the pose data, the motion information of the target object is determined.

7. The method according to claim 6, characterized in that, Acquire two consecutive frames of point cloud data of the target object, including: Obtain two consecutive frames of raw point cloud data of the target object; Obtain the location information corresponding to the two frames of original point cloud data; Using the location information corresponding to the two frames of original point cloud data, one frame of original point cloud data is transformed to the coordinate system of the other frame of original point cloud data, thereby obtaining the two frames of point cloud data.

8. The method according to claim 7, characterized in that, The process of using a deep learning model to simulate the ICP algorithm to process the two frames of point cloud data includes: Obtain the original dimensional features corresponding to each point in the two frames of point cloud data; The deep learning model is used to perform feature compilation on the original dimensional features corresponding to each point in the two frames of point cloud data, thereby obtaining the target dimensional features corresponding to each point in the two frames of point cloud data.

9. The method according to claim 8, characterized in that, The step of using the deep learning model to perform feature compilation on the original dimensional features corresponding to each point in the two frames of point cloud data includes: The original dimensional features corresponding to each point in the two frames of point cloud data are encoded using an encoding network to obtain the intermediate dimensional features of each point. Based on the intermediate dimension features of each point, a decoding network is used to decode the intermediate dimension features of each point to obtain the target dimension features corresponding to each point in the two frames of point cloud data, thereby obtaining the pose data in the two frames of point cloud data.

10. An electronic device, characterized in that, The device includes a memory and a processor that are coupled to each other, the processor being configured to execute program instructions stored in the memory to implement the point cloud data processing method according to any one of claims 1 to 5, or to implement the deep learning model training method according to claims 6 to 9.

11. A non-volatile computer-readable storage medium storing program instructions thereon, characterized in that, When the program instructions are executed by the processor, they implement the point cloud data processing method according to any one of claims 1 to 5, or the deep learning model training method according to claims 6 to 9.

Images