Point cloud completion method for automatic driving and related equipment

Abstract

The invention relates to the technical field of point cloud completion, and provides a point cloud completion method for automatic driving and related equipment. The method comprises the following steps: performing surface reconstruction based on original point cloud data to obtain a three-dimensional grid model, and performing multi-scale conversion on the three-dimensional grid model to obtain multi-scale model point cloud data; complementing the point cloud data of each model by using the point cloud complementing network to obtain initial complemented point cloud data of the point cloud data of each model; constructing a completion loss function according to all the initial completion point cloud data, and training the point cloud completion network based on the completion loss function; and performing point cloud completion on the to-be-completed point cloud data by using the trained point cloud completion network to obtain a plurality of initial completion point cloud data corresponding to the to-be-completed point cloud data, and combining the initial completion point cloud data to obtain final completion point cloud data. According to the method, the accuracy of point cloud completion can be improved under the condition that computing resources are limited.

Description

Technical Field

[0001] The present application relates to the technical field of point cloud completion, and in particular to a point cloud completion method for autonomous driving and related equipment. Background Art

[0002] With the rapid development of autonomous driving technology, point cloud data, as an important form of environmental perception, has become one of the core data for achieving high-precision positioning and dynamic scene understanding. Equipment such as lidar and millimeter-wave radar can generate high-density three-dimensional point cloud data to provide intuitive environmental information for autonomous driving vehicles. However, in actual application scenarios, due to the limitations of the resolution of scanning equipment, visual occlusion, complex road environment and the influence of dynamic objects, point cloud data often has serious incompleteness, manifested in sparsity, local missing or loss of geometric details. This data loss not only affects the accuracy of environmental perception, but also has an adverse impact on subsequent path planning and decision-making.

[0003] At present, point cloud completion methods based on deep learning have become a research hotspot. These methods usually use models such as generative adversarial networks, convolutional neural networks, or autoencoders to learn the distribution rules of completing missing parts through training on a large amount of point cloud data. However, existing methods often face the following problems: 1. Complex network structure and high computing resource requirements: Point cloud completion requires processing large-scale three-dimensional spatial data. Due to the large number of parameters, existing deep neural networks have a great dependence on computing resources in both training and reasoning stages, and are not suitable for resource-constrained autonomous driving platforms.

[0004] 2. Lack of geometric constraints, and difficulty in ensuring generated details: The generated results of point cloud completion often have problems such as inconsistent geometric shapes or lost details, which cannot fully meet the needs of high-precision autonomous driving.

[0005] It can be seen that the current point cloud completion method has the problem of low accuracy of point cloud completion when computing resources are limited. Summary of the invention

[0006] The present application provides a point cloud completion method and related equipment for autonomous driving, which can solve the problem of low accuracy of point cloud completion when computing resources are limited.

[0007] In a first aspect, an embodiment of the present application provides a point cloud completion method for autonomous driving, the point cloud completion method comprising: Obtaining original point cloud data of the environment surrounding the autonomous driving vehicle; the original point cloud data includes position data of multiple points; Surface reconstruction is performed based on the original point cloud data to obtain a three-dimensional mesh model, and the three-dimensional mesh model is converted into multiple scales to obtain model point cloud data of multiple scales; the number of point clouds per unit volume in model point cloud data of different scales is different; The point cloud data of each model is completed using a point cloud completion network to obtain initial completed point cloud data of each model point cloud data; the number of parameters in the point cloud completion network is less than a parameter number threshold; A completion loss function is constructed according to all the initial completion point cloud data, and the point cloud completion network is trained based on the completion loss function to obtain a trained point cloud completion network; the completion loss function is used to describe the accuracy of all the initial completion point cloud data; The trained point cloud completion network is used to complete the point cloud data to be completed, and multiple initial completion point cloud data corresponding to the point cloud data to be completed are obtained. The multiple initial completion point cloud data corresponding to the point cloud data to be completed are combined to obtain the final completion point cloud data.

[0008] Optionally, surface reconstruction is performed based on the original point cloud data to obtain a 3D mesh model, including: Calculate the normal vector of each point based on the position data of all points in the original point cloud data; Solve Poisson's equation based on all normal vectors to obtain the potential function; A triangular mesh is constructed according to the potential function, and the triangular mesh is smoothed and simplified to obtain a three-dimensional mesh model.

[0009] Optionally, the normal vector of each point is calculated based on the position data of all points in the original point cloud data, including: By formula: Calculate the Normal vector of a point ; in, Indicates Point relative to The offset vector of the point, Indicates The location data of the points, Indicates The location data of the points, represents the covariance matrix, Indicates The set of points in the area of ​​points, Indicates The number of points in the field of points, Indicates feature vectors, Indicates eigenvalues, , Indicates the number of points in the original point cloud data.

[0010] Alternatively, the Poisson equation is: in, represents the Laplace operator, represents the potential function, Represents the collection of volume density fields of all points in the original point cloud data, Represents the original point cloud data The volume density field of each point, , Represents the position data of the first point in the original point cloud data. Represents the original point cloud data The location data of each point.

[0011] Optionally, the 3D mesh model is converted to multiple scales to obtain model point cloud data of multiple scales, including: The three-dimensional grid model is uniformly divided into multiple scales to obtain multiple voxel grid models; each voxel grid model includes multiple voxel units, and the number of point clouds per unit volume in different voxel grid models is different; For each voxel mesh model, perform the following steps: Uniformly sampling each voxel unit in the voxel grid model to obtain multiple sampling points in each voxel unit; For each voxel unit, the position data of the centroid point is calculated according to the position data of all sampling points; The position data of the centroid points of all voxel units are combined to obtain model point cloud data of the scale corresponding to the voxel grid model.

[0012] Optionally, the position data of the centroid is calculated based on the position data of all sampling points, including: By formula: Calculate the location data of the centroid ; in, Represents the number of all sampling points in the voxel unit, represents the set of all sampling points in the voxel unit, Indicates The location data of the sampling points.

[0013] Optionally, the point cloud completion network is used to complete the point cloud data of each model to obtain the initial completed point cloud data of each model point cloud data, including: Extract features from the model point cloud data to obtain point cloud feature representation; The position data of the completion points corresponding to each centroid point in the point cloud data of the calculation model are represented according to the point cloud feature; The position data of all completed points are combined to obtain the initial completed point cloud data.

[0014] Optionally, feature extraction is performed on the model point cloud data to obtain point cloud feature representation, including: By formula: Computing point cloud feature representation ; in, represents a multi-layer perceptron, Indicates The characteristics of the centroid point, Represents the global features of the model point cloud data, Represents the characteristics of the first centroid point, Represents the characteristics of the second centroid point, Indicates The centroid features, Indicates The location data of the centroid points, , Indicates the number of centroid points; The position data of the completion points corresponding to each centroid point in the point cloud data of the calculation model is represented according to the point cloud feature, including: By formula: Calculate the The position data of the completion points corresponding to the centroid points .

[0015] Optionally, the completion loss function is: in, represents the value of the completion loss function, represents the weight, represents the adversarial comprehensive loss function obtained by weighted summation of the adversarial loss functions of each initial completed point cloud data, Represents the feature consistency loss function: in, represents the adversarial loss function, represents the true or false probability of the actual completed point cloud data corresponding to the initial completed point cloud data, represents the true or false probability of the initial completed point cloud data, Represents the fusion features of the real completed point cloud data corresponding to the initial completed point cloud data, represents the fusion features of the initial completed point cloud data, , Represents the collection of all initial completed point cloud data.

[0016] In a second aspect, an embodiment of the present application provides a point cloud completion device for autonomous driving, comprising: An acquisition module is used to acquire the original point cloud data of the environment surrounding the autonomous driving vehicle; the original point cloud data includes the position data of multiple points; The surface reconstruction module reconstructs the surface based on the original point cloud data to obtain a three-dimensional mesh model, and performs multi-scale transformation on the three-dimensional mesh model to obtain model point cloud data of multiple scales; the number of point clouds per unit volume in model point cloud data of different scales is different; The completion module uses the point cloud completion network to complete the point cloud data of each model to obtain the initial completed point cloud data of each model point cloud data; the number of parameters in the point cloud completion network is less than the parameter number threshold; A construction module is used to construct a completion loss function according to all the initial completion point cloud data, and the point cloud completion network is trained based on the completion loss function to obtain a trained point cloud completion network; the completion loss function is used to describe the accuracy of all the initial completion point cloud data; The point cloud completion module uses the trained point cloud completion network to complete the point cloud data to be completed, obtains multiple initial completed point cloud data corresponding to the point cloud data to be completed, and combines the multiple initial completed point cloud data corresponding to the point cloud data to be completed to obtain the final completed point cloud data.

[0017] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned point cloud completion method for autonomous driving when executing the above-mentioned computer program.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned point cloud completion method for autonomous driving is implemented.

[0019] The above solution of the present application has the following beneficial effects: In some embodiments of the present application, the original point cloud data of the surrounding environment of the autonomous driving vehicle is acquired, and then the surface is reconstructed based on the original point cloud data to obtain a three-dimensional mesh model, and the three-dimensional mesh model is transformed into multi-scale to obtain model point cloud data of multiple scales. Finally, the point cloud completion network is used to complete each model point cloud data to obtain the initial completed point cloud data of each model point cloud data, and then a completion loss function is constructed according to all the initial completed point cloud data, and the point cloud completion network is trained based on the completion loss function to obtain a trained point cloud completion network, and finally the trained point cloud completion network is used to complete the point cloud data to be completed, and multiple initial completed point cloud data corresponding to the point cloud data to be completed are obtained, and the multiple initial completed point cloud data corresponding to the point cloud data to be completed are combined to obtain the final completed point cloud data. Among them, surface reconstruction is performed based on the original point cloud data to obtain a three-dimensional mesh model, and then model point cloud data of multiple scales are obtained based on the three-dimensional mesh model, which improves the geometric accuracy of the model point cloud data and the information richness at multiple scales. Completion is performed based on the model point cloud data of multiple scales to obtain completed point cloud data of multiple scales, which improves the detail richness of the completed point cloud data. Compared with traditional neural networks, the point cloud completion network has fewer parameters and can be directly used in actual autonomous driving scenarios, greatly reducing the amount of calculation in actual applications, adapting to the environment with limited computing resources in autonomous driving, and improving the accuracy of point cloud completion under limited computing resources.

[0020] Other beneficial effects of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 A flowchart of a point cloud completion method for autonomous driving provided in one embodiment of the present application; Figure 2 A schematic diagram of the structure of a point cloud completion device for autonomous driving provided in one embodiment of the present application; Figure 3A schematic diagram of the structure of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0023] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0024] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0025] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0026] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0027] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0028] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0029] In response to the problem of low accuracy of point cloud completion under the existing situation of limited computing resources, an embodiment of the present application provides a point cloud completion method for autonomous driving. The point cloud completion method obtains original point cloud data of the surrounding environment of the autonomous driving vehicle, then reconstructs the surface based on the original point cloud data to obtain a three-dimensional mesh model, and performs multi-scale transformation on the three-dimensional mesh model to obtain model point cloud data of multiple scales. Finally, a point cloud completion network is used to complete each model point cloud data to obtain initial completed point cloud data for each model point cloud data. A completion loss function is then constructed based on all the initial completed point cloud data, and the point cloud completion network is trained based on the completion loss function to obtain a trained point cloud completion network. Finally, the trained point cloud completion network is used to complete the point cloud data to be completed to obtain multiple initial completed point cloud data corresponding to the point cloud data to be completed, and the multiple initial completed point cloud data corresponding to the point cloud data to be completed are combined to obtain the final completed point cloud data. Among them, surface reconstruction is performed based on the original point cloud data to obtain a three-dimensional mesh model, and then model point cloud data of multiple scales are obtained based on the three-dimensional mesh model, which improves the geometric accuracy of the model point cloud data and the information richness at multiple scales. Completion is performed based on the model point cloud data of multiple scales to obtain completed point cloud data of multiple scales, which improves the detail richness of the completed point cloud data. Compared with traditional neural networks, the point cloud completion network has fewer parameters and can be directly used in actual autonomous driving scenarios, greatly reducing the amount of calculation in actual applications, adapting to the environment with limited computing resources in autonomous driving, and improving the accuracy of point cloud completion under limited computing resources.

[0030] Next, an exemplary description is given of the point cloud completion method for autonomous driving provided in this application.

[0031] like Figure 1 As shown, the point cloud completion method for autonomous driving provided by this application includes the following steps: Step 11, obtaining the original point cloud data of the environment surrounding the autonomous driving vehicle.

[0032] The above-mentioned original point cloud data includes position data of multiple points.

[0033] In some embodiments of the present application, the radar and other equipment of the autonomous driving vehicle can be used to obtain original point cloud data of the environment around the autonomous driving vehicle, and the spatial range corresponding to the original point cloud data is the same as the radar scanning range of the autonomous driving vehicle.

[0034] It should be noted that after obtaining the original point cloud data, it needs to be preprocessed by normalization and denoising. The normalization steps include: 1. Centralized processing: Calculate the centroid of the original point cloud data Coordinates: in, Represents the number of points in the original point cloud data, Indicates The horizontal coordinate of the point, Indicates The vertical coordinate of the point, Indicates The vertical coordinates of the points are all coordinates in the three-dimensional coordinate system used by the radar and other equipment of the autonomous driving vehicle to obtain point cloud data (such as a three-dimensional coordinate system with the radar of the autonomous driving vehicle as the origin).

[0035] Translate the coordinates of each point to have the center of mass as the origin: in, represents the horizontal coordinate of the center of mass, represents the ordinate of the center of mass, Indicates the vertical coordinate of the centroid.

[0036] 2. Scaling processing: Calculate the maximum distance from each point to the centroid in the point cloud data : Based on the maximum distance, scale the point cloud to the unit range (e.g. ): in, Represents the scaled The location data of each point.

[0037] The denoising steps include: 1. Neighborhood search: For point , calculate its The number of neighboring points within : in, represents the location data set of all points after the normalization step above. Indicates The normalized position data of the points, the radius The value of is estimated by the average distance between points in the point cloud.

[0038] 2. Statistical filtering: Calculate the local neighborhood average distance of each point : Calculate the average neighborhood distance of all points and variance : 3. Denoising determination: Setting distance threshold , where Adjustable parameter, usually between 1 and 2. If the average neighborhood distance of a point , it is judged as a noise point and removed.

[0039] After the above preprocessing, the preprocessed original point cloud data is obtained, and the preprocessed original point cloud data is used as the original point cloud data in the subsequent steps.

[0040] It is worth mentioning that by preprocessing the original point cloud data, the quality and accuracy of the original point cloud data can be improved.

[0041] Step 12: reconstruct the surface based on the original point cloud data to obtain a three-dimensional mesh model, and perform multi-scale transformation on the three-dimensional mesh model to obtain model point cloud data of multiple scales.

[0042] Different models have different point cloud data with different number of point clouds per unit volume.

[0043] In some embodiments of the present application, the above-mentioned steps of performing surface reconstruction based on the original point cloud data to obtain a three-dimensional mesh model, and performing multi-scale transformation on the three-dimensional mesh model to obtain model point cloud data of multiple scales are specifically as follows: In the first step, the normal vector of each point is calculated based on the position data of all points in the original point cloud data.

[0044] Specifically, through the formula: Calculate the Normal vector of a point .

[0045] in, Indicates Point relative to The offset vector of the point, Indicates The location data of the points, Indicates The location data of the points, represents the covariance matrix, Indicates The set of points in the area of ​​points, Indicates The number of points in the field of points, Indicates feature vectors, Indicates eigenvalues, , Indicates the number of points in the original point cloud data.

[0046] In the second step, the Poisson equation is solved based on all normal vectors to obtain the potential function.

[0047] The above Poisson equation is used to describe the implicit surface of the point cloud.

[0048] Specifically, Poisson's equation is: in, represents the Laplace operator, represents the potential function, Represents the volume density field of all points in the original point cloud data, representing the The volume density field of each point, , Represents the position data of the first point in the original point cloud data. Represents the original point cloud data The location data of each point.

[0049] For example, the normal vectors of all points in the original point cloud data can be substituted into the Poisson equation, and the Poisson equation can be discretized using numerical methods (such as finite difference method, weighted least squares method, etc.) to solve the potential function. A common method is to iteratively solve the Poisson equation based on a discrete grid to obtain a potential function, which implicitly defines a surface in space.

[0050] The third step is to construct a triangular mesh according to the potential function, and then smooth and simplify the triangular mesh to obtain a three-dimensional mesh model.

[0051] Specifically, from the potential function Extract isosurfaces from: Among them, is a set threshold value, which can be 0. By extracting the isosurface, an implicitly defined three-dimensional surface can be obtained.

[0052] Then a triangular mesh is extracted from the implicitly defined three-dimensional surface, and the triangular mesh is smoothed and simplified to obtain a three-dimensional mesh model.

[0053] For example, the Marching Cubes algorithm, a classic algorithm for extracting three-dimensional isosurfaces, can be used to extract triangular meshes from implicitly defined three-dimensional surfaces. The algorithm extracts triangular meshes by traversing all boundary points of the mesh cube in three-dimensional space and creating triangles at isosurfaces. The triangular mesh can be smoothed using the Laplacian smoothing algorithm, and can be simplified using the Quadric Error Metric (QEM) simplification algorithm. The Laplacian smoothing algorithm expression is: in, is the vertex position data of the smoothed 3D mesh, is the location data of the neighborhood vertices, is the smoothing parameter, Represents a vertex location data, Represents a vertex 's neighborhood.

[0054] The expression of the 3D mesh simplification algorithm is: in, represents the position error of all vertices, represents the error matrix, which represents the position data set of all vertices of the three-dimensional mesh. represents the optimized translation vector, Represents a constant term.

[0055] The fourth step is to perform uniform spatial division of the three-dimensional grid model at multiple scales to obtain multiple voxel grid models.

[0056] Each voxel grid model includes multiple voxel units. The number of point clouds per unit volume in different voxel grid models is different, and the volume of all voxel units in the same voxel grid model is the same.

[0057] Exemplarily, the multiple scales are voxel units with side lengths of 3 cm, 6 cm, and 9 cm. After uniform spatial division, three voxel grid models are obtained, namely: a voxel grid model with a voxel unit side length of 3 cm, a voxel grid model with a voxel unit side length of 6 cm, and a voxel grid model with a voxel unit side length of 9 cm.

[0058] Step 5: For each voxel grid model, perform the following steps: First, each voxel unit in the voxel grid model is uniformly sampled to obtain multiple sampling points in each voxel unit.

[0059] Specifically, the number of sampling points is ,in, represents the distance between two sampling points, Represents the side length of the voxel unit.

[0060] Then, for each voxel unit, the position data of the centroid point is calculated based on the position data of all sampling points.

[0061] Specifically, through the formula: Calculate the location data of the centroid .

[0062] in, Represents the number of all sampling points in the voxel unit, represents the set of all sampling points in the voxel unit, Indicates The location data of the sampling points.

[0063] Finally, the position data of the centroid points of all voxel units are combined to obtain the model point cloud data of the scale corresponding to the voxel grid model.

[0064] Specifically, the position data of all centroid points are combined into one data to obtain the model point cloud data.

[0065] It is worth mentioning that through this step, surface reconstruction is performed based on the original point cloud data to obtain a three-dimensional mesh model, and then model point cloud data of multiple scales are obtained based on the three-dimensional mesh model, which improves the geometric accuracy of the model point cloud data and the information richness at multiple scales.

[0066] Step 13: Use the point cloud completion network to complete the point cloud data of each model to obtain the initial completed point cloud data of each model point cloud data.

[0067] The number of parameters in the point cloud completion network is less than a parameter number threshold, and the parameter number threshold may be the number of parameters of a traditional neural network (such as a convolutional neural network, a graph convolutional neural network, etc.).

[0068] In some embodiments of the present application, the above-mentioned step of using the point cloud completion network to complete each model point cloud data to obtain the initial completed point cloud data of each model point cloud data is specifically: For each model point cloud data, perform the following steps: The first step is to extract features from the model point cloud data to obtain point cloud feature representation.

[0069] Specifically, through the formula: Computing point cloud feature representation .

[0070] in, represents a multi-layer perceptron, Indicates The centroid features, Represents the global features of the model point cloud data, Represents the characteristics of the first centroid point, Represents the characteristics of the second centroid point, Indicates The characteristics of the centroid point, Indicates The location data of the centroid points, , Indicates the number of centroid points.

[0071] In the second step, the position data of the completion points corresponding to each centroid point in the point cloud data of the calculation model is represented according to the point cloud features.

[0072] Specifically, through the formula: Calculate the The position data of the completion points corresponding to the centroid points .

[0073] The third step is to combine the position data of all completed points to obtain the initial completed point cloud data.

[0074] Specifically, the position data of all the completed points are combined into one data to obtain the initial completed point cloud data.

[0075] It can be understood that the above operations of calculating point cloud feature representation, calculating completion point position data, and combining position data are all operations in the point cloud completion network.

[0076] Obviously, compared with traditional neural networks, the above point cloud completion network has fewer parameters and less computation, and can be deployed in environments with limited computing resources.

[0077] Step 14: construct a completion loss function based on all the initial completion point cloud data, and train the point cloud completion network based on the completion loss function to obtain a trained point cloud completion network.

[0078] The above completion loss function is used to describe the accuracy of all initial completed point cloud data.

[0079] Specifically, the completion loss function is: in, represents the value of the completion loss function, represents the weight, represents the adversarial comprehensive loss function obtained by weighted summation of the adversarial loss functions of each initial completed point cloud data, Represents the feature consistency loss function: in, represents the adversarial loss function, represents the true or false probability of the actual completed point cloud data corresponding to the initial completed point cloud data, represents the true or false probability of the initial completed point cloud data, Represents the fusion features of the real completed point cloud data corresponding to the initial completed point cloud data, represents the fusion features of the initial completed point cloud data, , Represents the collection of all initial completed point cloud data.

[0080] It should be noted that the process of training the point cloud completion network based on the completion loss function to obtain the trained point cloud completion network can be as follows: determine whether the value of the completion loss function is less than the preset loss value, if so, use the point cloud completion network as the trained point cloud completion network, otherwise, adjust the parameters in the point cloud completion network, and return to the step of using the point cloud completion network to complete each model point cloud data to obtain the initial completed point cloud data of each model point cloud data. The above-mentioned adversarial comprehensive loss function The weight coefficient in is set according to the corresponding initial completed point cloud data. For example, in the case of three initial completed point cloud data, The expression is: in, , and Both represent weight coefficients, Represents the adversarial loss function of the initial completed point cloud data with a scale of sparse point cloud, Represents the adversarial loss function of the initial completion point cloud data with a scale of medium sparse point cloud, Represents the adversarial loss function for the initial completion point cloud data at the scale of a highly dense point cloud.

[0081] Exemplarily, the above-mentioned real completed point cloud data can be generated using the Farthest Point Sampling (FPS) method, and the specific process is as follows:

[0082] 1. Initialization: Randomly select a point in the point cloud as the first sampling point: Where S is the set of sampling points, represents the sampling point, R is the set of remaining points, Represents the complete point cloud data corresponding to the original point cloud data.

[0083] 2. Iterative selection: In each iteration, select a point from the remaining points so that its minimum distance to the sampling point set S is maximized: Will Add the sampling point set S and remove it from the remaining point set: Termination condition: Stop when the number of points in the sampling point set S reaches the target number of points M: in, . S is the sampled point cloud, which means the point cloud .

[0084] By setting different target points, real completed point cloud data of different scales can be obtained.

[0085] It should be noted that the above true and false probabilities can be calculated by the discriminator in the generative adversarial network, and the expression is: in, Represents the local features of the point in the point cloud data (initial completed point cloud data or real completed point cloud data) input to the discriminator, Indicates The location data of the points, represents a multi-layer perceptron, Represents global features, represents the fusion feature, Indicates the true or false probability of point cloud data.

[0086] It is worth mentioning that due to the small number of parameters of the point cloud completion network, the number of parameters that need to be adjusted during the training of the point cloud completion network is small, which effectively reduces the training time and improves the training efficiency.

[0087] Step 15, using the trained point cloud completion network to complete the point cloud data to be completed, to obtain multiple initial completed point cloud data corresponding to the point cloud data to be completed, and to combine the multiple initial completed point cloud data corresponding to the point cloud data to be completed to obtain the final completed point cloud data.

[0088] The above-mentioned point cloud data to be completed is the point cloud data of the surrounding environment during the actual driving process of the autonomous driving vehicle.

[0089] In some embodiments of the present application, the above-mentioned step of using the trained point cloud completion network to complete the point cloud data to be completed, obtaining multiple initial completed point cloud data corresponding to the point cloud data to be completed, and combining the multiple initial completed point cloud data corresponding to the point cloud data to be completed to obtain the final completed point cloud data includes:

[0090] In the first step, the trained point cloud completion network is used to complete the point cloud data to be completed, and multiple initial completed point cloud data corresponding to the point cloud data to be completed are obtained.

[0091] Specifically, surface reconstruction is performed based on the point cloud data to be completed to obtain a three-dimensional mesh model, and the three-dimensional mesh model is transformed into multiple scales to obtain model point cloud data of multiple scales. Then, the trained point cloud completion network is used to complete each model point cloud data to obtain the initial completed point cloud data of each model point cloud data.

[0092] In the second step, multiple initial completed point cloud data corresponding to the point cloud data to be completed are combined to obtain the final completed point cloud data.

[0093] Specifically, all initial completed point cloud data are combined into the same three-dimensional coordinate system to obtain final completed point cloud data.

[0094] For example, the number of initial completed point cloud data is 3, and the number of points in the unit volume corresponds to sparse point cloud, medium sparse point cloud and high density point cloud respectively. When combined, they are gradually superimposed, and the initial completed point cloud data corresponding to the completed sparse point cloud, medium sparse point cloud and high density point cloud are gradually fused, and enhanced layer by layer with the skeleton-skin-surface architecture: sparse point cloud completes the skeleton structure, medium sparse point cloud completes the skin, and high density point cloud completes the detail surface, forming a complete final completed point cloud data.

[0095] It is worth mentioning that surface reconstruction is performed based on the original point cloud data to obtain a three-dimensional mesh model, and then model point cloud data of multiple scales are obtained based on the three-dimensional mesh model, which improves the geometric accuracy of the model point cloud data and the information richness at multiple scales. The model point cloud data of multiple scales are completed and combined to obtain completed point cloud data, which improves the detail richness of the completed point cloud data. Compared with traditional neural networks, the point cloud completion network has fewer parameters, which greatly reduces the amount of calculation in actual applications, adapts to the environment with limited computing resources in autonomous driving, and achieves improved accuracy of point cloud completion under limited computing resources.

[0096] In addition, the method of this application avoids the high resource requirements of complex deep network models. Specific problems solved include: Reduced resource consumption: Reduces the dependence of deep networks on computing resources and hardware, making them suitable for real-time scenarios of autonomous driving.

[0097] Model size reduction: Point cloud completion is performed through a simple network, effectively reducing the size of network parameters.

[0098] Balance between accuracy and efficiency: Providing high-precision geometric shapes, the point cloud completion network generates detailed completion data, finding a balance between efficiency and accuracy.

[0099] Applicable to specific scenarios: Point cloud completion methods are provided to meet specific completion requirements in autonomous driving scenarios.

[0100] The following is an exemplary description of the point cloud completion device for autonomous driving provided in this application.

[0101] like Figure 2 As shown, an embodiment of the present application provides a point cloud completion device for autonomous driving, and the point cloud completion device 200 for autonomous driving includes: An acquisition module 201 acquires original point cloud data of the environment surrounding the autonomous driving vehicle; the original point cloud data includes position data of multiple points; The surface reconstruction module 202 performs surface reconstruction based on the original point cloud data to obtain a three-dimensional mesh model, and performs multi-scale transformation on the three-dimensional mesh model to obtain model point cloud data of multiple scales; the number of point clouds per unit volume in the model point cloud data of different scales is different; The completion module 203 uses a point cloud completion network to complete each model point cloud data to obtain initial completed point cloud data of each model point cloud data; the number of parameters in the point cloud completion network is less than a parameter number threshold; A construction module 204 is used to construct a completion loss function according to all the initial completion point cloud data, and to train the point cloud completion network based on the completion loss function to obtain a trained point cloud completion network; the completion loss function is used to describe the accuracy of all the initial completion point cloud data; The point cloud completion module 205 uses the trained point cloud completion network to complete the point cloud data to be completed, obtains multiple initial completed point cloud data corresponding to the point cloud data to be completed, and combines the multiple initial completed point cloud data corresponding to the point cloud data to be completed to obtain the final completed point cloud data.

[0102] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0103] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0104] like Figure 3 As shown, an embodiment of the present application provides a terminal device. The terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 3 Only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 implements the steps of any of the above-mentioned method embodiments when executing the computer program D102.

[0105] Specifically, when the processor D100 executes the computer program D102, it obtains the original point cloud data of the surrounding environment of the autonomous driving vehicle, and then reconstructs the surface based on the original point cloud data to obtain a three-dimensional mesh model, and performs multi-scale transformation on the three-dimensional mesh model to obtain model point cloud data of multiple scales. Finally, each model point cloud data is completed using a point cloud completion network to obtain initial completed point cloud data for each model point cloud data, and then a completion loss function is constructed based on all the initial completed point cloud data, and the point cloud completion network is trained based on the completion loss function to obtain a trained point cloud completion network, and finally the trained point cloud completion network is used to complete the point cloud data to be completed, and multiple initial completed point cloud data corresponding to the point cloud data to be completed are obtained, and the multiple initial completed point cloud data corresponding to the point cloud data to be completed are combined to obtain the final completed point cloud data. Among them, surface reconstruction is performed based on the original point cloud data to obtain a three-dimensional mesh model, and then model point cloud data of multiple scales are obtained based on the three-dimensional mesh model, which improves the geometric accuracy of the model point cloud data and the information richness at multiple scales. Completion is performed based on the model point cloud data of multiple scales to obtain completed point cloud data of multiple scales, which improves the detail richness of the completed point cloud data. Compared with traditional neural networks, the point cloud completion network has fewer parameters and can be directly used in actual autonomous driving scenarios, greatly reducing the amount of calculation in actual applications, adapting to the environment with limited computing resources in autonomous driving, and improving the accuracy of point cloud completion under limited computing resources.

[0106] The processor D100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0107] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk, a smart memory card (SMC, SmartMedia Card), a secure digital (SD, Secure Digital) card, a flash card (Flash Card), etc. equipped on the terminal device D10. Further, the memory D101 may also include both an internal storage unit of the terminal device D10 and an external storage device. The memory D101 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory D101 may also be used to temporarily store data that has been output or is to be output.

[0108] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0109] An embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0110] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium that can carry the computer program code to the point cloud completion method device / terminal device of autonomous driving. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0111] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0112] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0113] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A point cloud completion method for autonomous driving, characterized in that: include: Obtain raw point cloud data of the environment surrounding the autonomous vehicle; The original point cloud data includes position data of a plurality of points; Performing surface reconstruction based on the original point cloud data to obtain a three-dimensional mesh model, and performing multi-scale transformation on the three-dimensional mesh model to obtain model point cloud data of multiple scales; The number of point clouds per unit volume in model point cloud data of different scales is different; Using a point cloud completion network to complete each of the model point cloud data, to obtain initial completed point cloud data of each of the model point cloud data; the number of parameters in the point cloud completion network is less than a parameter number threshold; Constructing a completion loss function according to all the initial completion point cloud data, and training the point cloud completion network based on the completion loss function to obtain a trained point cloud completion network; the completion loss function is used to describe the accuracy of all the initial completion point cloud data; The trained point cloud completion network is used to complete the point cloud data to be completed, so as to obtain a plurality of initial completed point cloud data corresponding to the point cloud data to be completed, and the plurality of initial completed point cloud data corresponding to the point cloud data to be completed are combined to obtain the final completed point cloud data.

2. The point cloud completion method according to claim 1, characterized in that: The surface reconstruction based on the original point cloud data to obtain a three-dimensional mesh model includes: Calculate the normal vector of each point according to the position data of all points in the original point cloud data; Solve Poisson's equation based on all normal vectors to obtain the potential function; A triangular mesh is constructed according to the potential function, and the triangular mesh is smoothed and simplified to obtain a three-dimensional mesh model.

3. The point cloud completion method according to claim 2, characterized in that: The step of calculating the normal vector of each point according to the position data of all points in the original point cloud data comprises: By formula: Calculate the Normal vector of a point ; in, Indicates point relative to the The offset vector of the point, Indicates the The location data of the points, Indicates the The location data of the points, represents the covariance matrix, Indicates the The set of neighborhood points of points represents the The number of points in the field of points, Indicates feature vectors, Indicates eigenvalues, , Indicates the number of points in the original point cloud data.

4. The point cloud completion method according to claim 3, characterized in that: The Poisson equation is: in, represents the Laplace operator, represents the potential function, Represents the collection of volume density fields of all points in the original point cloud data, Represents the original point cloud data The volume density field of each point, , Represents the position data of the first point in the original point cloud data. Represents the original point cloud data The location data of each point.

5. The point cloud completion method according to claim 1, characterized in that: The multi-scale transformation of the three-dimensional grid model to obtain model point cloud data of multiple scales includes: Performing uniform spatial division of multiple scales on the three-dimensional grid model to obtain multiple voxel grid models; each voxel grid model includes multiple voxel units, and the number of point clouds per unit volume in different voxel grid models is different; For each voxel mesh model, perform the following steps: Uniformly sampling each voxel unit in the voxel grid model to obtain a plurality of sampling points in each voxel unit; For each of the voxel units, respectively, calculate the position data of the centroid point according to the position data of all sampling points; The position data of the centroids of all voxel units are combined to obtain model point cloud data of a scale corresponding to the voxel grid model.

6. The point cloud completion method according to claim 5, characterized in that: The step of calculating the position data of the centroid point according to the position data of all sampling points includes: By formula: Calculate the location data of the centroid ; in, Represents the number of all sampling points in the voxel unit, represents the set of all sampling points in the voxel unit, Indicates The location data of the sampling points.

7. The point cloud completion method according to claim 1, characterized in that: The method of using a point cloud completion network to complete each of the model point cloud data to obtain initial completed point cloud data of each of the model point cloud data includes: For each of the model point cloud data, perform the following steps: Performing feature extraction on the model point cloud data to obtain point cloud feature representation; Calculate the position data of the completion points corresponding to each centroid point in the model point cloud data according to the point cloud feature representation; The position data of all completed points are combined to obtain the initial completed point cloud data.

8. The point cloud completion method according to claim 7, characterized in that: The step of extracting features from the model point cloud data to obtain a point cloud feature representation includes: By formula: Computing point cloud feature representation ; in, represents a multi-layer perceptron, Indicates The centroid features, Represents the global features of the model point cloud data, Represents the characteristics of the first centroid point, Represents the characteristics of the second centroid point, Indicates The centroid features, Indicates the The location data of the centroid points, , Indicates the number of centroid points; The calculating the position data of the completion points corresponding to each centroid point in the point cloud data of the model according to the point cloud feature representation includes: By formula: Calculate the The position data of the completion points corresponding to the centroid points .

9. The point cloud completion method according to claim 1, characterized in that: The completion loss function is: in, represents the value of the completion loss function, represents the weight, represents the adversarial comprehensive loss function obtained by weighted summation of the adversarial loss functions of each initial completed point cloud data, Represents the feature consistency loss function: in, represents the adversarial loss function, represents the true or false probability of the actual completed point cloud data corresponding to the initial completed point cloud data, represents the true or false probability of the initial completed point cloud data, Represents the fusion features of the real completed point cloud data corresponding to the initial completed point cloud data, represents the fusion features of the initial completed point cloud data, , Represents the collection of all initial completed point cloud data.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the point cloud completion method for autonomous driving is implemented as described in any one of claims 1 to 9.

Images