Point cloud data processing method and related device

Abstract

The invention discloses a point cloud data processing method and device and related equipment, and belongs to the field of computer vision, and the method comprises the steps: obtaining an original point cloud data set of a scene, the scene comprising a target object; augmenting the point cloud data of the target object to generate an augmented point cloud data set; the augmented point cloud data set is used for training a first model to obtain a trained first model, and the trained first model is used for executing a point cloud data processing task; training a second model by using the at least one original point cloud data set and the trained first model; obtaining a trained second model, wherein the trained second model is used for executing a point cloud data processing task. That is, the dense point cloud is obtained by augmenting the original point cloud, the models are obtained by training the dense point cloud and the original point cloud respectively, and the target model has the capability of extracting high-quality information from the sparse point cloud and executing a point cloud processing task by utilizing inter-model training, so that the processing effect of the point cloud data is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a point cloud data processing method, apparatus, computing device and storage medium. Background Art

[0002] Point cloud data is a collection of vectors in three-dimensional coordinates, which contains rich information, such as three-dimensional coordinates, colors, classification values, etc. Point cloud data can be used for a variety of downstream computer vision tasks, such as target segmentation and target detection.

[0003] Point cloud data is usually collected and generated by LiDAR or camera. The point cloud data collected by LiDAR is relatively sparse due to factors such as distance, occlusion and illumination. The point cloud data collected by camera needs to be generated by depth estimation algorithm, which is unstable and also accompanied by deformation, sparseness and vacancies. Sparse and empty point clouds will affect the accuracy of downstream tasks such as target segmentation and target detection.

[0004] In the related technology, point cloud data is augmented to enrich the point cloud information, and the augmented point cloud is used to train the model. However, this method does not solve the fundamental problem that the sparsity of point cloud data affects the execution of tasks. In addition, due to the shortcomings of the point cloud data augmentation method, the model training efficiency is low and the execution effect of the point cloud data processing task is poor. Summary of the invention

[0005] The present application provides a point cloud data processing method, apparatus, computing device and storage medium, which can augment the original sparse point cloud data, and use the augmented point cloud data to train the model, and then train the task model, thereby enhancing the task model's perception performance of the sparse point cloud data, and expanding the point cloud data augmentation method. The technical solution is as follows:

[0006] In a first aspect, a point cloud data processing method is provided, the method comprising: acquiring at least one original point cloud data set of a scene, the scene including a target object; augmenting the point cloud data of the target object included in the at least one original point cloud data set to generate at least one augmented point cloud data set corresponding to the at least one original point cloud data set; training a first neural network model using the at least one augmented point cloud data set to obtain a trained first neural network model, the trained first neural network model being used to perform a point cloud data processing task; training a second neural network model using the at least one original point cloud data set and the trained first neural network model; obtaining a trained second neural network model, the trained second neural network model being used to perform a point cloud data processing task.

[0007] In an embodiment of the present application, a dense point cloud with better quality is obtained by augmenting the original data set, and a first neural network capable of performing point cloud processing tasks is obtained by training the dense point cloud. The first neural network and the sparse point cloud are used to train the second neural network, so that the learning ability of the second neural network approaches that of the first neural network, and the second neural network has the ability to extract high-quality information from the sparse point cloud and perform point cloud processing tasks. After that, the trained first neural network can perform point cloud processing tasks more efficiently directly from the original sparse point cloud. For example, point cloud processing tasks can be point cloud data processing tasks such as target segmentation, target detection, target classification, and point cloud denoising.

[0008] Optionally, the second neural network model is trained using at least one original point cloud dataset and the trained first neural network model, including: using at least one original point cloud dataset to pre-train the second neural network model to obtain a pre-trained second neural network model, and using the trained first neural network model to train the pre-trained second neural network model; or, using at least one original point cloud dataset and the trained first neural network model to simultaneously train the second neural network model.

[0009] That is, in the embodiment of the present application, the original point cloud data set can be first used to input the second neural network model for pre-training, and then the first neural network model can be used to train the pre-trained second neural network model. In this case, the pre-training of the model by the original point cloud and the training of the second neural network by the first neural network occur successively; the second neural network model can also be trained simultaneously with the original point cloud data set and the first neural network model. In this case, the supervision loss of the first neural network for consistency training of the second neural network and the target task supervision loss of the second neural network using the original point cloud data as input will be simultaneously transmitted to the second neural network for training. Specifically, in the process of using the first neural network model to perform consistency training on the second neural network model, the first neural network is the teacher network and the second neural network is the student network.

[0010] That is, in both cases, the final trained second neural network model is trained with the original point cloud data set and the first neural network model. As a result, the second neural network learns to extract high-quality information close to the second neural network model to perform the target point cloud processing task in the scenario of taking the original low-quality point cloud data as input.

[0011] Optionally, the point cloud data of the target object included in at least one original point cloud data set are augmented, including: determining first point cloud data with a density lower than a sparse threshold from the point cloud data of the target object included in at least one original point cloud data set and densifying it, and at least one augmented point cloud data set includes the densified first point cloud data.

[0012] That is, the augmentation method of point cloud data is expanded. The point cloud data of the target object with lower density in the original point cloud data set is densified to obtain the augmented point cloud data set, which increases the density of the point cloud of the target object and improves the quality of the augmented point cloud data set, making it easier to use the augmented point cloud data set to train the second neural network model.

[0013] Optionally, the point cloud data of the target object included in at least one original point cloud data set are augmented, including: determining second point cloud data with a density higher than a density threshold from the point cloud data of the target object included in at least one original point cloud data set and copying the second point cloud data, and at least one augmented point cloud data set includes the copied second point cloud data.

[0014] That is, the augmentation method of point cloud data is expanded. The point cloud data of the target object with higher density in the original point cloud data set is copied to obtain the augmented point cloud data set. These higher density data are the point cloud data with better quality in the original point cloud data set. By copying these data, the number of high-quality point cloud data of the target object in the augmented point cloud data set is increased, and the quality of the augmented point cloud data set is improved, which is convenient for using the augmented point cloud data set to train the second neural network model.

[0015] Optionally, before using at least one original point cloud data set and the trained first neural network model to train the second neural network model, the method also includes: copying the second target point cloud data included in a certain original point cloud data set; and adding the second target point cloud data included in the copied original point cloud data set to the original point cloud data set.

[0016] That is, if the augmentation method of copying high-density target object point cloud data is adopted, the original point cloud dataset used to train the second neural network model also needs to add these high-density target object point clouds that have undergone the same copying operation, ensuring that the distribution of target objects in the datasets used to train the respective neural networks is the same, so as to improve the effectiveness and accuracy of the subsequent consistency training of the second neural network model using the first neural network model.

[0017] Optionally, at least one augmented point cloud dataset is used to train the first neural network model to obtain a trained first neural network model, including: using an augmented point cloud dataset corresponding to a certain original point cloud dataset to train the first neural network model to obtain a trained first neural network model, and obtaining a first task processing result of the original point cloud dataset; using at least one original point cloud dataset and the trained first neural network model to train the second neural network model, including: using the original point cloud dataset and the trained first neural network model to train the second neural network model to obtain a second task processing result of the original point cloud dataset; the method also includes: generating a visualization interface, the visualization interface is used to display the first task processing result and the second task processing result of the original point cloud dataset in pairs in the form of images or data; receiving a difficult example dataset confirmation instruction, the difficult example dataset confirmation instruction is used to indicate that the original point cloud dataset corresponding to the first task processing result or the second task processing result is a difficult example dataset.

[0018] That is, in the embodiment provided by the present application, the task processing results of the first neural network model and the second neural network model are displayed to the user through a visual interface. Specifically, the visual interface shows the user the processing results of a pair of original point cloud data sets and its corresponding augmented point cloud data sets under the second neural network model and the first neural network model, respectively, so as to intuitively demonstrate the processing capabilities of the two networks for the data. Optionally, if the two results are quite different, it means that the second neural network is insufficient in processing the point cloud data set, and the point cloud data set may be a difficult example data set. At this time, a confirmation instruction such as a click from the user is received to confirm that the data set corresponding to the processing result is a difficult example data set. Specifically, the task processing result may be the processing result of an intermediate process, such as a feature extraction result, or may be the processing result at the end of the process, such as a target segmentation mask or a target recognition probability, etc. The present application does not limit this processing result.

[0019] Optionally, the method further includes: augmenting the hard example dataset to generate at least one augmented hard example dataset; adding at least one augmented hard example dataset to at least one original point cloud dataset. That is, after confirming the hard example dataset, the hard example dataset is augmented and added to the original dataset, so that the neural network is trained based on more hard example datasets, which expands the augmentation method of the hard example dataset and improves the neural network's processing ability for hard example data.

[0020] Optionally, the scene also includes background objects, and the hard example dataset is augmented to generate at least one augmented hard example dataset, including: extracting point cloud data of the target object in the hard example dataset; merging the point cloud data of the target object in the hard example dataset with the point cloud data of the background objects in at least one original point cloud dataset to generate at least one augmented hard example dataset.

[0021] That is, an embodiment of the present application proposes a method for augmenting a hard example dataset. In some cases, a dataset becomes a hard example dataset because the point cloud data of the target object is relatively special. The point cloud data of the target objects in these hard example datasets are extracted and combined with the point cloud data of the background objects in the non-hard example data, thereby generating more point cloud datasets with relatively special point cloud data of the target objects as augmented hard example datasets, thereby expanding the method for augmenting hard example datasets.

[0022] In a third aspect, a point cloud data processing device is provided, wherein the point cloud data processing device has the function of implementing the point cloud data processing method in the first aspect. The point cloud data processing device includes at least one module, and the at least one module is used to implement the point cloud data processing method provided in the first aspect.

[0023] In a fourth aspect, a computing device cluster is provided, the computing device cluster comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the point cloud data processing method provided in the first aspect above.

[0024] Optionally, each computing device may further include a communication bus, which is used to establish a connection between the processor and the memory.

[0025] Optionally, each computing device may further include a communication interface, where the communication interface is used to implement communication between the computing device and other devices or a communication network.

[0026] In a fifth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the storage medium. When the instructions are executed on a computing device, the computing device executes the point cloud data processing method provided in the first aspect.

[0027] In a sixth aspect, a computer program product comprising instructions is provided, and when the instructions are executed on a computing device, the computing device executes the point cloud data processing method provided in the first aspect. In other words, a computer program is provided, and when the computer program is executed on a computing device, the computing device executes the point cloud data processing method provided in the first aspect.

[0028] The technical effects obtained in the above-mentioned second to sixth aspects are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0030] Figure 1 It is a structural schematic diagram of a point cloud data processing system provided in an embodiment of the present application;

[0031] Figure 2 It is a flowchart of a point cloud data processing method provided in an embodiment of the present application;

[0032] Figure 3 It is a schematic diagram of a point cloud data processing interactive interface provided in an embodiment of the present application;

[0033] Figure 4 This is another schematic diagram of a point cloud data processing interactive interface provided in an embodiment of the present application;

[0034] Figure 5a-5d This is a schematic diagram of a point cloud data augmentation effect provided in an embodiment of the present application;

[0035] Figure 6a-6d This is a training result comparison diagram provided in an embodiment of the present application;

[0036] Figure 7 This is another schematic diagram of a point cloud data processing interactive interface provided in an embodiment of the present application;

[0037] Figure 8 It is a flowchart of another point cloud data processing method provided in an embodiment of the present application;

[0038] Fig. 9 is a structural schematic diagram of a point cloud data processing device provided in an embodiment of the present application;

[0039] Fig.10 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0040] Fig.11 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0041] Fig.12 It is a schematic diagram of the network connection relationship of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application are described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0043] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0044] First, some of the terms and related technologies involved in this application are explained in conjunction with the accompanying drawings to facilitate understanding by those skilled in the art.

[0045] Deep Learning (DL): A type of machine learning technology based on deep neural network algorithms, whose main feature is the use of multiple nonlinear transformation structures to process and analyze data. It is mainly used in perception, decision-making and other scenarios in the field of artificial intelligence, such as image and speech recognition, natural language translation, computer games, etc.

[0046] Point Cloud (PC): is a three-dimensional data representation, where each point in the data represents a set of X, Y, Z geometric coordinates, color, intensity and other attribute values. This intensity value records the intensity of the return signal based on the reflectivity of the object surface. When these points are combined together, a point cloud is formed, which represents a collection of data points of a 3D shape or object in space. Three-dimensional point clouds can be obtained through devices such as LiDAR, RGB-D cameras, 3D scanners, etc., or they can be generated through computer graphics methods.

[0047] Object Detection (OD): One of the core issues in the field of computer vision, its task is to find all the objects of interest in the image and determine their categories and locations. Object detection algorithms are mainly divided into two categories: region-based methods (Two Stage) and regression-based methods (One Stage). Region-based methods first generate some candidate regions that may contain objects, and then classify and locate each region; regression-based methods directly predict the category and location of the target on the image, which is usually faster but less accurate.

[0048] Semantic Segmentation (SS): A typical computer vision task that involves taking some raw data (e.g., flat images) as input and converting them into masks with highlighted regions of interest. Its goal is to assign each pixel in the image to a category with semantic meaning, such as person, car, cat, etc. Semantic segmentation can help better understand the content and structure in the image.

[0049] Hard Sample: A sample that is relatively complex and difficult for the model to accurately identify or locate. For example, in object detection, it refers to objects that are difficult to accurately locate or identify. These samples have a large loss, which brings challenges to model training. In object segmentation, it refers to those target areas that are difficult to accurately segment. These samples may contain blurred boundaries, occlusions, or complex shapes.

[0050] Consistency training: the student model is trained using the consistency loss function between the teacher model and the student model. In consistency training, the outputs of the teacher model and the student model are regarded as two different perspectives, and the consistency loss function is used to ensure the consistency of the two perspectives. The goal of the consistency loss function is to make the output of the student model as consistent as possible with the output of the teacher model. This method can improve the generalization ability of the student model and reduce the computational cost of the model. Put the same batch of data into two models at the same time, use the predicted output of the teacher model as the soft label, and the true label as the hard label, calculate the two losses of the student model respectively, and finally take the weighted sum of the two losses as the final loss to update the network parameters. When predicting, only the student model is used.

[0051] Gold Standard: Refers to the Ground Truth Boxes or Predicted Boxes. The Ground Truth Boxes refer to the actual target locations marked in the image. For each target, we manually or automatically annotate a rectangular bounding box to indicate the location of the target. Predicted bounding boxes are bounding boxes generated by the model to represent the targets it believes exist in the image. The model generates these bounding boxes through a detection algorithm, usually using a Convolutional Neural Network (CNN) or other deep learning models.

[0052] Before explaining the point cloud data processing method provided by the embodiment of the present invention, the application scenario of the embodiment of the present invention is first introduced.

[0053] Point cloud data is a collection of points in a three-dimensional coordinate system. It contains rich information and is usually used to represent objects, scenes or environments in the real world. Each point in the point cloud data has a set of attributes, which may include: three-dimensional coordinates, color information, intensity value, classification value, timestamp, etc.

[0054] There are many ways to obtain point cloud data, among which the most common ones are laser radar and camera. Laser radar is currently the most commonly used perception device in the fields of autonomous driving, unmanned vehicles, augmented reality, and robots. As a common depth measurement device, its advantage is accurate depth. However, due to common factors such as distance, occlusion, and reflection, the point cloud data it collects is a set of sparse points in three-dimensional space. The density of these points is low and cannot form a continuous surface. The corresponding shape and structure are extremely incomplete, and the point cloud is even sparser at a distance, which brings difficulties to downstream tasks such as target detection and instance segmentation. Therefore, it is extremely difficult to rely on the sparse point cloud of laser radar for accurate autonomous obstacle avoidance and autonomous operation. Cameras are also commonly used sensors for perception. The collected visual signals are dense, high-resolution, rich in information, and low-cost. However, the shortcomings of perception algorithms based on pure vision are also very significant. The results of depth estimation are unstable and even accompanied by a large number of incomplete holes, which will cause deformation, sparseness, and vacancies of three-dimensional point clouds, which will affect downstream tasks.

[0055] In some embodiments, point cloud data can be used for a variety of downstream computer vision tasks, such as semantic segmentation, target detection, etc. In more specific scenarios, it can also be applied to autonomous driving systems to perceive the surrounding environment to improve safety and efficiency, and to three-dimensional reconstruction applications such as AR or VR to generate detailed and complete three-dimensional models, and then create virtual scenes, etc.

[0056] Usually, in related technologies, point cloud data is augmented to enhance the perception effect of downstream task models. The complete process of using point cloud data to perform related tasks is as follows: in the model training stage, first, the original point cloud data is collected and generated through devices such as lidar and cameras. Then, the original point cloud data is augmented to obtain augmented point cloud data, and the augmented point cloud data is used to train the downstream task model. Finally, the trained downstream task model is used for reasoning to perform the target task. However, due to the characteristics of point cloud data itself, such as rotation invariance and scaling invariance, related technologies generally have problems such as insufficient point cloud augmentation methods and diversity, which leads to low training efficiency and overfitting. At the same time, related technologies do not solve the fundamental problem of poor task execution due to sparsity.

[0057] Based on this, an embodiment of the present invention provides a point cloud data processing method, which uses original data and augmented data to train a neural network respectively, and uses a neural network with augmented data as data to train a neural network with sparse data as input, so that the output and task processing capabilities of the latter are close to those of the former, and a trained neural network model is obtained. In this way, when processing sparse original point cloud data, the network model can output better processing results, thereby improving the effect of executing point cloud processing tasks.

[0058] It should be understood that the above-mentioned example point cloud processing scenario is for the purpose of more clearly illustrating the technical solution of the embodiment of the present invention, and does not constitute a limitation on the technical solution provided by the embodiment of the present invention. A person of ordinary skill in the art can know that with the emergence of new point cloud data processing scenarios, the technical solution provided by the embodiment of the present invention is also applicable to similar technical problems.

[0059] To illustrate the composition of the point cloud processing device in the embodiment of the present invention, please refer to Figure 1 , Figure 1 is a schematic diagram of the structure of a point cloud data processing system provided by an embodiment of the present invention, such as Figure 1As shown, the schematic diagram shows the relationship between the point cloud processing device 10 and other components in the system and the constituent modules of the point cloud processing device 10. The schematic diagram includes a point cloud processing device 10, a terminal device 20 and a communication connection 30, wherein the point cloud processing device 10 includes an acquisition module 101, an augmentation module 102, a model training module 103, a difficult case data processing module 104, an export module 105, and an inference module 106. Specifically, the model training module 103 also includes a first training module 1031, a second training module 1032, an interface generation module 1033, and a difficult case data processing module 1034. The terminal device 20 is connected to the point cloud processing device 10 via a communication connection, and the user can send instructions to the point cloud processing device 10 through the terminal device 20. Specifically, the point cloud processing device 10 receives information or instructions input by the user at the terminal device 20 via the communication connection 20. In an embodiment provided by the present invention, the content available for user setting is obtained by an interface in the point cloud processing device 10, and the interface is, for example, an application programming interface (Application Programming Interface, API) or an API-based configuration interface, or other interactive interfaces that can interact with tenants, which is not limited by the embodiment of the present invention. The interactive interface generated by the point cloud processing device 10 can be displayed in the terminal device 20. Some or all of the modules in the device can be applied to computing instances, specifically, some or all of them can be applied to computing instances such as data centers, servers, virtual machines or containers, which is not limited by the present invention. Among them, preferably, the acquisition module 101, the augmentation module 102, the model training module 103, the difficult example data processing module 104, and the export module 105 can be deployed in at least one server, and the reasoning module 106 can be deployed in servers in other data centers or on terminal devices, which is not limited by the present invention.

[0060] Based on the point cloud data processing device 10 and the point cloud data processing system established above, the embodiment of the present invention further discloses a point cloud data processing method, please refer to Figure 2 , Figure 2 It is a flow chart of a point cloud data processing method provided by an embodiment of the present invention. The method can be applied to computing instances. Specifically, it can be partially or completely applied to computing instances such as data centers, servers, virtual machines or containers. The present invention does not limit the application scenarios of the point cloud data processing method. The method includes but is not limited to the following steps.

[0061] S101: Obtain an original point cloud dataset.

[0062] Specifically, the acquisition module 101 acquires at least one original point cloud data set. In an embodiment provided by the present invention, the point cloud processing device 10 includes a point cloud database. The original point cloud data set can be stored in the original point cloud database in advance, or can be uploaded to the point cloud processing device online by the user. For an embodiment of a user importing an original data point cloud, please refer to Figure 3 , Figure 3 is a schematic diagram of a point cloud data processing interactive interface provided by an embodiment of the present invention, such as Figure 3 As shown, the user imports the original point cloud data set in the original data import interface. For example, when collecting point cloud data on public roads, a 3D point cloud collection device such as a laser radar or a depth camera can be mounted on a mobile device. While the mobile device is moving, it collects a 3D point cloud data set at fixed time intervals. This 3D point cloud data set is the point cloud data set at that moment and the moving position. Specifically, the collection scene may include target objects and background objects. Therefore, a 3D point cloud data set may include point cloud data of target objects and point cloud data of background objects. After the collection is completed, a data packet including at least one 3D point cloud data set may be formed, and the data packet is the 3D point cloud data packet of the road section. The data packet includes point cloud data of road A section A1, road A section A2...road C intersection C3 on the roads in the area. Optionally, data at different times at the same location may also be included.

[0063] In one embodiment provided by the present invention, the original point cloud data set acquired by the acquisition module 101 is a data set that has been annotated. For example, the annotation process may be that the annotator uses the annotation software to select part or all of the objects in the collected 3D point cloud data, and the object may include the target object, and add labels or attributes to the selected object. Optionally, the object may be selected by a 3D detection box as a gold standard. Thus, the point cloud data corresponding to some objects in the original point cloud data set acquired by the acquisition module 101 have labels. Optionally, in other embodiments provided by the present invention, the point cloud data processing device may include a data annotation module, which is used to generate a data annotation interface, and the user can annotate the unlabeled point cloud data collected by it through the data annotation interface.

[0064] S102: Determine the target object point cloud in the original point cloud data set.

[0065] Specifically, the user uses a terminal device to send instructions to the point cloud processing device via a communication connection to determine the target object in the original point cloud data set. The target object can be the target object of a downstream task. For example, it can be a target object for target detection, or for semantic segmentation, etc. The present invention does not limit the specific downstream tasks. The original point cloud data set may include point cloud data representing a variety of labeled objects. The user can confirm at least one target object among the multiple objects by entering or clicking on keywords, attribute tags, etc., and the point cloud data corresponding to the at least one target object is the target point cloud data. In one possible implementation, the point cloud processing device includes a point cloud screening module, which selects an object bounding box based on keywords and corresponds the position of the bounding box to the position of the original sparse point cloud. In other embodiments provided by the present invention, this step can be omitted. In this scenario, the object labeled in the original point cloud data set obtained by the point cloud data device is the target object. Please refer to the specific implementation. Figure 4 , Figure 4 It is another schematic diagram of a point cloud data processing interactive interface provided by an embodiment of the present invention. The schematic diagram shows a target object confirmation interface, and the user can select the target object on the interface. In this schematic diagram, the user selects a vehicle as the target object, and the point cloud data corresponding to the tag or label of the vehicle is the target point cloud data for subsequent use.

[0066] S103: Augment the original point cloud dataset.

[0067] S1031: Densifying the sparse point cloud in the target point cloud dataset.

[0068] Specifically, the augmentation module 102 determines that the relatively sparse point cloud data in the target point cloud data is densified. For example, the augmentation module 102 can determine that the point cloud data with a point cloud density lower than a sparse threshold is a relatively sparse point cloud, and the unit of the point cloud density is, for example, the number of points per cubic meter. Optionally, the value of the sparse threshold can be input by the user, for example, the input parameter low_bound=80. The densification processing of point cloud data can also be called point cloud data completion. In some embodiments provided by the present invention, the densification processing can be performed in the following ways: deep learning-based methods: these methods use deep learning models to predict missing point cloud data, which usually require a large amount of training data, but they can produce high-quality results; interpolation-based methods: these methods use interpolation techniques to estimate missing point cloud data, which usually require dense sampling of point cloud data; geometry-based methods: these methods use the geometry of point cloud data to estimate missing point cloud data, which usually require dense sampling of point cloud data. The present invention does not limit the point cloud data densification method provided by the point cloud data processing device. A diagram of the effect of point cloud data densification can be seen in Figure 5b , Figure 5b FIG5 is a schematic diagram of a point cloud data augmentation effect provided by an embodiment of the present invention, as shown in FIG5 . Figure 5a is the original point cloud data set, which includes pedestrians, vehicles, trees and other objects. After confirming that the vehicle is the target object, car1, 2, 3, 4, and 5 are the point cloud data of the target object. Among them, car1 and car2 are target point clouds with point cloud density lower than the sparse threshold. The target point clouds of car1 and car2 are densified to increase their point cloud density, and the result is Figure 5b The dense point cloud data of car1 and car2.

[0069] As mentioned above, the point clouds collected by point cloud acquisition equipment have problems of sparseness and holes, while dense point clouds can make the execution of downstream point cloud processing tasks more efficient. Therefore, completing the sparse parts in the target point cloud data can improve the effects of point cloud processing tasks such as point cloud feature extraction.

[0070] S1032: Copy the dense point cloud in the target point cloud data.

[0071] Specifically, optionally, the augmentation module 102 copies the dense point cloud in the point cloud data of the target object. For example, the point cloud data with a point cloud density higher than a dense threshold can be determined as a dense point cloud. Optionally, the value of the dense threshold can be input by the user, for example, the input parameter up_bound=80. The augmentation module can perform random angle rotation and / or random distance translation on the copied point cloud to generate one or more copies of the dense point cloud, increase the number of dense point clouds in the augmented point cloud, and generate an augmented point cloud data set. A rendering of the point cloud data replication can be seen in Figure 5b , Figure 5b is a schematic diagram of a point cloud data augmentation effect provided by an embodiment of the present invention, wherein car3 and car4 are target point clouds with point cloud density higher than a dense threshold, and the target point clouds of car3 and car4 are copied to increase the number of these high-quality point cloud data, and the result is Figure 5b The point cloud data of car3 and car4 are copied and represented by dotted lines.

[0072] As mentioned above, duplicating a copy of the dense point cloud in the target point cloud can increase the number of high-quality target object point clouds in the point cloud dataset, thereby improving the effect of the point cloud processing task.

[0073] In an embodiment provided by the present invention, the dense point cloud may be cropped using a detection frame used to annotate the target object point cloud in the original point cloud data set.

[0074] Since the neural network trained by the sparse point cloud and the neural network trained by the dense point cloud need to be trained for consistency later, when the augmented point cloud dataset includes copied object point cloud data, the augmentation module 102 needs to perform the same operation on the same data in the original point cloud dataset. That is, the augmentation module 102 performs the same random move operation on the copied object point cloud data and adds it to the original point cloud dataset. In other embodiments provided by the present invention, the same operations can be performed on the dense point cloud data of the original point cloud dataset itself with reference to the operations on the dense point cloud in the augmented point cloud dataset, such as recording the movement angle and distance of the former, etc. The point cloud data can also be copied in the augmented point cloud dataset and added to the augmented point cloud dataset and the original point cloud dataset respectively. The present invention does not limit the method of augmenting the dense point cloud. A schematic diagram of performing the same operation on the original point cloud dataset can be seen in Figure 5d , that is, in the original point cloud dataset, Figure 5d The point cloud data of car3 and car4 are copied and represented by dotted lines.

[0075] S1033: Receive a point cloud data filtering instruction sent by the user, and delete the data according to the instruction.

[0076] In the scenario of the augmentation method of copying the dense point cloud in S1032, point cloud collisions of objects may occur, for example, point clouds after being copied and randomly moved, or point clouds of objects in the point cloud data may overlap. At this time, the point cloud processing device 10 may optionally generate an interactive interface for displaying the augmentation results of the point cloud data. For an implementation of the interactive interface, please refer to FIG5, where: Figure 5b To augment the point cloud dataset, there are two replicated and randomly moved vehicles overlapping in the lower right corner of the figure. The user can send instructions to the point cloud processing device, for example, Figure 5c , select the vehicle on the right and delete the corresponding point cloud, so that the point cloud data has a correct and realistic semantic expression. Optionally, in this step, the augmentation module 102 or other modules in the point cloud processing device 10 can automatically identify and delete overlapping or colliding point clouds to ensure that the semantics of the point cloud in the picture is reasonable.

[0077] In summary, Figure 5ais the original point cloud dataset. In the process from S102 to S103, the user selects the vehicle as the target object, wherein car1 and car2 are target point clouds with point cloud density lower than the sparse threshold, and car3 and car4 are target point clouds with point cloud density higher than the dense threshold. In S1031, the target point clouds of car1 and car2 are densified to increase their point cloud density. In S1032, the target point clouds of car3 and car4 are copied and randomly moved to increase the number of dense point clouds. At this time, the schematic diagram of the augmented point cloud dataset is as follows Figure 5b , the data set contains car1 and car2 after densification, and at least one car3 and car4 after replication and random movement; the schematic diagram of the original point cloud data set is as follows Figure 5d , the data set includes at least one car3 and car4 that have been copied and randomly moved. In other embodiments provided by the present invention, only step S1031 may be performed without performing other steps of S103, which is not limited.

[0078] S104: Train a first neural network model using the augmented point cloud dataset.

[0079] Specifically, the first training module 1031 uses the augmented point cloud data set to train the first neural network model, wherein the first neural network model is a neural network model for performing point cloud processing tasks. As mentioned above, the point cloud processing tasks may be target detection, target classification, target segmentation, point cloud registration, target denoising, and the like. As mentioned above, the augmented point cloud data set includes densified sparse point cloud data, and optionally, also includes dense point cloud data that is copied and randomly moved. The training process of the first neural network calculates the loss with the target task as supervision. The augmented point cloud data set is input into the first neural network model to obtain an output result as a predicted value, and the result error is calculated between the result and the true value and fed back to the network model for training.

[0080] Specifically, a neural network model structure for point cloud data processing can be composed of a backbone, a neck and a head, wherein the backbone is responsible for extracting features of the input image for subsequent processing and analysis; the main function of the neck is to reduce the dimension or adjust the features from the backbone to better adapt to the task requirements; the head generates the final output by inputting the features processed by the neck. The structure of the head varies according to the task. For example, for image classification tasks, a softmax classifier can be used; for target detection tasks, a bounding box regressor and a classifier can be used. Preferably, the neural network model of an embodiment of the present invention is a neural network model that includes all the above structures. In other embodiments provided by the present invention, some of the above-mentioned structures may also be included. The present invention does not limit the structure of the neural network model.

[0081] S105: Train a second neural network model using the original point cloud dataset and the trained first neural network model.

[0082] Specifically, the first implementation provided by the embodiment of the present invention can be that the second training module 1032 uses the sparse original point cloud data set and the trained first neural network model to simultaneously train the second neural network model, that is, while using the task supervision of the second neural network itself to calculate the loss, the trained first neural network model is used to supervise the consistency training of the second neural network model and calculate the loss, that is, the two are synchronously constrained to train the second neural network model; at this time, there are two losses for supervision in the training of the second neural network model, one is the loss of the first neural network for consistency training of the second neural network, and the other is the loss of the second neural network for performing the target task using the original point cloud data set. Specifically, in the process of using the first neural network model to perform consistency training on the second neural network model, the first neural network is the teacher network and the second neural network is the student network. The goal of the student network is to improve its performance by learning the knowledge of the teacher network, so it needs to be as close to the output of the teacher network as possible during the training process. In this process, the loss function usually consists of two parts: one is the distance between the prediction result of the student network and the prediction result of the teacher network, and the other is the distance between the prediction result of the student network and the true label. The weights of these two parts of the loss function can be adjusted according to the specific task. Among them, the distance between the prediction results of the student network and the prediction results of the teacher network can be calculated using methods such as mean square error (MSE) or cross entropy (CE), while the distance between the prediction results of the student network and the true label is usually calculated using cross entropy (CE). This application does not limit the specific implementation method of consistency training.

[0083] A second implementation method provided by an embodiment of the present invention can be that the second training module 1032 first uses a sparse original point cloud data set to pre-train a second neural network model, and then uses the first neural network model to train the pre-trained second neural network model. That is, the sparse original point cloud data set is first input into the second neural network model for pre-training. At this time, the loss is calculated based on the result of the second neural network using the original point cloud data to perform the target task. After that, the parameters of the trained first neural network model are fixed, and the trained first neural network model and the pre-trained second neural network model are trained for consistency together.

[0084] As a result, the features output by the second neural network model are made closer to the features output by the first neural network. The trained second neural network can extract high-quality point cloud features and output high-quality task execution results when taking a sparse original point cloud data set as input.

[0085] S106: Generate a visualization interface to display the processing results of the two neural networks.

[0086] Specifically, a visualization interface including a UI is generated, and the visualization interface displays the processing results of the trained first neural network and the second neural network. Preferably, the training results display the output results of a certain original point cloud data set on the two networks in pairs. Specifically, the output result referred to in the embodiment of the present invention can be the output of the backbone or backbone+neck, that is, the output result of the intermediate process, such as the result of feature extraction; it can also be the output of the head, such as the target classification result, bounding box or mask, etc. Preferably, the output result of the first neural network can be the output result when the training is about to end, or it can be the inference result of fixing the network parameters after the training is completed; the output result of the second neural network can be the output result when the training is stable, that is, the loss convergence stage. At this time, the second neural network is still in the training process, and each original point cloud data set may be iteratively trained and output the training result. The present invention does not limit the output stage of the result. For example, the visualization interface can display the point cloud data feature extraction results of the two networks for a certain data set in the form of a heat map; for another example, the visualization interface can display the target segmentation results of the two networks for a certain data set in the form of a black and white mask. The present invention does not limit the display form. A schematic diagram of the visualization interface can be seen in Figure 6a-6d , as shown in FIG6 , FIG6 is a schematic diagram of a training result comparison provided by an embodiment of the present invention, wherein, Figure 6a , Figure 6b is the output of the second neural network model, Figure 6c , Figure 6d is the output of the first neural network model, where Figure 6a , Figure 6c Corresponding to the same original point cloud dataset, Figure 6b , Figure 6d Corresponding to the same original point cloud dataset, if the two results corresponding to the same original point cloud dataset are quite different, then the training effect of the neural network on this dataset is not good, and the original point cloud dataset can be identified as a difficult example dataset.

[0087] S107: Receive a difficult example data set screening instruction sent by a user, and confirm the difficult example data set.

[0088] Specifically, by viewing the above-mentioned visualization interface, the user can compare the output results of the same data set under the two neural networks to intuitively demonstrate the processing capabilities of the two networks for the data. Optionally, if the two results are quite different, it means that the second neural network is not capable of processing the point cloud data set, and the point cloud data set may be a difficult data set. At this time, the difficult data processing module 104 receives the user's click and other instructions to confirm that the data set corresponding to the processing result is a difficult data set. If the two results differ slightly, it means that the ability of the second neural network to process the point cloud data set has approached the processing capability of the first neural network under the augmented point cloud data set, and the data set may not be a difficult data set. Therefore, the user can send a difficult data set screening instruction, specifically, one or more results with a large difference can be selected for the user, and then it can be confirmed that the data set corresponding to the result is a difficult data set.

[0089] S108: Augment the original point cloud dataset corresponding to the difficult example dataset and put it back into the original point cloud dataset.

[0090] Specifically, after the difficult data processing module 104 confirms that a certain original point cloud data set or its corresponding augmented point cloud data set is a difficult data set, it makes a small adjustment to the data set and then returns to multiple original point cloud data sets as training data to continue training the second neural network model. For example, in a possible implementation provided by an embodiment of the present invention, some difficult data sets appear because the morphology of the target point cloud is relatively rare. For example, in a traffic scene, a certain appearance of a vehicle is relatively rare, or a certain angle of a vehicle is relatively rare, which may cause the data set to be a difficult data set. Therefore, the point cloud data processing device makes a small adjustment to the target point cloud data in the difficult data set, such as a small displacement and / or rotation, and then merges it with other point cloud data in the difficult data set, or with other original point cloud data sets, or with background point cloud data in other point cloud data sets, to generate an augmented difficult data set containing the target point cloud. Optionally, a user's point selection instruction can be received to confirm that one or more target point cloud data in the difficult data set is the main reason for the appearance of the difficult data, and the target point cloud data is fine-tuned and augmented. In other embodiments of the present invention, other augmentation methods may be used to augment the difficult example data set, and the present invention does not limit the augmentation method of the difficult example data set.

[0091] Optionally, after the difficult example dataset flows back to the original point cloud dataset, the augmentation module 102 can continue to execute the original method of augmenting the original point cloud dataset to train different neural networks. Optionally, the original point cloud dataset and augmented point cloud dataset corresponding to the difficult example dataset can perform the same adjustment operation on a target point cloud that causes the difficult example to directly generate the original point cloud dataset and augmented point cloud dataset for continued training.

[0092] Preferably, when adding the augmented hard example dataset to the original point cloud dataset, the second neural network model is still in the training process, that is, the user can add the hard example dataset for training in real time during the training process to continuously optimize the training effect.

[0093] By adding more difficult datasets to the training data, the model can be helped to better learn the distribution of the data and improve the generalization ability of the model. In addition, difficult datasets can also help the model better learn the noise of the data, thereby improving the robustness of the model and enhancing the task execution results of the second neural network model under different data.

[0094] S109: Export the trained second neural network model.

[0095] Specifically, the second neural network model can be packaged into a file by the export module 105 for the user to download and deployed on the user's local computing instance, or it can be deployed on the same or different computing instance as other modules of the point cloud processing device, and provide a corresponding reasoning interaction interface for the user to further use the model for reasoning. The present invention does not limit the deployment location of the trained neural network model.

[0096] S110: Perform inference using the trained second neural network model.

[0097] Specifically, the user uploads a point cloud data set to the trained second neural network model, and the point cloud data set is the original data set that has not been augmented. The reasoning module 106 infers the execution result of the target task using the trained second neural network model. An interactive interface for this step can be referenced Figure 7 ,like Figure 7 As shown, Figure 7 This is another schematic diagram of a point cloud data processing interactive interface provided by an embodiment of the present invention. The user imports the original point cloud data set and inputs it into the trained second neural network for inference. When the point cloud data processing task is a target segmentation task, the inference result is a target segmentation mask for each original point cloud data set, and the result file can be downloaded by the user.

[0098] The point cloud data processing method and point cloud data processing device provided in the present invention can also be applied to 2D point clouds, and the present invention is not limited to this.

[0099] In other embodiments provided by the present invention, the point cloud data processing method and device provided by the embodiments of the present invention can be applied to point cloud data processing or analysis tasks such as point cloud target detection, point cloud target classification, and point cloud target segmentation, and can be applied to scenarios such as autonomous driving, robots, virtual reality, architectural design, and medical imaging. The present invention does not limit the application scenarios of the embodiments.

[0100] Therefore, through the point cloud data processing method provided by the embodiment of the present invention, in the process of point cloud data processing, augmented point cloud data of the original point cloud data can be obtained through data augmentation, a neural network can be trained with the augmented point cloud data, and then the original point cloud data and the aforementioned neural network can be used to train the final desired neural network for performing the target task. In this way, the neural network can obtain high-quality processing results when taking sparse original point cloud data as input, thereby solving the fundamental problem of poor execution effect of point cloud processing tasks caused by sparse point cloud, and at the same time expanding the augmentation method of point cloud data.

[0101] It should be noted that: when the point cloud data processing device provided in the above embodiment processes the original point cloud, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Figure 8 The point cloud data processing method embodiments shown belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0102] See also Fig. 9 , Fig. 9 is a schematic diagram of the structure of a point cloud data processing device provided by an embodiment of the present invention. The point cloud data processing device 700 can be implemented by software, hardware or a combination of both to become part or all of a point cloud data processing server. Fig. 9 The point cloud data processing device 700 includes: an acquisition module 701, an augmentation module 702, a model training module 703, an export module 704, and an inference module 705.

[0103] An acquisition module 701 is used to acquire at least one original point cloud data set of a scene, where the scene includes a target object;

[0104] An augmentation module 702, configured to augment the point cloud data of the target object included in at least one original point cloud data set to generate at least one augmented point cloud data set corresponding to the at least one original point cloud data set;

[0105] A model training module 703 is used to train a first neural network model using at least one augmented point cloud data set to obtain a trained first neural network model, where the trained first neural network model is used to perform a point cloud data processing task;

[0106] An export module 704, configured to train a second neural network model using at least one original point cloud data set and the trained first neural network model;

[0107] The reasoning module 705 is used to obtain a trained second neural network model, and the trained second neural network model is used to perform the point cloud data processing task.

[0108] Among them, the above-mentioned acquisition module 701, augmentation module 702, model training module 703, export module 704, and reasoning module 705 can all be implemented by software, or can be implemented by hardware. Exemplarily, the following takes the acquisition module 701 as an example to introduce the implementation method of the acquisition module 701. Similarly, the implementation methods of the augmentation module 702, the model training module 703, the export module 704, and the reasoning module 705 can refer to the implementation method of the acquisition module 701.

[0109] As an example of a software functional unit, the acquisition module may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the above-mentioned computing instance may be one or more. For example, the acquisition module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.

[0110] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0111] As an example of a hardware functional unit, the acquisition module may include at least one computing device, such as a server, etc. Alternatively, module A may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0112] The multiple computing devices included in the acquisition module 701 can be distributed in the same region or in different regions. The multiple computing devices included in the acquisition module can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the acquisition module can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0113] It should be noted that, in other embodiments, the acquisition module 701 can be used to execute any step in the point cloud data processing method, the augmentation module 702, the model training module 703, the export module 704, and the reasoning module 705 can be used to execute any step in the point cloud data processing method, and the steps that the acquisition module 701, the augmentation module 702, the model training module 703, the export module 704, and the reasoning module 705 are responsible for implementing can be specified as needed. By respectively implementing different steps in the point cloud data processing method through the acquisition module 701, the augmentation module 702, the model training module 703, the export module 704, and the reasoning module 705, all functions of the point cloud data processing device are realized.

[0114] In an embodiment of the present invention, during the point cloud data processing process, the augmented point cloud data of the original point cloud data can be obtained through data augmentation, a neural network can be trained with the augmented point cloud data, and then the original point cloud data and the aforementioned neural network can be used to train the neural network that is ultimately intended to perform the target task. In this way, the neural network can obtain a high-quality processing result when the sparse original point cloud data is used as input, which solves the fundamental problem that the sparse point cloud leads to poor execution of the point cloud processing task, and at the same time expands the augmentation method of the point cloud data. It should be noted that when the point cloud data processing device provided in the above embodiment processes the point cloud data, only the division of the above functional modules is used as an example. In actual applications, the above functional distribution can be completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the point cloud data processing device provided in the above embodiment belongs to the same concept as the point cloud data processing method embodiment. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0115] See also Fig.10 , Fig.109 is a schematic diagram of a computing device provided by an embodiment of the present invention, wherein the computing device 900 includes: a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate with each other via the bus 902. The embodiment of the present invention does not limit the number of processors and memories in the computing device 900.

[0116] When implementing the technical solution provided by the embodiment of the present invention, the computing device 900 may be a server that performs point cloud data processing tasks.

[0117] The bus 902 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 The bus 902 may include a path for transmitting information between various components of the computing device 900 (eg, the memory 906, the processor 904, and the communication interface 908).

[0118] The processor 904 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0119] The memory 906 may include a volatile memory, such as a random access memory (RAM). The processor 904 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0120] In the embodiment of the present invention, the memory 906 stores executable program codes, and the processor 904 executes the executable program codes to respectively implement the functions of the aforementioned acquisition module 701, augmentation module 702, model training module 703, export module 704, and reasoning module 705, thereby implementing Figure 2 That is, the memory 906 stores the method for executing the above-mentioned point cloud data processing method. Figure 2 Instructions for the point cloud data processing method shown.

[0121] The communication interface 908 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 900 and other devices or communication networks.

[0122] An embodiment of the present invention further provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server, for example, a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0123] like Fig.11 As shown, the computing device cluster includes at least one computing device 900. The memory 906 in one or more computing devices 900 in the computing device cluster may store the same Figure 3 Instructions for the point cloud data processing method shown.

[0124] In some possible implementations, the memory 906 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the point cloud data processing method. In other words, the combination of one or more computing devices 900 may jointly execute instructions for executing Figure 2 Instructions for the point cloud data processing method shown.

[0125] It should be noted that the memory 906 in different computing devices 900 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the point cloud data processing device. That is, the instructions stored in the memory 906 in different computing devices 900 can implement the functions of one or more of the aforementioned acquisition module 701, augmentation module 702, model training module 703, export module 704, and reasoning module 705.

[0126] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network, etc. Fig.12 A possible implementation is shown, such as Fig.12 As shown, two computing devices 900A and 900B are connected via a network. Specifically, they are connected to the network via a communication interface in each computing device.

[0127] Using computing device clusters to achieve the above Fig.12 As an example, the point cloud data processing method shown in Fig.12 In the implementation shown, the memory 906 in the computing device 900A stores instructions for executing the functions of the acquisition module 701, the augmentation module 702, and the model training module 703. At the same time, the memory 906 in the computing device 900B stores instructions for executing the functions of the export module 704 and the reasoning module 705.

[0128] Fig.12 The connection method between the computing device clusters shown can be based on the fact that the point cloud data processing method provided by the present invention may require a large amount of raw data for data processing and neural network training. Therefore, it is considered to implement multiple functions in different computing devices, and the functions implemented by the instruction receiving module 704, the estimation module 706, and the resource status acquisition module 705 are handed over to the computing device 900B for execution, so as to reduce the computing amount of the computing device 900A.

[0129] It should be understood that Fig.12 The functions of the computing device 900A shown in FIG. 9A may also be completed by multiple computing devices 900. Similarly, the functions of the computing device 900B may also be completed by multiple computing devices 900.

[0130] The embodiment of the present invention also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similarly referred to as Fig.11 and Fig.12 Connection mode of computing device cluster. The difference is that the memory 906 in one or more computing devices 900 in the computing device cluster may store the same instructions for executing the point cloud data processing method.

[0131] In some possible implementations, the memory 906 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the point cloud data processing method. In other words, the combination of one or more computing devices 900 may jointly execute instructions for executing the point cloud data processing method.

[0132] The embodiment of the present invention also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the above Figure 3 The point cloud data processing method shown.

[0133] The embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the above Figure 3 The point cloud data processing method shown.

[0134] It should be understood that the "multiple" mentioned herein refers to two or more than two. In the description of the embodiments of the present invention, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of the embodiments of the present invention, in the embodiments of the present invention, the words "first", "second" and the like are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not limit them to be necessarily different.

[0135] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of the present invention are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A point cloud data processing method, characterized in that: The method comprises: Acquire at least one raw point cloud data set of a scene, the scene including a target object; Augmenting the point cloud data of the target object included in the at least one original point cloud data set to generate at least one augmented point cloud data set corresponding to the at least one original point cloud data set; Using the at least one augmented point cloud data set to train a first neural network model to obtain a trained first neural network model, wherein the trained first neural network model is used to perform a point cloud data processing task; Training a second neural network model using the at least one original point cloud data set and the trained first neural network model; A trained second neural network model is obtained, and the trained second neural network model is used to perform the point cloud data processing task.

2. The method according to claim 1, characterized in that Training a second neural network model using the at least one original point cloud data set and the trained first neural network model comprises: Pre-training a second neural network model using the at least one original point cloud data set to obtain a pre-trained second neural network model, and training the pre-trained second neural network model using the trained first neural network model; or, The second neural network model is trained simultaneously using the at least one original point cloud data set and the trained first neural network model.

3. The method according to claim 1 or 2, characterized in that: The step of augmenting the point cloud data of the target object included in the at least one original point cloud data set comprises: First point cloud data with a density lower than a sparse threshold is determined from the point cloud data of the target object included in the at least one original point cloud data set to be densified, and the at least one augmented point cloud data set includes the densified first point cloud data.

4. The method according to any one of claims 1 to 3, characterized in that: The step of augmenting the point cloud data of the target object included in the at least one original point cloud data set comprises: Second point cloud data with a density higher than a density threshold is determined from the point cloud data of the target object included in the at least one original point cloud data set and copied, and the at least one augmented point cloud data set includes the copied second point cloud data.

5. The method according to claim 4, characterized in that Before training the second neural network model using the at least one original point cloud data set and the trained first neural network model, the method further includes: Performing the copying on the second target point cloud data included in a certain original point cloud data set; The second target point cloud data included in the copied original point cloud data set is added to the original point cloud data set.

6. The method according to any one of claims 1 to 5, characterized in that: The step of training the first neural network model using the at least one augmented point cloud data set to obtain the trained first neural network model comprises: Using an augmented point cloud data set corresponding to a certain original point cloud data set to train the first neural network model, to obtain the trained first neural network model, and to obtain a first task processing result of the original point cloud data set; The using the at least one original point cloud data set and the trained first neural network model to train the second neural network model comprises: Using the original point cloud data set and the trained first neural network model to train the second neural network model, to obtain a second task processing result of the original point cloud data set; The method further comprises: Generate a visualization interface, wherein the visualization interface is used to display the first task processing result and the second task processing result of the original point cloud data set in pairs in the form of images or data; A difficult example data set confirmation instruction is received, where the difficult example data set confirmation instruction is used to indicate that the original point cloud data set corresponding to the first task processing result or the second task processing result is a difficult example data set.

7. The method according to claim 6, characterized in that The method further comprises: Augmenting the hard example dataset to generate at least one augmented hard example dataset; The at least one augmented hard example dataset is added to the at least one original point cloud dataset.

8. The method according to claim 7, characterized in that The scene also includes background objects, and the step of augmenting the hard example data set to generate at least one augmented hard example data set includes: Extracting point cloud data of the target object in the difficult example dataset; The point cloud data of the target object in the hard example data set and the point cloud data of the background object in the at least one original point cloud data set are merged to generate the at least one augmented hard example data set.

9. A point cloud data processing device, characterized in that: The device comprises: An acquisition module, configured to acquire at least one original point cloud data set of a scene, wherein the scene includes a target object; An augmentation module, configured to augment the point cloud data of the target object included in the at least one original point cloud data set to generate at least one augmented point cloud data set corresponding to the at least one original point cloud data set; A first training module, configured to train a first neural network model using the at least one augmented point cloud data set to obtain a trained first neural network model, wherein the trained first neural network model is used to perform a point cloud data processing task; A second training module, configured to train a second neural network model using the at least one original point cloud data set and the trained first neural network model; An export module is used to obtain the trained second neural network model, and the trained second neural network model is used to perform the point cloud data processing task.

10. The apparatus according to claim 9, wherein the second training module is configured to: Pre-training a second neural network model using the at least one original point cloud data set to obtain a pre-trained second neural network model, and training the pre-trained second neural network model using the trained first neural network model; or, The second neural network model is trained simultaneously using the at least one original point cloud data set and the trained first neural network model.

11. The device according to claim 9 or 10, characterized in that The augmentation module is used to: First point cloud data with a density lower than a sparse threshold is determined from the point cloud data of the target object included in the at least one original point cloud data set to be densified, and the at least one augmented point cloud data set includes the densified first point cloud data.

12. The device according to any one of claims 9 to 11, wherein the augmentation module is configured to: Second point cloud data with a density higher than a density threshold is determined from the point cloud data of the target object included in the at least one original point cloud data set and copied, and the at least one augmented point cloud data set includes the copied second point cloud data.

13. The device according to claim 12, characterized in that The augmentation module is further used for: Copying the second target point cloud data included in a certain original point cloud data set; The second target point cloud data included in the copied original point cloud data set is added to the original point cloud data set.

14. The device according to any one of claims 9 to 13, characterized in that The first training module is used to: Using an augmented point cloud data set corresponding to a certain original point cloud data set to train the first neural network model, to obtain the trained first neural network model, and to obtain a first task processing result of the original point cloud data set; The second training module is used to: Using the original point cloud data set and the trained first neural network model to train the second neural network model, to obtain a second task processing result of the original point cloud data set; The device also includes: An interface generation module, used to generate a visualization interface, wherein the visualization interface is used to display the first task processing result and the second task processing result of the original point cloud data set in pairs in the form of images or data; The difficult data processing module is used to receive a difficult data set confirmation instruction, and the difficult data set confirmation instruction is used to indicate that the original point cloud data set corresponding to the first task processing result or the second task processing result is a difficult data set.

15. The device according to claim 14, characterized in that The difficult example data processing module further includes: A hard example data augmentation module, configured to augment the hard example data set to generate at least one augmented hard example data set; The hard example data adding module adds the at least one augmented hard example data set to the at least one original point cloud data set.

16. The device according to claim 15, characterized in that The scene also includes background objects, and the hard example data augmentation module is used to: Extracting point cloud data of the target object in the difficult example dataset; The point cloud data of the target object in the hard example data set and the point cloud data of the background object in the at least one original point cloud data set are merged to generate the at least one augmented hard example data set.

17. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the point cloud data processing method according to any one of claims 1 to 8.

18. A computer program product comprising instructions, characterized in that When the instruction is executed by a computing device cluster, the computing device cluster executes the point cloud data processing method as described in any one of claims 1 to 8.

19. A computer-readable storage medium, characterized in that: It includes computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the point cloud data processing method according to any one of claims 1 to 8.

Images