A three-dimensional model construction method, a neural network training method, and a device

Abstract

The application discloses a three-dimensional model construction method, a neural network training method and device in the field of artificial intelligence, which are used for realizing instance segmentation at the level of a graph element and obtaining a simplified three-dimensional model. The method comprises the following steps: firstly, acquiring point cloud data, wherein the point cloud data comprises a plurality of points and information corresponding to each point; then, inputting the point cloud data into a sub-manifold prediction network to obtain prediction results of the plurality of points, wherein the prediction results are used for identifying whether each point and a neighboring point in the plurality of points belong to a same sub-manifold, wherein the sub-manifold prediction network extracts features corresponding to each point from the point cloud data, and determines whether each point and the neighboring point belong to the same sub-manifold according to the features corresponding to each point; and filtering a plurality of corner points from the plurality of points according to the prediction results of the plurality of points, wherein the plurality of corner points comprise points on boundaries of each sub-manifold formed by the plurality of points; and constructing a three-dimensional model according to the plurality of corner points, wherein a mesh formed by the plurality of corner points constitutes a manifold in the three-dimensional model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method for constructing a three-dimensional model, a method for training a neural network, and an apparatus. Background Technology

[0002] The input 3D reconstruction data is reconstructed into a vectorized 3D model, which can be used for rapid rendering and interaction on the terminal. 3D models can be acquired in two ways: manual modeling and reconstruction using acquisition equipment. Reconstruction using acquisition equipment can realistically reproduce the scene, but the data is often noisy and massive. Manual modeling typically uses basic shapes (such as planes, cylinders, spheres, and cones) to abstractly fit the real environment; therefore, the model is smaller, less precise, but has a better structured representation.

[0003] For example, the PointNet++ network can be used to extract features from point clouds and output the instance ID, normal vector, and type of instance primitive for each point. However, this method can only identify and reconstruct a small number of objects. For scenarios with a large number of instances, the required cost is too high or even impossible, resulting in weak generalization ability. Summary of the Invention

[0004] This application provides a method for constructing a 3D model, a method for training a neural network, and an apparatus for performing instance segmentation at the primitive level to obtain a simplified 3D model.

[0005] In view of this, in a first aspect, this application provides a method for constructing a three-dimensional model, comprising: first, acquiring point cloud data, which includes multiple points and information corresponding to each point, wherein the information corresponding to each point may include information such as depth, pixel value, brightness value, or intensity value; subsequently, inputting the point cloud data into a submanifold prediction network to obtain prediction results for multiple points, wherein the prediction results are used to identify whether each point and its adjacent points belong to the same submanifold, wherein the submanifold prediction network extracts features from the point cloud data to obtain features corresponding to each point, and determines whether each point and its adjacent points belong to the same submanifold based on the features corresponding to each point; selecting multiple corner points from the multiple points based on the prediction results of the multiple points, wherein the multiple corner points include points on the boundaries of each submanifold formed by the multiple points; and constructing a three-dimensional model based on the multiple corner points, wherein the mesh formed by the multiple corner points constitutes the manifold in the three-dimensional model.

[0006] Therefore, in this embodiment, a submanifold prediction network can be used to predict whether point pairs are in the same submanifold. Then, based on the prediction results of all points, corner points on the boundary of the submanifold are selected from multiple points, and the shape of the submanifold is constructed based on the corner points, thereby combining them to obtain a simplified 3D model. Furthermore, more specifically, this application uses a trained submanifold prediction network to predict whether point pairs are in the same submanifold, thereby dividing multiple points in the point cloud data into different submanifolds or primitives. This achieves primitive-level instance segmentation, which can be very accurate, thus improving the final 3D model's performance. The 3D model is simplified while being richer in detail. Even with noise, the method provided in this application can adapt to different noise levels by training the submanifold prediction network, improving the accuracy of the output 3D model.

[0007] In one possible implementation, the aforementioned submanifold prediction network extracts features from point cloud data, which may include: extracting features from the point cloud data on a per-point basis and on a per-point basis and on a per-point basis a first preset number of adjacent points, to obtain local features corresponding to each point; downsampling the point cloud data to obtain downsampled data, wherein the resolution of the downsampled data is lower than the resolution of the point cloud data; extracting features from the downsampled data to obtain global features corresponding to each point; and fusing local features and global features to obtain features corresponding to each point among multiple points.

[0008] In this embodiment, local features that incorporate local information and global information that incorporates a wider range can be extracted, making the information included in the features corresponding to each point more complex and resulting in more complex features. This is equivalent to being able to more accurately describe each point and its surroundings, thereby making the subsequent prediction results more accurate.

[0009] In one possible implementation, the aforementioned downsampling of point cloud data may include: dividing the point cloud data to obtain multiple voxels, each voxel including at least one point and local features of each point in the corresponding at least one point; the aforementioned extraction of features from the downsampled data may include: extracting features by taking the points in each voxel in the multiple voxels and the points in adjacent second preset number of voxels as units to obtain global features, and the number of points in the second preset number of voxels is not less than the first preset number.

[0010] In this embodiment of the application, the range of extracted features can be expanded by downsampling to obtain global features with stronger correlation with the surrounding environment, thereby allowing each point to contain more information in its corresponding features.

[0011] In one possible implementation, the aforementioned determination of whether each point and its neighboring points belong to the same submanifold based on the characteristics of each point may include: determining the normal vector corresponding to each point based on the characteristics of each point; and determining whether each point and its neighboring points belong to the same submanifold based on the characteristics of each point, the normal vector of each point, and the normal vectors of its neighboring points.

[0012] Therefore, in the embodiments of this application, the offset of the normal vector can be used to identify whether a pair of points is in the same submanifold, which is equivalent to accurately identifying whether two points are in the same submanifold in a geometric way, thereby improving the accuracy of identification.

[0013] In one possible implementation, selecting multiple corner points from multiple points based on the prediction results includes: constructing a triangular mesh from the multiple points to form at least one triangular mesh; extracting boundaries belonging to the same submanifold from the at least one triangular mesh based on the prediction results; and extracting multiple corner points from the points on the boundaries belonging to the same submanifold extracted from the at least one triangular mesh.

[0014] Therefore, in this embodiment, multiple points in the point cloud data can be triangulated, and the boundary belonging to the same submanifold can be extracted from the triangular mesh according to the output of the submanifold prediction network. Points on the boundary can be extracted as corner points so as to construct a simplified three-dimensional model through the corner points.

[0015] In one possible implementation, constructing a three-dimensional model based on multiple corner points may include: constructing at least one Delaunay triangle mesh using multiple corner points and geodesic distances between the multiple corner points; merging at least one Delaunay triangle mesh to obtain a three-dimensional model.

[0016] Therefore, in the embodiments of this application, a Delaunay triangle mesh can be constructed based on geodesic distance, thereby obtaining a simplified three-dimensional model efficiently and accurately.

[0017] Secondly, this application provides a neural network training method, comprising: first, acquiring training data, the training data including multiple points and labels corresponding to each point, the labels corresponding to each point including an identifier indicating whether each point and its neighboring points belong to the same submanifold; using the multiple points as input to a submanifold prediction network to obtain prediction results for the multiple points, the prediction results including whether each point and its neighboring points belong to the same submanifold, wherein the submanifold prediction network extracts features from point cloud data to obtain features corresponding to each point among the multiple points, and determines whether each point and its neighboring points belong to the same submanifold based on the features corresponding to each point; calculating a loss value based on the prediction results and the labels corresponding to each point; and updating the submanifold prediction network based on the loss value to obtain an updated submanifold prediction network.

[0018] Therefore, in the embodiments of this application, a submanifold prediction network can be trained to output whether point pairs in the point cloud data are in the same submanifold, so that during inference, the boundary of the submanifold corresponding to the point cloud data can be constructed based on the prediction results of the submanifold prediction network, thereby constructing a simplified three-dimensional model, and improving the accuracy of the obtained three-dimensional model, making the submanifolds included in the three-dimensional model richer and more detailed.

[0019] In one possible implementation, the aforementioned submanifold prediction network extracts features from point cloud data, which may include: extracting features from the point cloud data on a per-point basis and on a first preset number of adjacent points to obtain local features; performing downsampling on the point cloud data at least once to obtain downsampled data, wherein the resolution of the downsampled data is lower than the resolution of the point cloud data; extracting features from the downsampled data to obtain global features; and fusing the local features and global features to obtain features corresponding to each point among multiple points.

[0020] In this embodiment, local features that incorporate local information and global information that incorporates a wider range can be extracted, making the information included in the features corresponding to each point more complex and resulting in more complex features. This is equivalent to being able to more accurately describe each point and its surroundings, thereby making the subsequent prediction results more accurate.

[0021] In one possible implementation, one of the aforementioned downsampling of the point cloud data at least once may include: dividing the point cloud data to obtain multiple voxels, each voxel including at least one point and local features of each point in the corresponding at least one point; extracting features from the downsampled data, including: performing at least one feature extraction on the basis of points in each voxel and points in adjacent second preset number of voxels to obtain global features, wherein the number of points in the second preset number of voxels is not less than the first preset number.

[0022] In this embodiment of the application, the range of extracted features can be expanded by downsampling to obtain global features with stronger correlation with the surrounding environment, thereby allowing each point to contain more information in its corresponding features.

[0023] In one possible implementation, the aforementioned determination of whether each point and its neighboring points belong to the same submanifold based on the features of each point may include: determining the predicted normal vector corresponding to each point based on the features corresponding to each point; and determining whether each point and its neighboring points belong to the same submanifold based on the features of each point, the predicted normal vector of each point, and the predicted normal vectors of its neighboring points.

[0024] Therefore, in the embodiments of this application, the offset of the normal vector can be used to identify whether a pair of points is in the same submanifold, which is equivalent to accurately identifying whether two points are in the same submanifold in a geometric way, thereby improving the accuracy of identification.

[0025] In one possible implementation, the prediction result also includes the normal vector corresponding to each point, and the label of each point also includes the ground truth normal vector corresponding to each point; the aforementioned calculation of the loss value based on the prediction result and the label corresponding to each point may include: calculating the loss value based on the normal vector corresponding to each point and the ground truth normal vector corresponding to each point.

[0026] In this embodiment of the application, the output of the trained submanifold prediction network can also include a normal vector by defining the prediction result as including a normal vector, so as to more accurately identify whether point pairs are in the same submanifold.

[0027] Thirdly, embodiments of this application provide a three-dimensional model building apparatus, which has the function of implementing the three-dimensional model building method described in the first aspect. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described function.

[0028] Fourthly, embodiments of this application provide a neural network training device that has the function of implementing the neural network training method described in the second aspect. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described function.

[0029] Fifthly, embodiments of this application provide a three-dimensional model building apparatus, including a processor and a memory, wherein the processor and the memory are interconnected via circuits, and the processor calls program code in the memory to execute processing-related functions in the three-dimensional model building method shown in any of the first aspects above. Optionally, the three-dimensional model building apparatus may be a chip.

[0030] Sixthly, embodiments of this application provide a neural network training apparatus, including: a processor and a memory, wherein the processor and the memory are interconnected via circuits, and the processor calls program code in the memory to execute processing-related functions in the neural network training method shown in any of the second aspects above. Optionally, the neural network training apparatus may be a chip.

[0031] In a seventh aspect, embodiments of this application provide a three-dimensional model building apparatus, which may also be referred to as a digital processing chip or a chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform processing-related functions as described in the first aspect or any optional embodiment of the first aspect.

[0032] Eighthly, embodiments of this application provide a neural network training device, which may also be referred to as a digital processing chip or a chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform processing-related functions as described in the second aspect or any optional embodiment of the second aspect above.

[0033] In a ninth aspect, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method in any of the optional embodiments of the first or second aspect described above.

[0034] In a tenth aspect, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the method in any of the optional embodiments of the first or second aspect described above. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of an artificial intelligence main framework applied in this application;

[0036] Figure 2 A schematic diagram of a system architecture is provided for this application;

[0037] Figure 3 Another system architecture diagram provided for an embodiment of this application;

[0038] Figure 4 A flowchart illustrating a neural network training method provided in an embodiment of this application;

[0039] Figure 5 A flowchart illustrating a three-dimensional model construction method provided in this application;

[0040] Figure 6 This is a schematic diagram of the structure of a submanifold prediction network provided in an embodiment of this application;

[0041] Figure 7 A schematic diagram illustrating the specific steps of the submanifold prediction network performed in an embodiment of this application;

[0042] Figure 8A flowchart illustrating another three-dimensional model construction method provided in this application embodiment;

[0043] Figure 9 A comparative schematic diagram showing the output of models in various ways provided in the embodiments of this application;

[0044] Figure 10 A schematic diagram of instance segmentation and three-dimensional model provided for an embodiment of this application;

[0045] Figure 11 This is a schematic diagram of the structure of a neural network training device provided in an embodiment of this application;

[0046] Figure 12 This is a schematic diagram of the structure of a three-dimensional model building device provided in an embodiment of this application;

[0047] Figure 13 A schematic diagram of another neural network training device provided in this application;

[0048] Figure 14 A schematic diagram of another three-dimensional model building device provided in this application;

[0049] Figure 15 The hardware execution flow of the server when executing the neural network training method provided in the embodiments of this application;

[0050] Figure 16 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0051] Figure 17 This is a schematic diagram of the structure of another terminal provided in an embodiment of this application;

[0052] Figure 18 This application provides a schematic diagram of the structure of a server according to an embodiment of the present application.

[0053] Figure 19 This is a schematic diagram of another server structure provided in an embodiment of this application;

[0054] Figure 20 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0056] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1 The diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed through technological means) to the industrial ecosystem of the system.

[0057] (1) Infrastructure

[0058] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips, such as hardware acceleration chips (CPUs, NPUs, GPUs, ASICs, or FPGAs); the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, which is then provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0059] (2) Data

[0060] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0061] (3) Data processing

[0062] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0063] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0064] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0065] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0066] (4) General ability

[0067] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0068] (5) Smart Products and Industry Applications

[0069] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0070] In certain scenarios, such as smart terminals, intelligent transportation, smart healthcare, autonomous driving, and smart cities, model building is required. For example, in building AR maps, point cloud data can be collected using lasers and then used to construct the AR map. Alternatively, in smart terminals, point cloud data of the currently captured scene can be collected using a camera, and a 3D model of the scene can be built based on this data. This model can then be applied to image processing or games on the terminal, thereby improving the user experience. In short, the method provided in this application has value in AI systems, terminal applications, and cloud services.

[0071] For example, after obtaining the point cloud data to be reconstructed, a mesh is reconstructed using various algorithms. For each point, the Quadric energy is calculated based on its nearest neighbor coordinates. Then, a priority queue is used to select the edge / point with the lowest energy, and the selected elements are deleted while maintaining the local topology to ensure that the local area remains a manifold. After deleting a sufficient number of points, a simplified mesh is output, which, when combined, forms the simplified 3D model. However, Quadric describes local energy, so it cannot guarantee that deleting the element with the lowest local energy will not disrupt the overall structure. When a high simplification rate is required, it is often necessary to disrupt the overall structure, resulting in a significant difference in geometry from the original data. This approach will be referred to as the first common method below.

[0072] For example, after obtaining the point cloud data to be reconstructed, a large plane is selected using a geometric plane fitting method. The extracted plane is then extended along the boundary until it intersects with other planes, and the space divided by the plane is tetrahedralized. Based on the normal information and spatial continuity of the point cloud, energy optimization based on graphcut is performed, and all faces simultaneously adjacent to the inner and outer tetrahedrons are extracted as the final mesh. However, this method assumes that the streams composed of the constructed point cloud are closed surfaces. In real-world applications, the 3D model may not consist entirely of closed surfaces. Furthermore, assuming the constructed point cloud is planar while curved surfaces may actually exist results in weak generalization ability and poor robustness when performing planar detection. This method will be referred to as the second common method below.

[0073] For example, after obtaining the point cloud data to be reconstructed, the Random Sample Consensus (RANSAC) algorithm is used to detect primitives, remove points that are close to the detected primitives, and obtain simplified primitives. If no new planes are detected, all simplified primitives are output to form a 3D model. However, when the point cloud data contains noise, the 3D model constructed in this way has poor robustness, and the difficulty of detecting curved surfaces depends on the accuracy of the normal vectors, making it difficult to accurately predict normal vectors in real-world data. This method will be referred to as the third common method below.

[0074] For example, after obtaining the point cloud data to be reconstructed, the PointNet++ network is used to extract the features of the point cloud data and output the instance, normal vector, and primitive type of each point. However, this method only reconstructs objects, has weak generalization ability, and is computationally intensive and difficult to implement in large scenes with a large number of instances. This method will be referred to as the fourth common method below.

[0075] Therefore, this application provides a neural network training method and a 3D model construction method, which selects points on the boundary of the submanifold by determining whether each point and its neighboring points are in the same submanifold, thereby constructing a simplified 3D model.

[0076] Specifically, a submanifold prediction network can be trained using a neural network training method provided in this application. This submanifold prediction network can be used to identify whether each point in the input point cloud data and its neighboring points are in the same submanifold. After training the submanifold prediction network using the neural network training method provided in this application, the prediction results of the submanifold prediction network can be used to perform 3D reconstruction of the point cloud data to obtain a simplified 3D model after reconstruction.

[0077] It can be understood that the neural network training method and the 3D model construction method provided in this application are respectively the training stage and the inference stage, and in the inference stage, a step of using the prediction results of the neural network to perform 3D reconstruction is added.

[0078] The neural network training method and 3D model construction method provided in this application can be applied to terminals, servers, or cloud platforms. For example, a submanifold prediction network can be trained on a server and then deployed on a terminal, where the terminal executes the 3D model construction method provided in this application; or a submanifold prediction network can be trained on a terminal and then deployed on the terminal, where the terminal executes the 3D model construction method provided in this application through the submanifold prediction network; or a submanifold prediction network can be trained on a server and then deployed on the server, where the server executes the 3D model construction method provided in this application, etc.

[0079] First, participate Figure 2 This paper introduces a system architecture provided in this application.

[0080] The system architecture includes a database 230, a client device 240, a training device 220, and an execution device 210. The data acquisition device 260 collects data and stores it in the database 230. The training device 220 trains the target model / rule 201 based on the data maintained in the database 230. The execution device 210 processes the data input to the client device 240 using the target model / rule 201 trained by the training device 220 and returns the output results to the client device 240.

[0081] Training device 220 can be used to train neural networks and output target model / rule 201.

[0082] The execution device 210 can call data, code, etc. in the data storage system 250, and can also store data, instructions, etc. in the data storage system 250.

[0083] Regarding how the training device 220 obtains the target model / rule 201 based on the data, the target model / rule 201 is the submanifold prediction network trained in the following embodiments of this application, please refer to the following for details. Figures 4-10 The relevant description in the document.

[0084] The execution device 210 may also include a calculation module 211 for processing the input data using the target model / rule 201.

[0085] Specifically, the target model / rule 201 obtained from training device 220 can be applied to different systems or devices, such as... Figure 2 As shown, the target model / rule 201 can be deployed on execution device 210. (See attached...) Figure 2 In this embodiment, the execution device 210 is equipped with a transceiver 212 (taking the I / O interface as an example) to interact with external devices. The "user" can input data to the I / O interface 212 through the client device 240. For example, in the following implementation of this application, the client device 240 can send point cloud data that needs to be reconstructed into a 3D model to the execution device 210.

[0086] Finally, transceiver 212 returns the 3D model of computing module 211 to client device 240, so that client device 240 or other devices can use the 3D model to perform other operations, such as image processing or application to games.

[0087] At a deeper level, the training device 220 can obtain corresponding target models / rules 201 based on different data for different tasks, so as to provide users with better results.

[0088] In the appendix Figure 2 In the scenario shown, the data input to the execution device 210 can be determined based on the user's input data. For example, the user can operate on the interface provided by the transceiver 212. Alternatively, the client device 240 can automatically input data to the transceiver 212 and obtain results. If the client device 240 needs user authorization to automatically input data, the user can set appropriate permissions on the client device 240. The user can view the results output by the execution device 210 on the client device 240; the specific presentation format can be display, sound, animation, etc. The client device 240 can also act as a data acquisition terminal, storing the acquired data associated with the target task into the database 230.

[0089] It should be noted that the appendix Figure 2 This is merely an exemplary schematic diagram of a system architecture provided in this application embodiment, and the positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in the appendix... Figure 2In this context, the data storage system 250 is an external memory relative to the execution device 210. In other scenarios, the data storage system 250 can also be placed within the execution device 210.

[0090] It is understandable that the training process of a neural network is essentially learning how to control spatial transformations, more specifically, learning the weight matrix. The goal of training a neural network is to make its output as close as possible to the expected value. Therefore, this can be achieved by comparing the current network's predicted value with the expected value, and then updating the weight vector of each layer in the neural network based on the difference between the two (of course, the weight vector can usually be initialized before the first update, i.e., pre-configured parameters for each layer in the deep neural network). For example, if the network's predicted value is too high, the values ​​of the weights in the weight matrix are adjusted to lower the predicted value. This process is repeated until the neural network's output value is close to or equal to the expected value. Specifically, the difference between the neural network's predicted value and the expected value can be measured using a loss function or an objective function. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and the training of the neural network can be understood as a process of minimizing the loss as much as possible. The process of updating the weights of the starting network and training the sequential network in the following embodiments of this application can be referred to in this process, and will not be repeated hereafter.

[0091] like Figure 2 As shown, the target model / rule 201 is trained by the training device 220. In this embodiment of the application, the target model / rule 201 may be the sub-flow prediction network mentioned in this application.

[0092] Optionally, the device used to train the submanifold prediction network and the device on which the submanifold is deployed may be the same device, i.e., as shown in the example below. Figure 2 The training device 220 and execution device 210 shown may be the same device or set in the same device. For example, the training device may be a terminal, the execution device may be a server, or the training device may be a server and the execution device may be the same server, etc.

[0093] The following provides an example of some possible system architectures provided in this application.

[0094] For example, such as Figure 3As shown, the neural network training method provided in this application can be executed by a server cluster 310, that is, to train the submanifold prediction network and send the trained submanifold prediction network to the terminal 301 via a communication network for deployment in the terminal 301. Point cloud data collected by the terminal's camera or LiDAR can be used as input to the submanifold prediction network. The terminal processes the output of the submanifold prediction network, outputting a simplified reconstructed 3D model. This 3D model can be used by the terminal for image processing to identify the types of various objects in the image, or the reconstructed 3D model can be applied to the terminal's AR games, allowing the AR games to be combined with the user's real-world environment, thus improving the user experience.

[0095] For example, the neural network training method provided in this application can be executed by a server, and the trained submanifold prediction network can be deployed on the server. This server can be used to execute the 3D model construction method provided in this application. The server can receive point cloud data sent by a client, or extract point cloud data from locally stored data, and then construct a simplified 3D model using the 3D model construction method provided in this application. If the point cloud data is sent to the server by the client, the server can feed back the simplified 3D model to the client.

[0096] This application involves some neural network-related applications. In order to better understand the solutions of this application, some neural network-related terms and concepts that may be involved in this application will be introduced below.

[0097] This application relates to applications in the fields of neural networks and images. To better understand the solutions of this application, the relevant terms and concepts of neural networks that may be involved in this application will be introduced below.

[0098] (1) Neural Network

[0099] Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit takes the intercept 1 as input, and its output can be shown in formula (1-1):

[0100]

[0101] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x sThe weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer; the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together; that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0102] (2) Convolutional Neural Network

[0103] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of a series of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are called the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The convolutional kernel can be initialized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is that it reduces the connections between layers in the CNN, while also reducing the risk of overfitting.

[0104] The network used for feature extraction mentioned below in this application may include one or more convolutional layers. For example, the network used for feature extraction may be implemented using a CNN.

[0105] (3) Loss function: Also known as the cost function, it is a metric that compares the predicted output of a machine learning model with the true value of the sample (also known as the supervised value). In other words, it measures the difference between the predicted output of the machine learning model and the true value of the sample. This loss function typically includes mean squared error, cross-entropy, logarithmic, exponential, and other loss functions. For example, mean squared error can be used as the loss function, defined as... The specific loss function can be selected based on the actual application scenario.

[0106] (4) Gradient: The vector of the derivative of the loss function with respect to the parameters.

[0107] (5) Stochastic gradient: In machine learning, the number of samples is very large, so the loss function is calculated from the randomly sampled data each time, and the corresponding gradient is called the stochastic gradient.

[0108] (6) Back propagation (BP): An algorithm for calculating the gradient of model parameters based on the loss function and updating the model parameters.

[0109] (7) Manifold: A geometric shape that is isomorphic to Euclidean space in any subset of a region.

[0110] (8) Submanifold: A subset of a manifold that has the structure of a manifold itself.

[0111] (9) Primitive: Basic shape, such as plane, cylinder, cone or sphere. A submanifold can include one or more primitives. For example, a primitive can be understood as a submanifold.

[0112] (10) Embedding: A mapping expression from a high-dimensional space to a continuous low-dimensional space.

[0113] (11) Point cloud: Data formed by multiple points, each point has corresponding information, such as depth, brightness or intensity.

[0114] This application provides a neural network training method and a 3D model construction method. It can be understood that the neural network training method and the 3D model construction method provided in this application are respectively a training phase and an inference phase. Furthermore, in the inference phase, a step is added to use the prediction results of the neural network for 3D reconstruction. The method provided in this application will be described below in different phases, including the training phase and the inference phase. The training phase refers to the neural network training method provided in this application, and the inference phase refers to the 3D model construction method provided in this application.

[0115] I. Training Phase

[0116] See Figure 4 The flowchart of a three-dimensional model construction method provided in this application is as follows.

[0117] 401. Obtain training data.

[0118] The training data may include point cloud data and labels corresponding to multiple points in the point cloud data.

[0119] This point cloud data can include data formed by multiple points. This point cloud data can be acquired by a camera, by a LiDAR scanner, or read from stored data.

[0120] The label corresponding to each point can include an indicator of whether the point and any other point form a point pair within the same submanifold. In other words, it indicates whether a point in the point cloud data belongs to the same submanifold as other points.

[0121] Optionally, the label corresponding to each point may also include information about the tangent plane of the submanifold to which each point is located, such as the normal vector.

[0122] For example, after acquiring point cloud data, ground truth data can be identified. This ground truth data might include: taking each point and a certain number of its neighboring points as point pairs, and using identifiers to mark whether each point pair is in the same submanifold. The ground truth data might also include: tangent plane information of the primitive corresponding to each point in the point cloud data. This primitive can include the primitive formed by each point and its nearest points; for example, each point can be represented by coordinates o. i and the corresponding normal vector n i .

[0123] Typically, the submanifold prediction network can be trained in a supervised manner using training data, so that the output of the submanifold prediction network is closer to the label of the point cloud data.

[0124] 402. Input the point cloud data from the training data into the submanifold prediction network and output the prediction results for multiple points.

[0125] The submanifold prediction network can be a pre-trained network that outputs a prediction result for each of multiple points. This prediction result can be used to indicate whether each point and its neighbors belong to the same submanifold.

[0126] Specifically, a submanifold prediction network can extract features from the input point cloud data and then identify whether each point and its neighbors belong to the same submanifold based on the extracted features. For example, features can be extracted on a per-point basis and a certain number of points around it, and then the extracted features can be used to identify whether each point and its neighbors belong to the same submanifold.

[0127] In one possible implementation, features can be extracted from point cloud data on a per-point basis and by taking each point and its adjacent first preset number of points as units to obtain the local features corresponding to each point. Alternatively, the point cloud data can be downsampled to obtain downsampled data with lower resolution. Features can be extracted from the downsampled data to obtain the global features of each point. By fusing the global features and local features, the features corresponding to each point can be obtained.

[0128] In the process of extracting global features, the point cloud data can be downsampled multiple times. Each downsampling can extract features from the feature map corresponding to the current downsampling, thus enabling iterative feature extraction. This allows the final features to reference more features from neighboring points, improving the accuracy and complexity of the features and increasing the implicit information included in the features.

[0129] Optionally, the downsampling method may include: dividing the point cloud data into multiple voxels, each voxel including at least one point and local features of each point. This is equivalent to dividing multiple points into multiple grids, each grid including one or more points, each point having corresponding local features. Then, feature extraction can be performed on the points within each voxel and a second preset number of adjacent voxels as units to obtain the global features corresponding to each point, where the number of points within the second preset number of voxels is not less than a first preset number.

[0130] Therefore, in this embodiment, features can be extracted over a larger range by downsampling, combining the feature maps of each point with those of more neighboring points, thereby increasing the information included in the features of each point in the global features extracted.

[0131] Optionally, the method of fusing local and global features can include concatenating the local and global features to form the features for each point. For example, if the local features are 16-dimensional and the global features are also 16-dimensional, they can be concatenated to obtain 32-dimensional features.

[0132] Therefore, in this embodiment, local and global features can be combined to obtain features that better describe the information of each point, increasing the information included in the features of each point and making the prediction results of each point more accurate.

[0133] In one possible implementation, the submanifold prediction network can also output the normal vector of the plane containing each point. For example, a portion of the extracted features can be used as information about the tangent plane corresponding to each point, so that after extracting the features of each point, the predicted tangent plane information can be obtained based on a portion of those features. This tangent plane information can be used to determine whether a pair of points is in the same submanifold. For example, the tangent plane information may include the normal vector of the tangent plane, and the offset between the normal vectors of the two points in the pair can be used to determine whether the pair of points is in the same submanifold.

[0134] 403. Update the submanifold prediction network based on the loss value between the prediction results and the label corresponding to each point.

[0135] After obtaining the prediction results, the loss value between the prediction result and the label of each point can be calculated. Then, the loss value is used to update the submanifold prediction network in reverse to obtain the updated submanifold prediction network.

[0136] Specifically, the loss function can be the mean squared error, cross-entropy, logarithmic, exponential, or other loss functions. For example, the loss between the predicted result and the true value can be calculated using the loss function, and then backpropagation can be performed based on this loss to calculate the gradient, which is equivalent to the inverse vector of the parameters of the submanifold prediction network. Then, the gradient is used to update the parameters of the submanifold prediction network.

[0137] 404. Determine whether the convergence condition is met. If yes, proceed to step 405; otherwise, proceed to step 402.

[0138] After obtaining the updated submanifold prediction network, it can be determined whether the convergence condition is met. If the convergence condition is met, the updated submanifold prediction network can be output, thus completing the training of the submanifold prediction network. If the convergence condition is not met, the training of the submanifold prediction network can continue, that is, step 402 can be repeated until the convergence condition is met.

[0139] The convergence condition may include one or more of the following: the number of training iterations of the submanifold prediction network reaches a preset number, or the output accuracy of the submanifold prediction network is higher than a preset accuracy value, or the average accuracy of the submanifold prediction network is higher than a preset average value, or the training time of the submanifold prediction network exceeds a preset time, etc.

[0140] 405. Output the updated submanifold prediction network.

[0141] Once the convergence condition is met, the updated submanifold prediction network can be output. For example, if the neural network training method provided in this application is executed by a server, the converged submanifold prediction network can be deployed on the server or terminal after obtaining it.

[0142] Therefore, in the embodiments of this application, a submanifold prediction network can be trained to output whether point pairs in the point cloud data are in the same submanifold, so that during inference, the boundary of the submanifold corresponding to the point cloud data can be constructed based on the prediction results of the submanifold prediction network, thereby constructing a simplified three-dimensional model, and improving the accuracy of the obtained three-dimensional model, making the submanifolds included in the three-dimensional model richer and more detailed.

[0143] II. Reasoning Stage

[0144] During the training phase, an updated submanifold prediction network is trained to output whether point pairs in the point cloud data belong to the same submanifold. During the inference phase, the trained submanifold prediction network is used to predict whether point pairs in the input point cloud data belong to the same submanifold. Then, based on the prediction results, the boundary of the submanifold formed by the point cloud data is determined, and a simplified 3D model is constructed based on the points on the submanifold boundary. The flow of the 3D model construction method provided in this application is described in detail below.

[0145] See Figure 5 The flowchart of a three-dimensional model construction method provided in this application is as follows.

[0146] 501. Obtain point cloud data.

[0147] The point cloud data includes data formed by multiple points. This point cloud data can be referenced from the point cloud data mentioned in step 401 above. The difference is that the point cloud data in this step does not have labels, which will not be elaborated here.

[0148] 502. Input the point cloud data into the submanifold prediction network and output the prediction results for multiple points.

[0149] The submanifold prediction network can be obtained through the aforementioned... Figure 4 The network trained using the method described above takes point cloud data as input to a submanifold prediction network and outputs prediction results for multiple points. These prediction results may include identifiers indicating whether pairs of points belong to the same submanifold.

[0150] Optionally, the prediction result may also include information about the tangent plane corresponding to each point, such as the normal vector of the tangent plane. This information about the tangent plane can be used to help identify whether a pair of points are in the same submanifold. For example, the normal vector can be used to identify whether the tangent planes of two points are the same or parallel, thereby helping to determine whether the two points are in the same submanifold.

[0151] 503. Select multiple corner points from multiple points based on the prediction results.

[0152] After determining whether each point and its neighboring points belong to the same submanifold based on the prediction results from the submanifold prediction network, multiple corner points can be selected from these multiple points. Corner points include points on the boundary of the submanifold formed by these multiple points.

[0153] Specifically, after obtaining the prediction results for multiple points, the points within the same submanifold can be identified, and points on the boundary of the submanifold can be selected as corner points. For example, if the submanifold is a polygon, the points at both ends of each edge of the polygon can be selected as corner points, and the connection between these corner points can form a simplified pattern of the submanifold.

[0154] In one possible implementation, the specific method for selecting corner points may include: constructing a triangular mesh from multiple points, that is, connecting three adjacent points to form one or more triangular meshes; selecting points on the boundary of the same submanifold based on the prediction results from one or more triangular meshes, and extracting corner points from them. For example, when the submanifold is a polygon, the corner points of the polygon can be extracted; when the submanifold is a circle, points can be selected as corner points from each distance along the boundary of the circle. Therefore, in this embodiment, a simplified shape of the submanifold can be obtained by selecting corner points, which facilitates the subsequent construction of a simplified 3D model.

[0155] 504. Construct a 3D model based on multiple corner points.

[0156] After obtaining multiple corner points, the shape of each submanifold can be determined based on these corner points, and then combined into a simplified 3D model.

[0157] In one possible implementation, one or more Delaunay triangle meshes can be constructed using the selected corner points and the geodesic distances between them. When multiple Delaunay triangle meshes exist, merging these meshes yields a simplified 3D model.

[0158] Therefore, in this embodiment, a submanifold prediction network can be used to predict whether point pairs are in the same submanifold. Then, based on the prediction results of all points, corner points on the boundary of the submanifold are selected from multiple points, and the shape of the submanifold is constructed based on the corner points, thereby combining them to obtain a simplified 3D model, which can efficiently construct a simplified 3D model. Furthermore, more specifically, this application uses a trained submanifold prediction network to predict whether point pairs are in the same submanifold, thereby dividing multiple points in the point cloud data into different submanifolds or primitives. This achieves primitive-level instance segmentation, which can be very accurate, thus improving the final 3D model's performance. The 3D model is simplified while being richer in detail. Even with noise, the method provided in this application can adapt to different noise levels by training the submanifold prediction network, improving the accuracy of the output 3D model.

[0159] The foregoing has introduced the process of the neural network training method and the 3D model construction method provided in this application. For ease of understanding, the process of the neural network training method and the 3D model construction method provided in this application will be introduced in more detail below with more specific examples.

[0160] First, exemplarily, the structure of a submanifold prediction network can be as follows: Figure 6 As shown.

[0161] The submanifold prediction network may include the PointNet++ network (hereinafter referred to as PN network) 601, sparse 3D convolution (spconv) 602, and boundary discrimination network 603.

[0162] PN networks can be used to extract features from point cloud data to obtain low-resolution local features.

[0163] Sparse 3D convolution can be used to extract global features based on low-resolution local features. For example, a UNet can be composed of one or more spconvs to perform multiple convolutions and corresponding deconvolutions on the input features, thereby combining the features of more points near each point and outputting the global features corresponding to each point.

[0164] The combination of global and local features can be used to obtain the features corresponding to each point.

[0165] For ease of understanding, the network formed by the PN network and spconv can be called a submanifold nested network. That is, the submanifold nested network can be used to extract features from point cloud data to obtain low-resolution local features. Based on the low-resolution local features, feature extraction is performed to obtain global features. By combining global features and local features, the features corresponding to each point are obtained.

[0166] The input to the boundary discrimination network includes the features corresponding to each point (i.e., features composed of local and global features), which are used to determine whether each point and the point pair formed by its neighboring points are in the same submanifold.

[0167] In combination with the above Figure 6 The submanifold prediction network is described in the reference section. Figure 7 The specific steps performed by the submanifold prediction network are illustrated below in conjunction with the training process. For ease of understanding, the output steps of the submanifold prediction network are divided into multiple steps, as shown below as S1-1 to S1-8.

[0168] S1-1:

[0169] First, the input point cloud data can include N points, which are then represented by {p i} represents an Nx3 tensor. Within this tensor, the point cloud data identifies the location of each point, and each point also contains corresponding information such as pixel value, brightness value, depth value, or intensity.

[0170] Point cloud data also includes corresponding ground truth data, such as manually labeled data. This includes taking the j neighboring points of each point i as point pairs, such as taking each point and its 16 neighboring points as point pairs, and marking whether each point pair is in the same primitive or submanifold, such as using True / False to indicate whether the point pair is in the same submanifold.

[0171] Optionally, the truth data may also include information about the tangent plane corresponding to the primitive or submanifold where each point is located, such as the coordinates of the nearest point o. i and the corresponding normal vector n i .

[0172] Form point pairs with each of the N points and their neighboring points, such as... Figure 7 The K-Nearest Neighbor (KNN) classification algorithm shown in the diagram forms a pair of points for each point and multiple neighboring points. For each point i, j neighboring points are selected to form a pair, resulting in a pair of points.<i,j> For ease of understanding, this embodiment of the application takes the 16 points surrounding each point to form 16 point pairs, that is, the point cloud data can form N*16 point pairs.

[0173] A PointNet network (PN) can consist of two layers (PointNet++, PN). In each PN layer, information about each point and its 16 nearest neighbors is obtained. The output features of each point in that layer are extracted using conv1D convolution, batch normalization, and maxpooling operations on the 16 nearest neighbors. Because the number of neighboring points is 16, the output dimension is 16. This is equivalent to extracting features from each point and its 16 nearest neighbors, obtaining the local features corresponding to each point.

[0174] S1-2:

[0175] Then, the N points are voxelized, dividing them into M voxels, each containing one or more points. This is equivalent to downsampling the N points to obtain M voxels with lower resolution. For example, the voxelization resolution can be set to r = 0.1 meters. For a point, such as the point with coordinates [x, y, z], it can be mapped to voxels of [floor(x / r), floor(y / r), floor(z / r)] (floor(d) represents the largest integer less than or equal to d). The feature of each voxel after voxelization can be set to the average of the features of the points it contains, or it can be set to the most widely distributed value, etc., depending on the actual application scenario. Assuming that N points are mapped to M voxels, the feature tensor after voxelization is M*16.

[0176] S1-3:

[0177] Then, a UNet composed of spconv outputs the features corresponding to each point. For example, the UNet can perform multiple convolutions and deconvolutions on the input features. For instance, it can perform a convolution operation on the input M*16 dimensional features to extract features from the input data, then perform a deconvolution operation on the extracted features, using the deconvolution operation as the input for the next convolution operation, and so on, ultimately outputting M*16 dimensional global features. This is equivalent to using the UNet to extract features from a larger number of points (i.e., a second preset number), obtaining global features that are more correlated with neighboring points.

[0178] S1-4:

[0179] Then, devoxalization, or de-voxalization, is performed, which involves feeding back the M*16 features to each point to obtain the features corresponding to each point. For example, in the aforementioned voxelization step, N points are divided into M voxels. After obtaining the M*16 features, the features of points within each voxel are set as the features of that voxel, and the N*16 global features are output.

[0180] S1-5:

[0181] Then, the global and local features are combined. The combination method can be concatenation, that is, concatenating the local features N*16 and the global features N*16 to obtain the final N*32 features.

[0182] S1-6:

[0183] Optionally, to further integrate local and global features, a two-layer multilayer perceptron (MLP) can be used to increase the feature dimension from 32 to 64, and then reduce it back to 32, outputting an updated N*32 feature set. This is equivalent to using an MLP to fit more complex data, increasing the parameters of the submanifold prediction network, thereby making the output of the trained submanifold prediction network more accurate.

[0184] Optionally, if the label of each point also includes information about the tangent plane of the submanifold it belongs to, then some features in the N*32 dimensions can also be used as the information of the predicted tangent plane output by the submanifold prediction network. Here, we take the first 6 dimensions of the N*32 features as an example to use the information of the predicted tangent plane output by the submanifold prediction network. The first 6 dimensions are defined as the predicted tangent plane information, i.e., the coordinates o of the nearest point to each point. i and the corresponding normal vector n i The latter 26 dimensions can be understood as the implicit feature vector X of the local information of the primitive corresponding to that point. iAccordingly, during the training phase, a loss function can be set to train the submanifold prediction network, making the first 6 dimensions of the N*32 dimensions output by the submanifold prediction network closer to or the same as the information of the tangent plane in the ground truth. This loss function can be, for example, a regression loss function (L2-Loss) or mean-square error (MSE). Essentially, it calculates the error between the features output by the submanifold prediction network and the normal vectors included in the ground truth data, then calculates and updates the gradient of the submanifold prediction network based on this error, and updates the parameters of the submanifold prediction network based on this gradient.

[0185] S1-7:

[0186] Then, the N*32 features are used as input to the boundary discrimination network, which outputs a prediction of whether each point pair belongs to the same submanifold. The specific steps performed by the boundary discrimination network are as follows: For example, N points in the point cloud data form N*16 point pairs. Through the aforementioned steps, the features corresponding to each point can be output, as shown below: <o i ,n i ,X i ,o j >,o i Indicates the position of the point, n i X represents the normal vector of the tangent plane of the primitive or submanifold to which each point is located. i This represents the feature corresponding to that point, o j Indicates with o i The position of the other point forming the point pair. Obtain the information to be judged, as represented as: <n i ,X i ,o j -o i > That is, for an orientation of n i And the characteristic is X i In a submanifold centered at the origin, determine the offset as o j -o i The function checks whether a point is within a submanifold. This can be understood as taking a point as the origin and drawing a certain range, such as using the distance between the point and the furthest point among the 16 neighboring points as the radius to form a submanifold or primitive. It then determines whether the remaining N-1 points are within this submanifold or primitive. For example, an output of 0 indicates that the point pair is not in the same submanifold, and an output of 1 indicates that the point pair is in the same submanifold; or, an output of 1 indicates that the point pair is not in the same submanifold, and an output of 0 indicates that the point pair is in the same submanifold.

[0187] S1-8:

[0188] During training, the output of the boundary discrimination network can be input into a three-layer MLP (the output dimensions of the three layers are 64, 32 and 2 respectively). The output is a score for whether each point is a boundary. The loss value is calculated by the boundary marked in the ground truth data and the binary cross-entropy. Then, the submanifold prediction network is updated in reverse based on the loss value.

[0189] The above steps S1-1 to S1-8 can be repeated until the submanifold prediction network meets the convergence condition. The convergence condition may include one or more of the following: the number of iterations reaches a preset number, the iteration time reaches a preset time, the output accuracy of the submanifold prediction network is higher than a first threshold, or the average output accuracy of the submanifold prediction network is higher than a second threshold. In this way, a submanifold prediction network that meets the requirements can be obtained to facilitate the subsequent construction of the 3D model.

[0190] Therefore, in this embodiment, combining PN and Spconv to construct a submanifold nested network is equivalent to a high-resolution feature extraction network, improving the recognition accuracy of the subsequent boundary discrimination network and making the predicted points belonging to the same submanifold more accurate. Furthermore, a large amount of artificially constructed real laser-acquired point cloud data and corresponding hand-drawn 3D models can be used as ground truth data. By analyzing the surface structure of the 3D model, it is equivalent to analyzing the primitive instances of the hand-drawn model. The submanifold prediction network trained in this way can accurately generate primitive-level prediction results for point cloud data using the nearest neighbor method, improving the accuracy of predicting whether each point belongs to the same submanifold, thereby improving the accuracy of subsequent 3D model construction and obtaining a more accurate and clearer 3D model.

[0191] The training process of the submanifold prediction network has been described exemplarily above. The inference process will be described exemplarily below.

[0192] See Figure 8 This application provides another flowchart illustrating the three-dimensional model construction method.

[0193] S0: Obtain the input point cloud data.

[0194] The point cloud data includes N points, and this point cloud data is similar to the aforementioned... Figure 7 The point cloud data input to the submanifold prediction network is similar, except that the point cloud data does not include ground truth data, which will not be elaborated here.

[0195] The point cloud data can include data collected by LiDAR or cameras, which includes multiple points that form various instances in the scene being collected.

[0196] For example, the 3D model construction method provided in this application can be executed by a terminal. The terminal can collect point cloud data from a LiDAR or a depth-sensing camera. This point cloud data includes N points, which form various instances in the current scene.

[0197] For example, the terminal could be a smart car equipped with multiple cameras and LiDAR. It can collect point cloud data of the surrounding environment through the cameras or LiDAR and construct a simplified 3D model through the following steps, thereby enabling the vehicle to quickly obtain information about the surrounding environment, such as the location and shape of obstacles, thereby improving the vehicle's driving safety.

[0198] S1: Submanifold nesting.

[0199] The submanifold prediction network determines whether pairs of points formed by N points belong to the same submanifold or the same primitive, which is equivalent to identifying whether the N points are in the same plane. The steps performed by the submanifold prediction network can be referred to steps S1-1 to S1-8 above. The difference is that the point cloud data does not have corresponding ground truth data, and there is no need to train the submanifold prediction network, that is, there is no need to calculate the loss value. Similar steps will not be described in detail here.

[0200] S2: Mesh construction and polyline simplification.

[0201] This step is equivalent to constructing a network based on N points, that is, connecting the N points to form multiple triangular meshes. Based on the prediction results of whether the point pairs formed by the N points output by the submanifold prediction network are in the same submanifold, the edges that are adjacent to at least two instances or only one triangle are selected from the multiple triangular meshes as the boundaries of the submanifold.

[0202] Then, the Ramer-Douglas-Peucker algorithm is used to extract simplified corner points. This is equivalent to extracting the endpoints on the boundary as corner points, or, if the submanifold is circular, selecting a point at regular intervals along the boundary of the circle as a corner point. The steps for extracting corner points can be as follows: 1. For N points that are connected end-to-end, initially only the last two points are selected. 2. Connect the currently selected points end-to-end to form a polyline. Calculate the farthest distance from the unselected points to the polyline. 3. If the farthest distance is greater than a certain threshold, the corresponding point is included in the selection, and step 2 is repeated; otherwise, the polyline formed by connecting the end-to-end points is the final simplified polyline, ensuring that the distance between the simplified polyline and the original polyline does not exceed the threshold.

[0203] This can be understood as follows: based on the prediction results of whether the pairs of points formed by the N points output by the submanifold prediction network are in the same submanifold, the points in the same submanifold can be identified, and which points are on the boundary of the submanifold can be determined. Thus, the boundary of the submanifold can be extracted or points inside the submanifold can be deleted, while the corner points on the boundary of the submanifold can be retained.

[0204] For example, in the point cloud data, point p i Move to the output of the submanifold prediction network o i An undirected graph is constructed using 16 nearest neighbors (i.e., each vertex is a node in the graph, and each node is connected to its 16 nearest neighbors by an edge). A boundary discrimination network within the submanifold prediction network is used to determine whether each edge of the undirected graph belongs to a different primitive instance, and edges that do not belong to the same instance are deleted. Next, the standard flood fill algorithm is used to partition the undirected graph into connected components. The set of vertices within each connected component is a primitive instance. A submanifold can include one or more primitives. This can be understood as follows: after determining the boundary of the submanifold, the set of vertices within the boundary constitutes a primitive.

[0205] S3: Geodesic distance triangulation.

[0206] Specifically, the triangular grid mentioned in step S2 can be treated as an undirected graph, with its vertex set consisting of all corner vertices and its edge set consisting of all edges. The distance between edges is defined as the Euclidean distance between the two vertices on the edge. For each primitive, the shortest path from any of the N points to the corner vertices extracted in S2 can be calculated, and the nearest corner vertices of each of the N points can be recorded, thus outputting a Voronoi diagram. For each triangular grid in the Voronoi diagram, if the nearest corner vertices of the three points are distinct, the corner vertices can be connected to form a Delaunay triangular grid based on geodesic distance.

[0207] The shortest path from any point among N points to the corner points extracted in S2 can be calculated using Dijkstra's algorithm. For example, given N points and the edges connecting them, find the shortest paths from each of the N points to each corner point. Set the shortest path of a corner point to 0 and include it in the completed point set. The shortest paths of the remaining points are set to infinity. Add the corner point to a priority queue. Continuously remove the point with the minimum distance from the queue, traverse all adjacent edges, and if the distance of the current point plus the edge distance is less than the distance of its adjacent points, update the distance of the adjacent points and add it to the priority queue. The algorithm terminates when the queue is empty.

[0208] S4: Plane combination.

[0209] After obtaining multiple Delaunay triangle meshes through simplified corner points in step S3, combining these multiple Delaunay triangle meshes yields the simplified 3D model.

[0210] S5: Output a vectorized 3D model.

[0211] Through the above S0-S4, a simplified 3D model can be obtained, and this simplified 3D model can be output as the final 3D model.

[0212] Therefore, in this embodiment, a submanifold prediction network can be used to determine whether point pairs formed by multiple points in point cloud data are in the same submanifold. This is equivalent to identifying points in the same submanifold, extracting primitive instances, determining the boundaries of each submanifold, and obtaining Delaunay triangulation meshes through geodesic distance partitioning. By combining the Delaunay triangulation meshes, a simplified 3D model based on primitives is obtained. In other words, a geometric analysis system that transforms point clouds into vectorized meshes, combined with tetrahedral meshing, boundary simplification, and geodesic distance Delaunay triangulation, yields a simplified 3D model based on primitives. Therefore, the method provided in this application can efficiently and accurately output simplified 3D models, achieving lightweight 3D models.

[0213] To facilitate further understanding, the output of the 3D model construction method provided in this application will be described in more detail below.

[0214] like Figure 9 The diagram shows a comparison of the model based on a commonly used 3D model with spatial partitioning simplification, a locally simplified model with geometric optimization, and the model output by the 3D model construction method provided in this application.

[0215] After obtaining point cloud data, it is common practice to structure the plane based on spatial partitioning to obtain a simplified 3D model, which is the aforementioned common method two. Alternatively, a simplified mesh can be reconstructed based on geometrically optimal local simplification to form a simplified 3D model, which is the aforementioned common method one.

[0216] This application uses a submanifold prediction network to predict whether pairs of points within a set of points belong to the same submanifold. It then connects these points to construct multiple triangular meshes and uses the output of the submanifold prediction network to select boundaries and corner points from those boundaries. This achieves simplification at the primitive level, resulting in a more simplified and accurate 3D model. Figure 9 As shown, the 3D model provided in this application can be significantly simplified and more accurately described. For example, the submanifold prediction network provided in this application can accurately achieve instance segmentation, such as... Figure 10As shown, instances formed by points in point cloud data can be identified through a submanifold prediction network. Combined with tetrahedral mesh construction, boundary simplification, and Delaunay triangulation, a simplified 3D model is obtained. Compared to the aforementioned common method three, the method provided in this application can further reduce the simplification ratio to 0.15% through a submanifold prediction network, thus obtaining a simplified yet accurate 3D model that describes the instances.

[0217] Furthermore, the method provided in this application can improve prediction performance in noisy environments even when point cloud data contains noise, by training the submanifold prediction network, thus enhancing the robustness of the proposed solution. In addition, even when point cloud data contains a large number of primitive instances, the method provided in this application, such as using the output of the submanifold prediction network and employing a flood fill algorithm to identify each primitive, can accurately construct a 3D model even with a large number of primitives, demonstrating strong generalization ability.

[0218] Furthermore, once a simplified 3D model is obtained, it can be applied to various scenarios.

[0219] For example, in autonomous driving scenarios, the surrounding environment information can be collected by the LiDAR installed in the vehicle, and a simplified 3D model can be constructed using the method provided in this application. The simplified 3D model can then be displayed on the vehicle's screen, allowing users to quickly view the vehicle's surrounding environment and improving the user experience.

[0220] For example, users can use AR maps on their devices. When users use AR maps for navigation, they can use the devices to take real-time pictures of the surrounding environment and quickly complete the construction of a 3D model using the method provided in this application. They can then quickly identify instances on the display screen and display the navigation path based on the identified instances, allowing users to follow the navigation path.

[0221] For example, the method provided in this application can be deployed in a cloud platform. When a user needs to build a simplified 3D model of a certain area, the user can send the data collected by the camera or LiDAR to the cloud platform through the client. The cloud platform can quickly build the simplified 3D model and feed it back to the client, so that the user can quickly obtain a simplified 3D model of a certain area.

[0222] The foregoing has provided a detailed description of the neural network training method and 3D model construction method provided in this application. The following section describes the training device and 3D model construction device provided in this application. This training device can be used to execute the aforementioned... Figures 4-8 The steps of the neural network training method mentioned above, the three-dimensional model building device can be used to perform the aforementioned steps. Figures 5-8The steps of the three-dimensional model construction method mentioned in the article.

[0223] See Figure 11 The present application provides a schematic diagram of the structure of a neural network training device, as shown below.

[0224] The neural network training device may include:

[0225] The acquisition module 1101 is used to acquire training data, which includes multiple points and the label corresponding to each point. The label corresponding to each point includes the true result of whether each point and its neighboring points belong to the same submanifold.

[0226] The output module 1102 is used to take multiple points as input to the submanifold prediction network and output the prediction results of multiple points. The prediction results include whether each point and its neighboring points belong to the same submanifold. The submanifold prediction network extracts features from the point cloud data to obtain the features corresponding to each point in the multiple points, and determines whether each point and its neighboring points belong to the same submanifold based on the features corresponding to each point.

[0227] Loss module 1103 is used to calculate the loss value based on the prediction results and the label corresponding to each point;

[0228] The update module 1104 is used to update the submanifold prediction network based on the loss value, so as to obtain the updated submanifold prediction network.

[0229] In one possible implementation, the output module 1102 is specifically used to: extract features from the point cloud data, taking each point and a first preset number of adjacent points as units, to obtain local features; perform downsampling on the point cloud data at least once to obtain downsampled data; extract features from the downsampled data to obtain global features; and fuse the local features and global features to obtain the features corresponding to each point among multiple points.

[0230] In one possible implementation, the output module 1102 is specifically used to: divide the point cloud data to obtain multiple voxels, each voxel including at least one point and local features of each point in the corresponding at least one point; and perform at least one feature extraction on each voxel in the multiple voxels and points in the adjacent second preset number of voxels as units to obtain global features.

[0231] In one possible implementation, the output module 1102 is specifically used to: determine the predicted normal vector corresponding to each point based on the features corresponding to each point; and determine whether each point and its neighboring points belong to the same submanifold based on the features of each point, the predicted normal vector of each point, and the predicted normal vectors of neighboring points.

[0232] In one possible implementation, the prediction result also includes the normal vector corresponding to each point, and the label of each point also includes the ground truth normal vector corresponding to each point; the loss module 1103 is specifically used to calculate the loss value based on the normal vector corresponding to each point and the ground truth normal vector corresponding to each point.

[0233] See Figure 12 This application provides a structural schematic diagram of a three-dimensional model building device.

[0234] The 3D model building device may include:

[0235] The transceiver module 1201 is used to acquire point cloud data, which includes data formed by multiple points;

[0236] The prediction module 1202 is used to input point cloud data into the submanifold prediction network and output the prediction results of multiple points. The prediction results are used to identify whether each point and its neighboring points belong to the same submanifold. The submanifold prediction network extracts features from the point cloud data to obtain the features corresponding to each point in the multiple points, and determines whether each point and its neighboring points belong to the same submanifold based on the features corresponding to each point.

[0237] The filtering module 1203 is used to filter out multiple corner points from multiple points based on the prediction results of multiple points. The multiple corner points include points on the boundaries of each submanifold formed by the multiple points.

[0238] Module 1204 is used to construct a 3D model based on multiple corner points. The mesh formed by the multiple corner points constitutes the manifold in the 3D model.

[0239] In one possible implementation, the prediction module 1202 is specifically used to perform the following steps through the sub-stream prediction network: extracting features from the point cloud data, taking each point and its adjacent first preset number of points as units, to obtain the local features corresponding to each point; downsampling the point cloud data to obtain downsampled data; extracting features from the downsampled data to obtain the global features corresponding to each point; and fusing the local features and global features to obtain the features corresponding to each point among multiple points.

[0240] In one possible implementation, the prediction module 1202 is specifically used to: divide the point cloud data to obtain multiple voxels, each voxel including at least one point and local features of each point in the corresponding at least one point; and extract features by taking each voxel in the multiple voxels and the points in the adjacent second preset number of voxels as units to obtain global features.

[0241] In one possible implementation, the prediction module 1202 is specifically used to: determine the normal vector corresponding to each point based on the features corresponding to each point; and determine whether each point and its neighboring points belong to the same submanifold based on the features of each point, the normal vector of each point, and the normal vectors of neighboring points.

[0242] In one possible implementation, the filtering module 1203 is specifically used to: construct a triangular mesh from multiple points to form at least one triangular mesh; extract the boundaries belonging to the same submanifold from the at least one triangular mesh according to the prediction results; and extract multiple corner points from the points on the boundaries belonging to the same submanifold extracted from the at least one triangular mesh.

[0243] In one possible implementation, the construction module 1204 is specifically used to: construct at least one Delaunay triangular mesh using multiple corner points and geodesic distances between the multiple corner points; and merge at least one Delaunay triangular mesh to obtain a three-dimensional model.

[0244] Please see Figure 13 The following is a schematic diagram of another neural network training device provided in this application.

[0245] The neural network training device may include a processor 1301 and a memory 1302. The processor 1301 and the memory 1302 are interconnected via a circuit. The memory 1302 stores program instructions and data.

[0246] The aforementioned are stored in memory 1302 Figures 4 to 10 The steps in the code include the corresponding program instructions and data.

[0247] Processor 1301 is used to perform the aforementioned Figures 4 to 10 The method steps performed by the training device shown in any of the embodiments.

[0248] Optionally, the neural network training device may also include a transceiver 1303 for receiving or sending data.

[0249] This application also provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform the aforementioned actions. Figures 4 to 10 The steps in the neural network training method described in the illustrated embodiment.

[0250] Alternatively, the aforementioned Figure 13 The neural network training device shown is a chip.

[0251] participate Figure 14 This application provides a schematic diagram of another three-dimensional model building device.

[0252] The 3D model building apparatus may include a processor 1401 and a memory 1402. The processor 1401 and the memory 1402 are interconnected via circuits. The memory 1402 stores program instructions and data.

[0253] The aforementioned are stored in memory 1402 Figures 4 to 10 The steps in the code include the corresponding program instructions and data.

[0254] Processor 1401 is used to perform the aforementioned Figures 4 to 10 The method steps performed by the three-dimensional model building apparatus shown in any of the embodiments.

[0255] Optionally, the 3D model building apparatus may also include a transceiver 1403 for receiving or sending data.

[0256] This application also provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform the aforementioned actions. Figures 4 to 10 The steps in the three-dimensional model construction method described in the illustrated embodiment.

[0257] Alternatively, the aforementioned Figure 14 The 3D model building device shown is a chip.

[0258] The methods and apparatus provided in this application have been described above. For ease of understanding, the steps executed by each piece of hardware when the methods and apparatus provided in this application are run on hardware will be described below by way of example.

[0259] I. Regarding neural network training methods,

[0260] The neural network training method can be deployed on a server or on a terminal. This application will use server deployment as an example for illustration below.

[0261] For example, the hardware execution flow of the server when executing the neural network training method is as follows: Figure 15 As stated above.

[0262] The server 1500 may include hardware for running neural networks, such as... Figure 15 The GPU / Ascend chip 1501, CPU 1502, and memory 1503 shown are among the components.

[0263] The server-side GPU / Ascend chip 1501 can be used to read ground truth data from memory during training and to train neural networks using the read ground truth data.

[0264] Specifically, the GPU / Ascend1501 can read ground truth data corresponding to point cloud data from the database via the CPU1502. The data for each point can be divided into tangent plane information (such as normal vectors) and discrimination information regarding whether point pairs belong to the same submanifold. The GPU / Ascend1501 can train the submanifold nested network 1504 using the tangent plane information from the ground truth data, and use the output of the submanifold nested network 1504 and the discrimination information regarding whether point pairs belong to the same submanifold from the ground truth data to train the boundary discrimination network 1505, thereby achieving the training of both the submanifold nested network and the boundary discrimination network.

[0265] The trained submanifold prediction network can be deployed on a server or terminal for use during the inference phase. However, the hardware deployment of the submanifold nested network 1504 and the boundary discrimination network 1505 depends on the application environment of the server or terminal: if inference is performed on the server, the submanifold nested network 1504 and the boundary discrimination network 1505 are deployed on the server's GPU / Ascend chip; if inference is performed on the terminal, the submanifold nested network 1504 and the boundary discrimination network 1505 are deployed on the terminal's GPU / D chip.

[0266] II. Methods for constructing 3D models

[0267] The 3D model construction method provided in this application can be deployed on a terminal or a server, and will be described by example below.

[0268] 1. Deployed on the terminal

[0269] like Figure 16 As shown, terminal 1600 may include GPU / D chip 1601 and CPU 1602.

[0270] The GPU / D chip 1601 can be used to run submanifold prediction networks, such as... Figure 16 The submanifold nesting network and boundary discrimination network shown in the figure output the prediction results of whether each pair of points in the point cloud data belongs to the same submanifold. This is equivalent to dividing multiple points in the point cloud data and assigning each point to the corresponding submanifold.

[0271] Based on the prediction results fed back by the GPU / D chip 1601, the CPU1602 can perform instance segmentation on the point cloud data and identify each submanifold in the point cloud data. The CPU1602 can also construct multiple triangular meshes using multiple points from the input point cloud data, then extract these triangular meshes as the boundaries of the submanifolds based on the instance segmentation results, and filter out the corner points. Then, based on the geodesic distance, the corner points are triangulated to form multiple Delaunay triangles, and the combined set yields a simplified 3D output model.

[0272] Once the terminal obtains the simplified 3D model, it can apply the simplified 3D model to AR maps, AR games, or other scenarios.

[0273] For example, such as Figure 17 As shown, visual or depth information can be acquired through a camera or laser (Dtof) set on the terminal. Then, the corresponding point cloud data is obtained through the SLAM algorithm. The GPU / D chip 1601 can then output the prediction results of whether each pair of points in the point cloud data belongs to the same submanifold through the submanifold prediction network, which is equivalent to outputting the instance segmentation results of the point cloud data.

[0274] The CPU1602 performs instance segmentation using the output of the submanifold prediction network. Specifically, it identifies points belonging to the same submanifold or the same primitive based on the network's output, and then obtains a vectorized 3D model using the 3D model construction method provided in this application. This 3D model can be directly fed into the GPU for rendering, while the instance segmentation results can be used as information for AR applications. For example, in AR games, the method described in this paper can automatically extract structural information (ground, walls, etc.) and build a reasonable AR game scene based on this information.

[0275] Therefore, in the embodiments of this application, the terminal can be endowed with more functions, providing direct vectorized 3D models and primitive-level instance segmentation capabilities for the terminal's 3D data, leading to more terminal applications and generating value. For example, users can use their mobile phones to provide furniture and interior modeling capabilities for ordinary users, allow users to share lightweight 3D data, and provide furniture design ideas or AR games based on environmental information, thereby improving the user experience.

[0276] 2. Deployed on a server

[0277] like Figure 18 As shown, the server may include a GPU / Ascend chip 1801 and a CPU 1802, etc. The server's processing flow is similar to that described above. Figure 16 The processing flow in the middle terminal is similar, the difference being that the steps executed by the GPU / D chip 1601 are replaced by the GPU / Ascend chip 1801, and the steps executed by the CPU 1602 are replaced by the CPU 1802. This will not be elaborated further here.

[0278] In addition, servers may also include transceivers, such as I / O interfaces, antennas, or other wired or wireless communication interfaces, such as... Figure 19 As shown, the transceiver 1803 can receive input point cloud data and output a simplified 3D model.

[0279] For example, the server can be a server on a cloud platform or one of the servers in a server cluster, equipped with a GPU / Ascend1801 to support neural network operations, and a CPU1802 to support instance segmentation and vectorization of 3D models. Users can transmit collected point cloud data to the cloud platform via a network interface. The GPU / Ascend1801 runs a submanifold prediction network deployed on the server to predict information about local submanifolds, such as points within the same submanifold, and transmits this information to the CPU. The CPU performs instance segmentation based on the submanifold information fed back from the GPU / Ascend and constructs a simplified vectorized 3D model. Then, according to user needs, it feeds back the instance segmentation results and / or the simplified 3D model to the user via the network interface. Essentially, the cloud platform can provide users with a server for primitive-level instance segmentation and 3D model vectorization, allowing users to conveniently and efficiently achieve primitive-level instance segmentation and 3D model vectorization through the cloud platform. For example, the cloud platform can provide automated 3D modeling services to 3D data producers. Manufacturers only need to upload the point cloud data reconstructed by laser scanning or photography to the cloud platform's server. The server can automatically complete the vectorization CAD algorithm process, thereby outputting a vectorized 3D model.

[0280] The following is in conjunction with the above. Figures 4-10 The method steps are illustrated by providing an example of the specific process for performing primitive-level instance segmentation and vectorization of 3D models on the server.

[0281] First, the server can receive point cloud data from the network interface, construct a tensor, and transmit the tensor to the GPU / Ascend1801. If the input point cloud data can include N points, it can be processed through {p i} indicates that an Nx3 tensor is created.

[0282] Then, the GPU / Ascend chip 1801 and CPU 1802 are used to perform primitive-level instance segmentation on the input point cloud data. Specifically, each point in the point set is selected to have 16 nearest neighbors, forming N*16 point pairs. Next, the GPU / Ascend 1801 inputs the point cloud data into a submanifold nested network to predict local submanifolds. Then, the GPU / Ascend 1801 inputs the submanifold features of each pair of nearest neighbors into a boundary discrimination network to determine whether it is a boundary, i.e., whether each edge of the undirected graph composed of N points belongs to different primitives or submanifolds. Edges that do not belong to the same primitive or submanifold are deleted, or points belonging to the same primitive or submanifold are selected. Subsequently, the CPU 1802 can use the boundary output by the submanifold prediction network and the points identified in the same submanifold or primitive to perform primitive-level instance segmentation on the point cloud using the flood fill algorithm.

[0283] Subsequently, the CPU1802 triangulates multiple points in the point cloud data to construct a mesh. Then, it extracts the boundary meshes from these triangular meshes, which are sets of boundaries where two different instances are adjacent or where an instance is adjacent to only one triangle. This set is then simplified using the Ramer-Douglas-Peucker algorithm to extract corner points.

[0284] Subsequently, the CPU1802 performs corner triangulation on each primitive based on the prediction results transmitted from the GPU / Ascend1801, along with the triangular mesh and corner points. For example, the Dijkstra algorithm is used to find the nearest corner point of any point in the mesh, and for each triangle with three distinct nearest corner points, the three related corner points are connected to form a triangle, thereby achieving corner triangulation and obtaining multiple Delaunay triangular meshes.

[0285] By merging the triangular meshes formed by the corner points, the final vectorized 3D model, such as a CAD model, can be output.

[0286] Therefore, in the embodiments of this application, the three-dimensional model construction method provided in this application can be deployed on the server, so that the server can convert the input point cloud data into a vectorized three-dimensional model, and can achieve primitive-level instance segmentation, which has high robustness and efficiently obtains a better three-dimensional model.

[0287] This application embodiment also provides a neural network training device, which can also be called a digital processing chip or a chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform the aforementioned... Figures 4-10 The method steps performed by the neural network training apparatus shown in any of the embodiments.

[0288] This application also provides a three-dimensional model building device, which can also be called a digital processing chip or a chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform the aforementioned... Figures 4-10 The method steps performed by the target detection device shown in any of the embodiments.

[0289] This application also provides a digital processing chip. This digital processing chip integrates circuitry for implementing the functions of the aforementioned processor 1301 / 1401, and one or more interfaces. When the digital processing chip integrates a memory, it can complete the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the actions performed by the neural network distillation device in the foregoing embodiments based on the program code stored in the external memory.

[0290] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned actions. Figures 4 to 10 The steps performed by the neural network training device in the method described in the illustrated embodiment.

[0291] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned actions. Figures 4 to 10 The steps performed by the three-dimensional model building apparatus in the method described in the illustrated embodiment.

[0292] The neural network training device or 3D model building device provided in this application embodiment can be a chip, which may include a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip in the server to perform the above-mentioned operations. Figures 4 to 10 The neural network training method or 3D model construction method described in the illustrated embodiments. Optionally, the storage unit is a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0293] Specifically, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0294] For example, please refer to Figure 20 , Figure 20 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 200. The NPU 200 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 2003, which is controlled by the controller 2004 to extract matrix data from the memory and perform multiplication operations.

[0295] In some implementations, the arithmetic circuit 2003 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 2003 is a two-dimensional pulsating array. The arithmetic circuit 2003 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 2003 is a general-purpose matrix processor.

[0296] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 2002 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 2001 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 2008.

[0297] Unified memory 2006 is used to store input and output data. Weight data is directly transferred to weight memory 2002 via direct memory access controller (DMAC) 2005. Input data is also transferred to unified memory 2006 via DMAC.

[0298] The bus interface unit (BIU) 2010 is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 2009.

[0299] The bus interface unit 2010 (BIU) is used by the instruction fetch memory 2009 to fetch instructions from external memory, and also by the memory access controller 2005 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0300] DMAC is mainly used to move input data from external memory DDR to unified memory 2006, or weight data to weight memory 2002, or input data to input memory 2001.

[0301] The vector computation unit 2007 includes multiple processing units that further process the output of the computation circuits as needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is primarily used for computations in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0302] In some implementations, the vector computation unit 2007 can store the processed output vector in the unified memory 2006. For example, the vector computation unit 2007 can apply linear and / or nonlinear functions to the output of the computation circuit 2003, such as performing linear interpolation on the feature planes extracted by the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 2007 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 2003, for example, for use in subsequent layers of the neural network.

[0303] The instruction fetch buffer 2009 connected to the controller 2004 is used to store the instructions used by the controller 2004;

[0304] Unified Memory 2006, Input Memory 2001, Weight Memory 2002, and Instruction Fetch Memory 2009 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0305] The operations of each layer in the recurrent neural network can be performed by the arithmetic circuit 2003 or the vector calculation unit 2007.

[0306] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more processors used to control the above. Figures 4-10 The method of program execution of integrated circuits.

[0307] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0308] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0309] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0310] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0311] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0312] Finally, it should be noted that the above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A method for constructing a three-dimensional model, characterized in that, include: Acquire point cloud data, which includes multiple points and information corresponding to each point; The point cloud data is input into the submanifold prediction network to obtain the prediction results of the multiple points. The prediction results are used to identify whether each point and its neighboring points belong to the same submanifold. The submanifold prediction network extracts features from the point cloud data to obtain the features corresponding to each point in the multiple points, and determines whether each point and its neighboring points belong to the same submanifold based on the features corresponding to each point. Based on the prediction results of the multiple points, multiple corner points are selected from the multiple points, and the multiple corner points include points on the boundaries of each submanifold formed by the multiple points; A three-dimensional model is constructed based on the multiple corner points, and the mesh formed by the multiple corner points constitutes the manifold in the three-dimensional model.

2. The method according to claim 1, characterized in that, The submanifold prediction network extracts features from the point cloud data, including: From the point cloud data, features are extracted for each point and a first preset number of adjacent points to obtain the local features corresponding to each point. The point cloud data is downsampled to obtain downsampled data, the resolution of which is lower than that of the point cloud data. Features are extracted from the downsampled data to obtain the global features corresponding to each point; By fusing the local features and the global features, the features corresponding to each of the multiple points are obtained.

3. The method according to claim 2, characterized in that, The downsampling of the point cloud data includes: The point cloud data is divided into multiple voxels, each voxel including at least one point and the local features of each point in the at least one point; The extraction of features from the downsampled data includes: The global features are obtained by extracting features from points within each of the multiple voxels and points within a second preset number of adjacent voxels, wherein the number of points within the second preset number of voxels is not less than the first preset number.

4. The method according to any one of claims 1-3, characterized in that, The step of determining whether each point and its neighboring points belong to the same submanifold based on the characteristics of each point includes: Determine the normal vector corresponding to each point based on the features corresponding to each point; Based on the characteristics of each point, the normal vector of each point, and the normal vectors of neighboring points, determine whether each point and its neighboring points belong to the same submanifold.

5. The method according to any one of claims 1-3, characterized in that, The step of selecting multiple corner points from the multiple points based on the prediction results includes: The points are then used to construct a triangular mesh, forming at least one triangular mesh. Based on the prediction results, the boundaries belonging to the same submanifold are extracted from the at least one triangular mesh; From the points on the boundary of the same submanifold extracted from the at least one triangular mesh, the plurality of corner points are extracted.

6. The method according to claim 5, characterized in that, The construction of the 3D model based on the multiple corner points includes: At least one Delaunay triangle grid is constructed using the plurality of corner points and the geodesic distances between the plurality of corner points; The three-dimensional model is obtained by merging at least one Delaunay triangle mesh.

7. A neural network training method, characterized in that, include: Acquire training data, which includes point cloud data and labels corresponding to each point in the point cloud data. The labels corresponding to each point include identifiers used to indicate whether each point and its neighboring points belong to the same submanifold. The multiple points are used as input to the submanifold prediction network to obtain the prediction results of the multiple points. The prediction results include whether each point and its neighboring points belong to the same submanifold. The submanifold prediction network extracts features from the point cloud data to obtain the features corresponding to each point in the multiple points, and determines whether each point and its neighboring points belong to the same submanifold based on the features corresponding to each point. Calculate the loss value based on the prediction results and the label corresponding to each point; The submanifold prediction network is updated based on the loss value to obtain the updated submanifold prediction network.

8. The method according to claim 7, characterized in that, The submanifold prediction network extracts features from the point cloud data, including: From the point cloud data, features are extracted on a per-point basis and on a first preset number of adjacent points to obtain local features; The point cloud data is downsampled at least once to obtain downsampled data, wherein the resolution of the downsampled data is lower than the resolution of the point cloud data. Features are extracted from the downsampled data to obtain global features; By fusing the local features and the global features, the features corresponding to each of the multiple points are obtained.

9. The method according to claim 8, characterized in that, One of the at least one downsampling operations performed on the point cloud data includes: The point cloud data is divided into multiple voxels, each voxel including at least one point and the local features of each point in the at least one point; The extraction of features from the downsampled data includes: At least one feature extraction is performed on each point within a voxel and the points within a second preset number of adjacent voxels to obtain the global features, wherein the number of points within the second preset number of voxels is not less than the first preset number.

10. The method according to any one of claims 7-9, characterized in that, The step of determining whether each point and its neighboring points belong to the same submanifold based on the characteristics of each point includes: The prediction normal vector corresponding to each point is determined based on the features corresponding to each point. Based on the features of each point, the predicted normal vector of each point, and the predicted normal vector of neighboring points, determine whether each point and its neighboring points belong to the same submanifold.

11. The method according to claim 10, characterized in that, The prediction result also includes the normal vector corresponding to each point, and the label of each point also includes the ground truth normal vector corresponding to each point; The step of calculating the loss value based on the prediction result and the label corresponding to each point includes: The loss value is calculated based on the normal vector corresponding to each point and the ground truth normal vector corresponding to each point.

12. A three-dimensional model construction device, characterized in that, include: The transceiver module is used to acquire point cloud data, which includes multiple points and information corresponding to each point; The prediction module is used to input the point cloud data into the submanifold prediction network to obtain the prediction results of the multiple points. The prediction results are used to identify whether each point and its neighboring points belong to the same submanifold. The submanifold prediction network extracts features from the point cloud data to obtain the features corresponding to each point in the multiple points, and determines whether each point and its neighboring points belong to the same submanifold based on the features corresponding to each point. A filtering module is used to filter out multiple corner points from the multiple points based on the prediction results of the multiple points, wherein the multiple corner points include points on the boundaries of each submanifold formed by the multiple points; A construction module is used to construct a three-dimensional model based on the multiple corner points, wherein the mesh formed by the multiple corner points constitutes the manifold in the three-dimensional model.

13. The apparatus according to claim 12, characterized in that, The prediction module is specifically used to perform the following steps through the sub-stream prediction network: From the point cloud data, features are extracted for each point and a first preset number of adjacent points to obtain the local features corresponding to each point. The point cloud data is downsampled to obtain downsampled data, the resolution of which is lower than that of the point cloud data. Features are extracted from the downsampled data to obtain the global features corresponding to each point; By fusing the local features and the global features, the features corresponding to each of the multiple points are obtained.

14. The apparatus according to claim 13, characterized in that, The prediction module is specifically used for: The point cloud data is divided into multiple voxels, each voxel including at least one point and the local features of each point in the at least one point; The global features are obtained by extracting features from points within each of the multiple voxels and points within a second preset number of adjacent voxels, wherein the number of points within the second preset number of voxels is not less than the first preset number.

15. The apparatus according to any one of claims 12-14, characterized in that, The prediction module is specifically used for: Determine the normal vector corresponding to each point based on the features corresponding to each point; Based on the characteristics of each point, the normal vector of each point, and the normal vectors of neighboring points, determine whether each point and its neighboring points belong to the same submanifold.

16. The apparatus according to any one of claims 12-14, characterized in that, The filtering module is specifically used for: The points are then used to construct a triangular mesh, forming at least one triangular mesh. Based on the prediction results, the boundaries belonging to the same submanifold are extracted from the at least one triangular mesh; From the points on the boundary of the same submanifold extracted from the at least one triangular mesh, the plurality of corner points are extracted.

17. The apparatus according to claim 16, characterized in that, The building module is specifically used for: At least one Delaunay triangle grid is constructed using the plurality of corner points and the geodesic distances between the plurality of corner points; The three-dimensional model is obtained by merging at least one Delaunay triangle mesh.

18. A neural network training device, characterized in that, include: The acquisition module is used to acquire training data, which includes point cloud data and a label corresponding to each point in the multiple points of the point cloud data. The label corresponding to each point includes an identifier indicating whether each point and its adjacent points belong to the same submanifold. The output module is used to take the plurality of points as input to the submanifold prediction network to obtain the prediction results of the plurality of points. The prediction results include whether each point and its neighboring points belong to the same submanifold. The submanifold prediction network extracts features from the point cloud data to obtain the features corresponding to each point in the plurality of points, and determines whether each point and its neighboring points belong to the same submanifold based on the features corresponding to each point. The loss module is used to calculate the loss value based on the prediction result and the label corresponding to each point. An update module is used to update the submanifold prediction network based on the loss value, so as to obtain the updated submanifold prediction network.

19. The apparatus according to claim 18, characterized in that, The output module is specifically used for: From the point cloud data, features are extracted on a per-point basis and on a first preset number of adjacent points to obtain local features; The point cloud data is downsampled at least once to obtain downsampled data, wherein the resolution of the downsampled data is lower than the resolution of the point cloud data. Features are extracted from the downsampled data to obtain global features; By fusing the local features and the global features, the features corresponding to each of the multiple points are obtained.

20. The apparatus according to claim 19, characterized in that, The output module is specifically used for: The point cloud data is divided into multiple voxels, each voxel including at least one point and the local features of each point in the at least one point; At least one feature extraction is performed on each point within a voxel and the points within a second preset number of adjacent voxels to obtain the global features, wherein the number of points within the second preset number of voxels is not less than the first preset number.

21. The apparatus according to any one of claims 18-20, characterized in that, The output module is specifically used for: The prediction normal vector corresponding to each point is determined based on the features corresponding to each point. Based on the features of each point, the predicted normal vector of each point, and the predicted normal vector of neighboring points, determine whether each point and its neighboring points belong to the same submanifold.

22. The apparatus according to claim 21, characterized in that, The prediction result also includes the normal vector corresponding to each point, and the label of each point also includes the ground truth normal vector corresponding to each point; The loss module is specifically used to calculate the loss value based on the normal vector corresponding to each point and the ground truth normal vector corresponding to each point.

23. A three-dimensional model construction device, characterized in that, The method includes a processor coupled to a memory storing a program, wherein the program instructions stored in the memory are executed by the processor to implement the method of any one of claims 1 to 6.

24. A neural network training device, characterized in that, The system includes a processor coupled to a memory storing a program, wherein the program instructions stored in the memory are executed by the processor to implement the method of any one of claims 7-11.

25. A computer-readable storage medium comprising a program, which, when executed by a processing unit, performs the method as claimed in any one of claims 1 to 6 or 7 to 11.

26. An apparatus, characterized in that, It includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and when the program instructions are executed by the processing unit, it implements the method of any one of claims 1 to 6 or 7 to 11.

Images