3D digital simulation spraying method, device, equipment and storage medium
Through 3D convolutional neural network and generative adversarial network, the high cost and inefficiency problems caused by manual intervention are solved, and automated 3D digital simulation spraying is realized.
Patent Information
- Application Number
- CN202411405945.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-10
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2044-10-10
AI Technical Summary
During the 3D digital simulation spraying process, manual intervention in selecting point clouds or patches that need to be sprayed leads to high cost and low efficiency.
Using 3D convolutional neural network and generative adversarial network, by training point cloud data and setting labels, the point cloud structure on the surface of the object is automatically dismantled, and automatic classification and simulation rendering are realized to avoid manual intervention.
Reduces labor costs, improves efficiency, and realizes an automated 3D digital simulation spraying process.
Smart Images
Figure CN118941689B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and more specifically, to a 3D digital simulation spraying method, device, equipment and storage medium. Background Art
[0002] When performing 3D digital simulation spraying on the surface of an object, manual intervention is usually required to select the point cloud or surface patch that needs to be sprayed on the object, which results in high labor costs and low efficiency. Summary of the invention
[0003] The purpose of the present invention is to provide a 3D digital simulation spraying method, device, equipment and storage medium, which does not require manual intervention to select point clouds or facets to be sprayed before simulation, thereby effectively reducing labor costs and improving efficiency.
[0004] In order to achieve the above object, the present invention provides the following technical solutions:
[0005] A 3D digital simulation spraying method, comprising:
[0006] Obtaining a preset amount of point cloud data collected from the surface of the object as training point cloud data;
[0007] A first label and a second label are respectively set for the training point cloud data, the first label is used to indicate that the corresponding training point cloud data belongs to the surface or boundary of the object, and the second label is used to indicate the surface of the object to which the corresponding training point cloud data belongs;
[0008] Sending the training point cloud data and the corresponding first labels to the first stage network of the initial preset classification model, so that the first stage network learns to predict the first labels of different point cloud data;
[0009] Using the first-stage network to predict the first label of the training point cloud data as the predicted label of the corresponding training point cloud data;
[0010] The training point cloud data and the corresponding predicted labels and the second labels are sent to the second-stage network of the initial preset classification model, so that the second-stage network learns to predict the second labels of different point cloud data, and obtains the target preset classification model;
[0011] Disassembly of different faces of the target object is realized based on the target preset classification model to obtain different disassembly faces of the target object;
[0012] Perform 3D digital simulation spraying based on the disassembled surface of the target object.
[0013] Preferably, both the first-stage network and the second-stage network are 3D convolutional neural networks;
[0014] In the first stage, the network learns to predict the first label of different point cloud data using a binary cross entropy loss function;
[0015] The second stage network learns to predict the second label of different point cloud data using the softmax loss function.
[0016] Preferably, before the step of setting the first label and the second label for the training point cloud data respectively, the method further includes:
[0017] Downsample the training point cloud data;
[0018] Normalize the training point cloud data obtained after downsampling;
[0019] The normalized training point cloud data is divided into 3D voxels.
[0020] Preferably, 3D digital simulation spraying is performed based on the disassembled surface of the target object, including:
[0021] Determine in sequence different disassembly surfaces of the target object as current disassembly surfaces;
[0022] Send the current disassembled surface to the pre-trained target rendering generation network to obtain the surface and texture information output by the target rendering generation network;
[0023] The face patches and corresponding texture information output by the target rendering generation network are embedded into the current disassembled surface to achieve simulated rendering of the current disassembled surface.
[0024] Preferably, the process of pre-training a target rendering generation network includes:
[0025] An initial rendering generation network is preset, the initial rendering generation network includes an embedded latent code matrix network and an initial adversarial generation network that are connected to each other, the embedded latent code matrix network is an embedded latent code matrix formed by multiple embedded latent codes, and different backbone layers in the initial generation adversarial network are connected to each embedded latent code in a one-to-one correspondence through a corresponding fully connected network;
[0026] Filter the data of the surface of the object from the training point cloud data as the target point cloud data;
[0027] Get the current input value embedded in the latent code;
[0028] Get the target point cloud data and simulate the surface patch according to the current input value;
[0029] Input the target point cloud data, simulated facets and simulation project labels into the initial generative adversarial network;
[0030] The initial generative adversarial network learning obtains the corresponding face and texture information based on the value simulation of the embedded latent code, and obtains the target rendering generation network.
[0031] Preferably, the loss function h_loss used when the initial generative adversarial network learns to obtain the corresponding patch based on the value simulation of the embedded latent code is:
[0032] h_loss=l2_loss(gram(g)-gram(t))*2;
[0033] Among them, l2_loss refers to the loss function that measures the difference between the patch obtained based on the initial generative adversarial network and the target patch, gram refers to the Gram matrix, g represents the patch obtained based on the initial generative adversarial network, and t represents the corresponding target patch.
[0034] Preferably, obtaining the current input value of the embedded latent code includes:
[0035] Each value embedded in the latent code is added to a corresponding piece of Gaussian noise to generate the current input value.
[0036] A 3D digital simulation spraying device, comprising:
[0037] An acquisition module is used to: acquire a preset amount of point cloud data collected from the surface of an object as training point cloud data;
[0038] A setting module, used to: respectively set a first label and a second label for the training point cloud data, the first label is used to mark the surface or boundary of the object to which the corresponding training point cloud data belongs, and the second label is used to mark the surface of the object to which the corresponding training point cloud data belongs;
[0039] A first training module is used to: send the training point cloud data and the corresponding first labels to the first stage network of the initial preset classification model, so that the first stage network learns to predict the first labels of different point cloud data;
[0040] A prediction module, used to: use the first-stage network to predict the first label of the training point cloud data as the predicted label of the corresponding training point cloud data;
[0041] The second training module is used to: send the training point cloud data and the corresponding predicted labels and the second labels to the second stage network of the initial preset classification model, so that the second stage network learns to predict the second labels of different point cloud data to obtain the target preset classification model;
[0042] The disassembly module is used to: disassemble different surfaces of the target object based on a preset classification model of the target, and obtain different disassembled surfaces of the target object;
[0043] The simulation module is used to: perform 3D digital simulation spraying based on the disassembled surface of the target object.
[0044] A 3D digital simulation spraying device includes a memory and a processor. When a program stored in the memory is executed by the processor, the steps of the 3D digital simulation spraying method described above can be implemented.
[0045] A computer-readable storage medium, wherein the program stored in the computer-readable storage medium is executed by a processor to implement the steps of the above-mentioned 3D digital simulation spraying method
[0046] The present invention obtains a preset amount of training point cloud data of the surface of an object, sets a first label indicating that the training point cloud data is an object surface or boundary, and a second label indicating which surface of the object it is, uses the training point cloud data and the first label to train the first stage network of the initial preset classification model, so that the first stage network can predict the first label content of different point cloud data, uses the first stage network to predict the first label of the training point cloud data, uses the training point cloud data, the second label and the predicted first label to train the second stage network of the initial preset classification model, so that the second stage network can predict the second label content of different point cloud data, obtains the target preset classification model, realizes the disassembly of different surfaces of the target object based on the target preset classification model, and performs 3D digital simulation spraying based on the disassembled surface obtained by disassembly. It can be seen that the present invention can automatically determine whether the points on the surface of the object belong to the surface or the boundary based on the classification model, and which surface it belongs to when it belongs to the surface, thereby realizing the disassembly of the point cloud structure on the surface of the object, classifying the point cloud data representing different surfaces, and there is no need to manually intervene to select the point cloud or surface patch that needs to be sprayed before simulation, thereby effectively reducing labor costs and improving efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0048] Figure 1 A flow chart of a 3D digital simulation spraying method provided by an embodiment of the present invention;
[0049] Figure 2 A schematic diagram of implementing point cloud data processing using a 3D convolutional neural network in a 3D digital simulation spraying method provided by an embodiment of the present invention;
[0050] Figure 3 A flowchart for implementing the generation of an adversarial network and embedding a latent code matrix in a 3D digital simulation spraying method provided by an embodiment of the present invention;
[0051] Figure 4A schematic structural diagram of a 3D digital simulation spraying device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0053] See also Figure 1 , which shows a flow chart of a 3D digital simulation spraying method provided by an embodiment of the present invention, which may specifically include:
[0054] S11: Obtain a preset amount of point cloud data collected from the surface of the object as training point cloud data.
[0055] The object in the embodiment of the present invention can be any object that needs to be sprayed with 3D digital simulation on the surface, such as automobile parts. In order to train the preset classification model (the initial preset classification model is the preset classification model before training, and the target initial classification model is the preset classification model after training), a preset amount of point cloud data collected from the surfaces of different objects can be obtained in advance as the data used for training the preset classification model, that is, training point cloud data. Among them, the larger the preset amount, the better the training effect, but the slower the training speed, so the preset amount can be set according to actual needs.
[0056] In addition, the objects for collecting training point cloud data can be the same type of objects or different types of objects. Generally speaking, the higher the classification accuracy of the trained model is, the more accurate it is. However, in order to reduce workload and improve efficiency, the objects can also be different types of objects. In this case, the trained model can be universal.
[0057] S12: setting a first label and a second label for the training point cloud data respectively, the first label is used to indicate that the corresponding training point cloud data belongs to a surface or boundary of an object, and the second label is used to indicate the surface of the object to which the corresponding training point cloud data belongs.
[0058] In the embodiment of the present invention, the surface of an object is the outer surface of the object, and the outer surface of the object includes a face and a boundary (the junction between faces is called a boundary).
[0059] A first label and a second label are set for each training point cloud data. The first label indicates whether the corresponding training point cloud data belongs to the surface of the object or the boundary of the object, that is, the value of the first label is the mark of the surface or the boundary; the second label indicates the specific surface of the corresponding training point cloud data in the object.
[0060] It should be noted that the second label can be set only for the training point cloud data belonging to the surface of the object, in which case the value of the second label is the label of the corresponding surface; the second label can also be set for all training point cloud data, in which case the value of the second label is the label of the corresponding surface or a label indicating that it does not belong to any surface.
[0061] Before training, the embodiment of the present invention assigns different labels to the training point cloud data (the point cloud data can be expressed as X, Y, Z) on different surfaces. Traditional classification models usually can only support classification of fixed categories. For example, a trained cat and dog classification model can only classify the target image into cats or dogs, and does not support the output of a third animal. However, when classifying the surface of an object, the number of faces of the object is uncertain. How to enable the classification model to support classification of different numbers of faces is a problem that needs to be addressed. The idea of solving this problem in this solution is to only find the boundaries of different faces, set the label (first label, Label1) value of the training point cloud data on the boundary to 0, and set the label (first label) value of the training point cloud data contained in the two connected faces to 1. In this way, based on human vision, after obtaining this classification result, it is relatively easy to determine which training point cloud data belongs to a face.
[0062] However, unlike 2D images, in training point cloud data, although it is easy for humans to observe, it is difficult to determine which points are divided into a surface by the boundary by setting specific rules in advance, so this work must be handed over to the neural network. After the first-stage network gives the boundary and surface labels 0 and 1, it can give each training point cloud data a second label Label2 according to the area size of each surface, and number each surface independently according to the area size, and integrate this set of labels into the training point cloud data (the data of each point becomes X, Y, Z, Label1, Label2), and send it to the second-stage network, so that the second-stage network can learn and give label2 for each point. In this way, the neural network can automatically complete the task of surface classification of different point clouds.
[0063] S13: Sending the training point cloud data and the corresponding first labels to the first stage network of the preset classification model, so that the first stage network learns to predict the first labels of different point cloud data.
[0064] In the embodiment of the present invention, a 3D convolutional neural network can be selected as a preset classification model, and a 3D Unet is used as the backbone of the neural network. If necessary, other large model structures or lightweight structures can also be selected. The 3D convolutional neural network can capture local and global features of point cloud data. The preset classification model includes a first-stage network and a second-stage network. The first-stage network and the second-stage network can be the same or different, and can both be 3D convolutional neural networks.
[0065] The training point cloud data and the corresponding first labels are sent to the first stage network of the initial preset classification model, so that the first stage network learns to predict the first label content of different point cloud data, thereby realizing the training of the first stage network.
[0066] S14: Using the first-stage network to predict the first label of the training point cloud data as the predicted label of the corresponding training point cloud data.
[0067] S15: Send the training point cloud data and the corresponding predicted labels and the second labels to the second stage network of the initial preset classification model, so that the second stage network learns to predict the second labels of different point cloud data to obtain the target preset classification model.
[0068] The first-stage network is used to predict the first label of the training point cloud data, and then the training point cloud data and the predicted first and second labels are sent to the second-stage network of the initial preset classification model, so that the second-stage network learns to predict the second labels of different point cloud data, thereby realizing the training of the second-stage network; after completing the first-stage network and the second-stage network, the initial preset classification model becomes the target preset classification model.
[0069] It should be noted that when the second label is set only for the training point cloud data belonging to the surface of the object, the training point cloud data and its related labels that are marked with the predicted first label as belonging to the surface can be sent to the second-stage network; when the second label is set for all training point cloud data, all training point cloud data and their related labels can be sent to the second-stage network.
[0070] After distinguishing the boundary and the surface, this embodiment of the present invention allows the preset classification model to learn to distinguish the size of the surface to which the point cloud data belongs and give labels for different surfaces based on this, ultimately achieving the purpose (distinguishing different surfaces). The overall process can be as follows Figure 2 As shown in the figure, the label of the input point cloud data consists of three parts: the double parallel line part is the boundary label 0, the white part within the double parallel lines is the surface label 1, and the numbers above the white part are independent labels for each surface, and the labels are given according to the surface area.
[0071] S16: Disassembling different surfaces of the target object based on the preset classification model of the target to obtain different disassembled surfaces of the target object.
[0072] S17: Perform 3D digital simulation spraying based on the disassembled surface of the target object.
[0073] The target object is an object that needs to be 3D digital simulated sprayed. After the target object is determined, the point cloud data of the surface of the target object can be collected, and then the collected point cloud data are sent to the target preset classification model to obtain the first label and the second label of the point cloud data. Based on the first label and the second label of the point cloud data, it can be obtained which point cloud data belongs to the surface of the object and which specific surface it belongs to, and which point cloud data belongs to the boundary of the object, thereby realizing the disassembly of different surfaces of the object, obtaining different disassembled surfaces of the target object, and finally realizing 3D digital simulated spraying of different disassembled surfaces.
[0074] The present invention obtains a preset amount of training point cloud data of the surface of an object, sets a first label indicating that the training point cloud data is an object surface or boundary, and a second label indicating which surface of the object it is, uses the training point cloud data and the first label to train the first stage network of the initial preset classification model, so that the first stage network can predict the first label content of different point cloud data, uses the first stage network to predict the first label of the training point cloud data, uses the training point cloud data, the second label and the predicted first label to train the second stage network of the initial preset classification model, so that the second stage network can predict the second label content of different point cloud data, obtains the target preset classification model, realizes the disassembly of different surfaces of the target object based on the target preset classification model, and performs 3D digital simulation spraying based on the disassembled surface obtained by disassembly. It can be seen that the present invention can automatically determine whether the points on the surface of the object belong to the surface or the boundary based on the classification model, and which surface it belongs to when it belongs to the surface, thereby realizing the disassembly of the point cloud structure on the surface of the object, classifying the point cloud data representing different surfaces, and there is no need to manually intervene to select the point cloud or surface patch that needs to be sprayed before simulation, thereby effectively reducing labor costs and improving efficiency.
[0075] In a 3D digital simulation spraying method provided by an embodiment of the present invention, both the first-stage network and the second-stage network can be 3D convolutional neural networks;
[0076] In the first stage, the network can use the binary cross entropy loss function when learning to predict the first label of different point cloud data;
[0077] The softmax loss function can be used when the second stage network learns to predict the second label of different point cloud data.
[0078] In the embodiment of the present invention, both the first-stage network and the second-stage network can be 3D convolutional neural networks. When training the first-stage network, a binary cross entropy loss function can be used. In fact, it only needs to give a binary label (first label) for each point; when training the second-stage network, a softmax loss function can be selected, and the number of faces contained in the point cloud data with the most faces in the training point cloud data is selected as the number of softmax classifications. Based on this selection, it can be ensured that the neural network can be used to achieve end-to-end distinction between different faces.
[0079] A 3D digital simulation spraying method provided by an embodiment of the present invention may further include, before the step of setting a first label and a second label for training point cloud data respectively:
[0080] Downsample the training point cloud data;
[0081] Normalize the training point cloud data obtained after downsampling;
[0082] The normalized training point cloud data is divided into 3D voxels.
[0083] After acquiring the training point cloud data, the embodiment of the present invention can also pre-process the training point cloud data, which may include sampling and normalization. Specifically, the training point cloud data can be down-sampled to reduce the point cloud density to reduce the amount of calculation. For example, three adjacent training point cloud data can be grouped together, and the center point of the group is used as a new training point cloud data to replace the original three adjacent training point cloud data. To normalize the training point cloud data, the center of gravity of the training point cloud data can be first calculated, and the center of gravity is used as the center of the cube, which is also the coordinate origin of the training point cloud data (i.e., the point cloud data center), and then the training point cloud data are normalized to the cube (the size of the cube can be set according to actual needs, such as 512*512*512). In addition, for the point cloud data collected from the surface of the target object in step S16, the above-mentioned pre-processing steps can also be performed first, and then sent to the target preset classification model for subsequent steps such as face decomposition.
[0084] In order to increase the step size and improve the training efficiency, the embodiment of the present invention can also perform point cloud segmentation (Voxelization) on the training point cloud data, dividing the training point cloud data into 3D voxels (Voxel), that is, dividing the three-dimensional space into grids of fixed size, each grid containing a group of points.
[0085] A 3D digital simulation spraying method provided by an embodiment of the present invention performs 3D digital simulation spraying based on a disassembled surface of a target object, which may include:
[0086] Determine in sequence different disassembly surfaces of the target object as current disassembly surfaces;
[0087] Send the current disassembled surface to the pre-trained target rendering generation network to obtain the surface and texture information output by the target rendering generation network;
[0088] The face patches and corresponding texture information output by the target rendering generation network are embedded into the current disassembled surface to achieve simulated rendering of the current disassembled surface.
[0089] After obtaining different disassembled surfaces of the target object (i.e., different surfaces of the target object obtained by disassembling), the embodiment of the present invention can sequentially send each disassembled surface into a pre-trained target rendering generation network, and the target rendering generation network can obtain corresponding face patches and texture information based on the sent-in disassembled surfaces. The face patches are the face patches of the surface on the target object obtained by restoring the disassembled surfaces, and the texture information is the information of the face style after the face patches are texture rendered. By embedding the face patches and texture information of each disassembled surface into the disassembled surface, simulated rendering of the disassembled surface can be realized, and visualization software can be used to view the effect diagram after simulated rendering.
[0090] In a 3D digital simulation spraying method provided by an embodiment of the present invention, the process of pre-training a target rendering generation network may include:
[0091] An initial rendering generation network is preset, the initial rendering generation network includes an embedded latent code matrix network and an initial adversarial generation network that are connected to each other, the embedded latent code matrix network is an embedded latent code matrix formed by multiple embedded latent codes, different embedded latent codes correspond to different simulation projects, and the value of each embedded latent code represents the simulation degree of the corresponding simulation project, and different backbone layers in the initial generation adversarial network are connected to each embedded latent code one by one through a corresponding fully connected network;
[0092] Filter the data of the surface of the object from the training point cloud data as the target point cloud data;
[0093] Get the current input value embedded in the latent code;
[0094] Get the target point cloud data and simulate the surface patch according to the current input value;
[0095] Input the target point cloud data, simulated facets and simulation project labels into the initial generative adversarial network;
[0096] The initial generative adversarial network learning obtains the corresponding face and texture information based on the value simulation of the embedded latent code, and obtains the target rendering generation network.
[0097] Wherein, obtaining the current input value of the embedded latent code may include: adding the value of each embedded latent code to a corresponding section of Gaussian noise to generate the current input value.
[0098] It should be noted that before training, the generative adversarial network (such as the GAN network) is the initial generative adversarial network, and the rendering generation network it belongs to is the initial rendering generation network. After training, it is the target generative adversarial network, and the rendering generation network it belongs to is the target rendering generation network. Figure 3 As shown, the entire implementation process of the embodiment of the present invention may include:
[0099] 1. Decompose the faces generated by the target preset classification model to obtain different faces of the corresponding objects, that is, Figure 3 The middle double parallel line section;
[0100] 2. Input the generated adversarial network to all point cloud data of a certain face;
[0101] 3. Figure 3 The dotted box on the left is an embedded latent code. Multiple such embedded latent codes form an embedded latent code matrix. Each embedded latent code corresponds to a simulation item (the simulation item corresponding to texture information, such as color, brightness, texture, etc.), and the value of each embedded latent code represents the degree of the corresponding simulation item, which can be a value from 0 to 9. After entering the value of the embedded latent code, the value will be added to a section of Gaussian noise to form a complete embedded latent code (i.e., the current input value). The complete embedded latent code will pass through a simple fully connected network and then enter the backbone layer of the generative adversarial network (each embedded latent code corresponds to a different backbone layer).
[0102] The embedded latent code is generated as follows:
[0103] y=x+n;
[0104] n=N(0,σ²);
[0105] Where x is a value ranging from 0 to 9, n is Gaussian noise that follows a normal distribution, and y is the generated embedded latent code matrix (embedded latent code sequence).
[0106] 4. Generate adversarial network output including the patch structure and the style of the surface after texture rendering.
[0107] To facilitate the preparation of training labels, only three levels of embedded latent codes can be input during training, such as 0, 4, and 9. At the same time, corresponding to the embedded latent codes of different simulation levels, the point cloud data can be given the corresponding simulation process patches and simulation project labels (corresponding texture, color, brightness, etc.). After the training of the generative adversarial network is completed, the effect of controlling the simulation requirements can be achieved based on this by embedding latent codes.
[0108] It should also be noted that if it is necessary to achieve simulated rendering of the disassembled surface of the target object based on a generative adversarial network, the point cloud data in the above steps 1 to 3 are the point cloud data of the target object, and there is no need to perform step 4; if it is necessary to train the generative adversarial network, the point cloud data in the above steps 1 to 3 are training point cloud data.
[0109] In a specific implementation, in order to make the simulation effect realistic, the process loss can be added to the loss function of the generative adversarial network to measure the difference between the simulated rendering (the patch obtained based on the generative adversarial network) and the target rendering (the corresponding target patch, that is, the patch to be formed). The loss function h_loss used by the generative adversarial network to learn the corresponding patch based on the value of the embedded latent code can be:
[0110] h_loss=l2_loss(gram(g)-gram(t))*2;
[0111] Among them, l2_loss refers to the loss function that measures the difference between the patch obtained based on the generative adversarial network and the target patch, gram refers to the Gram matrix, g represents the patch obtained based on the generative adversarial network, and t represents the corresponding target patch. That is, the Gram matrix of the simulated rendering and the target rendering is required to measure the hidden relationship between the features in the image, and then compare the differences in these relationships to measure the size of the process simulation loss.
[0112] After completing the above process, the target point cloud data can be collected from the surface of the object that needs to be simulated sprayed. First, the target point cloud data is sent to the preset classification model to decompose the surface, and then the different decomposed faces are sent to the generative adversarial network respectively, so that the generative adversarial network can give a suitable texture. Finally, the face patches and textures of each face are embedded in the original point cloud data, and the visualization software can be used to see the effect of the corresponding object after simulation rendering.
[0113] The present invention uses a 3D convolutional neural network to disassemble the point cloud structure of the object surface, classify the points representing different surfaces, and use an embedded latent code matrix and a generative adversarial network to simulate different spraying processes in a direction, generating end-to-end simulation effects after spraying. Using the technical solution of the present invention, there is no need for manual intervention to select and specify the effects of the point clouds or patches to be sprayed before simulation. Instead, such operations are used as content that the neural network needs to learn, and special training methods are used to allow the model to automatically complete this stage. Therefore, it is only necessary to fill in the latent code matrix of the desired generation effect, and the generation model can generate a variety of spraying simulation effects in a direction.
[0114] In a specific scenario, a 3D digital simulation spraying method provided by an embodiment of the present invention may include the following steps:
[0115] 1. 3D Point Cloud Surface Classification
[0116] 1. Preprocessing of training point cloud data;
[0117] 2. Training point cloud data blocks;
[0118] 3. 3D convolutional neural network: Use 3D convolutional neural network to process the divided training point cloud data.
[0119] 4. Surface classification:
[0120] 1. To label the training point cloud data, the first label needs to be: face label, boundary label 1; the second label is: face independent label.
[0121] 2. Send the training point cloud data and the first label to the first stage network, and let the model learn to predict the content of the first label (that is, whether a point is a boundary or a surface).
[0122] 3. After the first-stage network learns and predicts, input the training point cloud data, and the first-stage network will generate the first label; integrate the training point cloud data and the first label predicted by the first-stage network, and combine them with the second label to send them to the second-stage network, so that the model can learn to predict the independent label of each face.
[0123] 2. Embedding Latent Code Matrix and Generative Adversarial Network
[0124] 1. Disassemble the faces generated by the preset classification model;
[0125] 2. Input is all training point cloud data of a certain face;
[0126] 3. Multiple embedded latent codes form an embedded latent code matrix;
[0127] 4. The output is the style of the surface after including the patch structure and texture rendering:
[0128] Corresponding to the different degrees of embedded latent code matrix, the training point cloud data needs to be given the corresponding process patches and simulation project labels (corresponding texture color brightness, etc.). After the model training is completed, the effect of controlling the simulation requirements can be achieved by embedding the latent code.
[0129] 3. End-to-end integration
[0130] After completing the above process, for the target point cloud data, first send the target point cloud data to the preset classification model for surface decomposition, and then send the different decomposition surfaces to the generative adversarial network respectively, so that the generative adversarial network can give a suitable texture, and finally, embed the face patches and textures of each surface into the original point cloud data, and use the visualization software to see the effect of the corresponding object after simulation rendering.
[0131] The embodiment of the present invention also provides a 3D digital simulation spraying device, such as Figure 4 As shown, it may include:
[0132] The acquisition module 11 is used to: acquire a preset amount of point cloud data collected from the surface of the object as training point cloud data;
[0133] A setting module 12 is used to set a first label and a second label for the training point cloud data respectively, wherein the first label is used to indicate that the corresponding training point cloud data belongs to a surface or a boundary of an object, and the second label is used to indicate the surface of the object to which the corresponding training point cloud data belongs;
[0134] The first training module 13 is used to: send the training point cloud data and the corresponding first labels to the first stage network of the initial preset classification model, so that the first stage network learns to predict the first labels of different point cloud data;
[0135] Prediction module 14, used to: use the first-stage network to predict the first label of the training point cloud data as the predicted label of the corresponding training point cloud data;
[0136] The second training module 15 is used to: send the training point cloud data and the corresponding predicted labels and the second labels to the second stage network of the initial preset classification model, so that the second stage network learns to predict the second labels of different point cloud data to obtain the target preset classification model;
[0137] The disassembly module 16 is used to: disassemble different surfaces of the target object based on a preset classification model of the target, and obtain different disassembled surfaces of the target object;
[0138] The simulation module 17 is used to perform 3D digital simulation spraying based on the disassembled surface of the target object.
[0139] An embodiment of the present invention further provides a 3D digital simulation spraying device, which includes a memory and a processor. When a program stored in the memory is executed by the processor, the steps of the above-mentioned 3D digital simulation spraying method can be implemented.
[0140] An embodiment of the present invention further provides a computer-readable storage medium, and when a program stored on the computer-readable storage medium is executed by a processor, the steps of the above-mentioned 3D digital simulation spraying method are implemented.
[0141] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A 3D digital simulation spraying method, characterized in that: include: Obtaining a preset amount of point cloud data collected from the surface of the object as training point cloud data; respectively setting a first label and a second label for the training point cloud data, wherein the first label is used to indicate that the corresponding training point cloud data belongs to a surface or a boundary of an object, and the second label is used to indicate the surface of the object to which the corresponding training point cloud data belongs; Sending the training point cloud data and the corresponding first labels to the first stage network of the initial preset classification model, so that the first stage network learns to predict the first labels of different point cloud data; Using the first-stage network to predict the first label of the training point cloud data as the predicted label of the corresponding training point cloud data; Sending the training point cloud data and the corresponding predicted labels and the second labels to the second stage network of the initial preset classification model, so that the second stage network learns to predict the second labels of different point cloud data to obtain the target preset classification model; Disassembling different faces of the target object based on the preset classification model of the target to obtain different disassembled faces of the target object; Performing 3D digital simulation spraying based on the disassembled surface of the target object: sequentially determining different disassembled surfaces of the target object as current disassembled surfaces; sending the current disassembled surface to a pre-trained target rendering generation network to obtain a face patch and texture information output by the target rendering generation network; embedding the face patch and corresponding texture information output by the target rendering generation network into the current disassembled surface to achieve simulation rendering of the current disassembled surface; The process of pre-training the target rendering generation network includes: Preset an initial rendering generation network, the initial rendering generation network includes an embedded latent code matrix network and an initial generation adversarial network connected to each other, the embedded latent code matrix network is an embedded latent code matrix formed by multiple embedded latent codes, and different backbone layers in the initial generation adversarial network are connected to each of the embedded latent codes in a one-to-one correspondence through a corresponding fully connected network; Filtering data belonging to the surface of the object from the training point cloud data as target point cloud data; Obtaining a current input value of the embedded latent code; Acquire a surface patch obtained by simulating the target point cloud data according to the current input value; Inputting the target point cloud data, simulated facets and simulation project labels into the initial generative adversarial network; The initial generative adversarial network learning obtains corresponding facets and texture information based on the value simulation of the embedded latent code to obtain the target rendering generation network.
2. The 3D digital simulation spraying method according to claim 1, characterized in that: The first-stage network and the second-stage network are both 3D convolutional neural networks; The first stage network learning uses a binary cross entropy loss function when predicting the first label of different point cloud data; The second stage network uses a softmax loss function when learning to predict the second labels of different point cloud data.
3. The 3D digital simulation spraying method according to claim 2, characterized in that: Before the step of respectively setting the first label and the second label for the training point cloud data, the method further includes: Downsampling the training point cloud data; Normalize the training point cloud data obtained after downsampling; The normalized training point cloud data is divided into 3D voxels.
4. The 3D digital simulation spraying method according to claim 1, characterized in that: The loss function h_loss used by the initial generative adversarial network to learn the corresponding patch based on the value simulation of the embedded latent code is: h_loss = l2_loss(gram(g) - gram(t)) * 2; Among them, l2_loss refers to a loss function that measures the difference between a patch obtained based on the initial generative adversarial network and a target patch, gram refers to a Gram matrix, g represents a patch obtained based on the initial generative adversarial network, and t represents the corresponding target patch.
5. The 3D digital simulation spraying method according to claim 4, characterized in that: The obtaining of the current input value of the embedded latent code comprises: Each value of the embedded latent code is added to a corresponding Gaussian noise to generate the current input value.
6. A 3D digital simulation spraying device, characterized in that: The device includes a memory and a processor, and the program stored in the memory can implement the steps of the 3D digital simulation spraying method as described in any one of claims 1 to 5 when executed by the processor.
7. A computer-readable storage medium, characterized in that: When the program stored on the computer-readable storage medium is executed by the processor, the steps of the 3D digital simulation spraying method according to any one of claims 1 to 5 are implemented.