Three-dimensional model generation method based on text cues

Abstract

The invention discloses a three-dimensional model generation method based on a text cue word, and the method comprises the steps: predicting a distance sign function SDF through a constructed neural network according to a dynamically optimized random sampling camera posture and an illumination vector, and carrying out the volume rendering, and obtaining an initial rendered image; through a pre-trained diffusion model, the rendering image quality and the signed distance function prediction accuracy are improved according to the encoded text cue word information; and through the trained neural network, predicting rendering colors and a signed distance function according to known observation point coordinates, obtaining three-dimensional model surface point cloud data in an interpolation mode, and finally generating a three-dimensional model format file. According to the method, the problems that a professional needs to apply professional software to construct the three-dimensional model, time is long and the like are effectively solved, the three-dimensional model constructed through the method can be directly printed through three-dimensional printing equipment, and finally a three-dimensional object entity associated with the text cue word can be obtained.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and three-dimensional model reconstruction, and in particular to a three-dimensional model generation method based on text prompt words. Background Art

[0002] Artificial intelligence (AI) aims to simulate human intelligent behavior, including the ability to learn, reason, and make decisions. Since the 1950s, AI has undergone many evolutions, especially with the increase in computing power and the surge in data volume, machine learning and deep learning technologies have made significant progress. Deep learning uses multi-layer neural networks to automatically extract features from massive data and is widely used in image recognition, natural language processing, and speech recognition. Its high efficiency has made AI have a profound impact in industries such as healthcare, finance, and autonomous driving, and has promoted the rapid development of technology.

[0003] 3D printing, also known as additive manufacturing, is a technology that creates three-dimensional objects by adding materials layer by layer. The technology originated in the 1980s, but has developed rapidly in recent years with the advancement of materials science. The 3D printing process includes three stages: design, printing, and post-processing. The design stage uses computer-aided design (CAD) software to create a three-dimensional model, and the printing stage uses different materials and technologies (such as fused deposition modeling and selective laser sintering) to build it. Post-processing can further improve the quality and appearance of the finished product.

[0004] Generating 3D models from text prompts is an emerging technology that uses natural language processing and deep learning to convert text descriptions into 3D visualization models. This technology usually combines image generation with 3D modeling algorithms to generate corresponding 3D shapes by analyzing key concepts and relationships in the text. This process enables users to quickly obtain personalized 3D models through simple text descriptions. It is widely used in game development, virtual reality, education and other fields, providing great convenience for design and creation. As the technology continues to advance, its accuracy and scope of application are also expanding. Summary of the invention

[0005] The present invention provides a method for generating a three-dimensional model based on text prompt words, so as to solve the problem that the construction of a three-dimensional model in the related art requires professionals to use professional software and spend a long time. The three-dimensional model constructed by the method of the present application can be directly printed using a three-dimensional printing device, and finally a three-dimensional object associated with the text prompt word can be obtained.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] A method for generating a three-dimensional model based on text prompt words, characterized in that it comprises the following steps:

[0008] S1: Through the constructed neural network, the distance sign function SDF is predicted according to the dynamically optimized randomly sampled camera pose and illumination vector, and volume rendering is performed to obtain the initial rendered image;

[0009] S2: Using the pre-trained diffusion model, the quality of rendered images and the prediction accuracy of the signed distance function are improved based on the encoded textual cue information;

[0010] S3: Through the trained neural network, the rendering color and signed distance function are predicted according to the known observation point coordinates, and the surface point cloud data of the 3D model is obtained by interpolation, and finally a 3D model format file is generated;

[0011] S4: Slice the generated 3D model format file and store it in GCODE format, transmit it to the printer via the local area network, and automatically print the described object.

[0012] As a preferred technical solution of the present invention: in step S1, the method of predicting the signed distance function and performing volume rendering to create a rendered image includes:

[0013] S11, randomly initialize sampling coordinates

[0014] Sampling information about light and camera parameters, dynamically optimizing sampling points through spatial resolution, importance sampling, and feedback mechanisms;

[0015] S12. Create a neural network

[0016] The neural network contains two encoders and three multi-layer perceptrons. One encoder is used to encode the sampled coordinates, and two multi-layer perceptrons are used, one predicts the signed distance function SDF and calculates its gradient, and the other predicts the albedo. Another encoder is used to encode the sampled light rotation vector, and the third multi-layer perceptron is used to predict the background color.

[0017] S13. Calculate the transparency based on the signed distance function, and then obtain the ray weight

[0018] Combining albedo, ambient light, diffuse light and ray weights, we get the foreground color, and combining ray weights, foreground color and background color to get the rendered image to complete volume rendering.

[0019] As a preferred technical solution of the present invention: in step S2, the method for improving the quality of the rendered image and the prediction accuracy of the signed distance function includes:

[0020] S21, segmenting and encoding the input text information, copying the text information into four copies for four observation directions, using the pre-trained model to calculate the relevance of each copy of the text information with the corresponding observation direction, and deleting the part of the text information that is weakly related to the current observation direction;

[0021] S22. First, following the training process of the diffusion model, random noise that gradually increases with the time step is added to the rendered image. During the training process, the time step gradually decreases according to the rule of square annealing. The observation direction is determined according to the camera pose corresponding to the rendered image. The corresponding direction is extracted and semantic debiasing and encoded text prompt information are performed. Then, the text prompt information and the rendered image with random noise added are brought into the pre-trained diffusion model to predict the noise.

[0022] S23. The difference between the processed random noise and the predicted noise, and the difference between the original image and the image after the predicted noise is removed are truncated in a score debiasing manner, combined with the gradient of the predicted signed distance function and the variance of the z coordinate of the sampling point, to finally constitute the loss function of the trained neural network, and train the neural network.

[0023] As a preferred technical solution of the present invention: in step S3, the method for generating a three-dimensional model format file includes:

[0024] S31, reading a vertex data file including observation vertex coordinates and observation vertex serial numbers, using the observation vertex coordinates, substituting into a trained neural network model for signed distance function prediction, and obtaining a signed distance function corresponding to the observation vertex coordinates;

[0025] S32, constructing a tetrahedron that penetrates the surface of the object according to the observed vertex coordinates and the signed distance function, calculating the coordinates of the three-dimensional surface points by interpolation for the two vertex coordinates whose two end points of the line segment are respectively inside and outside the geometric body and whose signed distance functions have different signs, and establishing interpolation vertices, and recording the interpolation vertex serial numbers constituting each triangular facet;

[0026] S33, first, the interpolation vertex is brought into the trained albedo prediction neural network model to obtain the albedo corresponding to the interpolation vertex, then, the three-dimensional model is rasterized and the interpolation vertex is further interpolated according to the pixel height and width required for rendering to obtain the interpolation vertex that conforms to the pixel, and then brought into the trained albedo prediction neural network model again to obtain the albedo of the interpolation vertex distributed in the pixel;

[0027] S34. According to the interpolation vertices, the interpolation vertices of the pixel distribution, the corresponding triangle patch numbers, the albedo corresponding to the interpolation vertices, the albedo corresponding to the interpolation vertices of the pixel distribution, the requirements for the storage format of the three-dimensional model file are followed, and finally the three-dimensional model files in different formats are obtained.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. The threshold for obtaining 3D printing models is lowered:

[0030] Users no longer need to learn 3D modeling software and slicing software. They only need to enter a text description of the target object to directly obtain the required 3D model.

[0031] 2. High integration:

[0032] The whole solution of the present invention includes three steps: generating a 3D model from text based on a deep learning method, slicing the 3D model, and sending the sliced ​​file remotely to the printer to start automatic printing. All three steps can be implemented using Python and encapsulated in the same function package. The user only needs to enter the text description and some of the printing parameters to be modified to obtain the final physical object.

[0033] 3. The model is more vivid;

[0034] The 3D model generated by the deep learning method of the present invention aims to achieve photo-realism, using smoother curves and technology that integrates information such as color and texture, so that more vivid effects may be obtained in certain applications. This method can supplement the functions of traditional software such as SolidWorks and UG in specific scenarios, and enhance the realism and delicacy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The overall method flow chart of this application;

[0036] Figure 2 A flow chart of the method from text prompt words to generating a three-dimensional model;

[0037] Figure 3 A flow chart of the method from 3D model to slice printing;

[0038] Figure 4 Compute a structural diagram for generating a 3D model from a textual prompt. DETAILED DESCRIPTION

[0039] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0040] like Figure 1-4 As shown, the present invention proposes a method for generating a three-dimensional model based on text prompt words, comprising the following steps:

[0041] S1: Through the constructed neural network, the distance sign function SDF is predicted according to the dynamically optimized randomly sampled camera pose and illumination vector, and volume rendering is performed to obtain the initial rendered image;

[0042] Methods for predicting a signed distance function and performing volume rendering to create a rendered image include:

[0043] S11, randomly initialize sampling coordinates

[0044] Sampling information about light and camera parameters, dynamically optimizing sampling points through spatial resolution, importance sampling, and feedback mechanisms;

[0045] S12. Create a neural network

[0046] The neural network contains two encoders and three multi-layer perceptrons. One encoder is used to encode the sampled coordinates, and two multi-layer perceptrons are used, one predicts the signed distance function SDF and calculates its gradient, and the other predicts the albedo. Another encoder is used to encode the sampled light rotation vector, and the third multi-layer perceptron is used to predict the background color.

[0047] S13. Calculate the transparency based on the signed distance function, and then obtain the ray weight

[0048] Combining albedo, ambient light, diffuse light and ray weights to get the foreground color, combining ray weights, foreground color and background color to get the rendered image to complete volume rendering;

[0049] S2: Using the pre-trained diffusion model, the quality of rendered images and the prediction accuracy of the signed distance function are improved based on the encoded textual cue information;

[0050] Methods to improve the quality of rendered images and the accuracy of signed distance function predictions include:

[0051] S21, segmenting and encoding the input text information, copying the text information into four copies for four observation directions, using the pre-trained model to calculate the relevance of each copy of the text information with the corresponding observation direction, and deleting the part of the text information that is weakly related to the current observation direction;

[0052] S22. First, following the training process of the diffusion model, random noise that gradually increases with the time step is added to the rendered image. During the training process, the time step gradually decreases according to the rule of square annealing. The observation direction is determined according to the camera pose corresponding to the rendered image. The corresponding direction is extracted and semantic debiasing and encoded text prompt information are performed. Then, the text prompt information and the rendered image with random noise added are brought into the pre-trained diffusion model to predict the noise.

[0053] S23, truncating the difference between the processed random noise and the predicted noise, and the difference between the original image and the image after the predicted noise is removed in a score debiasing manner, combining the gradient of the predicted signed distance function and the variance of the z coordinate of the sampling point, and finally forming a loss function of the trained neural network, and training the neural network;

[0054] S3: Through the trained neural network, the rendering color and signed distance function are predicted according to the known observation point coordinates, and the surface point cloud data of the 3D model is obtained by interpolation, and finally a 3D model format file is generated;

[0055] Methods for generating 3D model format files include:

[0056] S31, reading a vertex data file including observation vertex coordinates and observation vertex serial numbers, using the observation vertex coordinates, substituting into a trained neural network model for signed distance function prediction, and obtaining a signed distance function corresponding to the observation vertex coordinates;

[0057] S32, constructing a tetrahedron that penetrates the surface of the object according to the observed vertex coordinates and the signed distance function, calculating the coordinates of the three-dimensional surface points by interpolation for the two vertex coordinates whose two end points of the line segment are respectively inside and outside the geometric body and whose signed distance functions have different signs, and establishing interpolation vertices, and recording the interpolation vertex serial numbers constituting each triangular facet;

[0058] S33, first, the interpolation vertex is brought into the trained albedo prediction neural network model to obtain the albedo corresponding to the interpolation vertex, then, the three-dimensional model is rasterized and the interpolation vertex is further interpolated according to the pixel height and width required for rendering to obtain the interpolation vertex that conforms to the pixel, and then brought into the trained albedo prediction neural network model again to obtain the albedo of the interpolation vertex distributed in the pixel;

[0059] S34, according to the interpolation vertex, the interpolation vertex of the pixel distribution, the corresponding triangle patch number, the albedo corresponding to the interpolation vertex, the albedo corresponding to the interpolation vertex of the pixel distribution, following the 3D model file storage format requirements, finally obtaining 3D model files in different formats, including 3D model files in various formats such as stl, obj, obj+mtl, etc.;

[0060] S4: Slice the generated 3D model format file and store it in GCODE format, transmit it to the printer via the local area network, and automatically print the described object.

[0061] The part from text description to generating a 3D model is mainly based on deep learning. The pre-trained transformers library and diffusers library in huggingface are used to encode the text and guide model optimization, and the 3D model mesh is created based on the distance signed function SDF.

[0062] First, the camera pose lighting vector with normal distribution and perturbation is added as the model input batch, and the lighting points are dynamically optimized using the nerfacc.OccGridEstimator.sampling function.

[0063] Secondly, create a neural network and use the tinycudann.Encoding encoder to encode the input lighting midpoint vector. Use two multi-layer perceptrons, one to receive the output of the encoder and predict the distance signed function SDF, and the other to receive the output of the encoder and extract features for subsequent volume rendering. For the predicted distance signed function SDF, interpolation and gradient calculation are performed using the finite difference method. On the one hand, the gradient of the signed distance function will be used as part of the neural network loss function, and the generated model will be smoother and more realistic by reducing the gradient of the signed distance function. On the other hand, based on the signed distance function SDF and its gradient, the cumulative density function CDF can be calculated to obtain the transparency of this part of the three-dimensional model.

[0064] The transparency calculation formula is as follows:

[0065]

[0066] Where rad is the radius, sam is the number of sampling points, and std is the dynamic inverse standard deviation of the neural network training. After obtaining the transparency, the nerfacc.render_weight_from_alpha function can be used to convert it into a ray weight and apply it to subsequent volume rendering.

[0067] Then, volume rendering is performed, and the previously extracted features are brought into the activation function to obtain the albedo. The albedo is integrated with the hyperparameters of ambient light and diffuse light, and then combined with the ray weights converted from transparency, and the foreground color is accumulated along the ray based on the nerfacc.accumulate_along_rays function. Another set of neural networks, also tinycudann.Encoding encoder, multi-layer perceptron and activation function, is used to predict a background color separately. Similarly, the function nerfacc.accumulate_along_rays is used to accumulate the ray weights along the ray direction to obtain the rendering weights, and the foreground color and background color are fused according to the rendering weights to obtain the rendered image.

[0068] Next is the rendering image optimization. The input text needs to be processed first. The T5Tokenizer and T5EncoderModel in the transformers library provided by huggingface are used to segment and encode the input text prompt information. The text is copied four times and distributed to four perspectives. The existing language model is used to identify the contradiction between the perspective prompt and the user prompt. Four text description codes that are strongly related to the four observation directions are retained. After that, the current direction can be determined according to the three-dimensional polar coordinates composed of the elevation angle, direction angle and lighting distance of the lighting point to extract the direction-specific text description code. The rendering image optimization is mainly based on the diffusion model, using the IFPipeline function in the diffusers library provided by huggingface and the pre-trained model DeepFloyd / IF-I-XL-v1.0. In the forward process, random noise is added to the rendered image, and it increases with the increase of time steps. In the reverse process, the pre-trained unet is used to predict the noise based on the rendered image and text prompt code after adding noise.

[0069] The original diffusion model loss function is based on the offset between the predicted noise and the random noise and follows the formula:

[0070]

[0071] On this basis, this scheme further introduces supervision of the diffusion model detection image, compares the original image with the image denoised according to the predicted noise and adds it to the loss function to obtain a new diffusion model loss function, following the formula:

[0072]

[0073] This improves the supervision effect on the generated image and accelerates the convergence of the model. Considering the multi-faceted Janus problem when generating three-dimensional models, this model introduces a score debiasing solution here, following the formula:

[0074]

[0075] Among them, Clip(x,c)=max(min(x,c),-c), which reduces the probability of artifacts by timely truncating the larger part of the noise prediction loss.

[0076] In addition, as the model is optimized, the time step of the forward diffusion process is set to be inversely proportional to the square root of the number of training iterations, and the random time step is gradually reduced by square time step annealing, thereby effectively improving the quality of model generation. The corresponding formula is written as:

[0077]

[0078] The last part of the loss function is to calculate the variance of the z coordinates of the ray sampling points. By reducing the variance of the z coordinates of the sampling point distribution, the generated geometric shape can obtain a sharper surface, thereby obtaining a more vivid 3D model generation effect. The calculation method of the Z variance loss follows the formula:

[0079]

[0080] The z variance is expressed as

[0081] Finally, following the UDP protocol, the gcode file is transmitted to the host computer connected to the printer through the local area network. After decoding, the M24 command is automatically called to start printing.

[0082] Based on the above technical solution, the present invention can achieve:

[0083] 1. The threshold for obtaining 3D printing models is lowered:

[0084] Users no longer need to learn 3D modeling software and slicing software. They only need to enter a text description of the target object to directly obtain the required 3D model.

[0085] 2. High integration:

[0086] The whole solution of the present invention includes three steps: generating a 3D model from text based on a deep learning method, slicing the 3D model, and sending the sliced ​​file remotely to the printer to start automatic printing. All three steps can be implemented using Python and encapsulated in the same function package. The user only needs to enter the text description and some of the printing parameters to be modified to obtain the final physical object.

[0087] 3. The model is more vivid;

[0088] The 3D model generated by the deep learning method of the present invention aims to achieve photo-realism, using smoother curves and technology that integrates information such as color and texture, so that more vivid effects may be obtained in certain applications. This method can supplement the functions of traditional software such as SolidWorks and UG in specific scenarios, and enhance the realism and delicacy of the model.

[0089] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. A method for generating a three-dimensional model based on text prompt words, characterized in that: The steps include: S1: Through the constructed neural network, the distance sign function SDF is predicted according to the dynamically optimized randomly sampled camera pose and illumination vector, and volume rendering is performed to obtain the initial rendered image; S2: Using the pre-trained diffusion model, the quality of rendered images and the prediction accuracy of the signed distance function are improved based on the encoded textual cue information; S3: Through the trained neural network, the rendering color and signed distance function are predicted according to the known observation point coordinates, and the surface point cloud data of the 3D model is obtained by interpolation, and finally a 3D model format file is generated; S4: Slice the generated 3D model format file and store it in GCODE format, transmit it to the printer via the local area network, and automatically print the described object.

2. The method for generating a three-dimensional model based on text prompt words according to claim 1, characterized in that: In step S1, the method of predicting a signed distance function and performing volume rendering to create a rendered image includes: S11, randomly initialize sampling coordinates Sampling information about light and camera parameters, dynamically optimizing sampling points through spatial resolution, importance sampling, and feedback mechanisms; S12. Create a neural network The neural network contains two encoders and three multi-layer perceptrons. One encoder is used to encode the sampled coordinates, and two multi-layer perceptrons are used, one predicts the signed distance function SDF and calculates its gradient, and the other predicts the albedo. Another encoder is used to encode the sampled light rotation vector, and the third multi-layer perceptron is used to predict the background color. S13. Calculate the transparency based on the signed distance function, and then obtain the ray weight Combining albedo, ambient light, diffuse light and ray weights, we get the foreground color, and combining ray weights, foreground color and background color to get the rendered image to complete volume rendering.

3. The method for generating a three-dimensional model based on text prompt words according to claim 1, characterized in that: In step S2, the method for improving the quality of the rendered image and the prediction accuracy of the signed distance function includes: S21, segmenting and encoding the input text information, copying the text information into four copies for four observation directions, using the pre-trained model to calculate the relevance of each copy of the text information with the corresponding observation direction, and deleting the part of the text information that is weakly related to the current observation direction; S22. First, following the training process of the diffusion model, random noise that gradually increases with the time step is added to the rendered image. During the training process, the time step gradually decreases according to the rule of square annealing. The observation direction is determined according to the camera pose corresponding to the rendered image. The corresponding direction is extracted and semantic debiasing and encoded text prompt information are performed. Then, the text prompt information and the rendered image with random noise added are brought into the pre-trained diffusion model to predict the noise. S23. The difference between the processed random noise and the predicted noise, and the difference between the original image and the image after the predicted noise is removed are truncated in a score debiasing manner, combined with the gradient of the predicted signed distance function and the variance of the z coordinate of the sampling point, to finally constitute the loss function of the trained neural network, and train the neural network.

4. The method for generating a three-dimensional model based on text prompt words according to claim 1, characterized in that: In step S3, the method for generating a three-dimensional model format file includes: S31, reading a vertex data file including observation vertex coordinates and observation vertex serial numbers, using the observation vertex coordinates, substituting into a trained neural network model for signed distance function prediction, and obtaining a signed distance function corresponding to the observation vertex coordinates; S32, constructing a tetrahedron that penetrates the surface of the object according to the observed vertex coordinates and the signed distance function, calculating the coordinates of the three-dimensional surface points by interpolation for the two vertex coordinates whose two end points of the line segment are respectively inside and outside the geometric body and whose signed distance functions have different signs, and establishing interpolation vertices, and recording the interpolation vertex serial numbers constituting each triangular facet; S33, first, the interpolation vertex is brought into the trained albedo prediction neural network model to obtain the albedo corresponding to the interpolation vertex, then, the three-dimensional model is rasterized and the interpolation vertex is further interpolated according to the pixel height and width required for rendering to obtain the interpolation vertex that conforms to the pixel, and then brought into the trained albedo prediction neural network model again to obtain the albedo of the interpolation vertex distributed in the pixel; S34. According to the interpolation vertices, the interpolation vertices of the pixel distribution, the corresponding triangle patch numbers, the albedo corresponding to the interpolation vertices, the albedo corresponding to the interpolation vertices of the pixel distribution, the requirements for the storage format of the three-dimensional model file are followed, and finally the three-dimensional model files in different formats are obtained.

Images