Three-dimensional model generation method, electronic equipment and storage medium
By training a denoising diffusion probability model and a detail prediction model, coarse and detailed feature parameters of a 3D model are generated, solving the problems of low efficiency and uncontrollability in existing technologies, and realizing efficient and controllable 3D model generation.
Patent Information
- Application Number
- CN202311121685.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-01
- Publication Date
- 2026-04-24
AI Technical Summary
Existing 3D model generation methods are inefficient, computationally expensive, and uncontrollable when dealing with high resolution and irregular topologies, and cannot generate 3D models according to user needs.
A denoising diffusion probability model and a detail prediction model are used to train parameters for extracting coarse and detailed features of a 3D model, respectively. The coarse feature parameters are obtained through the denoising diffusion probability model and combined with the detail prediction model to generate the target 3D model.
Based on existing computing power, the generation time of 3D models is greatly reduced, the generation efficiency and controllability are improved, and the generated 3D models are more in line with expectations.
Smart Images
Figure CN121921430A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of deep learning automatic generation of 3D models technology, and in particular relates to a 3D model generation method, electronic device and storage medium. Background Technology
[0002] With technological advancements, neural networks are increasingly being applied to the field of automated modeling. Artificial intelligence-based automated modeling can rapidly create 3D content, enriching computer graphics, vision, and virtual / augmented reality. Existing generative models primarily focus on 3D representations such as voxels, point clouds, and meshes. These methods are ill-suited for handling high resolution and irregular topologies, thus failing to produce high-fidelity results. Representing a 3D surface requires sampling a large number of spatial points for point clouds, necessitating significant computational resources. For fields, only regions near the model surface have distinct data values, but encoding samples in smoothly varying regions requires substantial effort, consuming significant computational resources and contributing little to model generation. Furthermore, the controllability of point clouds and fields during 3D model reconstruction is limited, making it impossible to generate 3D models tailored to user needs. Summary of the Invention
[0003] To overcome the shortcomings of the aforementioned three-dimensional model generation process, such as low efficiency, high computational cost, and uncontrollable complexity, this invention proposes a three-dimensional model generation method, electronic device, and storage medium.
[0004] The technical solution includes: a three-dimensional model generation method, comprising: training a denoising diffusion probability model for coupling input expected generation information into coarse features of a three-dimensional model and a detail prediction model for predicting detailed features of the three-dimensional model; inputting expected generation information into the denoising diffusion probability model to obtain coarse feature parameters of the target three-dimensional model; inputting the coarse feature parameters into the detail prediction model to obtain detailed feature parameters of the target three-dimensional model; and generating the target three-dimensional model based on the coarse feature parameters and the detailed feature parameters.
[0005] Based on the above-described method for generating three-dimensional models, this invention also discloses an electronic device, comprising: a memory storing a computer program thereon; and a processor for executing the computer program in the memory to generate a three-dimensional model according to the above-described method for generating three-dimensional models.
[0006] Based on the above-mentioned three-dimensional model generation method, the present invention also discloses a storage medium storing a program for the three-dimensional model generation method, the program being executed by a processor to implement the steps of the three-dimensional model generation method.
[0007] This invention trains a denoising diffusion probability model to couple coarse feature parameters of a 3D model based on desired generation information, and trains a detail prediction model to generate detailed feature parameters of the 3D model based on the coarse feature parameters. Finally, a target 3D model is generated based on both the coarse and detailed feature parameters. During this process, the coarse feature parameters can be adjusted and refined according to the desired generation information, thereby achieving matching and controllability of the final generated 3D model. Attached Figure Description
[0008] Figure 1 This is a schematic diagram of a method flow according to one embodiment of the present invention;
[0009] Figure 2 This is a flowchart illustrating one implementation of the denoising diffusion probability model of the present invention;
[0010] Figure 3 This is a flowchart illustrating one implementation of the three-dimensional model generation process of the present invention;
[0011] Figure 4 This is a flowchart illustrating one implementation of the detail prediction model of the present invention. Detailed Implementation
[0012] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0013] In the description of this application, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.
[0014] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.
[0015] The invention will now be described in further detail with reference to the accompanying drawings. This invention primarily focuses on Denoising Diffusion Probabilistic Models (DDPM), and reconstructs 3D models based on these models. Using this invention, the 3D model generation time can be significantly reduced and the controllable generation efficiency of 3D models can be improved with existing computing power.
[0016] Reference Figure 1 This invention discloses a method for generating a three-dimensional model, comprising: training a denoising diffusion probability model for coupling input expected generation information into coarse features of a three-dimensional model and a detail prediction model for predicting detailed features of the three-dimensional model; inputting expected generation information into the denoising diffusion probability model to obtain coarse feature parameters of a target three-dimensional model; inputting the coarse feature parameters into the detail prediction model to obtain detailed feature parameters of the target three-dimensional model; and generating a target three-dimensional model based on the coarse feature parameters and the detailed feature parameters.
[0017] Combination Figure 1-4 In this embodiment, a denoising diffusion probability model is trained to effectively extract useful information from the expected generated information to obtain coarse feature parameters of the 3D model. The expected generated information can be a rendered image of the 3D training model and / or a text description of the 3D training model. For example, if an image or text of a chair is input, relevant information elements are extracted from the input image or text. Based on these information elements, the denoising diffusion probability model obtains coarse feature parameters and the overall shape of the chair, such as the backrest, seat, and leg rest. Then, a trained detail prediction model obtains the detailed features of the aforementioned backrest, seat, and leg rest. Finally, a 3D mesh model of the chair is generated based on the coarse and detailed features.
[0018] Based on the above embodiments, the control variables include text and / or images. The desired generated information can be: inputting descriptive text alone, with the denoising diffusion probability model extracting relevant object 3D information from the text; or inputting an object image alone, with the denoising diffusion probability model extracting relevant object 3D information from the image; or inputting both an object image and descriptive text, with the denoising diffusion probability model extracting relevant object 3D information by cross-referencing the image and text information.
[0019] Based on one or more of the above embodiments, combined with Figure 2-4 In the process of training and reconstructing using a chair as an example, the denoising diffusion probability model training method includes: acquiring a 3D training model and converting it into directed distance field data; performing continuous wavelet transform on the directed distance field data to generate frequency domain data expressing the shape features of the 3D training model at multiple scales; and obtaining high-frequency coefficients expressing the shape detail features of the 3D training model at different scales and low-frequency coefficients expressing the shape coarseness features of the 3D training model based on the frequency domain data. In the above steps, the first step is data format conversion, transforming the 3D model file into a directed distance field. In this embodiment, to balance image clarity and computational efficiency, the resolution of the directed distance field is generally set to 256x256x256, and its values are ranged between [-0.1, 0.1]. After acquiring the directed distance field data, it is converted to the frequency domain using wavelet transform to obtain a multi-scale frequency domain coefficient set. The specific implementation is as follows: The directed distance field is denoted as S. Wavelet transform can decompose it into high-frequency detail information coefficients and low-frequency coarse information coefficients. The low-frequency coefficients are denoted as Si. By repeating the above process multiple times, high-frequency and low-frequency coefficients at different scales can be obtained.
[0020] To facilitate understanding, this embodiment further illustrates the concept. For example, if the original directed range field data size is 256*256*256, and assuming only one wavelet transform is performed, the original directed range field data size is 256*256*256. After the wavelet transform, the size is 192*192*192 resolution. The low-frequency coefficients corresponding to this wavelet transform are the low-frequency coefficients at the 192*192*192 resolution. The inverse transform at the 192 resolution results in 256, i.e., 256*256*256 - (inverse wavelet transform at 192*192*192). The high-frequency coefficients are the low-frequency coefficients at the 256 resolution. High-frequency coefficients; if further wavelet transform is performed, the low-frequency coefficients of the first transform change from 192 to 76, then the low-frequency coefficient resolution is 76, and the high-frequency coefficients are (the inverse transform of 192-76) at resolution 192; the reconstruction process is as follows: the diffusion model obtains low-frequency coefficients at resolution 76, the 76-resolution predicts high-frequency coefficients at resolution 192, the inverse transform of the high-frequency coefficients at resolution 192 + the 76 low-frequency coefficients yields the low-frequency coefficients at resolution 192 from the first transform, and 192 is used to obtain high-frequency coefficients of 256 through the prediction network, 256SDF = the inverse transform of the 256-size high-frequency + 192 low-frequency. The above is only one calculation process of the diffusion model of the present invention for ease of understanding, and the resolution and transformation method of the present invention are not limited to the above embodiment.
[0021] After the 3D model in this embodiment is converted into directed range field data, it can be converted into frequency domain data according to wavelet transform. The frequency domain data can also be used to regenerate directed range field data according to inverse transform. When the low-frequency and high-frequency information coefficients are sufficient, this embodiment can recover the directed range field data from the low-frequency and high-frequency information coefficients according to the inverse transform of wavelet decomposition.
[0022] The directed distance field of this invention, also known as the signed distance field (SDF), is generally abbreviated as SDF in the field of machine learning. The wavelet transform of this invention is also known in the art as wavelet function, wavelet function, wavelet analysis, or wavelet transform.
[0023] After the above-mentioned transformation of the directed distance field data, refer to Figure 3The denoising diffusion probability model is trained based on the low-frequency coefficients, the corresponding high-frequency coefficients, and the 3D training model, so that the denoising diffusion probability model acquires the probability distribution of low-frequency coefficients in the 3D training model. An attention mechanism is introduced during the training process of the denoising diffusion probability model, using the expected generation information as a control condition to couple the expected generation information with the low-frequency coefficients, so that the denoising diffusion probability model acquires low-frequency coefficients based on the expected generation information. The denoising diffusion probability model is selected as the generator network to learn the low-frequency information distribution of 3D objects, and it generates 3D models by iteratively correcting random noise input. To ensure controllable 3D model generation, control conditions need to be added during the model training process. Generally, an input image or text is selected as the condition. The image is the rendered image of the current 3D model, such as the chair in this embodiment; the text can be a text description of the current 3D object, such as the relevant feature description of the chair in this embodiment, including the backrest, legs, and seat. In the training of the denoising diffusion probability model, an attention mechanism is introduced. The encoded image / text, along with the low-frequency coefficients after adding noise, is used as input. The attention mechanism couples the low-frequency coefficients with the image / text information. This allows the shape of the generated 3D model to be controlled using the image / text. In this embodiment, the attention mechanism is the process of adding control components to the hidden layers during neural network training. The input content can be converted into control information that the neural network can recognize, including images and text descriptions. Images are encoded using a clip model and input to the attention mechanism, while text is encoded using a BERT model. The input conditions can be connected to the pre-trained model and passed as additional input information to the diffusion model.
[0024] In this invention, in order to extract detailed information from the input image, the image can be preprocessed. For example, the image outline can be extracted first, and the outline information can be used as shape information and input together with the image. Alternatively, specific descriptive text can be generated based on the image outline and input together with the image.
[0025] Based on one or more of the above embodiments, combined with Figure 4 The detailed prediction model training process includes: selecting corresponding high-frequency and low-frequency coefficients, using the low-frequency coefficients as input samples and the high-frequency coefficients as the expected output, and training the detailed prediction model so that the detailed prediction model can predict the high-frequency coefficients of the same region based on the low-frequency coefficients. The detailed prediction network is trained to predict high-frequency coefficients using pairs of low-frequency and high-frequency coefficients; inputting low-frequency coefficients, it predicts high-frequency coefficients. In this way, low-frequency coefficients can be obtained through a diffusion probability model, and then used as input to obtain high-frequency coefficients through the detailed prediction network. The directed distance field can be recovered from the low-frequency and high-frequency coefficients, and further, a 3D mesh can be obtained.
[0026] Based on one or more of the above embodiments, the target 3D model generation method includes: using the low-frequency coefficients obtained by the denoising diffusion probability model as coarse feature parameters of the target 3D model, using the corresponding high-frequency coefficients obtained by the detail prediction model as detail feature parameters of the target 3D model, performing inverse wavelet transform on the low-frequency coefficients and the corresponding high-frequency coefficients to obtain directed distance field data of the target 3D model, and then generating a mesh file of the target 3D model. The generation process in the above steps is as follows: first, a 3D model image or text description to be generated is provided, and it is fed into the denoising diffusion probability model along with random noise. Low-frequency coefficients are recovered through iterative denoising. High-frequency coefficients are recovered from the low-frequency coefficients through the detail prediction model. Further, the directed distance field is recovered from the low-frequency and high-frequency coefficients through inverse wavelet transform. Then, the directed distance field can be converted into a 3D mesh.
[0027] A storage medium storing a program for generating a three-dimensional model, the program being executed by a processor to perform the steps of the method as described in any of the above embodiments.
[0028] An electronic device includes: a memory storing a computer program thereon; and a processor for executing the computer program in the memory to implement the steps of the method described in any of the above embodiments.
[0029] The scope of protection of this invention is not limited thereto. Any variations or substitutions of technical solutions that can be conceived without inventive effort should be included within the scope of protection of this invention. The scope of protection of this invention is determined by the claims.
Claims
1. A method for generating a three-dimensional model, characterized in that, include: A denoising diffusion probability model for coupling the input expected generation information into coarse features of the 3D model and a detail prediction model for predicting the detailed features of the 3D model are trained separately. Input the desired generation information into the denoising diffusion probability model to obtain the coarse feature parameters of the target 3D model; Input the coarse feature parameters into the detail prediction model to obtain the detail feature parameters of the target 3D model; Generate the target 3D model based on the coarse feature parameters and the fine feature parameters.
2. The three-dimensional model generation method according to claim 1, characterized in that, The desired information to be generated includes text and / or image information.
3. The three-dimensional model generation method according to claim 1, characterized in that, The training method for the denoising diffusion probability model includes: Obtain a 3D training model and convert the 3D training model into directed distance field data; The directed distance field data is subjected to continuous wavelet transform to generate frequency domain data that expresses the shape features of the 3D training model at multiple scales. Based on the frequency domain data, obtain the high-frequency coefficients that express the shape detail features of the 3D training model at different scales and the low-frequency coefficients that express the shape coarseness features of the 3D training model; Based on the low-frequency coefficients, the corresponding high-frequency coefficients, and the 3D training model, the denoising diffusion probability model is trained so that the denoising diffusion probability model can obtain the probability distribution of the low-frequency coefficients in the 3D training model. An attention mechanism is introduced during the training of the denoising diffusion probability model. The expected generation information is used as a control condition to couple the expected generation information with the corresponding low-frequency coefficients, so that the denoising diffusion probability model can obtain the low-frequency coefficients according to the expected generation information.
4. The three-dimensional model generation method according to claim 3, characterized in that, The desired information to be generated is a rendered image of the 3D training model and / or a textual description of the 3D training model.
5. The three-dimensional model generation method according to claim 3, characterized in that, The resolution of the directed distance field data is 256x256x256, and its value ranges from [-0.1, 0.1].
6. The three-dimensional model generation method according to claim 3, characterized in that, The training process for the detailed prediction model includes: Select the corresponding high-frequency coefficients and low-frequency coefficients, use the low-frequency coefficients as input samples and the high-frequency coefficients as expected outputs, and train the detail prediction model so that the detail prediction model can predict the high-frequency coefficients of the same region based on the low-frequency coefficients.
7. The three-dimensional model generation method according to claim 6, characterized in that, The method for generating the target 3D model includes: The low-frequency coefficients obtained from the denoising diffusion probability model are used as coarse feature parameters of the target 3D model, and the corresponding high-frequency coefficients obtained from the detail prediction model are used as detail feature parameters of the target 3D model. Wavelet inverse transform is performed on the low-frequency coefficients and the corresponding high-frequency coefficients to obtain the directed distance field data of the target 3D model, thereby generating the mesh file of the target 3D model.
8. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-7.
9. A storage medium, characterized in that, The storage medium stores a program for a three-dimensional model generation method, which is executed by a processor to implement the steps of the three-dimensional model generation method as described in any one of claims 1 to 7.