A modular 3D object generation method for innovative product concept design

Abstract

The present invention discloses a combined three-dimensional object generation method for innovative product concept design. The method uses the target prompt word noise component to accurately express the overall information of the three-dimensional object and the positional relationship between objects. The present invention uses the text prompt word list noise component of each object to accurately express the detailed information of each object. Based on the above conditional noise component, the parameters of the three-dimensional model are updated using the fractional distillation sampling loss function to obtain a three-dimensional model that can accurately express complex objects and their relationships. Finally, it is converted into a three-dimensional model in a polygonal mesh format, which is compatible with standard 3D modeling software.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology and three-dimensional object generation, and in particular relates to a combined three-dimensional object generation method for product concept innovative design. Background Art

[0002] In the field of product concept innovation design, the traditional 3D modeling method has high requirements for designers, not only needing to master complex modeling commands, but also needing to perform tedious modeling operations. This not only raises the entry threshold of design, but also prolongs the cycle from product conceptualization to prototype and then to finished product realization.

[0003] However, in recent years, text-guided 3D object generation technology has brought a new solution for rapid prototyping. The progress of this technology is largely due to the rapid development of text-guided 2D image generation models. Existing methods optimize multiple randomly sampled 2D views of 3D objects by leveraging the knowledge accumulated by large-scale pre-trained text-guided image generation models to ensure that these views are consistent with the description of the input text.

[0004] The invention patent application with publication number CN116778061A discloses a method for generating three-dimensional objects based on non-realistic images. The method obtains the probability distribution of the generated image through a pre-trained diffusion model based on text prompt words and depth maps, and uses the KL divergence of the probability distribution of the generated image and the probability distribution of the target image to perform a loss function to update the parameters of the neural radiation field, thereby making the three-dimensional geometric model generated by the updated neural radiation field more accurate without relying on the depth estimator. The present invention will suppress the density of non-subject objects outside the semantic mask through a floating artifact loss function, that is, a loss function about the density map and the subject semantic mask, thereby eliminating the generation of floating artifacts, and encouraging the density growth within the semantic mask to form a more accurate three-dimensional geometric model.

[0005] The invention patent application with publication number CN117391934A discloses an image processing method, device, equipment and storage medium, which relates to the field of computer technology, including: obtaining an original face image, and extracting descriptive prompt words of the original face image; integrating the descriptive prompt words with user-defined prompt words to obtain image generation prompt words; generating an auxiliary face image based on the image generation prompt words through a portrait generation model; and constructing a three-dimensional face image based on the original face image and the auxiliary face image. The three-dimensional face image reconstructed by the embodiment of the present application has a high similarity with the user image, satisfying the user's pursuit of a beautiful image when using a digital avatar in the virtual world.

[0006] The above two patents and current text-guided 3D object generation technologies still face significant challenges in constructing 3D objects containing multiple objects and their complex semantics. This article refers to this task as combined 3D object generation. The main problem is that the existing technology relies too much on the optimization of a shared 3D representation, which is insufficient in the interaction between multiple objects and their precise positioning in space. This deficiency leads to defects in the generated 3D objects, which are manifested in the mismatch between objects and their attributes, incorrect relationships between multiple objects, and inaccurate representation of complex objects. Summary of the invention

[0007] The present invention provides a combined three-dimensional object generation method for innovative product concept design, which can accurately generate corresponding three-dimensional objects based on user text input.

[0008] The present invention provides a combined three-dimensional object generation method for product concept innovative design, comprising:

[0009] The target prompt word input by the user is split into a text prompt word list of multiple objects through a large language model. The text prompt word list of each object includes an object prompt word and an object prompt word with attributes. At the same time, a 3D bounding box corresponding to each object is generated, and the 3D bounding box is rendered to obtain a multi-view mask;

[0010] Input the target prompt word, the text prompt word list, the multi-view mask and the image with noise added after multi-view rendering of the three-dimensional model into the diffusion model to obtain the predicted noise, wherein the predicted noise is constructed by the unconditional noise, the target prompt word noise component and the text prompt word list noise component, wherein the target prompt word noise component is the difference between the target prompt word noise and the unconditional noise, and the text prompt word list noise component is obtained by the following steps: multiplying the difference between the noise under the text prompt word list condition of each object and the unconditional noise by the corresponding multi-view mask, and accumulating the calculation results of each object to obtain the text prompt word list noise component;

[0011] According to the predicted noise and the real noise, the fractional distillation sampling loss function is used to update the parameters of the three-dimensional model, the updated three-dimensional model is converted into a three-dimensional model compatible with the format of the 3D modeling software, and the format-compatible three-dimensional model is imported into the 3D modeling software to obtain the corresponding combined three-dimensional object.

[0012] Preferably, during each iteration of optimizing the three-dimensional model, the diffusion model Output target prompt word noise component for: ,in, The 3D model is rendered and noisy at a specific perspective. t The image obtained after the step ist represents the time step in the denoising process, For the target cue, Diffusion Model In the conditions The target prompt word conditional noise output is: Diffusion Model The unconditional prediction noise.

[0013] Preferably, the method for obtaining the target prompt word conditional noise includes:

[0014] The diffusion model includes an encoder, which encodes the image of the three-dimensional model after multi-view rendering and adding noise, and extracts the noise component of the target prompt word through the noise sampler based on the target prompt word.

[0015] Preferably, during each iteration of optimizing the three-dimensional model, the diffusion model Noise component of the output text prompt word list : ,in, It is the image obtained after rendering and adding noise to the 3D model at a specific viewing angle. t represents the time step in the denoising process, For the n A list of text prompt words for each object. Diffusion Model Output n The text prompt word list conditional noise for each object, Diffusion Model The unconditional noise at the output, N is the number of text prompt word lists of the object, For the n Multi-view masks corresponding to each object.

[0016] Preferably, the diffusion model Output n Conditional noise of text prompt word list for: ,in, is the scaling factor of the object attribute condition, and the text prompt word list of each object , containing object prompt words and prompt words for objects and their attributes , the text prompt word list conditional noise for each object Represented as object cue word conditional noise and the cue words of the object and its attributes A linear combination of .

[0017] Preferably, the parameters of the three-dimensional model are updated using a fractional distillation sampling loss function based on the predicted noise and the real noise at each iteration, wherein the fractional distillation sampling loss function is for: ,in, For expectations, is the real noise, are the parameters of the three-dimensional model, To control the weight of the parameter, Diffusion model for optimization iteration Rendering and noisy images based on 3D models , a list of text prompt words for the object and target cue words The output prediction noise is specifically defined as: ,in, is the noise component of the target cue word, is the noise component of the text prompt word list, Diffusion Model The unconditional noise at the output, is the original guidance parameter in Classifier-Free Guidance.

[0018] Preferably, at each iteration, a fractional distillation sampling loss function is used to update the parameters of the three-dimensional model based on the predicted noise and the real noise, and the three-dimensional model is a 3D neural radiation field.

[0019] Preferably, the updated three-dimensional model is converted into a three-dimensional model compatible with the format of the 3D modeling software by using a Marching Cube algorithm.

[0020] Preferably, the diffusion model comprises a noise sampler, through which the predicted noise is obtained by rendering the noisy image based on the input target prompt word, the text prompt word list of each object, the multi-view mask and the three-dimensional model.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The present invention utilizes the target prompt word noise component to accurately express the overall information of the three-dimensional object and the positional relationship between the objects. The present invention utilizes the text prompt word list noise component of each object to accurately express the detailed information of each object. The predicted noise constructed based on the unconditional noise, the target prompt word noise component and the text prompt word list noise component adopts the fractional distillation sampling loss function to update the parameters of the three-dimensional model to obtain a three-dimensional model that can accurately express complex objects and their relationships, and then converts it into a three-dimensional model that is compatible with the standard 3D modeling software format. The format-compatible three-dimensional model is imported into the 3D modeling software to more accurately obtain the corresponding combined three-dimensional object. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flowchart of a combined three-dimensional object generation method for product concept innovative design provided by a specific embodiment of the present invention;

[0024] Figure 2 A flowchart of a combined three-dimensional object generation method for innovative product concept design provided in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to solve the problems of mismatch between generated objects and their attributes, incorrect relationships between multiple objects, and inaccurate expression of complex objects, a specific embodiment of the present invention proposes a combined three-dimensional object generation method for product concept innovation design. This method can quickly generate combined three-dimensional objects based on text descriptions, such as Figure 1 and Figure 2 As shown, the specific steps are as follows:

[0026] S1. Construct a text prompt word list, a three-dimensional bounding box and a corresponding multi-view mask through a large language model based on the user's text description: The specific embodiment of the present invention uses large language models (LLM) to understand the user's text input, divide it into a text prompt word list of multiple objects, and generate corresponding three-dimensional bounding boxes. Specifically, the large language model will analyze the text input by the user according to the generation examples provided in the context prompt words (Context Prompt), and divide it into multiple shorter prompt words to obtain a text prompt word list. The list contains the prompt words for each object and the prompt words that describe the object attributes. Then, the large language model generates the three-dimensional coordinate values ​​and three-dimensional scale scaling size values ​​of each object in the three-dimensional space according to the coordinate format provided by the context prompt words. Then, by using the existing three-dimensional data processing tool library, the obtained six-dimensional parameters are converted into a three-dimensional bounding box (3D bounding box), and the three-dimensional bounding box is rendered to obtain a multi-view mask. In a specific embodiment, if Figure 2 As shown, the multi-view mask M n The masks M1 and M2 corresponding to the two objects are included as input to the subsequent generation model.

[0027] A specific embodiment of the present invention uses a large language model to split the target prompt word input by the user into text prompt word lists of multiple objects. The text prompt word list of each object includes object prompt words and object prompt words with attributes. At the same time, a three-dimensional bounding box corresponding to each object is generated, and the three-dimensional bounding box is rendered to obtain a multi-view mask.

[0028] Table 1 Specific examples of text segmentation and 3D bounding box generation based on large language model

[0029]

[0030] S2. Predicted noise is obtained through a diffusion model based on target prompt words, a text prompt word list, a multi-view mask, and a noisy image after multi-view rendering of a three-dimensional model. The three-dimensional model is updated based on a loss function constructed based on the predicted noise and the real noise. The updated three-dimensional model is formatted and the format-converted three-dimensional model is input into a 3D modeling software to obtain a corresponding three-dimensional object.

[0031] The specific embodiment of the present invention proposes a process based on Score Distillation Sampling (SDS), which uses a text prompt word list and a three-dimensional bounding box as conditions to achieve combined three-dimensional object generation. Specifically, unlike the prior art, the specific embodiment of the present invention first inputs the text prompt word list of each object obtained by the aforementioned method into the diffusion model as a condition, and each prompt word in the text prompt word list will participate in the noise estimation. In the denoising process, the noise estimation is constrained by the multi-view mask obtained by rendering the three-dimensional bounding box. Finally, based on the process of Score Distillation Sampling (SDS), the three-dimensional model is optimized using the prior predicted noise of the diffusion model to generate a three-dimensional model with appearance texture and fine details.

[0032] The specific embodiment of the present invention provides the following steps of inputting the target prompt word, the text prompt word list of each object, the multi-view mask and the rendered noisy image of the three-dimensional model into the diffusion model to obtain the predicted noise, and optimizing the current three-dimensional model:

[0033] The diffusion model provided by the specific embodiment of the present invention is Stable Diffusion, which includes a text encoder, a denoising network, a variational autoencoder, and a noise sampler. First, the model uses a pre-trained text encoder (such as the text encoder of CLIP) to convert the input target prompt word and text prompt word list into an embedding vector in the latent space, and then the target prompt word, the text prompt word list, the multi-view mask and the image of the three-dimensional model after multi-view rendering and noisy are input into the denoising network together. The U-Net receives the noisy input at each time step, and combines the text embedding, predicts the noise distribution of the current time step through the variational autoencoder and the noise sampler, and obtains the predicted noise through iterations of multiple time steps.

[0034] The prediction noise provided by the specific embodiment of the present invention is constructed by unconditional noise, a target prompt word noise component and a text prompt word list noise component of the object, wherein the target prompt word noise component is the difference between the target prompt word conditional noise and the unconditional noise, and the text prompt word list noise component is obtained by the following steps: for each object, the difference between the noise of the object under the text prompt word list condition and the unconditional noise is calculated, and then multiplied by the corresponding multi-view mask, and the calculation results of each object are accumulated to obtain the text prompt word list noise component.

[0035] The specific embodiment of the present invention updates the parameters of the 3D model based on the fractional distillation sampling (SDS) method. The method initializes a 3D model. In a specific embodiment, the 3D model is a neural scene representation model, and uses a pre-trained text-to-image diffusion model as a priori to optimize the 3D scene parameters to achieve 3D model synthesis based on text description. Specifically, the scene model is a parameterized function , which can generate images at a given camera pose .here is a differentiable volume renderer, Represents the parameters of a 3D scene. Diffusion model Through the score function Given a 3D model, rendered and denoised at a specific viewing angle t The image obtained after the time step , text prompt words and noise level This score function guides the update of scene parameters The gradient direction is calculated by fractional distillation sampling (SDS).

[0036] Different from the prior art, the specific embodiment of the present invention introduces a combined 3D object generation model of a text prompt word list and a 3D bounding box to solve the problem that when the prompt word contains multiple objects and each object has a complex description, the generated result may have mismatched attributes with the object or missing objects. Specifically, the user enters the target prompt word , the large language model divides it into Text prompt word list ,in for contains the number of objects, and the text prompt word list of each object contains the object prompt word and prompt words for objects and their attributes , defined as , a noise sampler using a diffusion model generates two separate denoised components.

[0037] The specific embodiment of the present invention provides a diffusion model when optimizing the iterative three-dimensional model. Output target prompt word noise component for: ,in, The 3D model is rendered and noisy at a specific perspective. t The image obtained after the time step, t represents the time step in the denoising process, For the target cue, Diffusion Model In the conditions The prediction noise under Diffusion Model The unconditional prediction noise and the conditional noise component of the target prompt word can accurately reflect the overall information of the three-dimensional object and the positional relationship between objects.

[0038] The specific embodiment of the present invention provides a diffusion model when optimizing the iterative three-dimensional model. Output of the noise component of the text prompt word list for each object for: in, The 3D model is rendered and noisy at a specific perspective. t The image obtained after the time step, t represents the time step in the denoising process, For the n A list of text prompt words for each object. Diffusion Model Output n The text prompt word list conditional noise for each object, Diffusion Model The unconditional noise at the output, N is the number of text prompt word lists of the object, For the n The multi-view mask corresponding to each object is used to generate a corresponding mask at a specified three-dimensional bounding box to prevent the denoised image from overlapping. The conditional noise component of the prompt word list provided in the specific embodiment of the present invention can accurately reflect the detailed information of the object.

[0039] When optimizing the iterative three-dimensional model provided by the specific embodiment of the present invention, the diffusion model Output of each object's text prompt word list conditional noise It can be expressed as object cue word conditional noise and the cue words of the object and its attributes The linear combination of is as follows: in, Represents the scaling factor of the object attribute condition, which is used to control the degree of influence of the attribute on the object.

[0040] In a specific embodiment of the present invention, a fractional distillation sampling loss function is used to update the parameters of the three-dimensional model based on the predicted noise and the real noise at each iteration. After reaching the number of iterations, an optimized three-dimensional model is obtained. A three-dimensional model in a triangular mesh format is generated through algorithms such as MarchingCubes, which is compatible with standard 3D modeling software.

[0041] The specific embodiment of the present invention provides that the parameters of the three-dimensional model are updated by using a fractional distillation sampling loss function based on the predicted noise and the real noise at each iteration, wherein the fractional distillation sampling loss function for: in, For expectations, is the real noise, are the parameters of the three-dimensional model, To control the weight of the parameter, Diffusion model for optimization iteration Rendering and noisy images based on 3D models , a list of text prompt words for the object and target cue words The output prediction noise is specifically defined as: ,in, is the noise component of the target cue word, is the noise component of the text prompt word list, Diffusion Model The unconditional noise at the output, It is the original guidance parameter in Classifier-Free Guidance (CFG), which is used to control the overall 3D model information and the detailed attribute information of each object.

[0042] A specific embodiment of the present invention converts the optimized 3D neural radiation field into a three-dimensional model in a polygonal mesh format through a Marching Cube algorithm, which is compatible with standard 3D modeling software, and imports the format-compatible three-dimensional model into the 3D modeling software to obtain a corresponding combined three-dimensional object.

Claims

1. A combined three-dimensional object generation method for product concept innovation design, characterized in that: include: The target prompt word input by the user is split into a text prompt word list of multiple objects through a large language model. The text prompt word list of each object includes an object prompt word and an object prompt word with attributes. At the same time, a 3D bounding box corresponding to each object is generated, and the 3D bounding box is rendered to obtain a multi-view mask; Input the target prompt word, the text prompt word list, the multi-view mask and the image with noise added after the multi-view rendering of the three-dimensional model into the diffusion model to obtain the predicted noise, wherein the predicted noise is constructed by the unconditional noise, the target prompt word noise component and the text prompt word list noise component, wherein the target prompt word noise component is the difference between the target prompt word conditional noise and the unconditional noise, and the text prompt word list noise component is obtained by the following steps: multiplying the difference between the text prompt word list conditional noise and the unconditional noise of each object by the corresponding multi-view mask, and accumulating the calculation results of each object to obtain the text prompt word list noise component; Using a fractional distillation sampling loss function to update the parameters of the 3D model according to the predicted noise and the real noise, converting the updated 3D model into a 3D model compatible with the format of the 3D modeling software, and importing the 3D model compatible with the format into the 3D modeling software to obtain the corresponding combined 3D object; During each iteration of optimizing the 3D model, the diffusion model Output target prompt word noise component for: ,in, The 3D model is rendered and noised at a specific perspective. t The image obtained after the step is t represents the time step in the denoising process, The target cue word, Diffusion Model In the conditions The target prompt word conditional noise output is: Diffusion Model Unconditional noise; During each iteration of optimizing the 3D model, the diffusion model Noise component of the output text prompt word list : ,in, It is the image obtained after rendering and adding noise to the 3D model at a specific viewing angle. t represents the time step in the denoising process, For the n A list of text prompt words for each object. Diffusion Model Output n The text prompt word list conditional noise for each object, Diffusion Model The unconditional noise at the output, N is the number of text prompt word lists of the object, For the n Multi-view masks corresponding to each object.

2. The combined three-dimensional object generation method for product concept innovation design according to claim 1 is characterized in that: The method for obtaining the target prompt word conditional noise comprises: The diffusion model includes an encoder, which encodes the image of the three-dimensional model after multi-view rendering and adding noise, and extracts the noise component of the target prompt word through a noise sampler based on the target prompt word.

3. The combined three-dimensional object generation method for product concept innovative design according to claim 1, characterized in that: Diffusion Model Output n Conditional noise of text prompt word list for: ,in, is the scaling factor of the object attribute condition, and a list of text prompt words for each object , containing object prompt words and prompt words for objects and their attributes , the text prompt word list conditional noise for each object Represented as object cue word conditional noise and the cue words of the object and its attributes A linear combination of .

4. The combined three-dimensional object generation method for product concept innovation design according to claim 1, characterized in that: At each iteration, the parameters of the 3D model are updated using the fractional distillation sampling loss function based on the predicted noise and the real noise. for: ,in, For expectations, is the real noise, are the parameters of the three-dimensional model, To control the weight of the parameter, Diffusion model for optimization iteration Rendering and noisy images based on 3D models , a list of text prompt words for the object and target cue words The output prediction noise is specifically defined as: ,in, is the noise component of the target cue word, is the noise component of the text prompt word list, Diffusion Model The unconditional noise at the output, is the original guidance parameter in Classifier-Free Guidance.

5. The combined three-dimensional object generation method for product concept innovative design according to claim 1, characterized in that: At each iteration, the parameters of the three-dimensional model are updated based on the predicted noise and the real noise using a fractional distillation sampling loss function, where the three-dimensional model is a 3D neural radiation field.

6. The combined three-dimensional object generation method for product concept innovative design according to claim 1, characterized in that: The updated 3D model is converted into a 3D model compatible with the 3D modeling software format through the Marching Cube algorithm.

7. The combined three-dimensional object generation method for product concept innovative design according to claim 1, characterized in that: The diffusion model includes a noise sampler, through which the noise sampler renders a noisy image based on an input target prompt word, a text prompt word list for each object, a multi-view mask and a three-dimensional model to obtain predicted noise.

Images