Text-to-3D content generation method aligned with human preferences

Abstract

The invention relates to a text-to-3D content generation method aligned with human preference, which comprises the following steps of: performing preference comparison feedback training of a direct three-dimensional preference optimization (D-3DPO) algorithm on a constructed multi-view diffusion model-based generation framework (DreamAlign) on the basis of a text-to-3D data set (HP3D) containing expert preference annotations in an offline stage; and in the online stage, the trained generation framework is adopted to generate 3D content. According to the method, the 3D content highly consistent with the input text can be generated in the whole stage from text to 3D generation, the satisfaction degree and the acceptability of a user to the 3D content are improved, the problem that the 3D content is not matched with human aesthetic preferences in the prior art is better solved, and therefore the method has higher practical value in practical application.

Description

Technical Field

[0001] The present invention relates to a technology in the field of image processing, and in particular to a method for generating 3D content from text aligned with human preferences. Background Art

[0002] 3D asset creation is widely used in many fields such as games, animation, simulation environments, chatbots, and art. Despite the numerous demands, creating high-quality 3D content requires artistic creativity and professional 3D modeling skills. Some recent studies have leveraged successful generative models and massive 3D datasets to efficiently generate 3D content. Optimization-based 2D lifting methods have attracted much attention due to their ability to create 3D assets based on text clues.

[0003] Although existing models are able to generate highly complex, view-consistent 3D content, it is difficult for these models to generate 3D content that is consistent with human preferences. The consistency here includes not only multi-viewpoint consistency and 3D text consistency, but also the consistency of attributes with human preferences, such as style, shading, geometry, and appearance. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a method for generating text-to-3D content aligned with human preferences. During the entire stage of text-to-3D generation, not only can 3D content that is highly consistent with the input text be generated, but also the generated 3D content can be ensured to conform to human aesthetic tendencies in terms of multiple key attributes, so that the ultimately generated 3D content not only meets the technical requirements, but also better fits human vision and aesthetic habits, thereby improving user satisfaction and acceptance of 3D content, and better solving the problem of mismatch with human aesthetic preferences in the prior art, thereby having higher practical value in actual applications.

[0005] The present invention is achieved through the following technical solutions:

[0006] The present invention relates to a method for generating text-to-3D content aligned with human preferences. In the offline stage, a construction of a generation framework (DreamAlign) based on a multi-view diffusion model is trained with preference contrast feedback using a direct three-dimensional preference optimization (D-3DPO) algorithm based on a text-to-3D dataset (HP3D) containing expert preference annotations. In the online stage, the trained DreamAlign is used to generate 3D content represented by an implicit neural network based on a fractional distillation method.

[0007] The text-to-3D dataset is a text-to-3D object dataset annotated with human preferences, which alleviates the limitation of insufficient data in human preference learning. Each sample in the dataset includes a text description and multiple groups of multi-view images, and each multi-view image is annotated with the degree of human preference.

[0008] The multi-view diffusion model generates a diffusion model of multiple orthogonal perspective images according to the input text prompt and perspective coordinates, and its framework includes: a text encoder, a U-Net module, and an image autoencoder, wherein: the text encoder encodes the input text information into a feature matrix, the U-Net module denoises the input latent space noise features according to the text feature matrix and perspective information, and the image autoencoder decodes the latent space noise features into multi-perspective images.

[0009] The generation framework based on the multi-view diffusion model generates a diffusion model of multiple orthogonal view images that conform to human aesthetics according to text. A Lora adapter is added to the U-Net module of the multi-view diffusion model to learn human aesthetic preferences.

[0010] The fractional distillation method described converts text into 3D content and uses a pre-trained image generation model to guide the training of another model.

[0011] The implicit neural network records the information of the 3D object through the parameters of the neural network and generates a 2D image of the 3D object at the specified viewing angle according to the viewing angle information, thereby representing the 3D content.

[0012] The direct three-dimensional preference optimization (D-3DPO) algorithm specifically includes:

[0013] Step 1: Sample a sample from HP3D, including a prompt text D, camera view c, and two sets of multi-view images {I w ,I l}, where: I w Than I l More in line with human aesthetics.

[0014] Step 2: Add Gaussian noise ∈ to the two sets of multi-view images to obtain images I that meet human aesthetic standards. w Input to DreamAlign, otherwise the picture that does not conform to human aesthetics I l Input to the original diffusion model.

[0015] Step 3, use L D-3DPO Fine-tune the model to make DreamAlign consistent with the data distribution that conforms to human aesthetics and away from the data distribution that does not conform to human aesthetics. Specifically: in: Indicates that the parameter is The noise of DreamAlign prediction, ∈ ref represents the noise predicted by the original diffusion model.

[0016] The preference contrast feedback training specifically includes:

[0017] Step i, sample the 3D content represented by the implicit neural network with parameter θ to obtain a multi-view image x 0 , after adding noise ∈, it is input into DreamAlign and the original multi-view diffusion model.

[0018] Step ii, based on the idea of ​​contrastive learning, we expect the sampled multi-view images to be close to the images reconstructed by DreamAlign and far away from the images reconstructed by the original diffusion model. PCL Conduct training, specifically: Technical Effects

[0019] The present invention solves the technical problem in the existing text-to-3D content generation that the generated 3D content is inconsistent with human preferences in multiple attributes such as text, shadow, shape, etc. due to the complexity and noise of the data set samples and the algorithm's neglect of human preference information. Compared with the prior art, the present invention introduces the direct three-dimensional preference optimization (D-3DPO) algorithm to fine-tune DreamAlign on the HP3D data set to ensure that the images generated from multiple perspectives not only have excellent performance in a single perspective, but also maintain high consistency and beauty between perspectives, thereby generating more coherent and realistic 3D content. In addition, the introduced preference comparison feedback training mechanism further enhances the visual appeal and detail performance of 3D content by optimizing the distribution of generated content. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of the present invention;

[0021] Figure 2 A schematic diagram of making an HP3D data set in an embodiment;

[0022] Figure 3 Schematic diagram of DreamAlign-2D of an embodiment;

[0023] Figure 4 Schematic diagram of DreamAlign-3D of an embodiment;

[0024] Figure 5 It is a schematic diagram of the effect of the present invention. DETAILED DESCRIPTION

[0025] like Figure 1 As shown, this embodiment relates to a method for generating 3D content from text aligned with human preferences, including:

[0026] Step 1: Figure 2 As shown, construct the HP3D data set, specifically including:

[0027] 1.1, Raw data filtering: To ensure the diversity of the selected 3D objects, we used the K-center algorithm and CLIP features to select 10,000 text-3D object pairs from the Cap3D dataset, and manually screened 376 high-quality objects that highly matched the text descriptions.

[0028] 1.2, Rendering Processing: The 3D objects obtained by the screening are densely rendered, and 32 multi-view images with a resolution of 512x512 are generated for each object. During the 360-degree orbital motion of the camera, the azimuth angle starts from the frontal view (90°), and the elevation angle is randomly selected from the range of 0 to 30 degrees, while the camera extrinsic parameters corresponding to each view are saved.

[0029] 1.3. Generate comparison samples: After using an advanced multimodal model (LLava) to generate text descriptions with the same semantics but different attributes, a multi-view diffusion model is used to generate 3D content with obvious preferences and easy to distinguish based on the text descriptions. Specifically, based on a single 3D object, multi-view images and their corresponding text descriptions are obtained after intensive rendering. LLava is used to generate 2 to 3 text descriptions with different attributes but the same semantics in terms of appearance, geometry and style based on the text descriptions. Based on these text descriptions with the same semantics but different attributes, a multi-view diffusion model is finally used to generate multi-view images with obvious preference differences to facilitate subsequent expert evaluation.

[0030] 1.4, Preference sorting: from the consistency of text and 3D objects, multi-view Figure 1 The 3D content is scored in terms of consistency and aesthetics, ranging from 0 to 7. The multi-view images are sorted according to these scores, and finally a HP3D dataset with annotated preferences is obtained.

[0031] Step 2: Construct a generation framework (DreamAlign) for a multi-view diffusion model: Introduce the LoRA adapter into the U-Net module of the diffusion model. During the preference optimization process, only the parameters of the LoRA adapter are trained, and other parameters are frozen. With a relatively low training cost, DreamAlign can generate multi-view images that are consistent with human preferences.

[0032] Step 3: Figure 3 As shown, based on the HP3D dataset obtained in step 1, direct three-dimensional preference optimization (D- The 3DPO algorithm trains the DreamAlign established in step 2, specifically including:

[0033] 3.1 Randomly select a dataset from the HP3D dataset that includes a text description D, a camera view c, and two corresponding sets of multi-view images I with preference sorting. w ,I l The sample {I w ,I l ,D,c}.

[0034] 3.2 Add random Gaussian noise ε to the two sets of multi-view images.

[0035] 3.3 The noise samples, camera view and text prompts are input into DreamAlign and the multi-view diffusion model respectively, and the predicted noise is and ∈ ref , and then calculate the loss function, take the minimum loss function as the optimization goal, and optimize the network parameters, specifically: Where: β is a hyperparameter used for regularization, represents the signal-to-noise ratio, ω(λ t ) is a predefined weighting function.

[0036] 3.4 Loop through step 3.3 on two V100 GPUs, with a total batch_size of 256, a local batch_size of 1, a gradient accumulation number of 128, and a learning rate of 1e-5. For the D-3DPO algorithm, β is set to 5000.

[0037] Step 4: Figure 4 As shown in the figure, based on DreamAlign obtained in step 3, preference contrast feedback learning is used to realize the generation of 3D content from text to human aesthetics, including:

[0038] 4.1 Randomly sample orthogonal camera perspectives from the 3D content represented by the implicit neural network, then obtain orthogonal multi-view images and add random Gaussian noise ∈.

[0039] 4.2 Input the image after noise addition in step 4.2 into DreamAlign and the multi-view diffusion model used as a reference respectively to perform noise prediction and generate a new image.

[0040] 4.3 Based on the DreamAlign obtained in step 4.2, which can generate images that meet human aesthetic standards, the fractional distillation method is used to generate text to 3D content. In order to further improve the aesthetics of the generated results, the preference contrast loss function is introduced, and the implicit neural network parameters are optimized with the minimum loss function as the optimization goal, specifically: Where: x=g(θ) represents the image x sampled from the 3D implicit neural network representation g with parameter θ.

[0041] Step 5: In the online stage, the text is converted to 3D content through step 4 to obtain a similar Figure 5 The output result of . CLIP and ImageReward are further used as quantitative indicators to evaluate the consistency of 3D content text and the aesthetics of 3D content. CLIP is used to extract text and image features and calculate cosine similarity to evaluate the consistency of 3D content and text, and the consistency of multi-view, and the scores are 35.85 and 78.29 respectively. The scalar result output by ImageReward is used to evaluate the aesthetics of 3D content, and the score is 0.29. This shows that the 3D content generated by this method can better meet user preferences in terms of vision and aesthetics.

[0042] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A method for generating text-to-3D content aligned with human preferences, characterized in that In the offline stage, the constructed generation framework (DreamAlign) based on the multi-view diffusion model is trained with preference contrast feedback of the direct 3D preference optimization (D-3DPO) algorithm based on the text-to-3D dataset (HP3D) containing expert preference annotations. In the online stage, the trained DreamAlign is used to generate 3D content represented by an implicit neural network based on the fractional distillation method.

2. The method for generating text to 3D content aligned with human preferences according to claim 1, characterized in that: The text-to-3D dataset is a text-to-3D object dataset annotated with human preferences, which alleviates the limitation of insufficient data in human preference learning. Each sample in the dataset includes a text description and multiple groups of multi-view images, and each multi-view image is annotated with the degree of human preference.

3. The method for generating text to 3D content aligned with human preferences according to claim 1, characterized in that: The multi-view diffusion model generates a diffusion model of multiple orthogonal view images according to the input text prompt and view coordinates, and its framework includes: a text encoder, a U-Net module, and an image autoencoder, wherein: the text encoder encodes the input text information into a feature matrix, the U-Net module denoises the input latent space noise features according to the text feature matrix and view information, and the image autoencoder decodes the latent space noise features into a multi-view picture.

4. The method for generating text to 3D content aligned with human preferences according to claim 1, characterized in that: The generation framework based on the multi-view diffusion model generates a diffusion model of multiple orthogonal view images that conform to human aesthetics according to text. A Lora adapter is added to the U-Net module of the multi-view diffusion model to learn human aesthetic preferences.

5. The method for generating text to 3D content aligned with human preferences according to claim 1, characterized in that: The fractional distillation method described converts text into 3D content and uses a pre-trained image generation model to guide the training of another model.

6. The method for generating text to 3D content aligned with human preferences according to claim 1, characterized in that: The implicit neural network records the information of the 3D object through the parameters of the neural network and generates a 2D image of the 3D object at the specified viewing angle according to the viewing angle information, thereby representing the 3D content.

7. The method for generating text to 3D content aligned with human preferences according to claim 1, characterized in that: The direct three-dimensional preference optimization (D-3DPO) algorithm specifically includes: Step 1: Sample a sample from HP3D, including a prompt text D, camera view c, and two sets of multi-view images {I w ,I l }, where: I w Than I l More in line with human aesthetics; Step 2: Add Gaussian noise ∈ to the two sets of multi-view images to obtain images I that meet human aesthetic standards. w Input to DreamAlign, otherwise the picture that does not conform to human aesthetics I l Input into the original diffusion model; Step 3, use L D-3DPO Fine-tune the model to make DreamAlign consistent with the data distribution that conforms to human aesthetics and away from the data distribution that does not conform to human aesthetics. Specifically: in: represents the noise predicted by DreamAlign with parameter φ, ∈ ref represents the noise predicted by the original diffusion model.

8. The method for generating text to 3D content aligned with human preferences according to claim 1, characterized in that: The preference contrast feedback training specifically includes: Step i, sample the 3D content represented by the implicit neural network with parameter θ to obtain a multi-view image x0, add noise ∈ and input it into DreamAlign and the original multi-view diffusion model; Step ii, based on the idea of ​​contrastive learning, we expect the sampled multi-view images to be close to the images reconstructed by DreamAlign and far away from the images reconstructed by the original diffusion model. PCL Conduct training, specifically:

Images