Rhatier culture 3D virtual digital human AIGC generation method based on zero sample

Abstract

The invention discloses a Rhatish culture 3D virtual digital human AIGC generation method based on zero samples, and particularly relates to the technical field of computer graphics and artificial intelligence. The method comprises the following steps: initializing implicit representation of a three-dimensional human body through a neural network to construct a basic model; a pre-trained depth conditional diffusion model is adopted, text prompts containing Rhatish culture features are received and serve as conditional input, and a neural implicit field is guided through a cross attention mechanism to generate a three-dimensional geometric model conforming to semantic description; and further, de-noising reduction is carried out based on a diffusion model, high-precision texture map reasoning and multi-scale iterative optimization are carried out in combination with texture priori provided by a geometric model, and finally, a three-dimensional virtual digital human with high reality and cultural characteristics is generated. According to the method, the dependence of a traditional method on large-scale data and manual manufacturing is effectively reduced, and the generation efficiency and the model diversity are improved.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics and artificial intelligence, and more specifically, to a method for generating 3D virtual digital humans (AIGC) based on zero-shot data from Luling culture. Background Technology

[0002] Virtual digital human technology is a cutting-edge field that intersects computer graphics and artificial intelligence. Its development has evolved from an early, high-cost, and long-cycle production model relying on manual modeling and skeletal rigging to automated generation based on traditional machine learning methods. In recent years, with breakthroughs in generative artificial intelligence (AIGC) technology, especially the rise of powerful generative models such as diffusion models, text-driven, high-quality, and efficient virtual human generation has become a significant trend in this field. This aims to significantly lower the creative threshold and promote its application in broader scenarios such as cultural heritage preservation and virtual interaction.

[0003] However, existing text-driven virtual human generation methods still have shortcomings. First, mainstream methods heavily rely on large-scale, high-quality 3D human datasets for supervised training, resulting in extremely high data acquisition and annotation costs, and limited model generalization ability, making it difficult to generate unique images outside the training set distribution. Second, existing technologies lack sufficient precision in understanding and controlling text prompts, especially when generating detailed features with specific cultural connotations (such as clothing and patterns from the Luling culture), often resulting in missing features, biases, or semantic inconsistencies. Furthermore, most methods separate geometry generation from texture mapping, leading to mismatches between geometric and texture details, requiring extensive post-processing to reach usable standards, resulting in overall low efficiency.

[0004] Therefore, this paper proposes a zero-shot AIGC-based method for generating 3D virtual digital humans of Luling culture. The aim is to address the following core issues: how to overcome data bottlenecks without relying on large-scale, specific 3D training data; how to improve the model's understanding and execution of detailed textual instructions (especially culturally relevant descriptions) to ensure the accuracy and cultural fit of generated features; and how to optimize the collaborative generation process of geometry and texture to ensure the final model has high realism and visual consistency, while reducing production costs and technical barriers. Summary of the Invention

[0005] In order to overcome the above-mentioned defects of the prior art, the embodiments of the present invention provide a method for generating 3D virtual digital human AIGC based on zero samples of Luling culture, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for generating 3D virtual digital human AIGC based on zero-shot Luling culture, comprising the following steps: S1. Initialize the implicit representation of the 3D human body through a neural network to construct an initial 3D human body model; S2. A pre-trained deep conditional diffusion model is used to receive text prompts as conditional input, and conditional guidance and diffusion optimization are performed on the neural implicit field corresponding to the initial three-dimensional human body model to generate a three-dimensional virtual digital human geometric model that conforms to the text description. S3. Perform the denoising and restoration process based on the diffusion model, and combine the texture prior information from the geometric model of the three-dimensional virtual digital human to perform high-precision texture map reasoning and updating; S4. Optimize the texture representation through iterative refinement to generate a final 3D virtual digital human model with highly realistic textures.

[0007] Alternatively, the implicit representation of the 3D human body initialized by the neural network adopts a coordinate-based implicit function representation. This implicit function maps the 3D spatial coordinates to a symbolic distance value or an occupancy probability value through a multilayer perceptron, thereby implicitly expressing the geometry of the 3D human body.

[0008] Alternatively, the deep conditional diffusion model is a noise prediction network based on the U-Net architecture. In the denoising process, it integrates the embedded representation of text prompt information through a cross-attention mechanism to conditionally guide the generation process of the neural implicit field, thereby achieving diversified control over the geometry of the three-dimensional virtual digital human.

[0009] Alternatively, the texture prior information includes geometric features and initial texture features extracted from the geometric model of the three-dimensional virtual digital human. The high-precision texture map inference and update process adopts a texture generator based on a diffusion model, which progressively infers and optimizes diffuse reflection, normals and specular maps in a denoising manner.

[0010] Alternatively, the texture generator employs a multi-scale feature fusion mechanism during inference to concatenate the texture prior information with the intermediate layer features of the diffusion model, thereby enhancing the consistency and realism of texture details.

[0011] Multiple options are available. The iterative refinement process includes alternating the execution of texture diffusion optimization under geometric consistency constraints and geometric fine-tuning based on texture feedback until the geometry and texture of the three-dimensional virtual digital human model both reach a preset quality threshold.

[0012] Alternatively, the text prompt information may include descriptions of specific clothing, decorations, or cultural symbols related to Luling culture, and the deep conditional diffusion model may generate a three-dimensional virtual digital human model with cultural characteristics based on these descriptions.

[0013] Alternatively, the method employs zero-shot learning in both the training and inference phases, eliminating the need for model training or fine-tuning for specific 3D virtual digital humans. New 3D virtual digital human models can be generated solely based on pre-trained models and text input.

[0014] The technical effects and advantages of this invention are as follows: Compared to existing technologies, this invention introduces a zero-shot text-driven generation method based on a diffusion model. It utilizes a pre-trained deep conditional diffusion model to parse text prompts and directly guides the generation of 3D geometry from a neural implicit field, eliminating the need for model training for specific virtual humans or the collection of large amounts of 3D data. This approach achieves high-quality model generation from scratch, reduces reliance on large-scale labeled 3D data, effectively solves the data bottleneck problem, and simultaneously improves the diversity and automation of the generated models.

[0015] Compared to existing technologies, this invention innovatively utilizes prior texture information provided by a preliminary geometric model to guide a high-precision texture diffusion model for iterative refinement by studying a texture inference and update mechanism based on a diffusion model. This method achieves high adaptability and detail enhancement between texture and underlying geometry, overcoming the problem of texture-geometry mismatch in traditional methods. It generates highly realistic and detailed texture maps, reducing the workload of post-processing manual adjustments.

[0016] Compared to existing technologies, this invention embeds Luling cultural characteristics into textual prompts and leverages the powerful semantic understanding and control capabilities of a diffusion model to transform abstract cultural descriptions into concrete visual elements. This approach ensures that the generated virtual digital avatars accurately reflect the unique features of Luling culture in details such as clothing and accessories, effectively improving the accuracy and expressiveness of cultural dissemination and providing a new technological path for the digital inheritance and innovation of cultural heritage. Attached Figure Description

[0017] Figure 1 This is a system framework diagram of the present invention.

[0018] Figure 2 This is a flowchart of the process of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Example 1: As attached Figures 1-2The method for generating 3D virtual digital humans based on zero-shot Luling culture using AIGC is characterized by the following steps: An implicit representation of the three-dimensional human body is constructed using a multilayer perceptron neural network architecture. This network maps three-dimensional spatial coordinates to symbolic distance values, forming an initial three-dimensional human body model. A deep conditional diffusion model based on the U-Net architecture is adopted. This model receives text prompts containing Luling cultural features, converts the prompts into embedding vectors through a text encoder, and uses a cross-attention mechanism to fuse the text embedding vectors with the feature representation of the neural implicit field. During the diffusion process, the neural implicit field is optimized by adding and removing noise through multiple iterations to generate a three-dimensional virtual digital human geometric model that conforms to the text description. The texture generation process is performed based on a denoising diffusion probability model. Geometric features and initial texture features are extracted from the geometric model of the 3D virtual digital human as prior information. A texture generator with a multi-scale convolutional neural network architecture is used for high-precision texture map inference. The generator optimizes the diffuse map, normal map and specular map step by step through multiple denoising steps. The geometric and texture representations are updated alternately through an iterative optimization algorithm. In each iteration, the geometric model is first fixed and the texture map is optimized. Then the texture map is fixed and the geometric details are fine-tuned, ultimately generating a three-dimensional virtual digital human model with highly realistic textures.

[0021] The implicit representation of the 3D human body initialized by the neural network adopts a coordinate-based implicit function representation method. Specifically, it is implemented as follows: a multilayer perceptron network with 8 hidden layers is constructed, each layer contains 256 neurons, and the ReLU activation function is used. The network input is the 3D spatial coordinate value, and the output is the signed distance value of the coordinate point relative to the human body surface. Point sampling is performed in 3D space by combining uniform sampling and importance sampling. The sampling points include surface points and internal and external points. The mathematical expression for this is f(x,y,z)=MLP(γ(x,y,z)); x, y, z: represent the coordinates of a point in three-dimensional space.

[0022] γ: Represents the position encoding function, which maps the original coordinates to a high-dimensional space to better learn high-frequency details.

[0023] MLP stands for Multilayer Perceptron Neural Network.

[0024] f: represents the output function of the neural implicit field, whose output value represents the signed distance from the point (x,y,z) to the model surface (positive for the outside, negative for the inside, and zero for the surface).

[0025] This formula defines an implicit representation of the 3D human body learned through a multilayer perceptron (MLP) neural network. It maps 3D spatial coordinates to a continuous scalar value (such as the signed distance value SDF), thus implicitly describing the geometry of the human body. This is the core mathematical expression for constructing the initial 3D human body model.

[0026] A gradient-based optimization algorithm is used to minimize the difference between the predicted symbol distance value and the true value. During the training process, position encoding is used to enhance the input coordinates at high frequencies, with encoding frequencies ranging from 1 to 10 in 10 frequency bands. After the network training is completed, the moving cube algorithm is used to extract the explicit triangular mesh representation from the learned implicit field, with the mesh resolution set to 512 x 512 x 512.

[0027] The specific implementation of the deep conditional diffusion model includes: using a text encoder based on the Transformer architecture to convert text prompt information into a 768-dimensional embedding vector; constructing a U-Net network with an encoder-decoder structure, wherein the encoder part contains 4 downsampling stages, each stage using 2 residual convolutional blocks and an attention module, and the decoder part symmetrically contains 4 upsampling stages; A cross-attention mechanism is introduced in the intermediate layer of U-Net to fuse text embedding vectors with image features, with the number of attention heads set to 8. The diffusion process adopts a 1000-step linear noise scheduling, gradually adding Gaussian noise to the data during the forward process and gradually denoising through a noise prediction network during the backward process. During training, the mean squared error loss function is used to optimize the noise prediction error, and the Adam optimizer is used for model training with a learning rate set to 0.0001.

[0028] The texture prior information extraction and texture generation process specifically includes: extracting geometric features such as vertex position, normal direction and curvature from the three-dimensional virtual digital human geometric model; using a pre-trained convolutional autoencoder to extract initial texture features, the encoder containing 5 convolutional layers, each followed by batch normalization and LeakyReLU activation function; The texture generator adopts the U-Net architecture, which includes an encoding path and a decoding path. The encoding path extracts features step by step through four downsampling blocks, each of which contains two convolutional layers. The decoding path restores the resolution step by step through four upsampling blocks, each of which uses a transposed convolution to perform a 2x upsampling. Skip connections are added between the encoder and decoder to preserve detail information; the diffusion process employs a 50-step accelerated sampling algorithm, predicting noise residuals and updating texture features in each denoising step.

[0029] The specific implementation of the multi-scale feature fusion mechanism is as follows: a feature pyramid network with four scales is constructed, each scale corresponding to 1 / 2, 1 / 4, 1 / 8 and 1 / 16 of the original resolution; features are extracted using a 3x3 convolutional layer at each scale, and then the low-resolution features are upsampled to high resolution through bilinear upsampling; The attention gate mechanism is used to perform weighted fusion of features at different scales, and the attention weights are learned. After feature fusion, a 1x1 convolutional layer is used for feature recombination. Deformable convolution is used in the multi-scale feature extraction process to adapt to texture features of different shapes. Finally, the sigmoid activation function is used to limit the output value to between 0 and 1 to generate the final texture map.

[0030] The iterative refinement process specifically includes: establishing a geometry-texture joint optimization framework and setting the maximum number of iterations to 10; in each iteration, first fixing the geometric model, using the texture diffusion model to generate diffuse maps, normal maps and specular maps, and using geometric features as conditional inputs during the texture generation process; Its optimization process is guided by the joint objective function: L total =λ1L tex +λ2L render ; L total : Represents the overall loss function.

[0031] L tex : Represents the loss in the texture generation process, such as the reconstruction loss or adversarial loss in a diffusion model.

[0032] L render : Represents the differentiable rendering loss, which calculates the difference between the current texture map rendering result and the expected target (such as L1 loss, perceptual loss).

[0033] λ1, λ2: These represent the weighting hyperparameters that balance the contributions of the two loss terms.

[0034] This formula defines the overall objective function for the optimization process during the iterative refinement phase. It consists of a weighted average of texture generation loss and rendering-based geometric consistency loss. By minimizing this joint loss, alternating optimization of geometric and texture representations is driven until the model achieves high realism.

[0035] Then, with the texture map fixed, a differentiable renderer is used to generate multi-view rendered images, and the consistency loss between the rendered image and the target texture is calculated. Based on the rendering loss, the vertex positions of the geometric model are updated using the backpropagation algorithm. The geometric update uses a gradient-based optimization method with a learning rate of 0.001. The convergence condition is set to the change in the loss function being less than 0.0001 in three consecutive iterations. The iteration is terminated early after the convergence condition is met.

[0036] The specific processing of Luling cultural features includes: constructing a text description template library containing Luling cultural elements, with templates containing keywords such as traditional clothing, architectural decoration, and cultural symbols; and using a pre-trained language model to semantically expand the text descriptions and generate diverse text prompts. Add a cultural feature recognition module to the conditional diffusion model. This module uses an attention mechanism to highlight text tags that are related to cultural features. During texture generation, feature enhancement techniques are used to improve the generation quality of specific cultural elements such as traditional patterns and color schemes; cultural feature consistency constraints are established to ensure that the generated virtual human model maintains the stability of cultural features from different perspectives.

[0037] The specific implementation of the zero-shot learning setting includes: pre-training a neural implicit representation network using a large-scale 3D human body dataset, which contains 3D scan data of various body types and poses; The conditional diffusion model was pre-trained using a text-image pair dataset, the training data of which contained millions of text descriptions and their corresponding images; During the inference phase, the pre-trained model is used directly to process new text prompts without the need for fine-tuning for specific virtual characters; text input is optimized using prompt word engineering, and combined prompts and negative prompts are used to guide the generation process. The influence of textual conditions is controlled by adjusting the classifier's free-guided weights, with weight values ​​set between 3.0 and 7.0. A multi-round inference strategy is used, first generating low-resolution results and then gradually optimizing to high resolution.

[0038] Example 2 The present invention describes a zero-shot-based AIGC generation method for 3D virtual digital human of Luling culture. The detailed workflow is as follows.

[0039] The workflow of this invention begins with a three-dimensional implicit representation network of the human body constructed by a multilayer perceptron, which maps spatial coordinates to symbolic distance values ​​to form an initial geometric basis of the human body.

[0040] Subsequently, the system receives a text description containing Luling cultural features and converts it into an embedding vector through a pre-trained text encoder. This vector, along with the neural implicit field features with added noise, is input into a deep conditional diffusion model based on the U-Net architecture. The cross-attention mechanism in the model is used to achieve deep fusion of text conditions and geometric generation processes. Through multi-step iterative denoising, a high-precision, semantically consistent 3D digital human geometric model is generated.

[0041] Next, based on the geometric model, texture prior information such as vertex normals and curvature is extracted to drive another texture diffusion model for inference: This model adopts a multi-scale convolution structure, which splices and fuses geometric priors and diffusion features during the denoising process, and gradually generates and optimizes diffuse reflection, normals and specular maps.

[0042] Finally, the system enters the geometry-texture joint iterative optimization stage, alternately fixing one side and optimizing the other side. Through differentiable rendering, the multi-view consistency loss is calculated, driving the model to fine-tune until convergence. The final output is a 3D virtual digital human with highly consistent texture and geometry, cultural characteristics, and visual realism. The entire process is completed under zero-sample settings without additional training.

[0043] Finally, the following points should be noted: First, in the description of this application, it should be noted that, unless otherwise specified and limited, the terms "installation", "connection", and "linkage" should be interpreted broadly, and can be mechanical or electrical connections, or internal connections between two components, or direct connections. "Up", "down", "left", "right", etc. are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may change. Secondly: The accompanying drawings of the embodiments disclosed in this invention only involve the structures involved in the embodiments disclosed in this invention. Other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of this invention can be combined with each other. In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A zero-shot-based AIGC generation method for 3D virtual digital human of Luling culture, characterized in that, Includes the following steps: S1. Initialize the implicit representation of the 3D human body through a neural network to construct an initial 3D human body model; S2. A pre-trained deep conditional diffusion model is used to receive text prompts as conditional input, and conditional guidance and diffusion optimization are performed on the neural implicit field corresponding to the initial three-dimensional human body model to generate a three-dimensional virtual digital human geometric model that conforms to the text description. S3. Perform the denoising and restoration process based on the diffusion model, and combine the texture prior information from the geometric model of the three-dimensional virtual digital human to perform high-precision texture map reasoning and updating; S4. Optimize the texture representation through iterative refinement to generate a final 3D virtual digital human model with highly realistic textures.

2. The method for generating 3D virtual digital human AIGC based on zero-shot Luling culture according to claim 1, characterized in that, The implicit representation of the 3D human body initialized by the neural network adopts a coordinate-based implicit function representation. This implicit function maps the 3D spatial coordinates into symbolic distance values ​​or occupancy probability values ​​through a multilayer perceptron, thereby implicitly expressing the geometry of the 3D human body.

3. The method for generating 3D virtual digital human AIGC based on zero-shot Luling culture according to claim 1, characterized in that, The deep conditional diffusion model is a noise prediction network based on the U-Net architecture. In the denoising process, it integrates the embedded representation of text prompt information through a cross-attention mechanism to conditionally guide the generation process of the neural implicit field, thereby achieving diversified control over the geometry of the three-dimensional virtual digital human.

4. The method for generating 3D virtual digital human AIGC based on zero-shot Luling culture according to claim 1, characterized in that, The texture prior information includes geometric features and initial texture features extracted from the geometric model of the three-dimensional virtual digital human. The high-precision texture map inference and update process adopts a texture generator based on a diffusion model. This generator infers and optimizes diffuse reflection, normals and specular maps step by step in a denoising manner.

5. The method for generating 3D virtual digital human AIGC based on zero-shot Luling culture according to claim 4, characterized in that, The texture generator employs a multi-scale feature fusion mechanism during inference to combine the prior texture information with the intermediate layer features of the diffusion model, thereby enhancing the consistency and realism of texture details.

6. The method for generating 3D virtual digital human AIGC based on zero-shot Luling culture according to claim 1, characterized in that, The iterative refinement process includes alternating the execution of texture diffusion optimization under geometric consistency constraints and geometric fine-tuning based on texture feedback, until the geometry and texture of the three-dimensional virtual digital human model both reach a preset quality threshold.

7. The method for generating 3D virtual digital human AIGC based on zero-shot Luling culture according to claim 1, characterized in that, The text prompts contain descriptions of specific clothing, decorations, or cultural symbols related to Luling culture, and the deep conditional diffusion model generates a three-dimensional virtual digital human model with cultural characteristics based on these descriptions.

8. The method for generating 3D virtual digital human AIGC based on zero-shot Luling culture according to claim 1, characterized in that, The method employs zero-shot learning in both the training and inference phases, eliminating the need for model training or fine-tuning for specific 3D virtual digital humans. New 3D virtual digital human models can be generated solely based on pre-trained models and text input.

Images