A Method and System for Generating Digital Human Images Based on Dynamic Interaction of 3D Digital Twins

Abstract

This disclosure provides a method and system for generating digital human images based on 3D digital twin dynamic interaction, relating to the fields of 3D reconstruction and virtual digital human technology. The method includes: acquiring a single multi-view image; extracting features from the multi-view image; outputting intermediate attribute features for each vertex; adding noise to the intermediate attribute features and inputting them into a diffusion model for inverse denoising, outputting a set of 3D attribute points; embedding the 3D attribute point set into a reference configuration without bias in pose and expression; using the attribute points under the reference configuration, implementing pose-driven behavior through multiple local spatial transformations, and superimposing local deformation offsets according to expression control parameters to generate dynamic facial changes; for attribute points with directional features, synchronous updates are performed by controlling rotation relationships; finally, a natural and stable digital human image is output using a transparency-hybrid 3D point rendering method. This disclosure can meet the application requirements of real-time interactive digital humans.

Description

Technical Field

[0001] This disclosure relates to the field of 3D reconstruction and virtual digital human technology, specifically to a method and system for generating digital human images based on 3D digital twin dynamic interaction. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] With the widespread application of digital human technology in scenarios such as virtual social interaction, content generation, remote conferencing, and intelligent services, the demand for real-time responsiveness and realism of digital humans is constantly increasing.

[0004] Traditional methods still have many limitations in pose-driven rendering, facial expression generation, and cross-frame stability, especially when relying on neural networks to generate the entire rendering result, which can easily lead to problems such as structure drift, inter-frame jumps, or response lag. Furthermore, traditional mesh- or voxel-based representations require complex topology maintenance and computational resources, making them unsuitable for operation on resource-constrained devices. Representations based on 3D point sets, due to their lightweight, controllable, and easy-to-render advantages, have become a new direction for development.

[0005] Most existing technologies rely on image features to drive 3D reconstruction when building digital twins. However, the generated 3D structure is difficult to maintain stability between consecutive frames, and slight drifts often occur between local details and the overall shape, resulting in a lack of consistency in dynamic performance. On the other hand, existing driving mechanisms cannot achieve detailed dynamic responses to facial expressions and postures, and are prone to geometric inconsistencies and abnormal deformations under large movements or complex expressions. Directional 3D attributes cannot obtain correct spatial updates when posture changes, causing unreasonable distortions and inconsistencies in appearance as the viewpoint or movement changes.

[0006] In summary, the main problems faced by existing solutions include the difficulty in maintaining structural consistency of point sets under pose changes, the lack of precise local control in expression-driven approaches, and the inability of directional attributes to adjust naturally with changes in movement. Summary of the Invention

[0007] To address the aforementioned issues, this disclosure proposes a method and system for generating digital human images based on dynamic interaction of 3D digital twins. It maintains the structural consistency of the three-dimensional attribute point set under a reference configuration and achieves pose changes, expression changes, and directional attribute updates in a controllable manner. It outputs natural and stable digital human images using a transparency-hybrid three-dimensional point rendering method to meet the application requirements of real-time interactive digital humans.

[0008] According to some embodiments, the present disclosure adopts the following technical solutions:

[0009] A digital human image generation method based on 3D digital twin dynamic interaction includes:

[0010] Acquire a single multi-view image and preprocess it;

[0011] Feature extraction is performed on the preprocessed multi-view images, and feature interaction is performed based on a preset set of structured query points to output the intermediate attribute features of each vertex.

[0012] The intermediate attribute features are denoised to generate noisy attribute features. The noisy attribute features are then input into the diffusion model for reverse denoising, and the output is a set of three-dimensional attribute points.

[0013] The set of three-dimensional attribute points is embedded in a reference configuration with no bias in pose and expression. Based on the attribute points under the reference configuration, multiple local spatial transformations are used to achieve pose-driven operation. Local deformation offsets are superimposed according to expression control parameters to generate dynamic facial changes. For attribute points with directional features, synchronous updates are performed by controlling rotation relationships. Finally, a natural and stable digital human image is output using a transparency-hybrid three-dimensional point rendering method.

[0014] According to some embodiments, the present disclosure adopts the following technical solutions:

[0015] A digital human image generation system based on 3D digital twin dynamic interaction includes:

[0016] The image acquisition module is used to acquire a single multi-view image and preprocess it;

[0017] The feature attribute point construction module is used to extract features from the preprocessed multi-view image, and perform feature interaction based on the preset structured query point set, outputting the intermediate attribute features of each vertex; adding noise to the intermediate attribute features to generate noisy attribute features, inputting the noisy attribute features into the diffusion model for reverse denoising, and outputting a three-dimensional attribute point set.

[0018] The dynamic interaction generation module is used to embed a set of 3D attribute points into a reference configuration with no bias in pose and expression. Based on the attribute points under the reference configuration, multiple local spatial transformations are used to achieve pose-driven operation, and local deformation offsets are superimposed according to expression control parameters to generate dynamic facial changes. For attribute points with directional features, synchronous updates are performed by controlling rotation relationships. Finally, a natural and stable digital human image is output using a transparency-hybrid 3D point rendering method.

[0019] According to some embodiments, the present disclosure adopts the following technical solutions:

[0020] A computer program product includes a computer program that, when executed by a processor, implements the digital human image generation method based on 3D digital twin dynamic interaction.

[0021] According to some embodiments, the present disclosure adopts the following technical solutions:

[0022] A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the digital human image generation method based on 3D digital twin dynamic interaction.

[0023] According to some embodiments, the present disclosure adopts the following technical solutions:

[0024] An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the digital human image generation method based on 3D digital twin dynamic interaction.

[0025] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0026] This disclosed method for generating digital human images based on 3D digital twin dynamic interaction first establishes a set of three-dimensional attribute points under a reference configuration without bias in pose and expression. Then, it generates appearance parameters and geometric attributes of the points through multimodal input. Subsequently, it achieves pose-driven processing using spatial transformations of multiple local control units, and superimposes local deformation offsets based on expression control parameters to generate dynamic facial changes. For attributes with directional characteristics, they are updated synchronously through the rotation relationship of the control units. Finally, a natural and stable digital human image is output using a hybrid transparency 3D point rendering method. This invention has the advantages of high structural consistency, natural movements and expressions, and adaptability to real-time interaction.

[0027] This disclosed method for generating digital human images based on dynamic interaction of 3D digital twins utilizes a Transformer network to extract features from preprocessed multi-view images and performs feature interaction based on a preset set of structured query points, outputting intermediate attribute features for each vertex. Noise is added to these intermediate attribute features to generate noisy attribute features, which are then input into a diffusion model for reverse denoising, outputting a set of three-dimensional attribute points. The method combines the multi-scale features extracted by the Transformer with the asymptotic denoising mechanism of the diffusion model to replace noise-sensitive MLP decoding, reducing drift and distortion in attribute prediction. The input conditions for the diffusion module are constructed to simultaneously include the original Transformer features, noise markers (time steps), and attribute features to guide the denoising process, ensuring the generation of accurate Gaussian attributes.

[0028] This disclosure presents a digital human image generation method based on 3D digital twin dynamic interaction. It ensures that the Gaussian attributes generated by diffusion can be seamlessly integrated into geometric deformation driven by local control units, local adjustment mechanisms based on facial expression parameters, and a transparency-cumulative rendering process based on a set of 3D attribute points, without relying on additional neural networks. The driving signals include facial expression control parameters describing changes in motion in local facial regions. This disclosure achieves performance optimization for different scenarios through optional noise strategies, diffusion steps, or model distillation.

[0029] This disclosed method for generating digital human images based on dynamic interaction of 3D digital twins utilizes a multi-step denoising mechanism of a diffusion model to significantly reduce the accumulation of errors in a single regression, such as a significant reduction in Gaussian position drift and effective reduction in covariance estimation errors. The generated Gaussian attributes are directly compatible with traditional pipelines such as LBS and splatting rendering, without requiring modifications to the graphics engine interface. By adjusting the number of diffusion steps (e.g., 4 steps vs. 20 steps) or enabling a distillation model, an inference latency of ≤50ms can be achieved on mobile devices while maintaining rendering quality. The attributes generated by diffusion exhibit higher internal consistency in dimensions such as lighting and shape (performing better in terms of structural stability and appearance consistency, especially maintaining smoother geometric changes and visual effects under complex motion conditions).

[0030] This disclosed method for generating digital human images based on 3D digital twin dynamic interaction converts directional attributes into equivalent representations of corresponding postures according to the rotation relationship of the control unit on which the point depends. This ensures that the appearance structure of the point maintains its correct shape without distortion or blurring during actions such as turning the head or raising the head. Attached Figure Description

[0031] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0032] Figure 1 This is a schematic diagram of the process for generating digital human images based on 3D digital twin dynamic interaction, according to an embodiment of the present disclosure.

[0033] Figure 2 This is a schematic diagram of the diffusion model structure according to an embodiment of the present disclosure. Detailed Implementation

[0034] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0035] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0036] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0037] Example 1

[0038] One embodiment of this disclosure provides a method for generating digital human images based on dynamic interaction of 3D digital twins, the method steps of which include:

[0039] Step 1: Acquire a single multi-view image and preprocess it;

[0040] Step 2: Extract features from the preprocessed multi-view images, and perform feature interaction based on a preset set of structured query points to output the intermediate attribute features of each vertex;

[0041] Step 3: Add noise to the intermediate attribute features to generate noisy attribute features. Input the noisy attribute features into the diffusion model for reverse denoising and output a set of three-dimensional attribute points.

[0042] Step 4: Embed the set of 3D attribute points into a reference configuration with no bias in pose and expression. Based on the attribute points under the reference configuration, use multiple local spatial transformations to achieve pose-driven operation, and superimpose local deformation offsets according to expression control parameters to generate dynamic facial changes. For attribute points with directional features, synchronous updates are performed by controlling rotation relationships. Finally, a natural and stable digital human image is output using a transparency-blended 3D point rendering method.

[0043] As one embodiment, the digital human image generation method based on 3D digital twin dynamic interaction disclosed herein, and the 3D digital twin digital human interaction method based on single image input and fusion diffusion refinement mechanism, are applicable to real-time interactive application scenarios such as online meetings, virtual social networking, intelligent customer service, and mobile AR / VR. It can maintain the structural consistency of the three-dimensional attribute point set under a reference configuration and achieve pose changes, expression changes, and directional attribute updates in a controllable manner to meet the application requirements of real-time interactive digital humans. The specific implementation process is as follows:

[0044] Step 1: Acquire a single multi-view image and preprocess it;

[0045] Specifically, acquiring a single multi-view image includes: simultaneously acquiring N images from different perspectives (e.g., 3-8 perspectives, covering the front and several sides) within a short period of time, or using a single camera to perform slight panning and continuous shooting. This can significantly reduce geometric ambiguity based on a single perspective and improve the accuracy of 3D point sets and the stability of directional estimation. Since details are easily lost or artifacts are generated during reconstruction, details such as hair, mouth, and teeth need to be prioritized for protection or enhancement; different shooting conditions will affect the consistency of appearance attribute prediction and rendering, so a unified color space is required; line-of-sight occlusion or cluttered backgrounds will destroy the quality of feature interaction, so semantic segmentation and occlusion recovery are required.

[0046] Furthermore, the preprocessing process includes: quality cleaning, color and distortion correction of the acquired multi-view images, detection and cropping of the face and head and shoulder regions, semantic segmentation to separate the background, hair and local occlusion, and extraction and geometric alignment of key points to obtain images with uniform scale, clean and structurally normalized, providing stable input for subsequent feature extraction.

[0047] Step 2: Extract features from the preprocessed multi-view images, and perform feature interaction based on the preset set of structured query points to output the intermediate attribute features of each vertex;

[0048] Feature extraction from multi-view images is performed using a Transformer network. Based on a pre-defined set of structured query points, feature interaction is performed. Through multi-head attention and interaction with multi-scale image features, the intermediate attribute features of each vertex are output, including position, appearance, and shape embedding features.

[0049] Input a single RGB image The ResNet50 encoder in the Transformer network extracts multi-scale features. The dimensions are 256×H / 4×W / 4, 512×H / 8×W / 8, and 1024×H / 16×W / 16, respectively. A pre-defined set of standardized 3D reference points is used to establish a stable correspondence between the input image and the internal structure of the model. A bottleneck layer is set to compress and reorganize the global semantics, avoid redundant feature propagation, and act as an "information barrier" in the diffusion process, isolating noise and effective features, and providing controllable cross-scale feature fusion for subsequent diffusion reconstruction. The Transformer uses 8-head attention, and through the interaction of multi-head attention and multi-scale image features, it outputs the intermediate attribute features of each vertex. ,in N Where is the number of vertices, and D is the attribute dimension, including position, appearance, and shape embedding features.

[0050] Specifically, a uniformly defined set of 3D reference points is first introduced as a standardized benchmark for the model's internal space. This set of points is pre-fixed under a reference configuration without bias in pose or expression, ensuring that all input images can be projected onto the same 3D semantic coordinates in a consistent manner. Based on this structured set of reference points, the Transformer utilizes eight parallel attention heads to decouple and aggregate image features from different spatial levels: some attention heads are responsible for capturing local textures and edge details, some focus on large-scale shape contours, and others are used for cross-scale alignment and global semantic fusion.

[0051] Step 3: Add noise to the intermediate attribute features to generate noisy attribute features. Input the noisy attribute features into the diffusion model for reverse denoising and output a set of three-dimensional attribute points.

[0052] Specifically, controllable Gaussian noise is applied to intermediate attribute features to generate noisy attribute features, and the diffusion time step is recorded. The noisy attribute features, time step encoding, and original intermediate attribute features are used as joint conditional inputs to a diffusion model based on the U-Net architecture. Inverse denoising is then used to generate a final Gaussian three-dimensional attribute point set. This three-dimensional attribute point set describes the spatial location, directional expansion parameters, and visualization-related parameters of the points, including:

[0053] After the Transformer generates point / vertex level features, these features are first subjected to controlled noise addition. Specifically, the noise addition formula is as follows:

[0054]

[0055] in, Initial features; For the first t Noisy characteristics of time steps; For random disturbance terms; This is the noise attenuation coefficient; This represents the decay magnitude of the original features. During training, During reasoning t ∈{4,8}.

[0056] Furthermore, the noisy attribute features Noise time step marker Original intermediate attribute features As a joint conditional input, a diffusion model (diffusion head, based on U-Net architecture) is used for inverse denoising. The original intermediate attribute features are introduced through residual connections. The time-step embedding (256-dimensional) is fused with the attribute features through cross attention. The original features participate in the calculation of each layer through residual connections. The diffusion model outputs the final three-dimensional attribute point set. The three-dimensional attribute point set is used to describe the spatial location, directional extension parameters, and visualization-related parameters of the points.

[0057] Specifically, the diffusion model adopts a hierarchical encoding-decoding structure, which achieves gradual refinement of noisy attributes through layer-by-layer feature scaling and cross-layer correlation. Its specific depth and width can be flexibly configured according to the performance of different devices.

[0058] In the feature fusion process, the diffusion stage identifiers are first continuously mapped to obtain embedding vectors that can be directly processed by the network. Then, the basic features and diffusion inference results are integrated in two parallel ways: one is to preserve the original form of the basic features and add them directly to the diffusion output; the other is to project the basic features through a learnable linear mapping and then merge them with the diffusion output. These two paths can be selectively activated according to different task requirements, achieving stable and controllable feature compensation effects.

[0059] The diffusion head outputs predicted noise in the diffusion model. The following equation is used to denoise and update the features:

[0060]

[0061] in, It is a noisy attribute feature. The noise predicted by the diffuser head. For noise intensity, such as linear scheduling , The diffusion coefficient is... ; This is posterior noise, following a standard normal distribution. For maximum noise intensity, To minimize noise intensity, T This represents the total time step.

[0062] As one embodiment, the noise injection intensity is controlled according to a preset monotonic variation curve, and different shapes of variation functions can be selected according to task requirements.

[0063] Finally, the obtained attribute points are post-processed and constrained. Geometric constraints (such as positive covariance definiteness) and illumination consistency constraints are applied to the Gaussian attribute points to ensure physical rationality.

[0064] Step 4: Embed the set of 3D attribute points into a reference configuration with no bias in pose and expression. Based on the attribute points under the reference configuration, use multiple local spatial transformations to achieve pose driving, and superimpose local deformation offsets according to the expression control parameters to generate dynamic facial changes. For attribute points with directional features, update synchronously by controlling the rotation relationship.

[0065] Specifically, after generating the set of 3D attribute points refined by diffusion, to enable these attributes to change with the user's posture and expression, achieving real-time interaction, this disclosure introduces a spatially consistent deformation mechanism based on a reference configuration. This mechanism comprises the following three parts:

[0066] (1) Attribute initialization under reference configuration;

[0067] First, under the basic configuration where both posture and expression are in an unbiased state, all three-dimensional attribute points (including position, directional descriptors, and appearance-related parameters) are uniformly defined.

[0068] Specifically, a standardized 3D head model with a neutral expression, facing forward, and without head rotation or offset is selected to form a reference frame that does not change with the input content. Based on the model's needs, dense point sets are placed in key regions, and sparse points are placed in semantically weak regions. Then, 3D coordinates, normal vectors, semantic labels, and hierarchical indexes are defined for each reference point, ultimately resulting in a uniformly defined reference point template.

[0069] This reference configuration serves as the global initial state, maintaining a consistency benchmark in all subsequent dynamic deformations, ensuring that attribute changes between different actions do not result in structural offsets.

[0070] (2) Spatial deformation based on multiple local control units;

[0071] To achieve a dynamic transition from the reference configuration to the target attitude, this disclosure pre-defines several spatial control units and sets a combined weight relationship between each attribute point and the control unit. When the input attitude changes, each control unit provides a local rigid or quasi-rigid spatial transformation; the final position of the attribute point is obtained by linearly superimposing the local transformations according to the point weights.

[0072] Specifically, under the reference configuration, the overall region is divided into several local functional areas based on the topology of the face or head. Each functional area defines a local spatial control unit, and each control unit includes a local coordinate center, an initial orientation frame, and a local influence radius. Gaussian weights are calculated by calculating the distance between the attribute point and the centers of all control units, and then normalized, resulting in a fixed set of combined weights for each attribute point.

[0073] When the input attitude information changes, these inputs are converted into the local target attitude for each control unit. For each control unit, its local transformation matrix from the reference configuration to the target attitude is calculated, which includes the rotation matrix and translation vector.

[0074] As one example, this approach based on "linear combination of multiple control units" ensures that:

[0075] A. Geometric consistency across a wide range of attitude changes;

[0076] B. The overall structure of the point set remains smooth and continuous;

[0077] C. It requires reliance on any additional neural networks.

[0078] This mechanism is essentially equivalent to the multi-segment spatial transformation field in traditional computer graphics, but it has stronger interpretability and controllability.

[0079] (3) Additional deformation driven by facial expressions;

[0080] To reflect the subtle changes in local details caused by facial expressions, this disclosure introduces an additional offset of expression control parameters on top of posture deformation. This offset is obtained by weighted superposition of a set of preset local deformation directions and is used to describe subtle changes in mouth opening and closing, eyebrow and eye movement, and cheek and corner of the mouth, including:

[0081] The system pre-defines several local deformation direction vectors, each corresponding to a type of independently controllable facial expression, such as mouth opening and closing, eyebrow raising, eyelid closing, cheek lifting, and corner of the mouth pulling. For any attribute point, the system assigns a set of weight parameters corresponding to the deformation direction based on its semantic region, to characterize the degree to which the point is affected by various facial expressions. When an expression control signal is input, the corresponding deformation directions are superimposed according to their weights to form the local expression offset of the attribute point.

[0082] By linearly superimposing these offset terms, attribute points can exhibit natural and continuous geometric changes during facial expression changes.

[0083] (4) Directional properties are updated with rotation;

[0084] Some three-dimensional attributes possess directional characteristics, such as geometric descriptors controlling the appearance diffusion range and lighting response. To ensure these directional parameters remain consistent with the local surface after attitude changes, this disclosure, based on the correspondence between points and control units, maps the rotation of the control unit onto these directional descriptions, generating their equivalent form under the target attitude, including:

[0085] A weighted mapping relationship is established between the directional description of each attribute point and its corresponding spatial control unit. After the control unit generates a local rotation matrix under the attitude input, the system applies the rotation amount of each control unit to the directional vector of that point according to the weight of the attribute point, and performs weighted superposition and normalization on the rotation results to generate the equivalent form of the directional description under the target attitude. Through this one-to-one rotation mapping method, the directional attribute can be ensured to remain continuous and stable under large movements, avoiding artifacts such as surface distortion, blurring, and stretching, thereby ensuring the quality of dynamic rendering.

[0086] This one-to-one rotation update method ensures that: the spatial orientation of Gaussian points is not distorted unreasonably; the appearance shape remains consistent even with large movements; and artifacts such as blurring and stretching are avoided during dynamic rendering.

[0087] Step 5: Finally, a natural and stable digital human image is output using a transparency-blended 3D point rendering method.

[0088] Specifically, after completing the aforementioned geometric and directional updates, a transparency-based blending rendering method using a set of 3D attribute points is used to achieve the final digital human visualization output. This rendering process does not rely on any neural network, and the process is as follows:

[0089] The 3D attribute points are projected onto the camera's view plane using the following formula:

[0090]

[0091]

[0092] in, For a point in three-dimensional space, For points projected onto a 2D screen, For the projection matrix, For the color information of the corresponding point, For point-to-pixel coverage, This is the final pixel color.

[0093] This disclosure calculates the screen coverage based on the shape and direction of the points, sorts and accumulates the transparency according to the front-to-back relationship, and synthesizes the final color and appearance result.

[0094] This ultra-lightweight rendering method can run directly on mobile GPUs and ordinary graphics interfaces, offering high real-time performance and high compatibility.

[0095] As one example, performance is optimized by balancing quality and inference speed through model distillation, few-step diffusion, or feature caching.

[0096] Example 2

[0097] One embodiment of this disclosure provides a digital human image generation system based on 3D digital twin dynamic interaction, comprising:

[0098] The image acquisition module is used to acquire a single multi-view image and preprocess it;

[0099] The feature attribute point construction module is used to extract features from the preprocessed multi-view image, and perform feature interaction based on the preset structured query point set, outputting the intermediate attribute features of each vertex; adding noise to the intermediate attribute features to generate noisy attribute features, inputting the noisy attribute features into the diffusion model for reverse denoising, and outputting a three-dimensional attribute point set.

[0100] The dynamic interaction generation module is used to embed a set of 3D attribute points into a reference configuration with no bias in pose and expression. Based on the attribute points under the reference configuration, multiple local spatial transformations are used to achieve pose-driven operation, and local deformation offsets are superimposed according to expression control parameters to generate dynamic facial changes. For attribute points with directional features, synchronous updates are performed by controlling rotation relationships. Finally, a natural and stable digital human image is output using a transparency-hybrid 3D point rendering method.

[0101] Example 3

[0102] One embodiment of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the digital human image generation method based on 3D digital twin dynamic interaction.

[0103] Example 4

[0104] One embodiment of this disclosure provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the digital human image generation method based on 3D digital twin dynamic interaction.

[0105] Example 5

[0106] One embodiment of this disclosure provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute the digital human image generation method based on 3D digital twin dynamic interaction.

[0107] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0108] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0109] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A method for generating digital human images based on 3D digital twin dynamic interaction, characterized in that, include: Acquire a single multi-view image and preprocess it; Feature extraction is performed on the preprocessed multi-view images, and feature interaction is performed based on a preset set of structured query points to output the intermediate attribute features of each vertex. The intermediate attribute features are denoised to generate noisy attribute features. The noisy attribute features are then input into the diffusion model for reverse denoising, and the output is a set of three-dimensional attribute points. The three-dimensional attribute point set is embedded in a reference configuration with no bias in pose and expression. Based on the attribute points under the reference configuration, multiple local spatial transformations are used to achieve pose driving. Local deformation offsets are superimposed according to the expression control parameters to generate dynamic facial changes. For attribute points with directional features, synchronous updates are performed by controlling the rotation relationship. Finally, a natural and stable digital human image is output by using a transparency-hybrid three-dimensional point rendering method. The process of establishing a set of three-dimensional attribute points under a reference configuration with no bias in pose and expression includes: First, under the basic reference configuration where both posture and expression are in an unbiased state, all three-dimensional attribute points are uniformly defined; the basic reference configuration serves as the global initial state and maintains a consistency benchmark in all dynamic deformations, ensuring that attribute changes between different actions do not produce structural offsets. The attribute points based on the reference configuration utilize multiple local spatial transformations to achieve pose-driven behavior, and generate dynamic facial changes by superimposing local deformation offsets based on expression control parameters, including: Several local spatial control units are preset, and a combined weight relationship is established between each attribute point and the local control unit; When the input attitude changes, each local control unit provides a local rigid or quasi-rigid spatial transformation to achieve attitude deformation; the final position of the attribute point is obtained by linearly superimposing the local transformations according to the point weights. An additional offset for expression control parameters is introduced on the basis of posture deformation. This offset is obtained by superimposing a set of preset local deformation directions according to weights. It is used to describe the subtle changes in mouth opening and closing, eyebrow and eye movement, cheek and corner of mouth. By linearly superimposing the offset terms, the attribute points present natural and continuous geometric changes in the process of expression changes.

2. The digital human image generation method based on 3D digital twin dynamic interaction as described in claim 1, characterized in that, The process involves extracting features from the preprocessed multi-view images and performing feature interaction based on a preset set of structured query points, outputting the intermediate attribute features of each vertex, including: The Transformer network is used to extract features from multi-view images. Based on a pre-defined set of structured query points, feature interaction is performed. Through multi-head attention and interaction with multi-scale features of the image, the intermediate attribute features of each vertex are output, including position, appearance and shape embedding features.

3. The digital human image generation method based on 3D digital twin dynamic interaction as described in claim 1, characterized in that, The process involves adding noise to intermediate attribute features to generate noisy attribute features, inputting these noisy attribute features into a diffusion model for reverse denoising, and outputting a set of three-dimensional attribute points, including: Controllable Gaussian noise is applied to intermediate attribute features to generate noisy attribute features, and the diffusion time step is recorded; The diffusion model based on the U-Net architecture uses noisy attribute features, time-step encoding, and original intermediate attribute features as joint conditional inputs. The final Gaussian three-dimensional attribute point set is generated through inverse denoising. The three-dimensional attribute point set is used to describe the spatial location, directional extension parameters, and visualization-related parameters of the points.

4. The digital human image generation method based on 3D digital twin dynamic interaction as described in claim 1, characterized in that, For attribute points with directional characteristics, synchronous updates are performed by controlling rotation relationships. Finally, a natural and stable digital human image is output using a transparency-blended 3D point rendering method, including: For some attribute points with directional characteristics, these include the control of the appearance diffusion range and the geometric description of the illumination response. Based on the correspondence between the attribute points and the control unit, the rotation of the control unit is mapped onto the directional description to generate its equivalent form under the target posture. After completing the geometry and orientation updates, a transparency-based blending rendering method based on a set of 3D attribute points is used to achieve the final digital human visualization output.

5. A digital human image generation system based on 3D digital twin dynamic interaction, specifically implementing the digital human image generation method based on 3D digital twin dynamic interaction as described in any one of claims 1-4, characterized in that, include: The image acquisition module is used to acquire a single multi-view image and preprocess it; The feature attribute point construction module is used to extract features from the preprocessed multi-view images, and to perform feature interaction based on a preset set of structured query points, outputting the intermediate attribute features of each vertex. The intermediate attribute features are denoised to generate noisy attribute features. The noisy attribute features are then input into the diffusion model for reverse denoising, and the output is a set of three-dimensional attribute points. The dynamic interaction generation module is used to embed a set of 3D attribute points into a reference configuration with no bias in pose and expression. Based on the attribute points under the reference configuration, multiple local spatial transformations are used to achieve pose-driven operation, and local deformation offsets are superimposed according to expression control parameters to generate dynamic facial changes. For attribute points with directional features, synchronous updates are performed by controlling rotation relationships. Finally, a natural and stable digital human image is output using a transparency-hybrid 3D point rendering method.

6. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the digital human image generation method based on 3D digital twin dynamic interaction as described in any one of claims 1-4.

7. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the digital human image generation method based on 3D digital twin dynamic interaction as described in any one of claims 1-4.

8. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the digital human image generation method based on 3D digital twin dynamic interaction as described in any one of claims 1-4.

Images