A three-dimensional gaussian digital human generation system, method and electronic device

Abstract

The application provides a three-dimensional Gaussian digital human generation system, method and electronic equipment, and relates to the technical field of augmented reality, and comprises the following steps: a data acquisition unit is used for acquiring visual depth data, dynamic behavior data and environment perception data, and generating point cloud data and multi-modal data accordingly; a rendering adjustment unit is used for monitoring a gaze area in real time, and determining a rendering strategy; according to real-time computing power parameters, the rendering specification is determined; a driving unit is used for extracting features from the multi-modal data, obtaining multi-modal features, and generating facial driving parameters and limb driving parameters of a digital human model; a modeling unit is used for generating a basic grid of the digital human model according to the point cloud data and an initial digital human model, and performing expression rendering and action rendering on the basic grid according to the rendering strategy and the rendering specification, in combination with the facial driving parameters and the limb driving parameters, to obtain a rendered digital human model. The application improves the fluency, authenticity and naturalness of the display effect.

Description

Technical Field

[0001] This invention relates to the field of augmented reality technology, and more specifically, to a three-dimensional Gaussian digital human generation system, method, and electronic device. Background Technology

[0002] With the development of augmented reality technology, 3D digital humans have been applied in many fields as an important carrier for virtual and real interaction. Among them, 3D Gaussian Splatting (3DGS) technology has been widely used in the field of digital human modeling due to its balance between high-fidelity rendering and real-time performance. 3DGS technology achieves high-precision facial modeling through multi-view geometric reconstruction, laying the foundation for the application of 3D digital humans on augmented reality devices.

[0003] In related technologies, the digital human models generated by 3DGS technology require large storage volumes. For example, processing a high-fidelity digital human requires 4 million Gaussian points, resulting in a file size exceeding 1GB. Therefore, it is difficult to deploy directly on augmented reality devices with limited storage resources, such as AR glasses. Secondly, the low computing power of augmented reality devices leads to insufficient real-time rendering efficiency, resulting in noticeable lag and affecting the user's augmented reality interactive experience. Summary of the Invention

[0004] The problem addressed by this invention is how to improve the applicability and display effect of 3D Gaussian digital humans on low-computing-power devices.

[0005] To address the above problems, this invention provides a three-dimensional Gaussian digital human generation system, method, and electronic device.

[0006] In a first aspect, the present invention provides a three-dimensional Gaussian digital human generation system, comprising:

[0007] The data acquisition unit is used to acquire the user's visual depth data, dynamic behavior data, and environmental perception data; and to generate point cloud data and multimodal data based on the visual depth data, the dynamic behavior data, and the environmental perception data.

[0008] The rendering adjustment unit is used to monitor the user's gaze area in real time using eye-tracking technology, and determine the rendering strategy based on the gaze area; and determine the rendering specifications based on the real-time computing power parameters of the user's augmented reality device.

[0009] A driving unit is used to extract features from the multimodal data to obtain multimodal features, wherein the multimodal features include speech features, facial expression features, and posture features; and to generate facial driving parameters and limb driving parameters of the digital human model based on the speech features, facial expression features, and posture features, respectively.

[0010] The modeling unit is used to generate the base mesh of the digital human model based on the point cloud data and the initial digital human model, and to perform expression rendering and motion rendering on the base mesh according to the rendering strategy and the rendering specifications, combined with the facial driving parameters and the limb driving parameters, to obtain the rendered digital human model.

[0011] Optionally, the data acquisition unit is specifically used for:

[0012] By acquiring the user's multi-view RGB image and depth information, the visual depth data is generated, and initial point cloud data is generated based on the visual depth data. The RGB values ​​in the multi-view RGB image are mapped to the initial point cloud data to obtain the point cloud data.

[0013] The dynamic behavior data is generated by acquiring the user's time-series dynamic data;

[0014] By using simultaneous localization and mapping (MRT) technology, the spatial location, orientation, and ambient lighting parameters of the user's augmented reality device are obtained, and the environmental perception data is generated based on the spatial location, orientation, and ambient lighting parameters.

[0015] The visual depth data, the dynamic behavior data, and the environmental perception data are fused to obtain the multimodal data.

[0016] Optionally, the rendering adjustment unit is specifically used for:

[0017] The user's gaze point is determined using eye-tracking technology;

[0018] The user's gaze area is determined by expanding the gaze point according to a preset field of view rule.

[0019] Determine the user's non-focused area based on the user's gaze area;

[0020] The fogging region and the non-fogging region are assigned corresponding rendering levels, wherein the rendering level of the fogging region is higher than that of the non-fogging region;

[0021] The rendering strategy is generated based on the rendering levels of the foveated region and the non-foveated region.

[0022] Optionally, the driving unit is specifically used for:

[0023] The computing power mode of the augmented reality device is determined based on the relationship between the real-time computing power parameters of the augmented reality device and the preset computing power threshold.

[0024] Based on the computing power mode, the number of Gaussian points and the rendering resolution of the digital human model are determined, and the number of Gaussian points and the rendering resolution are used as the rendering specifications.

[0025] Optionally, the driving unit is specifically used for:

[0026] The multimodal data is extracted using a lightweight speech encoder to obtain Mel-frequency features and fundamental frequency information, and the speech features are determined based on the Mel-frequency features and fundamental frequency information.

[0027] The multimodal data is extracted using MobileFaceNet to obtain facial key points and FACS expression coefficients, and the expression features are determined based on the facial key points and the FACS expression coefficients.

[0028] The multimodal data is processed to obtain the limb skeleton rotation matrix, and then the posture features are determined based on the limb skeleton rotation matrix.

[0029] The speech features, facial expression features, and posture features are fused to obtain a fused feature vector, and the fused feature vector is input into the emotion fusion network to obtain the emotion state vector of the multimodal data. Then, the facial driving parameters are obtained based on the emotion state vector.

[0030] The joint operating angles of the digital human model are determined based on the posture features using an inverse kinematics algorithm, and then the limb driving parameters are obtained based on the joint operating angles.

[0031] Optionally, the modeling unit is specifically used for:

[0032] The initial digital human model is updated based on the point cloud data using a mesh generation algorithm to generate the basic mesh of the digital human model.

[0033] Initialize one or more Gaussian points at each vertex of the base mesh, and establish a mapping relationship between the vertices of the base mesh and the Gaussian points;

[0034] Based on the rendering strategy and the rendering specifications, and in conjunction with the mapping relationship, the face and limbs of the digital human model are rendered respectively to obtain the rendered digital human model.

[0035] Optionally, the modeling unit is further used for:

[0036] Assign corresponding motion amplitude weights to the facial expression rendering and the motion rendering respectively, wherein the Gaussian dominant weight in the motion amplitude weight of the facial expression rendering is greater than the mesh dominant weight in the motion amplitude weight, and the mesh dominant weight in the motion amplitude weight of the motion rendering is greater than the Gaussian dominant weight in the motion rendering.

[0037] Based on the facial driving parameters and the limb driving parameters, and according to the motion amplitude weights of the facial expression rendering and the motion amplitude weights of the motion rendering, the vertex positions and Gaussian point parameters of the facial region in the base mesh are adjusted respectively to obtain the rendered digital human model.

[0038] Optionally, a decompression unit may also be included;

[0039] The decompression unit is used to semantically group the initial digital human model to obtain multiple components of the initial digital human model, and to extract the Gaussian parameter matrix of each component.

[0040] By selecting multiple anchor points through sampling at the farthest point, the Gaussian parameter matrix of each component is compressed to the anchor point to obtain the compressed model of the initial digital human model;

[0041] When the digital human model needs to be rendered, the decoding accuracy is determined according to the real-time computing power parameters of the augmented reality device; the compressed model is decompressed according to the decoding accuracy to obtain the initial digital human model.

[0042] Secondly, the present invention provides a method for generating a three-dimensional Gaussian digital human, comprising:

[0043] Acquire user's visual depth data, dynamic behavior data, and environmental perception data; generate point cloud data and multimodal data based on the visual depth data, dynamic behavior data, and environmental perception data;

[0044] The user's gaze area is monitored in real time using eye-tracking technology, and a rendering strategy is determined based on the gaze area; the rendering specifications are determined based on the real-time computing power parameters of the user's augmented reality device.

[0045] Feature extraction is performed on the multimodal data to obtain multimodal features, wherein the multimodal features include speech features, facial expression features, and posture features; facial driving parameters and limb driving parameters of the digital human model are generated based on the speech features, facial expression features, and posture features, respectively.

[0046] Based on the point cloud data and the initial digital human model, a base mesh for the digital human model is generated. Then, based on the rendering strategy and the rendering specifications, and in conjunction with the facial driving parameters and the limb driving parameters, facial expression rendering and motion rendering are performed on the base mesh to obtain the rendered digital human model.

[0047] Thirdly, the present invention provides an electronic device, including a memory and a processor;

[0048] The memory is used to store computer programs;

[0049] The processor is configured to implement the three-dimensional Gaussian digital human generation method described above when executing the computer program.

[0050] The three-dimensional Gaussian digital human generation system, method, and electronic device of the present invention acquire user visual depth data, dynamic behavior data, and environmental perception data through a data acquisition unit, and generate point cloud data and multimodal data accordingly. This provides rich basic data for the subsequent generation and rendering of the digital human model, ensuring that the construction of the digital human has complete data support, enabling the digital human to more realistically reflect the user's characteristics and environment, and laying the foundation for improving the display effect.

[0051] The rendering adjustment unit utilizes eye-tracking technology to monitor the user's gaze area in real time. Based on different gaze areas, it determines corresponding rendering strategies, ensuring detailed rendering of the digital human in key areas of user focus, guaranteeing display quality in critical areas. In areas where the user pays little or no attention, the rendering precision is appropriately reduced, saving computing resources and avoiding unnecessary computational overhead. Furthermore, it determines the rendering specifications based on the real-time computing power parameters of the augmented reality device. This allows for dynamic adjustment of the rendering complexity according to the device's actual capabilities, preventing stuttering caused by insufficient computing power. Ultimately, this improves the smoothness of the digital human's display on the augmented reality device and enhances the user's interactive experience.

[0052] The driving unit extracts features from multimodal data, obtaining multimodal features including voice features, facial expression features, and posture features. Based on these features, it generates facial and limb driving parameters for the digital human model. This enables the digital human to react and move in response to the user's dynamic behavior, enhancing its interactivity and realism. It allows the digital human to simulate the user's speech and actions in real time, improving the naturalness and vividness of the display.

[0053] The modeling unit generates a basic mesh for the digital human model based on point cloud data and an initial digital human model. Then, combining rendering strategies, rendering specifications, and facial and limb driving parameters, it performs expression and motion rendering on the basic mesh, ultimately obtaining the rendered digital human model. For example, based on the user's voice and facial expression characteristics, it adjusts facial muscles and skeletal structure to render facial expressions synchronized with the user's speech; based on the user's posture characteristics, it adjusts limb joints and muscles to render corresponding postures. This process realizes a complete workflow from data acquisition to final model generation, ensuring that the digital human model visually conforms to the user's actual characteristics while adapting to the performance limitations of augmented reality devices, thereby improving the display effect. In this process, the generation of the basic mesh guarantees the basic form of the digital human model, and the integration of rendering strategies, specifications, and driving parameters allows the digital human to adapt to the characteristics of augmented reality devices while maintaining visual quality, and to display expressions and movements consistent with the user's characteristics, thus comprehensively improving the display effect of the 3D Gaussian digital human.

[0054] In summary, through the collaborative work of each unit, the problems of large storage requirements and poor display effects of 3D Gaussian digital human models on low-computing-power augmented reality devices have been effectively solved, improving the smoothness, realism, and naturalness of the display effects and enhancing the user's interactive experience in the augmented reality environment. Attached Figure Description

[0055] Figure 1 This is a structural block diagram of the three-dimensional Gaussian digital human generation system according to an embodiment of the present invention;

[0056] Figure 2 This is a flowchart of the three-dimensional Gaussian digital human generation method according to an embodiment of the present invention;

[0057] Figure 3 This is a flowchart of the compression model of the compression unit in an embodiment of the present invention;

[0058] Figure 4 This is a flowchart illustrating the rendering process of a 3D Gaussian digital human according to an embodiment of the present invention. Detailed Implementation

[0059] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0060] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0061] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to"; the term "based on" means "at least partially based on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; and the term "optionally" means "optional embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first," "second," etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0062] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0063] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0064] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties. The collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0065] Combination Figure 1 As shown, an embodiment of the present invention provides a three-dimensional Gaussian digital human generation system, comprising:

[0066] The data acquisition unit is used to acquire the user's visual depth data, dynamic behavior data, and environmental perception data; and to generate point cloud data and multimodal data based on the visual depth data, the dynamic behavior data, and the environmental perception data.

[0067] Specifically, visual depth data is collected using the binocular cameras and structured light sensors built into the AR glasses to capture depth information of the user's face and surroundings, resulting in depth maps. These depth maps capture the user's facial contours, the position of facial features, and spatial relationships from different perspectives, reflecting the three-dimensional shape of the face. Subsequently, the system uses multi-view geometry principles to fuse multiple depth maps, constructing accurate 3D point cloud data, laying the foundation for digital human facial modeling, where each point has xyz coordinates. Dynamic behavior data is recorded using a high frame rate color camera to capture user actions and shoot color videos. The videos capture dynamic information such as the user's facial expressions and body movements, providing rich visual material for subsequent analysis. Then, using human pose estimation algorithms (such as OpenPose), the joint positions of the user's limbs are accurately identified from the video frames to obtain pose feature vectors, reflecting changes in human posture, including limb bending, extension, rotation, and other movement features. Environmental perception data is collected through environmental sensors (such as light sensors and spatial sensors) built into the AR glasses to collect information about the surrounding environment. Among them, the light sensor measures the ambient light intensity and color, providing the spatial sensor with structural information about the user's physical space, such as the size, layout, and obstructions of the space, and constructs a three-dimensional map of the environment through SLAM technology.

[0068] During the data fusion phase, the system integrates the collected depth maps, color videos, pose feature vectors, and 3D environmental maps. First, the depth maps and color videos are aligned pixel-by-pixel. An image registration algorithm maps each depth point to a corresponding pixel in the color image, thus assigning color attributes to the point cloud data and creating a 3D point cloud with color information. Next, pose feature vectors are combined to associate limb movements from dynamic behavior data with the point cloud data, ensuring that the point cloud not only contains static shape and color but also reflects the user's limb movement trajectory and posture changes. Finally, environmental perception data is incorporated. Based on the 3D environmental map, the coordinate system of the point cloud data is adjusted to match the user's actual environment, generating multimodal data associated with the environment, including visual appearance, dynamic behavior, and environmental context.

[0069] The rendering adjustment unit is used to monitor the user's gaze area in real time using eye-tracking technology, and determine the rendering strategy based on the gaze area; and determine the rendering specifications based on the real-time computing power parameters of the user's augmented reality device.

[0070] Specifically, the eye-tracking component is installed inside the AR glasses lenses, close to the user's eyes. Utilizing high-precision infrared eye-tracking technology, it emits infrared light and captures the reflected light from the user's eyes to capture the eye's movement trajectory and gaze position in real time. The system acquires eye images at a certain frame rate (e.g., 60fps or higher) and uses eye-tracking algorithms to accurately calculate the user's gaze coordinates (x, y) within the digital human display area. This determines the part of the digital human the user is currently focusing on, such as the face, body, or background. The system then dynamically adjusts the priority of the gaze area based on factors such as gaze duration and frequency. If the user continuously gazes at the digital human's eyes, the system determines this as the user's focus and marks it as a key rendering area; conversely, if the user only briefly glances at a body part, its priority is reduced, flexibly adapting to changes in the user's visual attention.

[0071] Based on the gaze area information, the system assigns different rendering levels (e.g., high, medium, and low). For the gaze area, a high rendering level is allocated, enabling high-resolution textures, complex shading models, and detailed lighting calculations to ensure a delicate and realistic digital human display in the area of ​​user attention. For non-gaze areas, the rendering level is reduced, texture details are decreased, and lighting processing is simplified, optimizing the allocation of computing resources and improving overall rendering efficiency. When the user's gaze shifts, the rendering level is quickly reclassified and resources are reallocated to ensure that the area of ​​user attention always receives high-quality rendering effects, achieving efficient utilization of rendering resources and an optimal balance between display effects, thus enhancing the user's visual experience.

[0072] The real-time computing power monitoring module is integrated into the system's underlying layer. Closely connected to the device's hardware drivers, it reads key computing power metrics (such as CPU utilization, GPU memory usage, and temperature) in real time, acquiring data multiple times per second and analyzing their trends to accurately grasp the device's real-time computing power status. Based on the device's real-time computing power parameters, the system dynamically adjusts the rendering specifications. When the device has sufficient computing power (e.g., CPU utilization below 60% and GPU memory usage below 70%), high rendering specifications are enabled, supporting 4K resolution, high frame rates (60fps+), and complex lighting effects to ensure optimal digital human display. When the device's computing power is strained (e.g., CPU utilization above 80% and GPU memory usage above 90%), it automatically switches to low rendering specifications, reducing the resolution to 1080p and the frame rate to 30fps, simplifying lighting effects, preventing stuttering, and ensuring a smooth user experience.

[0073] The driving unit is used to extract features from the multimodal data to obtain multimodal features, wherein the multimodal features include speech features, facial expression features, and posture features; and to generate facial driving parameters and limb driving parameters of the digital human model based on the speech features, facial expression features, and posture features, respectively.

[0074] Specifically, after acquiring the speech signal, the system preprocesses to remove noise. Then, using algorithms such as Mel-frequency cepstral coefficients (MFCC), it extracts speech features such as pitch, timbre, and speech rate to form a speech feature vector, providing a basis for facial expression driving. Key point detection is performed on the facial image to identify the position and change trends of key points in areas such as the eyes, eyebrows, and mouth. Combined with expression recognition algorithms, the system determines the current expression category (e.g., smiling, frowning) and quantifies the expression intensity to obtain an expression feature vector, reflecting real-time changes in the user's expression. Based on the posture estimation results, the system calculates the position, angle, and velocity of each joint in the body to form a posture feature vector, describing the user's limb movements and providing data support for limb movement driving. The speech feature vector is input into the speech driving model to generate mouth shape and facial muscle movement parameters synchronized with the speech. Combining the expression feature vector and expression mapping rules, corresponding expression driving parameters are obtained to control the contraction and relaxation of facial muscles, achieving synchronization between mouth shape and speech, resulting in natural and rich expressions. By analyzing posture feature vectors and using kinematic models to calculate joint rotation angles and limb movement trajectories, limb drive parameters are generated to control limb swinging, bending, and extension, making limb movements natural and smooth.

[0075] The modeling unit is used to generate the base mesh of the digital human model based on the point cloud data and the initial digital human model, and to perform expression rendering and motion rendering on the base mesh according to the rendering strategy and the rendering specifications, combined with the facial driving parameters and the limb driving parameters, to obtain the rendered digital human model.

[0076] Specifically, a digital human model is a virtual projection of a user in virtual space, used to enhance the user's interactive experience. The modeling unit's role is to present the user's appearance features and behavioral dynamics in the form of a digital human model within an augmented reality device. Specifically, a basic mesh is constructed based on point cloud data. First, noise points are removed through point cloud preprocessing (denoising, filtering, etc.). Then, a Poisson surface reconstruction algorithm is used to fit the overall shape of the point cloud, generating an initial triangular mesh. Finally, the mesh topology and geometric details are optimized, the triangle distribution is adjusted to uniformly fit the shape, and key areas (such as eyes and mouth) are refined, initially forming the three-dimensional basic mesh of the digital human model, reflecting the user's static appearance outline. Then, facial expressions and movements are rendered on the basic mesh. The rendering module receives the basic mesh, rendering strategy, rendering specifications, and facial and limb driving parameters, and adjusts the mesh vertex positions and shapes according to the driving parameters to simulate changes in facial expressions and limb movements. Meanwhile, based on the rendering strategy and specifications, the rendering effects and performance are optimized: high-precision lighting and high-resolution textures are enabled in the foveated area to improve detail; the precision is appropriately reduced in the non-foveated area to reduce the amount of computation; and the rendering parameters (such as resolution, frame rate, and effect complexity) are dynamically adjusted according to the device's computing power parameters to ensure a smooth and stable display effect, ultimately generating a natural and realistic rendered digital human model.

[0077] The three-dimensional Gaussian digital human generation system of the present invention acquires the user's visual depth data, dynamic behavior data, and environmental perception data through a data acquisition unit, and generates point cloud data and multimodal data accordingly. This provides rich basic data for the subsequent generation and rendering of the digital human model, ensuring that the construction of the digital human has complete data support, enabling the digital human to more realistically reflect the user's characteristics and environment, and laying the foundation for improving the display effect.

[0078] The rendering adjustment unit utilizes eye-tracking technology to monitor the user's gaze area in real time. Based on different gaze areas, it determines corresponding rendering strategies, ensuring detailed rendering of the digital human in key areas of user focus, guaranteeing display quality in critical areas. In areas where the user pays little or no attention, the rendering precision is appropriately reduced, saving computing resources and avoiding unnecessary computational overhead. Furthermore, it determines the rendering specifications based on the real-time computing power parameters of the augmented reality device. This allows for dynamic adjustment of the rendering complexity according to the device's actual capabilities, preventing stuttering caused by insufficient computing power. Ultimately, this improves the smoothness of the digital human's display on the augmented reality device and enhances the user's interactive experience.

[0079] The driving unit extracts features from multimodal data, obtaining multimodal features including voice features, facial expression features, and posture features. Based on these features, it generates facial and limb driving parameters for the digital human model. This enables the digital human to react and move in response to the user's dynamic behavior, enhancing its interactivity and realism. It allows the digital human to simulate the user's speech and actions in real time, improving the naturalness and vividness of the display.

[0080] The modeling unit generates a basic mesh for the digital human model based on point cloud data and an initial digital human model. Then, combining rendering strategies, rendering specifications, and facial and limb driving parameters, it performs expression and motion rendering on the basic mesh, ultimately obtaining the rendered digital human model. For example, based on the user's voice and facial expression characteristics, it adjusts facial muscles and skeletal structure to render facial expressions synchronized with the user's speech; based on the user's posture characteristics, it adjusts limb joints and muscles to render corresponding postures. This process realizes a complete workflow from data acquisition to final model generation, ensuring that the digital human model visually conforms to the user's actual characteristics while adapting to the performance limitations of augmented reality devices, thereby improving the display effect. In this process, the generation of the basic mesh guarantees the basic form of the digital human model, and the integration of rendering strategies, specifications, and driving parameters allows the digital human to adapt to the characteristics of augmented reality devices while maintaining visual quality, and to display expressions and movements consistent with the user's characteristics, thus comprehensively improving the display effect of the 3D Gaussian digital human.

[0081] In summary, through the collaborative work of each unit, the problems of large storage requirements and poor display effects of 3D Gaussian digital human models on low-computing-power augmented reality devices have been effectively solved, improving the smoothness, realism, and naturalness of the display effects and enhancing the user's interactive experience in the augmented reality environment.

[0082] Optionally, the data acquisition unit is specifically used for:

[0083] By acquiring the user's multi-view RGB image and depth information, the visual depth data is generated, and initial point cloud data is generated based on the visual depth data. The RGB values ​​in the multi-view RGB image are mapped to the initial point cloud data to obtain the point cloud data.

[0084] The dynamic behavior data is generated by acquiring the user's time-series dynamic data;

[0085] By using simultaneous localization and mapping (MRT) technology, the spatial location, orientation, and ambient lighting parameters of the user's augmented reality device are obtained, and the environmental perception data is generated based on the spatial location, orientation, and ambient lighting parameters.

[0086] The visual depth data, the dynamic behavior data, and the environmental perception data are fused to obtain the multimodal data.

[0087] Specifically, multiple RGB cameras and depth sensors are deployed around the user to capture images and depth maps from different perspectives, accurately capturing facial details and spatial structure. The depth sensors simultaneously acquire depth information, ensuring each RGB image pixel has a corresponding depth value reflecting its distance from the camera. The depth map data is converted into a 3D point cloud; based on the depth values, the spatial coordinates (x, y, z) of each pixel are calculated, forming a large set of points that outline the 3D contours of the user's face and body. The initial point cloud data retains detailed 3D information about the user's appearance. The color values ​​of the multi-view RGB images are mapped to the initial point cloud data; through image registration algorithms, the RGB images from different perspectives are accurately aligned with the point cloud data, determining the RGB value corresponding to each point cloud point, giving the point cloud color attributes, and generating point cloud data with rich color information, providing a realistic visual effect for the digital human model.

[0088] A high frame rate color camera is used to continuously capture user behavior, acquiring temporal dynamic data. The high frame rate ensures the capture of rapid movements and facial expressions, forming a coherent sequence of actions. Keyframes are extracted from the video, reducing data volume while preserving key nodes of actions and expressions. The temporal dynamic data is analyzed to extract dynamic behavioral features. Optical flow is used to calculate the motion vectors of adjacent frame pixels, obtaining the direction and velocity of motion. Combined with a human pose estimation algorithm, changes in limb posture are identified, generating dynamic behavioral data that describes the user's behavioral patterns and action details.

[0089] Simultaneous Localization and Mapping (SLAM) technology is used on augmented reality devices to track the device's spatial position and orientation in real time, constructing a 3D map of the environment to provide a basis for digital human positioning and environmental interaction. Simultaneously, environmental sensors measure ambient light intensity and color parameters, providing lighting information for rendering. Environmental perception data, including spatial position, orientation, and lighting parameters, is generated based on the information acquired from SLAM technology and environmental sensors. This data is then fused with visual depth data and dynamic behavior data to obtain multimodal data containing user appearance, behavior, and environmental information, providing comprehensive data support for generating realistic 3D Gaussian digital human models.

[0090] Visual depth data, dynamic behavior data, and environmental perception data are synchronized in time and registered spatially to ensure alignment within the same temporal and spatial reference frame. This allows the generation and rendering of the digital human model to accurately reflect the user's actual state and environment. A multi-layered fusion architecture is employed. At the feature layer, the geometric features of visual depth data are combined with the motion features of dynamic behavior data. At the decision layer, illumination and spatial information from environmental perception data are fused to generate multimodal data that comprehensively considers geometric, motion, and environmental factors, improving data integrity and consistency. During the fusion process, the data is optimized and enhanced. Filtering algorithms remove noise and improve data quality; data augmentation methods (such as interpolation) supplement detailed information, enriching data content and further enhancing the quality and usability of the multimodal data.

[0091] In this optional embodiment, multimodal data fusion integrates visual, dynamic behavior, and environmental perception information to provide a more comprehensive and accurate data foundation for digital human generation, making the model more realistic and natural in appearance, behavior, and environmental interaction. The various data acquisition methods complement each other: multi-view RGB images and depth information generate precise point cloud data; dynamic behavior data endows the model with realistic movements and expressions; and environmental perception data enhances environmental adaptability. The combined effect results in a more accurate and visually appealing digital human model. Accurate environmental perception data allows the digital human model to seamlessly integrate with the real environment, improving the immersion and realism of augmented reality applications. Multimodal data fusion enhances the system's robustness to data loss or noise. When a piece of data is incomplete or interfered with, the system can use other data to supplement and correct it, ensuring the stability and reliability of digital human generation.

[0092] Optionally, the rendering adjustment unit is specifically used for:

[0093] The user's gaze point is determined using eye-tracking technology;

[0094] The user's gaze area is determined by expanding the gaze point according to a preset field of view rule.

[0095] Determine the user's non-focused area based on the user's gaze area;

[0096] The fogging region and the non-fogging region are assigned corresponding rendering levels, wherein the rendering level of the fogging region is higher than that of the non-fogging region;

[0097] The rendering strategy is generated based on the rendering levels of the foveated region and the non-foveated region.

[0098] Specifically, the eye-tracking component is installed inside the AR glasses. Using infrared eye tracking, it emits infrared light and captures the reflected light from the user's eyes, acquiring the eye movement trajectory and gaze coordinates in real time. This accurately reflects the user's visual focus, providing a basis for subsequent differentiated rendering. The system expands the gaze range based on the gaze coordinates and preset field-of-view rules (such as a fan-shaped or circular area centered on the gaze point with a specific angle and radius), comprehensively considering the peripheral vision perception capabilities of the human eye to ensure coverage of the user's visual focus and provide complete visual information. Furthermore, on the digital human display screen, the gaze area is marked as the focus area, and the remaining areas are non-gaze areas, thus dividing the screen area and clearly defining the range of different rendering levels. A high rendering level is assigned to the gaze area, employing high-resolution textures, complex shaders, and detailed lighting calculations to ensure a delicate and realistic digital human display at the user's attention point; a low rendering level is assigned to the non-gaze area, appropriately reducing texture details and simplifying lighting processing, minimizing computational resource consumption while maintaining overall visual quality.

[0099] Simultaneously, considering factors such as the rendering levels of both foveated and non-foveated regions, device computing power parameters, and multimodal data characteristics, an optimal rendering strategy is formulated. This strategy clearly defines the rendering sequence, resource allocation ratios, and rendering parameters for each region, providing guidance for subsequent rendering processes.

[0100] In this optional embodiment, different rendering levels are assigned to the gazing region and the non-gazing region to achieve a reasonable allocation of rendering resources. Computational resources are concentrated on the gazing region that the user is focused on, performing high-precision rendering, while the non-gazing region uses simplified rendering. This improves resource allocation efficiency, enhances rendering efficiency, reduces stuttering, and provides a smoother interactive experience. High-precision rendering of the gazing region ensures that the digital human displayed in the user's focus area is rich in detail and realistic, improving the user's visual experience. For example, the detailed presentation of facial expressions and hand movements makes the digital human more vivid and natural, enhancing the user's immersion in augmented reality applications. The rendering strategy is matched to the device's performance by comprehensively considering the real-time computing power parameters of the augmented reality device. When the device's computing power is limited, the rendering precision of the non-gazing region is reduced to ensure the smooth completion of the overall rendering task, prevent device overload, and ensure stable application operation.

[0101] Optionally, the driving unit is specifically used for:

[0102] The computing power mode of the augmented reality device is determined based on the relationship between the real-time computing power parameters of the augmented reality device and the preset computing power threshold.

[0103] Based on the computing power mode, the number of Gaussian points and the rendering resolution of the digital human model are determined, and the number of Gaussian points and the rendering resolution are used as the rendering specifications.

[0104] Specifically, the device's built-in performance monitoring tools acquire computing power parameters in real time, including CPU utilization, GPU load, video memory usage, and memory bandwidth usage. These parameters comprehensively reflect the device's current operating status and available computing power resources. The acquired real-time computing power parameters are compared item by item with pre-set computing power thresholds. The deviation of each parameter from the threshold is calculated, and the deviation is comprehensively evaluated based on preset weighting coefficients. A comparison with preset computing power settings ultimately determines the device's computing power mode: high-performance mode, standard mode, or low-power mode. Preset ranges for the number of Gaussian points are defined for different computing power modes. In high-performance mode, the device has sufficient computing power to process a high number of Gaussian points, generating high-fidelity, detailed digital human models. In standard mode, the number of Gaussian points is appropriately reduced to balance rendering effects and performance. In low-power mode, the number of Gaussian points is significantly reduced to simplify the model and ensure smooth operation. The preset rendering resolution corresponds to the computing power mode. The high-performance mode supports high resolution (such as 4K) to ensure image detail and clarity; the standard mode uses medium resolution (such as 1080p) to provide better visual effects; and the low-power mode selects a lower resolution (such as 720p) to reduce computing power and memory usage and improve rendering efficiency.

[0105] In a preferred embodiment of the present invention, real-time computing power parameters (such as floating-point operation capability and video memory bandwidth) are obtained through the GPU driver of the AR glasses, and three levels of rendering thresholds are set, including: high computing power mode (>1.5GFLOPS): using 100,000 Gaussian points, 2K resolution rendering; medium computing power mode (0.8-1.5GFLOPS): downsampling to 50,000 points, 1080P resolution; low computing power mode (<0.8GFLOPS): enabling Gaussian point clustering, using 20,000 points, 720P resolution. Gaze-based rendering optimization: the user's gaze area is determined through eye-tracking data, and full Gaussian rendering is applied to the 30° field of view around the gaze point. The number of Gaussian points in non-gaze areas is reduced by 30%, reducing the computational load.

[0106] In this optional embodiment, the computing power mode and rendering specifications are dynamically adjusted based on real-time computing power parameters to ensure efficient resource utilization of the device under different conditions. For example, when the device's computing power is insufficient, the number of Gaussian points is automatically reduced and the rendering resolution is lowered to avoid stuttering and latency, ensuring the smoothness of the digital human display; when the computing power is sufficient, a high-precision mode is enabled to improve display quality and provide users with a better visual experience.

[0107] This embodiment also considers the performance differences of various augmented reality devices, employing an adaptive adjustment mechanism to ensure its widespread applicability across multiple devices. Whether it's a high-performance AR headset or a lower-spec mobile AR device, it can find suitable rendering specifications based on its own computing power, ensuring the compatibility and scalability of the 3D Gaussian digital human application. This helps promote the technology's widespread adoption and application across different devices. While maintaining display quality, by reasonably reducing the number of Gaussian points and rendering resolution, it significantly reduces computational load and memory usage, substantially lowers power consumption, and effectively extends battery life, providing users with a longer-lasting user experience. This is particularly suitable for mobile devices, meeting users' needs for extended use.

[0108] Optionally, the driving unit is specifically used for:

[0109] The multimodal data is extracted using a lightweight speech encoder to obtain Mel-frequency features and fundamental frequency information, and the speech features are determined based on the Mel-frequency features and fundamental frequency information.

[0110] The multimodal data is extracted using MobileFaceNet to obtain facial key points and FACS expression coefficients, and the expression features are determined based on the facial key points and the FACS expression coefficients.

[0111] The multimodal data is processed to obtain the limb skeleton rotation matrix, and then the posture features are determined based on the limb skeleton rotation matrix.

[0112] The speech features, facial expression features, and posture features are fused to obtain a fused feature vector, and the fused feature vector is input into the emotion fusion network to obtain the emotion state vector of the multimodal data. Then, the facial driving parameters are obtained based on the emotion state vector.

[0113] The joint operating angles of the digital human model are determined based on the posture features using an inverse kinematics algorithm, and then the limb driving parameters are obtained based on the joint operating angles.

[0114] Specifically, speech features primarily originate from dynamic behavioral data, which includes the user's speech signal. A lightweight speech encoder processes this signal to extract Mel-frequency spectral features and fundamental frequency information, thereby determining the speech features. The lightweight speech encoder processes the speech signal from multimodal data, extracting Mel-frequency spectral features and fundamental frequency information. Mel-frequency spectral features reflect the energy distribution of speech across different frequency bands and are closely related to timbre, loudness, etc.; fundamental frequency information corresponds to the vocal cord vibration frequency, determining pitch. The combination of these two features comprehensively describes the speech characteristics, providing crucial data for subsequent generation. Based on the Mel-frequency spectral features and fundamental frequency information, speech processing algorithms are used to determine the speech feature vector, quantifying attributes such as timbre, pitch, and speech rate of the speech signal. For example, the MFCC algorithm is used to calculate the cepstral coefficients of the Mel-frequency spectrum, extracting key feature parameters of the speech signal to form a speech feature vector, which serves as the basis for generating lip movements and facial expression changes.

[0115] Facial expression features are primarily derived from visual depth data, which includes facial image information of the user. MobileFaceNet analyzes these images to extract facial keypoints and FACS expression coefficients, thereby determining facial expression features. MobileFaceNet extracts features from facial images in multimodal data. On one hand, it accurately locates facial keypoints (such as eyes, eyebrows, and mouth) using keypoint detection technology to determine their positional coordinates. On the other hand, it combines the Facial Expression Coding System (FACS) to calculate the intensity coefficients of facial muscle movements, i.e., FACS expression coefficients, quantifying the activation level of different expression units. Based on facial keypoints and FACS expression coefficients, an expression feature vector is determined. The expression feature vector describes subtle changes in facial expressions. For example, by analyzing changes in the position of keypoints such as the degree of eye squinting, the degree of eyebrow raising, and the angle of mouth raising, combined with the corresponding FACS expression coefficients, the current expression category (such as smiling, surprise, etc.) is determined, and a corresponding expression feature vector is generated to drive facial expression changes in the digital human model.

[0116] Posture features are also derived from visual depth data. By capturing and processing the user's limb movements, the rotation matrix of the limb skeleton is obtained, thereby determining the posture features. Dynamic behavioral data from multimodal datasets is processed to extract the limb skeleton rotation matrix. Human posture estimation algorithms (such as deep learning-based human posture estimation models) are used to identify the position and posture of the limb skeletons, calculating the rotation matrix of each skeleton. The rotation matrix describes the rotational state of the limb skeleton in three-dimensional space, reflecting the direction and angle of limb movement. Based on the limb skeleton rotation matrix, posture feature vectors are determined. These vectors describe the rotation angles and motion states of various joints in the body, driving the limb movements of the digital human model. For example, by analyzing the rotation matrices of joints such as the arms and legs, the corresponding rotation angles and motion trajectories are extracted to generate posture feature vectors, achieving natural simulation of limb movements.

[0117] Voice feature vectors, facial expression feature vectors, and posture feature vectors are fused to generate a fused feature vector. Feature concatenation or fusion algorithms are used to concatenate the three feature vectors dimensionally or fuse them in the feature space, forming a fused feature vector that comprehensively reflects the features of multimodal data. This fused feature vector integrates voice, facial expression, and posture information to more comprehensively describe the user's state and emotions. The fused feature vector is input into an emotion fusion network, which processes it through a multi-layer neural network structure to analyze the emotional information. The network learns the mapping relationship between voice, facial expression, and posture features and emotional states, outputting an emotion state vector that quantifies the user's current emotional state. The emotion state vector represents the user's emotions such as joy, anger, sorrow, and happiness, providing a basis for generating more emotionally expressive facial driving parameters. Based on the emotion state vector, facial driving parameters are generated through expression mapping rules and a driving model. Different emotional states correspond to specific facial expression patterns; for example, joy corresponds to smiling and squinting. The system adjusts the contraction and relaxation of facial muscles based on the value of the emotional state vector, generating corresponding facial driving parameters to drive changes in the facial expressions of the digital human model, making its expressions more closely resemble the user's emotional state and enhancing the emotional driving capability of expression generation. The system uses an inverse kinematics algorithm to calculate the joint rotation angles of the digital human model based on the posture feature vector. The inverse kinematics algorithm solves the problem of calculating the rotation angles of each joint point given the position and posture of the limb's end. Based on the joint point positions and motion trajectory information in the posture feature vector, the inverse kinematics algorithm solves for the rotation angle of each joint point to achieve the desired limb movement posture. Based on the calculated joint rotation angles, limb driving parameters are generated. These parameters guide the movement of the digital human model's limbs, controlling the rotation angle and movement speed of each joint point, thereby driving the natural and smooth display of limb movements. In a preferred embodiment of the present invention, facial driving can be based on the binding relationship between FLAME mesh and Gaussian points (each FLAME triangle corresponds to a Gaussian point), and the Gaussian position offset can be adjusted through emotion vectors to achieve wrinkle details at the 0.1mm level; limb driving can use inverse kinematics algorithm to calculate joint angles, and pass them to the limb Gaussian model through Gaussian-mesh binding technology, with a rotation error of <1 degree; at the same time, a cross-modal attention mechanism is introduced to calculate the time alignment factor of speech and posture, and control the multimodal latency to within 20ms.

[0118] In this optional embodiment, facial driving parameters are generated by fusing voice, facial expression, and posture features to drive changes in the digital human's facial expressions, making them more natural and aligned with the user's emotional state, thereby enhancing the digital human's emotional expression capabilities and strengthening the emotional resonance of the interaction. By generating emotional state vectors and driving facial expressions, the digital human can display rich emotions, making the interaction more interesting and vivid, and improving the user experience. Using inverse kinematics algorithms to generate limb driving parameters can simulate natural limb movements, enhancing the realism of the digital human in augmented reality and improving immersion. Leveraging lightweight voice encoders and models such as MobileFaceNet, along with efficient algorithms, features can be quickly extracted and driving parameters generated on low-computing-power devices, ensuring real-time system response and guaranteeing smooth interaction.

[0119] Optionally, the modeling unit is specifically used for:

[0120] The initial digital human model is updated based on the point cloud data using a mesh generation algorithm to generate the basic mesh of the digital human model.

[0121] Initialize one or more Gaussian points at each vertex of the base mesh, and establish a mapping relationship between the vertices of the base mesh and the Gaussian points;

[0122] Based on the rendering strategy and the rendering specifications, and in conjunction with the mapping relationship, the face and limbs of the digital human model are rendered respectively to obtain the rendered digital human model.

[0123] Specifically, a mesh generation algorithm is used to update the initial digital human model based on point cloud data, generating a basic mesh. The point cloud data contains a set of 3D points representing the user's appearance. The algorithm first preprocesses these points, removing noise and smoothing them. Then, a surface mesh of the 3D point cloud is constructed, forming a triangular or quadrilateral mesh structure. For example, the Delaunay triangulation algorithm is used to triangulate the point cloud to generate the initial mesh. Subsequently, the mesh structure is optimized, adjusting vertex positions and connectivity to improve mesh quality and detail, generating a basic mesh that accurately reflects the user's appearance. The vertex positions and connectivity of the basic mesh determine the geometry of the digital human model, providing a foundation for subsequent rendering.

[0124] One or more Gaussian points are initialized at each vertex of the base mesh. These Gaussian points contain parameters such as position, orientation, size, and color, used to describe the local appearance features of the base mesh vertex. The number and parameters of Gaussian points at each vertex are determined based on the vertex's geometry and appearance attributes. A Gaussian point is initialized at each vertex, its position is the same as the vertex's position, its orientation is determined by the vertex's normal direction, its size is determined by the vertex's local geometric features (such as curvature), and its color is assigned by the vertex's color attribute. The number and parameters of Gaussian points affect rendering quality and computational cost; proper initialization ensures rendering accuracy and efficiency. A mapping relationship between base mesh vertices and Gaussian points is established by assigning a unique identifier to each vertex and recording its corresponding vertex identifier during Gaussian point initialization. This mapping relationship is crucial for efficient rendering of the digital human model. During rendering, the corresponding Gaussian point is quickly located using the vertex's index for rendering calculations.

[0125] Based on the rendering strategy and specifications, and combined with the mapping relationship, the face and limbs of the digital human model are rendered separately. For the facial and limb parts in the gaze area, a high rendering level is used, and precise rendering is performed using detailed parameters of Gaussian points. The contribution of each Gaussian point to the pixel is calculated to simulate lighting and shadow effects and generate realistic images. For non-gaze areas, a low rendering level is used to simplify calculations and reduce resource consumption. Finally, the rendering results are integrated to obtain the complete rendered digital human model.

[0126] In a preferred embodiment of the present invention, a basic mesh is first generated based on point cloud data, namely a simplified topological structure of the human body or object. The simplified topological structure can be a human body skinned mesh with skeleton, defining vertices, bone weights, and topological connections as a global framework for non-rigid deformation. Then, based on the FLAME mesh, a 3D Gaussian is initialized at the centroid of each triangle, establishing a projection mapping relationship φ:V→A between mesh vertices and Gaussian centers, where V is the set of vertices and A is the set of Gaussian centers. The digital human's motion is decomposed into rigid transformations (joint rotations) and non-rigid details (skin wrinkles). The rigid parts are processed through mesh skinning, and the non-rigid details are predicted by Gaussian parameter offsets through an MLP network to ensure that the non-rigid deformation conforms to the motion law of the human body or object and avoids unnatural distortion. In areas such as joint bending and muscle contraction, the radius and density of the Gaussian are dynamically adjusted through predefined rules or machine learning models. For example, when the elbow is bent, the inner Gaussian radius is reduced to simulate compression. Simultaneously, temporal filtering is applied to the Gaussian parameters of consecutive frames to avoid jitter during skin wrinkling deformation; subtle deformations that are difficult to represent with traditional meshes are captured, and the dynamic realism of the surface is enhanced through dynamic adjustment of Gaussian parameters. Finally, physical properties are added to the mesh and Gaussian model to ensure that non-rigid deformations conform to physical laws. The Gaussian model can be used as a lightweight collider to perform collision detection with the environment or its own mesh (such as the collision between clothing and the body). During collision, the position or radius of the Gaussian model is dynamically adjusted to avoid penetration.

[0127] In this optional embodiment, the geometric structure and appearance features of the digital human model are accurately modeled through mesh generation algorithms and Gaussian point initialization. Simultaneously, efficient rendering is achieved using mapping relationships and rendering strategies, reducing computational load and memory consumption, improving rendering efficiency, and ensuring smooth display. The initialization and mapping relationships of Gaussian points at the vertices of the basic mesh enable the rendering to finely represent the model's local details and appearance features, such as facial expressions and body movements, enhancing realism and immersion. Rendering is performed in conjunction with rendering strategies and specifications, ensuring display quality in critical areas while reducing resource consumption in non-critical areas, optimizing overall resource allocation, and improving system operating efficiency.

[0128] Optionally, the modeling unit is further used for:

[0129] Assign corresponding motion amplitude weights to the facial expression rendering and the motion rendering respectively, wherein the Gaussian dominant weight in the motion amplitude weight of the facial expression rendering is greater than the mesh dominant weight in the motion amplitude weight, and the mesh dominant weight in the motion amplitude weight of the motion rendering is greater than the Gaussian dominant weight in the motion rendering.

[0130] Based on the facial driving parameters and the limb driving parameters, and according to the motion amplitude weights of the facial expression rendering and the motion amplitude weights of the motion rendering, the vertex positions and Gaussian point parameters of the facial region in the base mesh are adjusted respectively to obtain the rendered digital human model.

[0131] Specifically, the contribution weights of the mesh and Gaussian transforms are automatically adjusted based on the amplitude of the movement. Small facial expressions (such as blinking) are primarily driven by Gaussian detail, while large movements (such as waving) are dominated by mesh rigid transformations, ensuring natural deformation. In facial expression rendering, Gaussian points are more advantageous in subtly expressing changes in facial expressions, such as better simulating micro-expressions of the skin and subtle movements of muscles. Therefore, the Gaussian dominance weight is allocated greater than the mesh dominance weight, allowing Gaussian points to play a greater role in facial expression rendering and ensuring the subtlety of facial expressions. In motion rendering, mesh vertices are more critical for representing large limb movements. Therefore, the mesh dominance weight is allocated greater than the Gaussian dominance weight, allowing the base mesh to play a dominant role in rendering limb movements and ensuring the smoothness and naturalness of limb movements.

[0132] The facial driving parameters adjust the vertex positions and Gaussian point parameters of the facial region. These parameters guide the modification of vertex positions; for example, when the facial expression is a smile, the relevant parameters trigger the movement of the corners of the mouth and eye vertices to form a smiling shape. Simultaneously, the direction, size, and color parameters of the Gaussian points are adjusted to enhance the realism of the expression. The limb driving parameters determine the rotation angles and motion trajectories of the limb joints, guiding the adjustment of vertex positions to simulate natural limb movements. For example, the arm-raising motion is determined by the rotation angles of the shoulder and elbow joints; the driving parameters adjust the vertex positions of the shoulder and elbow accordingly, making the arm movement natural and smooth. During the adjustment process, motion amplitude weights are integrated, allowing the Gaussian points and mesh vertices to work synergistically in expression and motion rendering, generating a natural and coherent rendered digital human model.

[0133] In summary, combining Figure 4 As shown, in a preferred embodiment of the invention, the FLAME mesh is initialized with Gaussian points through the centroid of a triangle, establishing a binding relationship φ: V→A. The weight control module allocates weights according to the amplitude of the movement: Gaussian dominance weight is 0.8 for small expressions, and mesh dominance weight is 0.9 for large movements. The binding relationship is divided into rigid transformation and non-rigid deformation. Rigid transformation is transmitted to the limb joints through weight allocation and connected to the joint skin, used to handle limb changes during large movements. Non-rigid deformation is used to capture subtle deformation details, predicting parameter offsets through MLP to generate skin wrinkles, thereby simulating subtle changes such as facial expressions.

[0134] In this optional embodiment, by reasonably allocating the weights of motion amplitude, the advantages of Gaussian points and the basic mesh are fully utilized, making the rendering of facial expressions and body movements more accurate and natural. For example, Gaussian-dominated expression rendering can delicately present changes in facial muscles, while mesh-dominated motion rendering ensures smooth body movements, significantly improving the realism and visual effect of the digital human model. The weight-based parameter adjustment method in this embodiment allows the digital human model to flexibly adjust the rendering method under different scenarios and needs, adapting to various application scenarios and improving the model's versatility and practicality. Allocating motion amplitude weights concentrates computational resources on key parts, namely, Gaussian points for expression rendering and meshes for motion rendering, avoiding resource waste and improving computational efficiency. For example, in expression rendering, reducing complex calculations on the mesh and concentrating resources to optimize Gaussian point parameters speeds up rendering and reduces system load.

[0135] Optionally, a decompression unit may also be included;

[0136] The decompression unit is used to semantically group the initial digital human model to obtain multiple components of the initial digital human model, and to extract the Gaussian parameter matrix of each component.

[0137] By selecting multiple anchor points through sampling at the farthest point, the Gaussian parameter matrix of each component is compressed to the anchor point to obtain the compressed model of the initial digital human model;

[0138] When the digital human model needs to be rendered, the decoding accuracy is determined according to the real-time computing power parameters of the augmented reality device; the compressed model is decompressed according to the decoding accuracy to obtain the initial digital human model.

[0139] Specifically, based on the semantic information of the initial digital human model (such as head, arms, torso, etc.), it is decomposed into multiple parts; deep learning algorithms (such as segmentation networks) are used to automatically identify and group the semantic regions of the model; the semantic information of the model is usually stored in the model file in the form of labels or attributes. By reading this information, the model is segmented into independent parts, which facilitates subsequent processing and optimization. The Gaussian parameter matrix (containing parameters such as the position, orientation, size, and color of Gaussian points) is extracted for each part. The Gaussian parameter matrix is ​​a key data structure describing the surface appearance and geometric features of the part; for example, for the head part, its Gaussian parameter matrix contains detailed parameters of each Gaussian point on the face, used to render facial expressions and details.

[0140] Multiple anchor points are selected from the Gaussian parameter matrix of a component using a farthest-point sampling algorithm. The algorithm ensures that the anchor points are evenly distributed, preserving the component's geometric and appearance features to the greatest extent possible. For example, in a component's Gaussian point set, an initial point is selected as the anchor point, and then the points furthest from the selected anchor point are selected sequentially until a preset number of anchor points are reached. These anchor points are used for subsequent compression processing. The Gaussian parameter matrix of each component is compressed to the anchor points, generating a compressed model. By recording the average value or main eigenvalues ​​of the Gaussian parameters at each anchor point, key appearance features of the component are preserved. For example, for each Gaussian point surrounding an anchor point, the average values ​​of its position, orientation, and other parameters are calculated and used as the compression parameters for that anchor point. This method significantly reduces the amount of data, achieving efficient compression.

[0141] Before rendering the digital human model, the decoding precision is determined based on the real-time computing power parameters of the augmented reality device (such as CPU utilization and GPU load). When computing power is sufficient, a high decoding precision is selected to decompress more details and generate a high-fidelity model. When computing power is limited, the decoding precision is reduced to decrease the amount of computation and ensure smooth rendering. The compressed model is decompressed according to the decoding precision to restore the initial digital human model. Under high decoding precision, Gaussian point parameters between anchor points are restored through interpolation and other methods to generate a Gaussian parameter matrix close to the original. Under low decoding precision, only key parameters at anchor points are retained to reduce details and generate a simplified model.

[0142] In a preferred embodiment of the present invention, the digital human Gaussian model is first divided into semantic components such as face, limbs, and clothing, and the Gaussian parameter matrix P={position, color, opacity, covariance} of each component is extracted. For each semantic group, the ZPressor cross-attention mechanism is applied, and anchor points are selected through farthest point sampling, compressing Gaussian-supporting information into the anchor points, achieving a compression ratio of up to 20:1. Based on an attention-based decoding network, the decoding accuracy is dynamically adjusted according to the computing power of the AR glasses during decompression, for example, by combining... Figure 3 As shown, the initial Gaussian point cloud is first grouped into semantic components, dividing the digital human model into semantic components such as face, limbs, and clothing, and extracting the Gaussian parameter matrix for each component. Next, an information bottleneck compression technique is applied, using the ZPressor's cross-attention mechanism for each semantic group. Anchor Gaussian points are selected through farthest point sampling, compressing Gaussian-supporting information into the anchor points. Then, anchor point Gaussian selection further reduces the data volume. Through cross-attention encoding, the Gaussian parameter matrix of each component is encoded into the anchor points. Finally, through progressive decoding, the compressed model is obtained, reducing its size to 55MB while maintaining a PSNR ≥ 35dB. This effectively reduces storage requirements and optimizes the data format for subsequent processing. On a 2.3 GFLOPS computing device, a 1GB model can be compressed to 55MB while maintaining a PSNR ≥ 35dB.

[0143] In this optional embodiment, semantic grouping and farthest-point sampling compression effectively reduce the data volume of the initial digital human model. For example, the number of Gaussian points is compressed from millions to thousands, significantly reducing the model file size (e.g., from over 1GB to tens of MB), lowering storage requirements, and facilitating deployment and transmission on augmented reality devices with limited storage resources. Decoding precision is dynamically adjusted based on the device's real-time computing power to optimize rendering efficiency. When computing power is sufficient, high decoding precision generates a high-fidelity model, providing an excellent visual experience; when computing power is limited, low decoding precision generates a simplified model, reducing computation and memory usage, avoiding stuttering, and ensuring a smooth interactive experience. Farthest-point sampling uses anchor points and Gaussian parameter matrix compression methods to preserve key visual features of the model and reduce detail loss. During decompression, interpolation and other methods are used to restore details, ensuring that the model maintains good visual quality at different decoding precisions, meeting the needs of various application scenarios.

[0144] Combination Figure 2 As shown, the present invention provides a method for generating a three-dimensional Gaussian digital human, comprising:

[0145] Acquire user's visual depth data, dynamic behavior data, and environmental perception data; generate point cloud data and multimodal data based on the visual depth data, dynamic behavior data, and environmental perception data;

[0146] The user's gaze area is monitored in real time using eye-tracking technology, and a rendering strategy is determined based on the gaze area; the rendering specifications are determined based on the real-time computing power parameters of the user's augmented reality device.

[0147] Feature extraction is performed on the multimodal data to obtain multimodal features, wherein the multimodal features include speech features, facial expression features, and posture features; facial driving parameters and limb driving parameters of the digital human model are generated based on the speech features, facial expression features, and posture features, respectively.

[0148] Based on the point cloud data and the initial digital human model, a base mesh for the digital human model is generated. Then, based on the rendering strategy and the rendering specifications, and in conjunction with the facial driving parameters and the limb driving parameters, facial expression rendering and motion rendering are performed on the base mesh to obtain the rendered digital human model.

[0149] The present invention also provides an electronic device, including a memory and a processor;

[0150] The memory is used to store computer programs;

[0151] The processor is configured to implement the three-dimensional Gaussian digital human generation method described above when executing the computer program.

[0152] The computer-readable electronic device of the present invention has the same advantages over the prior art as the above-mentioned three-dimensional Gaussian digital human generation method over the prior art, and will not be repeated here.

[0153] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A three-dimensional Gaussian digital human generation system, characterized in that, include: The data acquisition unit is used to acquire the user's visual depth data, dynamic behavior data, and environmental perception data. Point cloud data and multimodal data are generated based on the visual depth data, the dynamic behavior data, and the environmental perception data. The rendering adjustment unit is used to monitor the user's gaze area in real time using eye-tracking technology, and determine the rendering strategy based on the gaze area; and determine the rendering specifications based on the real-time computing power parameters of the user's augmented reality device. A driving unit is configured to extract features from the multimodal data to obtain multimodal features, wherein the multimodal features include speech features, facial expression features, and posture features; and to generate facial driving parameters and limb driving parameters for the digital human model based on the speech features, facial expression features, and posture features, respectively; specifically including: The multimodal data is extracted using a lightweight speech encoder to obtain Mel-frequency features and fundamental frequency information, and the speech features are determined based on the Mel-frequency features and fundamental frequency information. The multimodal data is extracted using MobileFaceNet to obtain facial key points and FACS expression coefficients, and the expression features are determined based on the facial key points and the FACS expression coefficients. The multimodal data is processed to obtain the limb skeleton rotation matrix, and then the posture features are determined based on the limb skeleton rotation matrix. The speech features, facial expression features, and posture features are fused to obtain a fused feature vector, and the fused feature vector is input into the emotion fusion network to obtain the emotion state vector of the multimodal data. Then, the facial driving parameters are obtained based on the emotion state vector. The joint running angles of the digital human model are determined based on the posture features using an inverse kinematics algorithm, and the limb driving parameters are obtained based on the joint running angles. The modeling unit is used to generate a basic mesh of the digital human model based on the point cloud data and the initial digital human model, and to perform expression rendering and motion rendering on the basic mesh according to the rendering strategy and the rendering specifications, combined with the facial driving parameters and the limb driving parameters, to obtain the rendered digital human model; specifically, it includes: updating the initial digital human model based on the point cloud data through a mesh generation algorithm to generate the basic mesh of the digital human model; Initialize one or more Gaussian points at each vertex of the base mesh, and establish a mapping relationship between the vertices of the base mesh and the Gaussian points; Based on the rendering strategy and the rendering specifications, and in conjunction with the mapping relationship, the face and limbs of the digital human model are rendered respectively to obtain the rendered digital human model.

2. The three-dimensional Gaussian digital human generation system according to claim 1, characterized in that, The data acquisition unit is specifically used for: By acquiring the user's multi-view RGB image and depth information, the visual depth data is generated, and initial point cloud data is generated based on the visual depth data. The RGB values ​​in the multi-view RGB image are mapped to the initial point cloud data to obtain the point cloud data. The dynamic behavior data is generated by acquiring the user's time-series dynamic data; By using simultaneous localization and mapping (MRT) technology, the spatial location, orientation, and ambient lighting parameters of the user's augmented reality device are obtained, and the environmental perception data is generated based on the spatial location, orientation, and ambient lighting parameters. The visual depth data, the dynamic behavior data, and the environmental perception data are fused to obtain the multimodal data.

3. The three-dimensional Gaussian digital human generation system according to claim 1, characterized in that, The rendering adjustment unit is specifically used for: The user's gaze point is determined using eye-tracking technology; The user's gaze area is determined by expanding the gaze point according to a preset field of view rule. Determine the user's non-focused area based on the user's gaze area; The fogging region and the non-fogging region are assigned corresponding rendering levels, wherein the rendering level of the fogging region is higher than that of the non-fogging region; The rendering strategy is generated based on the rendering levels of the foveated region and the non-foveated region.

4. The three-dimensional Gaussian digital human generation system according to claim 1, characterized in that, The driving unit is specifically used for: The computing power mode of the augmented reality device is determined based on the relationship between the real-time computing power parameters of the augmented reality device and the preset computing power threshold. Based on the computing power mode, the number of Gaussian points and the rendering resolution of the digital human model are determined, and the number of Gaussian points and the rendering resolution are used as the rendering specifications.

5. The three-dimensional Gaussian digital human generation system according to claim 1, characterized in that, The modeling unit is further used for: Assign corresponding motion amplitude weights to the facial expression rendering and the motion rendering respectively, wherein the Gaussian dominant weight in the motion amplitude weight of the facial expression rendering is greater than the mesh dominant weight in the motion amplitude weight, and the mesh dominant weight in the motion amplitude weight of the motion rendering is greater than the Gaussian dominant weight in the motion rendering. Based on the facial driving parameters and the limb driving parameters, and according to the motion amplitude weights of the facial expression rendering and the motion amplitude weights of the motion rendering, the vertex positions and Gaussian point parameters of the facial region in the base mesh are adjusted respectively to obtain the rendered digital human model.

6. The three-dimensional Gaussian digital human generation system according to claim 1, characterized in that, It also includes a decompression unit; The decompression unit is used to semantically group the initial digital human model to obtain multiple components of the initial digital human model, and to extract the Gaussian parameter matrix of each component. By selecting multiple anchor points through sampling at the farthest point, the Gaussian parameter matrix of each component is compressed to the anchor point to obtain the compressed model of the initial digital human model; When the digital human model needs to be rendered, the decoding accuracy is determined according to the real-time computing power parameters of the augmented reality device; the compressed model is decompressed according to the decoding accuracy to obtain the initial digital human model.

7. A method for generating a three-dimensional Gaussian digital human, characterized in that, include: Acquire user visual depth data, dynamic behavior data, and environmental perception data; Point cloud data and multimodal data are generated based on the visual depth data, the dynamic behavior data, and the environmental perception data. The user's gaze area is monitored in real time using eye-tracking technology, and a rendering strategy is determined based on the gaze area; the rendering specifications are determined based on the real-time computing power parameters of the user's augmented reality device. Feature extraction is performed on the multimodal data to obtain multimodal features, wherein the multimodal features include speech features, facial expression features, and posture features; facial driving parameters and limb driving parameters of the digital human model are generated based on the speech features, facial expression features, and posture features, respectively; specifically including: The multimodal data is extracted using a lightweight speech encoder to obtain Mel-frequency features and fundamental frequency information, and the speech features are determined based on the Mel-frequency features and fundamental frequency information. The multimodal data is extracted using MobileFaceNet to obtain facial key points and FACS expression coefficients, and the expression features are determined based on the facial key points and the FACS expression coefficients. The multimodal data is processed to obtain the limb skeleton rotation matrix, and then the posture features are determined based on the limb skeleton rotation matrix. The speech features, facial expression features, and posture features are fused to obtain a fused feature vector, and the fused feature vector is input into the emotion fusion network to obtain the emotion state vector of the multimodal data. Then, the facial driving parameters are obtained based on the emotion state vector. The joint running angles of the digital human model are determined based on the posture features using an inverse kinematics algorithm, and the limb driving parameters are obtained based on the joint running angles. Based on the point cloud data and the initial digital human model, a basic mesh of the digital human model is generated. Then, based on the rendering strategy and the rendering specifications, and combined with the facial driving parameters and the limb driving parameters, the basic mesh is used for expression rendering and motion rendering to obtain the rendered digital human model. Specifically, this includes updating the initial digital human model based on the point cloud data using a mesh generation algorithm to generate the basic mesh of the digital human model. Initialize one or more Gaussian points at each vertex of the base mesh, and establish a mapping relationship between the vertices of the base mesh and the Gaussian points; Based on the rendering strategy and the rendering specifications, and in conjunction with the mapping relationship, the face and limbs of the digital human model are rendered respectively to obtain the rendered digital human model.

8. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement the three-dimensional Gaussian digital human generation method as described in claim 7 when executing the computer program.

Images