Three-dimensional Gaussian digital human generation system and method and electronic equipment

Abstract

The invention provides a three-dimensional Gaussian digital human generation system and method and electronic equipment, and relates to the technical field of augmented reality, and the system comprises a data obtaining unit which is used for obtaining visual depth data, dynamic behavior data and environmental perception data, and generating point cloud data and multi-modal data according to the visual depth data, the dynamic behavior data and the environmental perception data; the rendering adjustment unit is used for monitoring the fixation area in real time and determining a rendering strategy; according to the real-time computing power parameter, determining a rendering specification; the driving unit is used for performing feature extraction on the multi-modal data to obtain multi-modal features and generating face driving parameters and limb driving parameters of the digital human model; and the modeling unit is used for generating a basic grid of the digital human model according to the point cloud data and the initial digital human model, and performing expression rendering and action rendering on the basic grid according to a rendering strategy and a rendering specification in combination with the face driving parameters and the limb driving parameters to obtain a rendered digital human model. The fluency, authenticity and naturalness of the display effect are improved.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and in particular to a three-dimensional Gaussian digital human generation system, method and electronic equipment. Background Art

[0002] With the development of augmented reality technology, three-dimensional digital humans are used in many fields as an important carrier for the interaction between virtual and reality. Among them, 3D Gaussian Splatting (3DGS) technology has been widely used in the field of digital human modeling due to its balanced advantages of high-fidelity rendering and real-time performance. 3DGS technology achieves high-precision facial modeling through multi-view geometric reconstruction, laying the foundation for the application of three-dimensional digital humans on augmented reality devices.

[0003] In related technologies, 3DGS technology requires large amounts of storage for digital human models. For example, processing a high-fidelity digital human requires 4 million Gaussian points, corresponding to a file size exceeding 1GB. Therefore, it is difficult to directly deploy on AR devices with limited storage resources, such as AR glasses. Furthermore, the low computing power of AR devices makes real-time rendering inefficient, resulting in noticeable lag and hindering the user's AR interactive experience. Summary of the Invention

[0004] The problem solved by the present invention is how to improve the applicability and display effect of a three-dimensional Gaussian digital human on a low-computing-power device.

[0005] To solve the above problems, the present invention provides a three-dimensional Gaussian digital human generation system, method and electronic equipment.

[0006] In a first aspect, a three-dimensional Gaussian digital human generation system of the present invention comprises: A data acquisition unit, configured to acquire visual depth data, dynamic behavior data, and environmental perception data of the user; and generate point cloud data and multimodal data based on the visual depth data, the dynamic behavior data, and the environmental perception data; A rendering adjustment unit, configured to monitor the user's gaze area in real time using eye tracking technology, and determine a rendering strategy based on the gaze area; and determine rendering specifications based on real-time computing power parameters of the user's augmented reality device; A driving unit is configured to extract features from the multimodal data to obtain multimodal features, wherein the multimodal features include voice features, expression features, and posture features; and to generate facial driving parameters and limb driving parameters of the digital human model based on the voice features, expression features, and posture features. A modeling unit is configured to generate a base mesh of the digital human model based on the point cloud data and the initial digital human model, and perform expression rendering and action rendering on the base mesh in combination with the facial drive parameters and the limb drive parameters according to the rendering strategy and the rendering specifications, to obtain the rendered digital human model.

[0007] Optionally, the data acquisition unit is specifically configured to: Generating the visual depth data by acquiring the multi-view RGB image and depth information of the user, generating initial point cloud data based on the visual depth data, and mapping the RGB values ​​in the multi-view RGB image to the initial point cloud data to obtain the point cloud data; Generating the dynamic behavior data by acquiring the user's time-series dynamic data; Acquiring the spatial position, spatial orientation, and ambient lighting parameters of the augmented reality device of the user through simultaneous positioning and mapping technology, and generating the environmental perception data based on the spatial position, the spatial orientation, and the ambient lighting parameters; The visual depth data, the dynamic behavior data, and the environmental perception data are fused to obtain the multimodal data.

[0008] Optionally, the rendering adjustment unit is specifically configured to: Determining the user's gaze point through eye tracking technology; Expanding the gaze point according to a preset field of view rule to determine the gaze area of ​​the user; Determining a non-gaze area of ​​the user according to the gaze area of ​​the user; assigning corresponding rendering levels to the attention area and the non-attention area, wherein the rendering level of the attention area is higher than the rendering level of the non-attention area; The rendering strategy is generated according to the rendering levels of the gaze area and the non-gaze area.

[0009] Optionally, the driving unit is specifically configured to: Determining a computing power mode of the augmented reality device according to a relationship between the real-time computing power parameter of the augmented reality device and a preset computing power threshold; According to the computing power mode, the number of Gaussian points and the rendering resolution of the digital human model are determined, and the number of Gaussian points and the rendering resolution are used as the rendering specifications.

[0010] Optionally, the driving unit is specifically configured to: Extracting the multimodal data using a lightweight speech encoder to obtain mel spectrum features and fundamental frequency information, and determining the speech features based on the mel spectrum features and the fundamental frequency information; Extracting the multimodal data through MobileFaceNet to obtain facial key points and FACS expression coefficients, and determining the expression features based on the facial key points and the FACS expression coefficients; Solving the multimodal data to obtain a limb bone rotation matrix, and then determining the posture feature based on the limb bone rotation matrix; fusing the speech features, the expression features, and the posture features to obtain a fused feature vector, inputting the fused feature vector into an emotion fusion network to obtain an emotion state vector of the multimodal data, and then obtaining the facial driving parameters based on the emotion state vector; The joint running angles of the digital human model are determined according to the posture features through an inverse kinematics algorithm, and the limb driving parameters are then obtained according to the joint running angles.

[0011] Optionally, the modeling unit is specifically configured to: Using a mesh generation algorithm, the initial digital human model is updated according to the point cloud data to generate the basic mesh of the digital human model; Initializing one or more Gaussian points at each vertex of the base mesh, and establishing a mapping relationship between the vertices of the base mesh and the Gaussian points; According to the rendering strategy and the rendering specification, combined with the mapping relationship, the face and limbs of the digital human model are rendered respectively to obtain the rendered digital human model.

[0012] Optionally, the modeling unit is further configured to: Assigning corresponding motion amplitude weights to the expression rendering and the motion rendering, respectively, wherein the Gaussian dominant weight in the motion amplitude weight of the expression rendering is greater than the grid dominant weight in the motion amplitude weight, and the grid dominant weight in the motion amplitude weight of the motion rendering is greater than the Gaussian dominant weight of the motion rendering; According to the facial driving parameters and the limb driving parameters, and in accordance with the motion amplitude weights of the expression rendering and the motion amplitude weights of the motion rendering, the vertex positions and Gaussian point parameters of the facial area in the basic mesh are adjusted respectively to obtain the rendered digital human model.

[0013] Optionally, a decompression unit is also included; The decompression unit is used to semantically group the initial digital human model to obtain multiple components of the initial digital human model, and extract each component to obtain a Gaussian parameter matrix of each component; Selecting multiple anchor points by farthest point sampling, compressing the Gaussian parameter matrix of each component to the anchor point, and obtaining a compressed model of the initial digital human model; When the digital human model needs to be rendered, the decoding accuracy is determined according to the real-time computing power parameters of the augmented reality device; the compressed model is decompressed according to the decoding accuracy to obtain the initial digital human model.

[0014] In a second aspect, the present invention provides a method for generating a three-dimensional Gaussian digital human, comprising: Acquire visual depth data, dynamic behavior data, and environmental perception data of the user; generate point cloud data and multimodal data based on the visual depth data, the dynamic behavior data, and the environmental perception data; The user's gaze area is monitored in real time through eye tracking technology, and a rendering strategy is determined based on the gaze area; and rendering specifications are determined based on the real-time computing power parameters of the user's augmented reality device; Performing feature extraction on the multimodal data to obtain multimodal features, wherein the multimodal features include voice features, expression features, and posture features; and generating facial drive parameters and limb drive parameters of the digital human model based on the voice features, the expression features, and the posture features, respectively; A base mesh of the digital human model is generated based on the point cloud data and the initial digital human model, and expression rendering and action rendering are performed on the base mesh in combination with the facial drive parameters and the limb drive parameters according to the rendering strategy and the rendering specifications to obtain the rendered digital human model.

[0015] In a third aspect, the present invention provides an electronic device comprising a memory and a processor; The memory is used to store computer programs; The processor is configured to implement the above-mentioned method for generating a three-dimensional Gaussian digital human when executing the computer program.

[0016] In the 3D Gaussian digital human generation system, method, and electronic device of the present invention, the data acquisition unit obtains the user's visual depth data, dynamic behavior data, and environmental perception data, and generates point cloud data and multimodal data based on this data, providing rich basic data for the subsequent generation and rendering of the digital human model. This ensures that the construction of the digital human has complete data support, enables the digital human to more realistically reflect the user's characteristics and environment, and lays the foundation for improving the display effect.

[0017] The rendering adjustment unit utilizes eye-tracking technology to monitor the user's gaze area in real time, determining corresponding rendering strategies based on the different gaze areas. This allows the digital human to be rendered with precision in key areas of user focus, ensuring display quality in critical areas. Rendering accuracy is appropriately reduced in areas where the user pays little or no attention, saving computing resources and avoiding unnecessary computational overhead. Furthermore, the rendering specifications are determined based on the AR device's real-time computing power. This allows the rendering complexity to be dynamically adjusted based on the device's actual capabilities, avoiding lags caused by insufficient computing power. This improves the smoothness of the digital human's display on the AR device and enhances the user's interactive experience.

[0018] The drive unit extracts features from multimodal data, obtaining multimodal features including speech, facial expressions, and posture. Based on these features, it generates facial and limb drive parameters for the digital human model. This enables the digital human to adapt and move according to the user's dynamic behavior, enhancing its interactivity and authenticity. This allows the digital human to simulate the user's speech and behavior in real time, making the display more natural and vivid.

[0019] The modeling unit generates a base mesh for the digital human model based on the point cloud data and the initial digital human model. Then, combining rendering strategies, rendering specifications, and facial and limb driver parameters, it performs expression and motion rendering on this base mesh, ultimately producing the rendered digital human model. For example, facial muscles and bone structure are adjusted based on the user's voice and facial expressions to render facial expressions that sync with the user's voice; limb joints and muscles are adjusted based on the user's posture characteristics to render corresponding motion poses. This completes the entire process from data acquisition to final model generation, ensuring that the visual effects of the digital human model both match the user's actual characteristics and adapt to the performance limitations of the augmented reality device, thereby improving display quality. The generation of the base mesh ensures the basic form of the digital human model. The integration of rendering strategies, specifications, and driver parameters allows the digital human to maintain visual quality while adapting to the characteristics of the augmented reality device and displaying expressions and movements that match the user's characteristics, thus comprehensively improving the display quality of the 3D Gaussian digital human.

[0020] In summary, through the collaborative work of various units, the problems of large storage capacity of three-dimensional Gaussian digital human models and poor display effects on low-computing power augmented reality devices have been effectively solved, the smoothness, authenticity and naturalness of the display effects have been improved, and the user's interactive experience in the augmented reality environment has been enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1This is a structural block diagram of a three-dimensional Gaussian digital human generation system according to an embodiment of the present invention; Figure 2 This is a flow chart of a method for generating a three-dimensional Gaussian digital human according to an embodiment of the present invention; Figure 3 A flow chart of a compression model of a compression unit according to an embodiment of the present invention; Figure 4 This is a flowchart of rendering a three-dimensional Gaussian digital human according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0023] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0024] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to"; the term "based on" means "based at least in part on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments"; the term "optionally" means "optional embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0025] It should be noted that the modifications of "one" and "multiple" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0026] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0027] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0028] Combine Figure 1 As shown, an embodiment of the present invention provides a three-dimensional Gaussian digital human generation system, including: The data acquisition unit is used to acquire the user's visual depth data, dynamic behavior data and environmental perception data; and generate point cloud data and multimodal data based on the visual depth data, the dynamic behavior data and the environmental perception data.

[0029] Specifically, visual depth data uses the AR glasses' built-in binocular cameras and structured light sensors to collect depth information about the user's face and surroundings, generating depth maps. These depth maps capture the user's facial contours, facial features, and spatial relationships from different perspectives, representing the three-dimensional form of the face. The system then utilizes multi-view geometry to fuse these multiple depth maps to construct a precise 3D point cloud, laying the foundation for digital human facial modeling. Each point in the cloud has x, y, and z coordinates. Dynamic behavioral data uses a high-frame-rate color camera to record the user's movements and capture color video. This video captures dynamic information such as the user's expressions and body movements, providing rich visual material for subsequent analysis. Human pose estimation algorithms (such as OpenPose) are then used to accurately identify the joints of the user's limbs from the video frames, generating pose feature vectors that reflect changes in human posture, including flexion, extension, rotation, and other motion characteristics. Environmental perception data uses the AR glasses' built-in environmental sensors (such as light sensors and spatial sensors) to collect information about the surrounding environment. Among them, the light sensor measures the intensity and color of ambient light, and provides the spatial sensor with structural information of the user's physical space, such as the size, layout, and obstructions of the space, and constructs a three-dimensional map of the environment through SLAM technology.

[0030] During the data fusion phase, the system integrates the depth map, color video, posture feature vectors, and three-dimensional map of the environment. First, the depth map and color video are aligned at the pixel level. Using an image registration algorithm, each depth point is mapped to the corresponding pixel of the color image, thereby assigning color attributes to the point cloud data and forming a three-dimensional point cloud with color information. Next, combined with the posture feature vectors, the limb movements in the dynamic behavior data are associated with the point cloud data, so that the point cloud not only contains static shapes and colors, but also reflects the movement trajectory and posture changes of the user's limbs. Finally, the environmental perception data is integrated into it. According to the three-dimensional map of the environment, the coordinate system of the point cloud data is adjusted to match the actual environment of the user, generating multimodal data associated with the environment, including multi-dimensional information such as visual appearance, dynamic behavior, and environmental context.

[0031] A rendering adjustment unit is used to monitor the user's gaze area in real time through eye tracking technology, and determine a rendering strategy based on the gaze area; and determine rendering specifications based on the real-time computing power parameters of the user's augmented reality device.

[0032] Specifically, the eye tracking component is installed inside the AR glasses' lenses, close to the user's eyes. It uses high-precision infrared eye tracking technology to capture eye movement and gaze position in real time by emitting infrared light and capturing the reflected light from the user's eyes. The system captures eye images at a certain frame rate (e.g., 60fps or higher) and uses an eye-tracking algorithm to accurately calculate the user's gaze position coordinates (x, y) within the digital human display area. This determines the part of the digital human the user is currently focusing on, such as the face, body, or background. The system then dynamically adjusts the priority of the gaze area based on factors such as gaze duration and frequency. If the user continuously gazes at the digital human's eyes, the system determines this as the user's focus and marks that area as a key rendering area. Conversely, if the user only briefly glances at a body part, the system lowers its priority, flexibly adapting to changes in the user's visual attention.

[0033] Based on the gaze area information, the system assigns different rendering levels (such as high, medium, and low). For the gaze area, a high rendering level is assigned, enabling high-resolution textures, complex shading models, and sophisticated lighting and shadow calculations to ensure that the digital human in the area the user is focused on appears delicate and realistic. For the non-gaze area, the rendering level is lowered, reducing texture detail and simplifying lighting and shadow processing, optimizing computing resource allocation and improving overall rendering efficiency. When the user's gaze shifts, the rendering level is quickly re-assigned and resources reallocated to ensure that the area the user is focused on always receives high-quality rendering effects, achieving an optimal balance between efficient utilization of rendering resources and display effects, and enhancing the user's visual experience.

[0034] The real-time computing power monitoring module is integrated into the underlying system layer. By closely connecting to the device's hardware driver, it reads the device's key computing power indicators (such as CPU usage, GPU memory usage, and temperature) in real time. It obtains data multiple times per second and analyzes its changing trends to accurately grasp the device's real-time computing power status. Based on the device's real-time computing power parameters, the system dynamically adjusts rendering specifications. When the device's computing power is sufficient (e.g., CPU usage below 60% and GPU memory usage below 70%), high rendering specifications are enabled, supporting 4K resolution, high frame rates (60fps+), and complex lighting and shadow effects to ensure optimal digital human display. When the device's computing power is insufficient (e.g., CPU usage exceeding 80% and GPU memory usage exceeding 90%), the system automatically switches to low rendering specifications, reducing the resolution to 1080p and the frame rate to 30fps, simplifying lighting and shadow effects, preventing lag, and ensuring a smooth user experience.

[0035] A driving unit is used to extract features from the multimodal data to obtain multimodal features, wherein the multimodal features include voice features, expression features, and posture features; and to generate facial driving parameters and limb driving parameters of the digital human model based on the voice features, the expression features, and the posture features.

[0036] Specifically, after acquiring the speech signal, the system preprocesses it to remove noise. It then uses algorithms such as Mel-Frequency Cepstral Coefficients (MFCC) to extract speech characteristics such as pitch, timbre, and speaking rate, generating a speech feature vector that provides a basis for driving facial expressions. Keypoint detection is performed on the facial image to identify the locations and changing trends of key points in areas such as the eyes, eyebrows, and mouth. Combined with an expression recognition algorithm, the system determines the current expression category (e.g., smile, frown), quantifies the intensity of the expression, and generates an expression feature vector that reflects the user's facial expressions in real time. Based on the pose estimation results, the position, angle, and velocity of each body joint are calculated to generate a pose feature vector that describes the user's limb movements and provides data support for limb movement driving. The speech feature vector is input into the speech-driven model to generate speech-synchronized lip shape and facial muscle movement parameters. Combining the expression feature vector with expression mapping rules, the corresponding expression driving parameters are generated to control the contraction and relaxation of facial muscles, achieving lip shape synchronization with speech and creating natural and rich expressions. Analyze the posture feature vector, use the kinematic model to calculate the rotation angle of the joint and the motion trajectory of the limb, generate the limb driving parameters, control the swing, bending and extension of the limb, and make the limb movement natural and smooth.

[0037] A modeling unit is configured to generate a base mesh of the digital human model based on the point cloud data and the initial digital human model, and perform expression rendering and action rendering on the base mesh in combination with the facial drive parameters and the limb drive parameters according to the rendering strategy and the rendering specifications, to obtain the rendered digital human model.

[0038] Specifically, the digital human model is a virtual projection of the user in a virtual space, enhancing the user's interactive experience. The modeling unit's role is to present the user's appearance and behavioral dynamics in the form of a digital human model within the augmented reality device. Specifically, a base mesh is constructed from point cloud data. First, point cloud preprocessing (such as denoising and filtering) removes noise points. Then, a Poisson surface reconstruction algorithm is used to fit the overall shape of the point cloud, generating an initial triangular mesh. Finally, the mesh's topology and geometric details are optimized, the triangle distribution is adjusted to ensure a uniform fit, and key areas (such as the eyes and mouth) are refined. This initially forms the three-dimensional base mesh of the digital human model, reflecting the user's static appearance. Expression and motion rendering are then performed on this base mesh. The rendering module receives the base mesh, rendering strategy, rendering specifications, and facial and limb driving parameters. It adjusts the mesh's vertex positions and shapes according to these driving parameters to simulate facial expressions and limb movements. At the same time, according to the rendering strategy and specifications, the rendering effect and performance are optimized: high-precision lighting and shadow and high-resolution textures are enabled in the gaze area to improve the detail performance; the precision is appropriately reduced in the non-gaze area to reduce the amount of calculation; according to the device computing power parameters, the rendering parameters (such as resolution, frame rate, special effect complexity) are dynamically adjusted to ensure smooth and stable display effects, and finally a natural and realistic rendered digital human model is generated.

[0039] In the 3D Gaussian digital human generation system of the present invention, the data acquisition unit obtains the user's visual depth data, dynamic behavior data, and environmental perception data, and generates point cloud data and multimodal data based on this data. This provides rich basic data for the subsequent generation and rendering of the digital human model, ensuring that the construction of the digital human has complete data support, enabling the digital human to more realistically reflect the user's characteristics and environment, and laying the foundation for improving the display effect.

[0040] The rendering adjustment unit utilizes eye-tracking technology to monitor the user's gaze area in real time, determining corresponding rendering strategies based on the different gaze areas. This allows the digital human to be rendered with precision in key areas of user focus, ensuring display quality in critical areas. Rendering accuracy is appropriately reduced in areas where the user pays little or no attention, saving computing resources and avoiding unnecessary computational overhead. Furthermore, the rendering specifications are determined based on the AR device's real-time computing power. This allows the rendering complexity to be dynamically adjusted based on the device's actual capabilities, avoiding lags caused by insufficient computing power. This improves the smoothness of the digital human's display on the AR device and enhances the user's interactive experience.

[0041] The drive unit extracts features from multimodal data, obtaining multimodal features including speech, facial expressions, and posture. Based on these features, it generates facial and limb drive parameters for the digital human model. This enables the digital human to adapt and move according to the user's dynamic behavior, enhancing its interactivity and authenticity. This allows the digital human to simulate the user's speech and behavior in real time, making the display more natural and vivid.

[0042] The modeling unit generates a base mesh for the digital human model based on the point cloud data and the initial digital human model. Then, combining rendering strategies, rendering specifications, and facial and limb driver parameters, it performs expression and motion rendering on this base mesh, ultimately producing the rendered digital human model. For example, facial muscles and bone structure are adjusted based on the user's voice and facial expressions to render facial expressions that sync with the user's voice; limb joints and muscles are adjusted based on the user's posture characteristics to render corresponding motion poses. This completes the entire process from data acquisition to final model generation, ensuring that the visual effects of the digital human model both match the user's actual characteristics and adapt to the performance limitations of the augmented reality device, thereby improving display quality. The generation of the base mesh ensures the basic form of the digital human model. The integration of rendering strategies, specifications, and driver parameters allows the digital human to maintain visual quality while adapting to the characteristics of the augmented reality device and displaying expressions and movements that match the user's characteristics, thus comprehensively improving the display quality of the 3D Gaussian digital human.

[0043] In summary, through the collaborative work of various units, the problems of large storage capacity of three-dimensional Gaussian digital human models and poor display effects on low-computing power augmented reality devices have been effectively solved, the smoothness, authenticity and naturalness of the display effects have been improved, and the user's interactive experience in the augmented reality environment has been enhanced.

[0044] Optionally, the data acquisition unit is specifically configured to: Generating the visual depth data by acquiring the multi-view RGB image and depth information of the user, generating initial point cloud data based on the visual depth data, and mapping the RGB values ​​in the multi-view RGB image to the initial point cloud data to obtain the point cloud data; Generating the dynamic behavior data by acquiring the user's time-series dynamic data; Acquiring the spatial position, spatial orientation, and ambient lighting parameters of the augmented reality device of the user through simultaneous positioning and mapping technology, and generating the environmental perception data based on the spatial position, the spatial orientation, and the ambient lighting parameters; The visual depth data, the dynamic behavior data, and the environmental perception data are fused to obtain the multimodal data.

[0045] Specifically, multiple RGB cameras and depth sensors are deployed around the user to capture images and depth maps from different perspectives, accurately capturing the user's facial details and spatial structure. The depth sensors simultaneously acquire depth information, ensuring that each RGB image pixel has a corresponding depth value reflecting its distance from the camera. The depth map data is converted into a three-dimensional point cloud. The spatial coordinates (x, y, z) of each pixel are calculated based on the depth value, forming a large collection of points that outline the three-dimensional contours of the user's face and body. The initial point cloud data retains detailed three-dimensional information about the user's appearance. The color values ​​of the multi-view RGB images are mapped to the initial point cloud data. Using an image registration algorithm, the RGB images from different perspectives are accurately aligned with the point cloud data, and the RGB value corresponding to each point in the point cloud is determined, giving the point cloud color attributes. This generates point cloud data with rich color information, providing realistic visual effects for the digital human model.

[0046] A high-frame-rate color camera continuously captures user behavior to acquire time-series dynamic data. The high frame rate ensures the capture of rapid movements and changes in expression, forming a coherent behavioral sequence. Keyframes are extracted from the video to reduce data volume while retaining key nodes of movement and expression. This time-series dynamic data is analyzed to extract dynamic behavior features. Optical flow is used to calculate motion vectors of pixels in adjacent frames to determine movement direction and speed. Human posture estimation algorithms are combined to identify changes in body posture and generate dynamic behavior data that describes the user's behavioral patterns and movement details.

[0047] Simultaneous Localization and Mapping (SLAM) technology, running on an augmented reality device, tracks the device's spatial position and orientation in real time, constructing a 3D map of the environment. This provides a basis for the digital human's positioning and interaction with the environment. Simultaneously, environmental sensors measure ambient light intensity and color parameters, providing lighting information for rendering. Based on this information from SLAM technology and environmental sensors, environmental perception data is generated, including spatial position, orientation, and lighting parameters. This data is then integrated with visual depth data and dynamic behavior data to generate multimodal data encompassing the user's appearance, behavior, and environment, providing comprehensive data support for generating realistic 3D Gaussian digital human models.

[0048] Visual depth data, dynamic behavior data, and environmental perception data are temporally synchronized and spatially registered to ensure alignment within the same temporal and spatial reference frames, enabling the generation and rendering of digital human models to accurately reflect the user's actual state and environment. A multi-layered fusion architecture is employed. At the feature layer, the geometric features of visual depth data are combined with the motion features of dynamic behavior data. At the decision layer, the lighting and spatial information of environmental perception data are integrated to generate multimodal data that comprehensively considers geometric, motion, and environmental factors, improving data integrity and consistency. During the fusion process, the data is optimized and enhanced. Filtering algorithms remove noise and improve data quality. Data enhancement methods (such as interpolation) supplement detailed information and enrich the data content, further improving the quality and usability of multimodal data.

[0049] In this optional embodiment, multimodal data fusion integrates visual, dynamic behavior, and environmental perception information to provide a more comprehensive and accurate data foundation for digital human generation, making the model more realistic and natural in appearance, behavior, and environmental interaction. The various data acquisition methods complement each other: multi-view RGB images and depth information generate precise point cloud data, dynamic behavior data imparts realistic movements and expressions to the model, and environmental perception data enhances environmental adaptability. The resulting digital human model, brought together, is more accurate and visually appealing. Accurate environmental perception data seamlessly integrates the digital human model with the real environment, enhancing the immersiveness and realism of augmented reality applications. Multimodal data fusion enhances the system's robustness to missing or noisy data. When a particular data point is incomplete or interfered with, the system can use other data to supplement and correct it, ensuring the stability and reliability of digital human generation.

[0050] Optionally, the rendering adjustment unit is specifically configured to: Determining the user's gaze point through eye tracking technology; Expanding the gaze point according to a preset field of view rule to determine the gaze area of ​​the user; Determining a non-gaze area of ​​the user according to the gaze area of ​​the user; assigning corresponding rendering levels to the attention area and the non-attention area, wherein the rendering level of the attention area is higher than the rendering level of the non-attention area; The rendering strategy is generated according to the rendering levels of the gaze area and the non-gaze area.

[0051] Specifically, the eye tracking component is installed inside the AR glasses and uses infrared eye tracking to emit infrared light and capture the light reflected from the user's eyes. This method captures the eye movement trajectory and gaze point coordinates in real time, accurately reflecting the user's visual focus and providing a basis for subsequent differentiated rendering. The system then expands the gaze range based on the gaze point coordinates and preset field of view rules (such as a sector or circular area centered on the gaze point with a specific angle and radius), taking into account the human eye's peripheral vision to ensure that the user's visual focus is covered and complete visual information is provided. On the digital human display screen, the gaze area is marked as the focus area, while the rest is the non-gaze area. This divides the screen area and clearly defines the range of areas with different rendering levels. The gaze area is assigned a high rendering level, using high-resolution textures, complex shaders, and sophisticated lighting and shading calculations to ensure the digital human display is detailed and realistic where the user is focused. The non-gaze area is assigned a low rendering level, with appropriately reduced texture detail and simplified lighting and shading processing, ensuring overall visual quality while reducing computing resource consumption.

[0052] At the same time, the optimal rendering strategy is developed by comprehensively considering multiple factors, including the rendering level of the gaze and non-gaze areas, device computing parameters, and multimodal data characteristics. The rendering strategy specifies the rendering order, resource allocation ratio, and rendering parameters for each area, providing guidance for subsequent rendering processing.

[0053] In this optional embodiment, by assigning different rendering levels to the gaze area and the non-gaze area, a reasonable allocation of rendering resources is achieved. Computing resources are concentrated on the gaze area that the user is paying attention to, and high-precision rendering is performed, while simplified rendering is used in the non-gaze area. Resource allocation efficiency is improved, rendering efficiency is improved, jamming is reduced, and the interactive experience is smoother. High-precision rendering of the gaze area ensures that the digital human in the area where the user is paying attention is rich in details and realistic, improving the user's visual experience. For example, the detailed presentation of facial expressions and hand movements makes the digital human more vivid and natural, enhancing the user's immersion in the augmented reality application. The real-time computing power parameters of the augmented reality device are comprehensively considered to match the rendering strategy with the device performance. When the computing power of the device is limited, by reducing the rendering accuracy of the non-gaze area, the smooth completion of the overall rendering task is ensured, the device is prevented from being overloaded, and the stable operation of the application is guaranteed.

[0054] Optionally, the driving unit is specifically configured to: Determining a computing power mode of the augmented reality device according to a relationship between the real-time computing power parameter of the augmented reality device and a preset computing power threshold; According to the computing power mode, the number of Gaussian points and the rendering resolution of the digital human model are determined, and the number of Gaussian points and the rendering resolution are used as the rendering specifications.

[0055] Specifically, computing power parameters, including CPU usage, GPU load, video memory occupancy, and memory bandwidth usage, are obtained in real time through the device's built-in performance monitoring tool. These parameters comprehensively reflect the device's current operating status and available computing power resources. The acquired real-time computing power parameters are compared item by item with the pre-set computing power threshold, and the degree of deviation of each parameter from the threshold is calculated. The degree of deviation of each parameter is comprehensively evaluated based on the preset weight coefficient, and the computing power mode of the device is finally determined to be high-performance mode, standard mode, or low-power mode by comparing with the preset computing power preset. The corresponding range of Gaussian point numbers under different computing power modes is preset. In high-performance mode, the device has sufficient computing power to process a high number of Gaussian points and generate high-fidelity, fine digital human models. In standard mode, the number of Gaussian points is appropriately reduced to balance rendering effects and performance. In low-power mode, the number of Gaussian points is greatly reduced to simplify the model and ensure smooth operation. The rendering resolution is preset to correspond to the computing power mode. The high-performance mode supports high resolution (such as 4K) to ensure image details and clarity; the standard mode uses medium resolution (such as 1080p) to provide better visual effects; the low-power mode selects a lower resolution (such as 720p) to reduce computing power and memory usage, and improve rendering efficiency.

[0056] In a preferred embodiment of the present invention, real-time computing parameters (such as floating-point computing power and video memory bandwidth) are obtained through the AR glasses' GPU driver, and three rendering thresholds are set. These include: High computing power mode (>1.5GFLOPS): using 100,000 Gaussian points, rendering at 2K resolution; Medium computing power mode (0.8-1.5GFLOPS): downsampling to 50,000 points, 1080P resolution; Low computing power mode (<0.8GFLOPS): enabling Gaussian point clustering, using 20,000 points, 720P resolution. Foveated rendering optimization: Eye tracking data is used to determine the user's gaze area, and full Gaussian rendering is applied to the 30° field of view surrounding the gaze point. The number of Gaussian points in the non-gaze area is reduced by 30%, reducing the computational load.

[0057] In this optional embodiment, the computing power mode and rendering specifications are dynamically adjusted based on real-time computing power parameters to ensure efficient resource utilization in different device states. For example, when the device's computing power is insufficient, the number of Gaussian points and rendering resolution are automatically reduced to avoid lags and delays, ensuring smooth display of the digital human. When computing power is sufficient, high-precision mode is enabled to improve display quality and provide users with a better visual experience.

[0058] At the same time, this embodiment takes into account the performance differences of different augmented reality devices and adopts an adaptive adjustment mechanism to enable it to be widely used in a variety of devices. Whether it is a high-performance AR headset or a low-configuration mobile AR device, it can find the appropriate rendering specifications based on its own computing power, ensuring the compatibility and scalability of the three-dimensional Gaussian digital human application, and helping to promote the popularization and application of technology on different devices. While ensuring the display effect, by reasonably reducing the number of Gaussian points and rendering resolution, the amount of calculation and memory usage are greatly reduced, power consumption is significantly reduced, and battery life is effectively extended, bringing users a more lasting user experience. It is particularly suitable for mobile devices and meets the needs of users for long-term use.

[0059] Optionally, the driving unit is specifically configured to: Extracting the multimodal data using a lightweight speech encoder to obtain mel spectrum features and fundamental frequency information, and determining the speech features based on the mel spectrum features and the fundamental frequency information; Extracting the multimodal data through MobileFaceNet to obtain facial key points and FACS expression coefficients, and determining the expression features based on the facial key points and the FACS expression coefficients; Solving the multimodal data to obtain a limb bone rotation matrix, and then determining the posture feature based on the limb bone rotation matrix; fusing the speech features, the expression features, and the posture features to obtain a fused feature vector, inputting the fused feature vector into an emotion fusion network to obtain an emotion state vector of the multimodal data, and then obtaining the facial driving parameters based on the emotion state vector; The joint running angles of the digital human model are determined according to the posture features through an inverse kinematics algorithm, and the limb driving parameters are then obtained according to the joint running angles.

[0060] Specifically, speech features are primarily derived from dynamic behavioral data, which includes the user's speech signal. A lightweight speech encoder processes the speech signal to extract mel-spectrographic features and fundamental frequency information, which are then used to determine speech features. The lightweight speech encoder processes the speech signal from multimodal data, extracting mel-spectrographic features and fundamental frequency information. Mel-spectrographic features reflect the energy distribution of speech across different frequency bands and are closely related to timbre, loudness, and other factors. The fundamental frequency corresponds to the frequency of vocal cord vibration and determines pitch. The combination of these two comprehensively describes speech characteristics, providing key data for subsequent generation. Based on these mel-spectrographic features and fundamental frequency information, a speech processing algorithm is applied to determine speech feature vectors, quantifying speech signal attributes such as timbre, intonation, and speaking rate. For example, the MFCC algorithm calculates the cepstral coefficients of the mel-spectrographic spectrum, extracting key characteristic parameters of the speech signal and forming a speech feature vector, which serves as the basis for generating changes in mouth shape and facial expression.

[0061] Expression features are primarily derived from visual depth data, which includes facial image information. MobileFaceNet analyzes these images to extract facial key points and FACS expression coefficients, thereby determining expression features. MobileFaceNet extracts features from multimodal facial images. Firstly, it uses key point detection technology to accurately locate facial key points (such as the eyes, eyebrows, and mouth) and determine their coordinates. Secondly, it uses the Facial Expression Coding System (FACS) to calculate the intensity coefficient of facial muscle movement, namely the FACS expression coefficient, to quantify the activation level of different expression units. Based on the facial key points and FACS expression coefficients, an expression feature vector is determined. The expression feature vector describes subtle changes in facial expression. For example, by analyzing the positional changes of key points such as the degree of eye squinting, the amplitude of eyebrow raising, and the angle of mouth corner lifting, combined with the corresponding FACS expression coefficients, the current expression category (such as smile, surprise) is determined and the corresponding expression feature vector is generated to drive the facial expression changes of the digital human model.

[0062] Posture features are also derived from visual depth data. By capturing and solving the user's limb movements, the limb skeleton rotation matrix is ​​obtained, which is then used to determine posture features. Dynamic behavior data in the multimodal data is solved to extract the limb skeleton rotation matrix. Using a human posture estimation algorithm (such as a deep learning-based human posture estimation model), the position and posture of the limb skeleton are identified, and the rotation matrix of each bone is calculated. The rotation matrix describes the rotation state of the limb skeleton in three-dimensional space, reflecting the direction and angle of movement of the limb. Based on the limb skeleton rotation matrix, a posture feature vector is determined. The posture feature vector describes the rotation angle and motion state of each joint of the body and is used to drive the limb movements of the digital human model. For example, by analyzing the rotation matrix of joints such as the arms and legs, the corresponding rotation angle and motion trajectory are extracted, and a posture feature vector is generated to achieve natural simulation of limb movements.

[0063] The speech feature vector, expression feature vector, and posture feature vector are fused to generate a fused feature vector. Using a feature concatenation or fusion algorithm, the three feature vectors are concatenated dimensionally or fused in feature space to form a fused feature vector that comprehensively reflects the characteristics of multimodal data. The fused feature vector integrates speech, expression, and posture information to more comprehensively describe the user's state and emotion. The fused feature vector is input into the emotion fusion network, which processes the fused feature vector using a multi-layer neural network structure to analyze the emotional information contained therein. The network learns the mapping relationship between speech, expression, and posture features and emotional state, and outputs an emotion state vector that quantifies the user's current emotional state. The emotion state vector represents the user's emotional information, such as joy, anger, sadness, and happiness, providing a basis for generating more expressive facial driving parameters. Based on the emotion state vector, facial driving parameters are generated using expression mapping rules and a driving model. Different emotional states correspond to specific facial expression patterns. For example, joy corresponds to facial features such as smiling and squinting. Based on the value of the emotional state vector, the system adjusts the contraction and relaxation of facial muscles, generates corresponding facial drive parameters, and drives the digital human model's facial expressions, making them more closely aligned with the user's emotional state and enhancing the emotional drive capability of expression generation. The inverse kinematics algorithm calculates the digital human model's joint operating angles based on the posture eigenvector. The inverse kinematics algorithm solves the problem of calculating the rotation angle of each joint given the position and posture of the limb end. Based on the joint point position and motion trajectory information in the posture eigenvector, the inverse kinematics algorithm solves the rotation angle of each joint to achieve the desired limb movement posture. Based on the calculated joint operating angles, the limb drive parameters are generated. The limb drive parameters guide the movement of the digital human model's limbs, controlling the rotation angle and movement speed of each joint, thereby driving the natural and smooth display of limb movements. In a preferred embodiment of the present invention, facial driving can be based on the binding relationship between the FLAME grid and the Gaussian points (each FLAME triangle corresponds to a Gaussian), and the Gaussian position offset can be adjusted through the emotion vector to achieve 0.1mm level wrinkle details; limb driving can use the inverse kinematics algorithm to calculate the joint angle, and pass it to the limb Gaussian model through the Gaussian-grid binding technology, with a rotation error of <1 degree; at the same time, a cross-modal attention mechanism is introduced to calculate the time alignment factor of speech and posture, and control the multimodal delay within 20ms.

[0064] In this optional embodiment, facial drive parameters are generated by fusing speech, expression, and posture features to drive the digital human's facial expressions, making them more natural and in line with the user's emotional state, improving the digital human's emotional expression ability, and enhancing the emotional resonance of the interaction. By generating emotional state vectors and driving facial expressions, the digital human can display rich emotions, making interactions more interesting and vivid, and improving the user experience. Using inverse kinematics algorithms to generate limb drive parameters can simulate natural limb movements, enhance the digital human's sense of reality in augmented reality, and improve immersion. With the help of lightweight speech encoders and models such as MobileFaceNet and efficient algorithms, features can be quickly extracted and drive parameters generated on low-computing power devices, ensuring real-time system response and smooth interaction.

[0065] Optionally, the modeling unit is specifically configured to: Using a mesh generation algorithm, the initial digital human model is updated according to the point cloud data to generate the basic mesh of the digital human model; Initializing one or more Gaussian points at each vertex of the base mesh, and establishing a mapping relationship between the vertices of the base mesh and the Gaussian points; According to the rendering strategy and the rendering specification, combined with the mapping relationship, the face and limbs of the digital human model are rendered respectively to obtain the rendered digital human model.

[0066] Specifically, a mesh generation algorithm is used to update the initial digital human model based on point cloud data to generate a base mesh. Point cloud data contains a collection of 3D points representing the user's appearance. The algorithm first preprocesses these points, removing noise and smoothing them. A surface mesh of the 3D point cloud is then constructed, forming a triangular or quadrilateral mesh structure. For example, the point cloud is triangulated using the Delaunay triangulation algorithm to generate an initial mesh. The mesh structure is then optimized, adjusting vertex positions and connectivity to improve mesh quality and detail, generating a base mesh that accurately reflects the user's appearance. The vertex positions and connectivity of the base mesh determine the geometric shape of the digital human model, providing the foundation for subsequent rendering.

[0067] One or more Gaussian points are initialized at each vertex of the base mesh. These Gaussian points contain parameters such as position, orientation, size, and color, describing the local appearance characteristics of the base mesh vertex. The number and parameters of Gaussian points at each vertex are determined based on the vertex's geometric and appearance properties. A Gaussian point is initialized at each vertex, with its position being the same as the vertex's position, its orientation determined by the vertex's normal, its size determined by the vertex's local geometric characteristics (such as curvature), and its color determined by the vertex's color attribute. The number and parameters of Gaussian points affect rendering quality and computational complexity. Proper initialization of Gaussian points ensures rendering accuracy and efficiency. A mapping relationship is established between base mesh vertices and Gaussian points. Each vertex is assigned a unique identifier and its corresponding vertex identifier is recorded during Gaussian point initialization. This association is crucial for efficient rendering of the digital human model. During rendering, the corresponding Gaussian point is quickly found using the vertex's index for rendering calculations.

[0068] Based on the rendering strategy and specifications, combined with the mapping relationship, the face and limbs of the digital human model are rendered separately. For the face and limbs in the gaze area, a high rendering level is used, using the detailed parameters of the Gaussian points for precise rendering, calculating the contribution of each Gaussian point to the pixel, simulating lighting and shadow effects, and generating realistic images. For the non-gaze areas, a low rendering level is used to simplify calculations and reduce resource consumption. Finally, the rendering results are integrated to obtain the complete rendered digital human model.

[0069] In a preferred embodiment of the present invention, a base mesh is first generated based on point cloud data, i.e., a simplified topological structure of a human body or object. The simplified topological structure can be a human skinned mesh with bones, defining vertices, bone weights, and topological connectivity as a global framework for non-rigid deformation. Then, based on the FLAME mesh, a 3D Gaussian is initialized at the centroid of each triangle, and a projection mapping relationship φ:V→A between the mesh vertices and the Gaussian centers is established, where V is the vertex set and A is the Gaussian center set. The digital human's movements are decomposed into rigid transformations (joint rotations) and non-rigid details (skin wrinkles). The rigid parts are processed through mesh skinning, while the non-rigid details are predicted using an MLP network to offset Gaussian parameters, ensuring that the non-rigid deformation conforms to the motion laws of the human body or object and avoids unnatural distortion. In areas such as joint bending and muscle contraction, the radius and density of the Gaussian are dynamically adjusted using predefined rules or machine learning models. For example, when the elbow is bent, the inner Gaussian radius decreases to simulate squeezing. At the same time, the Gaussian parameters of consecutive frames are temporally filtered to avoid jitter during skin wrinkle deformation. This captures subtle deformations that are difficult to represent with traditional meshes, and dynamic Gaussian parameter adjustment enhances the dynamic realism of the surface. Finally, physical properties are added to the mesh and Gaussian model to ensure that non-rigid deformations conform to physical laws. The Gaussian model can be used as a lightweight collision body to detect collisions with the environment or its own mesh (such as clothing colliding with the body). During collisions, the Gaussian position or radius is dynamically adjusted to prevent penetration.

[0070] In this optional embodiment, the geometric structure and appearance features of the digital human model are accurately modeled through a mesh generation algorithm and Gaussian point initialization. At the same time, mapping relationships and rendering strategies are used to achieve efficient rendering, reduce computational complexity and memory usage, improve rendering efficiency, and ensure smooth display. The initialization and mapping relationship of Gaussian points at the vertices of the base mesh enable the rendering to accurately represent the local details and appearance features of the model, such as facial expressions and body movements, enhancing realism and immersion. Rendering is performed in combination with rendering strategies and specifications, while ensuring the display quality of key areas, reducing resource consumption in non-key areas, optimizing overall resource allocation, and improving system operating efficiency.

[0071] Optionally, the modeling unit is further configured to: Assigning corresponding motion amplitude weights to the expression rendering and the motion rendering, respectively, wherein the Gaussian dominant weight in the motion amplitude weight of the expression rendering is greater than the grid dominant weight in the motion amplitude weight, and the grid dominant weight in the motion amplitude weight of the motion rendering is greater than the Gaussian dominant weight of the motion rendering; According to the facial driving parameters and the limb driving parameters, and in accordance with the motion amplitude weights of the expression rendering and the motion amplitude weights of the motion rendering, the vertex positions and Gaussian point parameters of the facial area in the basic mesh are adjusted respectively to obtain the rendered digital human model.

[0072] Specifically, the contribution weights of the mesh and Gaussian are automatically adjusted based on the amplitude of the movement. Small expressions (such as blinking) are primarily driven by Gaussian details, while large movements (such as waving) are dominated by the mesh's rigid transformation, ensuring natural deformation. In expression rendering, Gaussian points are more advantageous in expressing subtle changes in facial expression, for example, they can better simulate micro-expressions of the skin and subtle movements of muscles. Therefore, the Gaussian's dominant weight is assigned to be greater than the mesh's dominant weight, allowing Gaussian points to play a greater role in expression rendering and ensuring the subtlety of facial expressions. In motion rendering, mesh vertices are more critical for expressing large body movements. Therefore, the mesh's dominant weight is assigned to be greater than the Gaussian's dominant weight, allowing the base mesh to play a dominant role in rendering body movements, ensuring the smoothness and naturalness of body movements.

[0073] The vertex positions and Gaussian point parameters of the facial area are adjusted according to the facial driving parameters. The facial driving parameters guide the modification of the vertex positions. For example, when the facial expression is smiling, the relevant parameters trigger the movement of the corners of the mouth and the vertices of the eyes to form a smiling shape. At the same time, the direction, size and color parameters of the Gaussian points are adjusted to enhance the realism of the expression. The limb driving parameters determine the rotation angle and motion trajectory of the limb joints, and guide the adjustment of the vertex positions to simulate natural limb movements. For example, the arm raising action is determined by the rotation angle of the shoulder joint and the elbow joint. The driving parameters adjust the shoulder and elbow vertex positions accordingly to make the arm movement natural and smooth. During the adjustment process, the action amplitude weights are integrated to make the Gaussian points and mesh vertices work together in the expression and action rendering, generating a natural and coherent rendered digital human model.

[0074] In summary, combined with Figure 4 As shown, in a preferred embodiment of the present invention, the FLAME mesh initializes the Gaussian points through the triangle centroid to establish a binding relationship φ: V→A. The weight control module assigns weights according to the amplitude of the movement: the Gaussian dominant weight is 0.8 for small expressions, and the mesh dominant weight is 0.9 for large movements. The binding relationship is divided into rigid transformation and non-rigid deformation. The rigid transformation is transmitted to the limb joints through weight distribution and connected to the joint skin to process the limb changes of large movements. Non-rigid deformation is used to capture subtle deformation details, and the parameter offset is predicted by MLP to generate skin wrinkles, thereby simulating subtle changes such as facial expressions.

[0075] In this optional embodiment, by reasonably allocating action amplitude weights, the advantages of Gaussian points and basic grids are fully utilized, making the rendering of facial expressions and body movements more accurate and natural. For example, Gaussian-dominated expression rendering can delicately present facial muscle changes, while grid-dominated action rendering ensures smooth body movements, significantly improving the realism and visual effects of the digital human model. The weight-based parameter adjustment method of this embodiment enables the digital human model to flexibly adjust the rendering method under different scenarios and requirements, adapt to various application scenarios, and improve the versatility and practicality of the model. Allocating action amplitude weights concentrates computing resources on key parts, namely Gaussian points for expression rendering and grids for action rendering, avoiding resource waste and improving computing efficiency. For example, in expression rendering, complex calculations on the grid are reduced, resources are concentrated to optimize Gaussian point parameters, speeding up rendering speed and reducing system burden.

[0076] Optionally, a decompression unit is also included; The decompression unit is used to semantically group the initial digital human model to obtain multiple components of the initial digital human model, and extract each component to obtain a Gaussian parameter matrix of each component; Selecting multiple anchor points by farthest point sampling, compressing the Gaussian parameter matrix of each component to the anchor point, and obtaining a compressed model of the initial digital human model; When the digital human model needs to be rendered, the decoding accuracy is determined according to the real-time computing power parameters of the augmented reality device; the compressed model is decompressed according to the decoding accuracy to obtain the initial digital human model.

[0077] Specifically, based on the semantic information of the initial digital human model (such as the head, arms, torso, etc.), it is decomposed into multiple components; deep learning algorithms (such as segmentation networks) are used to automatically identify and group the semantic regions of the model; the semantic information of the model is usually stored in the form of labels or attributes in the model file. By reading this information, the model is divided into independent components for subsequent processing and optimization. The Gaussian parameter matrix of each component (including parameters such as the position, orientation, size, and color of the Gaussian points) is extracted. The Gaussian parameter matrix is ​​a key data structure that describes the surface appearance and geometric features of the component; for example, for the head component, its Gaussian parameter matrix contains detailed parameters for each Gaussian point on the face, which is used to render facial expressions and details.

[0078] Multiple anchor points are selected from the Gaussian parameter matrix of the component through the farthest point sampling algorithm; the algorithm ensures that the anchor points are evenly distributed, preserving the geometric and appearance features of the component to the greatest extent possible. For example, in the Gaussian point set of a component, a point is initially selected as the anchor point, and then the point farthest from the selected anchor point is selected in sequence until the preset number of anchor points is reached. These anchor points are used for subsequent compression processing. The Gaussian parameter matrix of each component is compressed to the anchor point to generate a compression model. By recording the average value or main eigenvalue of the Gaussian parameters at each anchor point, the key appearance features of the component are retained. For example, for each Gaussian point around the anchor point, the average value of its position, orientation and other parameters is calculated as the compression parameter of the anchor point. This method significantly reduces the amount of data and achieves efficient compression.

[0079] Before rendering the digital human model, the decoding accuracy is determined based on the real-time computing power parameters of the augmented reality device (such as CPU usage and GPU load). When computing power is sufficient, high decoding accuracy is selected to decompress more details and generate a high-fidelity model. When computing power is limited, decoding accuracy is reduced to reduce the amount of calculation and ensure smooth rendering. The compressed model is decompressed based on the decoding accuracy to restore the original digital human model. At high decoding accuracy, the Gaussian point parameters between the anchor points are restored through interpolation and other methods to generate a Gaussian parameter matrix close to the original. At low decoding accuracy, only the key parameters at the anchor points are retained, reducing details and generating a simplified model.

[0080] In a preferred embodiment of the present invention, the digital human Gaussian model is first divided into semantic components such as face, limbs, and clothing, and the Gaussian parameter matrix P = {position, color, opacity, covariance} of each component is extracted respectively; the cross-attention mechanism of ZPressor is applied to each semantic group, and the anchor point is selected by farthest point sampling, and the information supporting Gaussian is compressed into the anchor point, with a compression ratio of up to 20:1; the decoding network based on attention dynamically adjusts the decoding accuracy according to the computing power of AR glasses during decompression, for example, combined with Figure 3 As shown, the initial Gaussian point cloud is first grouped into semantic components, dividing the digital human model into semantic components such as face, limbs, and clothing. The Gaussian parameter matrices of each component are then extracted. Information bottleneck compression technology is then applied, using the cross-attention mechanism of ZPressor for each semantic group. Anchor Gaussian points are selected through farthest point sampling, compressing the supporting Gaussian information into the anchor points. Anchor Gaussian selection is then performed to further streamline the data volume. Cross-attention encoding is used to encode the Gaussian parameter matrix of each component into the anchor points. Finally, progressive decoding is performed to obtain the compressed model, reducing its size to 55MB while maintaining a PSNR ≥ 35dB. This effectively reduces storage requirements and optimizes the data format for subsequent processing. On a 2.3GFLOPS computing power device, a 1GB model can be compressed to 55MB while maintaining a PSNR ≥ 35dB.

[0081] In this optional embodiment, the amount of data in the initial digital human model is effectively reduced through semantic grouping and farthest point sampling compression. For example, by compressing the number of Gaussian points from millions to thousands, the model file size is significantly reduced (such as from more than 1GB to tens of MB), reducing storage requirements and facilitating deployment and transmission on augmented reality devices with limited storage resources. The decoding accuracy is dynamically adjusted according to the real-time computing power of the device to optimize rendering efficiency. When computing power is sufficient, high decoding accuracy generates a high-fidelity model to provide an excellent visual experience; when computing power is limited, low decoding accuracy generates a simplified model to reduce the amount of calculation and memory usage, avoid freezes, and ensure a smooth interactive experience. The farthest point sampling selects anchor points and Gaussian parameter matrix compression methods to retain the key visual features of the model and reduce detail loss. During decompression, details are restored through interpolation and other methods to ensure that the model maintains good visual quality at different decoding accuracies to meet the needs of various application scenarios.

[0082] Combine Figure 2 As shown, the present invention provides a method for generating a three-dimensional Gaussian digital human, comprising: Acquire visual depth data, dynamic behavior data, and environmental perception data of the user; generate point cloud data and multimodal data based on the visual depth data, the dynamic behavior data, and the environmental perception data; The user's gaze area is monitored in real time through eye tracking technology, and a rendering strategy is determined based on the gaze area; and rendering specifications are determined based on the real-time computing power parameters of the user's augmented reality device; Performing feature extraction on the multimodal data to obtain multimodal features, wherein the multimodal features include voice features, expression features, and posture features; and generating facial drive parameters and limb drive parameters of the digital human model based on the voice features, the expression features, and the posture features, respectively; A base mesh of the digital human model is generated based on the point cloud data and the initial digital human model, and expression rendering and action rendering are performed on the base mesh in combination with the facial drive parameters and the limb drive parameters according to the rendering strategy and the rendering specifications to obtain the rendered digital human model.

[0083] The present invention also provides an electronic device, comprising a memory and a processor; The memory is used to store computer programs; The processor is configured to implement the above-mentioned method for generating a three-dimensional Gaussian digital human when executing the computer program.

[0084] The advantages of the computer-readable electronic device of the present invention over the prior art are the same as the advantages of the above-mentioned three-dimensional Gaussian digital human generation method over the prior art, and will not be repeated here.

[0085] Although the present invention is disclosed as above, the scope of protection disclosed by the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A three-dimensional Gaussian digital human generation system, characterized in that: include: A data acquisition unit, used to acquire the user's visual depth data, dynamic behavior data, and environmental perception data; generating point cloud data and multimodal data based on the visual depth data, the dynamic behavior data, and the environmental perception data; A rendering adjustment unit, configured to monitor the user's gaze area in real time using eye tracking technology, and determine a rendering strategy based on the gaze area; and determine rendering specifications based on real-time computing power parameters of the user's augmented reality device; A driving unit is configured to extract features from the multimodal data to obtain multimodal features, wherein the multimodal features include voice features, expression features, and posture features; and to generate facial driving parameters and limb driving parameters of the digital human model based on the voice features, expression features, and posture features. A modeling unit is configured to generate a base mesh of the digital human model based on the point cloud data and the initial digital human model, and perform expression rendering and action rendering on the base mesh in combination with the facial drive parameters and the limb drive parameters according to the rendering strategy and the rendering specifications, to obtain the rendered digital human model.

2. The three-dimensional Gaussian digital human generation system according to claim 1, characterized in that: The data acquisition unit is specifically used to: Generating the visual depth data by acquiring the multi-view RGB image and depth information of the user, generating initial point cloud data based on the visual depth data, and mapping the RGB values ​​in the multi-view RGB image to the initial point cloud data to obtain the point cloud data; Generating the dynamic behavior data by acquiring the user's time-series dynamic data; Acquiring the spatial position, spatial orientation, and ambient lighting parameters of the augmented reality device of the user through simultaneous positioning and mapping technology, and generating the environmental perception data based on the spatial position, the spatial orientation, and the ambient lighting parameters; The visual depth data, the dynamic behavior data, and the environmental perception data are fused to obtain the multimodal data.

3. The three-dimensional Gaussian digital human generation system according to claim 1, characterized in that: The rendering adjustment unit is specifically used to: Determining the user's gaze point through eye tracking technology; Expanding the gaze point according to a preset field of view rule to determine the gaze area of ​​the user; Determining a non-gaze area of ​​the user according to the gaze area of ​​the user; assigning corresponding rendering levels to the attention area and the non-attention area, wherein the rendering level of the attention area is higher than the rendering level of the non-attention area; The rendering strategy is generated according to the rendering levels of the gaze area and the non-gaze area.

4. The three-dimensional Gaussian digital human generation system according to claim 1, characterized in that: The driving unit is specifically used for: Determining a computing power mode of the augmented reality device according to a relationship between the real-time computing power parameter of the augmented reality device and a preset computing power threshold; According to the computing power mode, the number of Gaussian points and the rendering resolution of the digital human model are determined, and the number of Gaussian points and the rendering resolution are used as the rendering specifications.

5. The three-dimensional Gaussian digital human generation system according to claim 1, characterized in that: The driving unit is specifically used for: Extracting the multimodal data using a lightweight speech encoder to obtain mel spectrum features and fundamental frequency information, and determining the speech features based on the mel spectrum features and the fundamental frequency information; Extracting the multimodal data through MobileFaceNet to obtain facial key points and FACS expression coefficients, and determining the expression features based on the facial key points and the FACS expression coefficients; Solving the multimodal data to obtain a limb bone rotation matrix, and then determining the posture feature based on the limb bone rotation matrix; fusing the speech features, the expression features, and the posture features to obtain a fused feature vector, inputting the fused feature vector into an emotion fusion network to obtain an emotion state vector of the multimodal data, and then obtaining the facial driving parameters based on the emotion state vector; The joint running angles of the digital human model are determined according to the posture features through an inverse kinematics algorithm, and the limb driving parameters are then obtained according to the joint running angles.

6. The three-dimensional Gaussian digital human generation system according to claim 1, characterized in that: The modeling unit is specifically used to: Using a mesh generation algorithm, the initial digital human model is updated according to the point cloud data to generate the basic mesh of the digital human model; Initializing one or more Gaussian points at each vertex of the base mesh, and establishing a mapping relationship between the vertices of the base mesh and the Gaussian points; According to the rendering strategy and the rendering specification, combined with the mapping relationship, the face and limbs of the digital human model are rendered respectively to obtain the rendered digital human model.

7. The three-dimensional Gaussian digital human generation system according to claim 6, characterized in that: The modeling unit is further configured to: Assigning corresponding motion amplitude weights to the expression rendering and the motion rendering, respectively, wherein the Gaussian dominant weight in the motion amplitude weight of the expression rendering is greater than the grid dominant weight in the motion amplitude weight, and the grid dominant weight in the motion amplitude weight of the motion rendering is greater than the Gaussian dominant weight of the motion rendering; According to the facial driving parameters and the limb driving parameters, and in accordance with the motion amplitude weights of the expression rendering and the motion amplitude weights of the motion rendering, the vertex positions and Gaussian point parameters of the facial area in the basic mesh are adjusted respectively to obtain the rendered digital human model.

8. The three-dimensional Gaussian digital human generation system according to claim 1, characterized in that: Also includes a decompression unit; The decompression unit is used to semantically group the initial digital human model to obtain multiple components of the initial digital human model, and extract each component to obtain a Gaussian parameter matrix of each component; Selecting multiple anchor points by farthest point sampling, compressing the Gaussian parameter matrix of each component to the anchor point, and obtaining a compressed model of the initial digital human model; When the digital human model needs to be rendered, the decoding accuracy is determined according to the real-time computing power parameters of the augmented reality device; the compressed model is decompressed according to the decoding accuracy to obtain the initial digital human model.

9. A method for generating a three-dimensional Gaussian digital human, characterized in that: include: Obtain user's visual depth data, dynamic behavior data, and environmental perception data; generating point cloud data and multimodal data based on the visual depth data, the dynamic behavior data, and the environmental perception data; The user's gaze area is monitored in real time through eye tracking technology, and a rendering strategy is determined based on the gaze area; and rendering specifications are determined based on the real-time computing power parameters of the user's augmented reality device; Performing feature extraction on the multimodal data to obtain multimodal features, wherein the multimodal features include voice features, expression features, and posture features; and generating facial drive parameters and limb drive parameters of the digital human model based on the voice features, the expression features, and the posture features, respectively; A base mesh of the digital human model is generated based on the point cloud data and the initial digital human model, and expression rendering and action rendering are performed on the base mesh in combination with the facial drive parameters and the limb drive parameters according to the rendering strategy and the rendering specifications to obtain the rendered digital human model.

10. An electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the three-dimensional Gaussian digital human generation method according to claim 9 when executing the computer program.

Images