A real-time modeling system based on three-dimensional gridding data of dynamic capture of facial expressions

Abstract

The application discloses a kind of three-dimensional gridding data real-time modeling system based on facial expression dynamic capture, it is related to three-dimensional modeling and computer vision technical field, the system includes: data acquisition and pretreatment module real-time acquisition face image sequence, output basic expression grid and regional complexity information;Modeling and enhancement module is based on regional complexity and real-time network state dynamic decision, carries out high-precision geometric enhancement processing to high-detail demand area, generates corresponding enhancement data;Self-adapting streaming module encapsulates streaming incremental data packet, dynamically schedules data transmission, ensures efficient low-latency transmission, each module constitutes closed-loop collaborative mechanism, reconstructs and outputs three-dimensional expression grid by real-time fusion rendering unit in user terminal.The application considers high-fidelity real-time modeling, low-latency processing and efficient transmission demand, reduces terminal computing power and bandwidth dependence, with good practicability and adaptability.

Description

Technical Field

[0001] This invention relates to the field of 3D modeling and computer vision technology, specifically to a real-time 3D mesh data modeling system based on dynamic capture of human facial expressions. Background Technology

[0002] Facial expressions are a core nonverbal carrier of human emotions and intentions, and their accurate digitization is a key technology driving the development of virtual reality (VR), augmented reality (AR), remote real-time communication, and digital human applications. In recent years, with the deep integration of artificial intelligence and 3D modeling technology, markerless facial expression capture via cameras has become a mainstream research direction. This technology aims to capture facial images using ordinary cameras and, using computer vision and deep learning algorithms, generate in real-time a sequence of 3D facial meshes that can drive virtual characters.

[0003] To achieve high-fidelity 3D facial expression reconstruction, researchers have proposed various approaches. One mainstream method is 4D (3D + temporal) scanning and reconstruction based on high-quality, dense camera arrays. For example, some existing technologies utilize compact, synchronous camera arrays to acquire data and employ progressive retopology techniques to generate dynamic facial mesh sequences with consistent topological structures and artist presets. This method can generate high-fidelity facial animations of film-quality standards, but it heavily relies on expensive and fixed acquisition equipment and computing power clusters, making it unsuitable for mobile, distributed, or consumer applications.

[0004] Another type of research focuses on achieving real-time or near real-time 3D modeling and simulation with limited computing resources. For example, patent publication number CN1299244C discloses a system and method for dynamic modeling and real-time simulation of 3D scenes. This system establishes a virtual scene model in a central processing unit (CPU) and uses a scene update unit to dynamically update and redraw the model to achieve real-time simulation of dynamic 3D scenes. Although this type of approach focuses on real-time performance, its modeling objects are usually macroscopic scenes. For highly dynamic and detailed objects such as facial expressions that require handling microscopic muscle movements, skin deformation, and extremely high data consistency, its general architecture is difficult to apply directly.

[0005] Existing technologies reveal a key challenge in the field of 3D dynamic facial expression modeling: how to simultaneously ensure high-fidelity detail in expression reconstruction, low-latency end-to-end data processing, and efficient data transmission under limited network bandwidth in distributed, networked application scenarios (such as mobile real-time communication and cloud rendering). Specifically, existing technologies, such as the solution proposed by ZeyuTian et al., prioritize extremely high precision at the expense of system portability and remote real-time capabilities; while general real-time modeling solutions, such as CN1299244C, struggle to meet the required level of detail and data consistency for facial expression reconstruction. More critically, existing system architectures often treat data acquisition, high-precision computation, and network transmission as separate modules, lacking global collaborative optimization from the source of perception to the end of transmission. This results in an inability to dynamically and intelligently balance detail quality, computational load, and transmission latency when mobile network bandwidth fluctuates, hindering the support for high-quality, low-latency remote real-time interactive applications.

[0006] Therefore, an innovative system architecture is needed to solve the problem of achieving high-fidelity real-time modeling, efficient encoding, and smooth transmission of 3D facial expression meshes under the constraints of dynamic network environments and limited terminal computing power. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a real-time modeling system for three-dimensional mesh data based on dynamic capture of facial expressions. Through a terminal-cloud collaborative architecture and closed-loop collaborative mechanism, combined with regional complexity assessment, dynamic computing power allocation and streaming incremental transmission technology, it can achieve real-time modeling, efficient transmission and seamless integration of high-fidelity three-dimensional mesh facial expressions in a limited terminal computing power and dynamic network environment. This solves the problem that it is difficult to balance high fidelity with portability, real-time performance and low bandwidth transmission in existing technologies.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a real-time modeling system for three-dimensional mesh data based on dynamic capture of facial expressions. The system comprises: a data acquisition and preprocessing module deployed on a user terminal, a modeling and enhancement module deployed on a cloud computing node, and an adaptive streaming transmission module connecting the user terminal and the cloud computing node. The data acquisition and preprocessing module is used to acquire and analyze facial image sequences in real time, and output basic expression grids and region complexity information based on expression dynamics. The modeling and enhancement module is used to receive image data of the high-detail region specified by the region complexity information and generate high-precision geometric enhancement data of the corresponding region. The adaptive streaming module is used to dynamically schedule the transmission of the image data and the high-precision geometric enhancement data between the user terminal and the cloud computing node according to the real-time network status, and to fuse the high-precision geometric enhancement data and the basic expression mesh in real time at the user terminal to output the final three-dimensional expression mesh sequence.

[0009] By constructing a collaborative mechanism that integrates data acquisition and preprocessing, modeling and enhancement, and adaptive streaming, real-time capture, high-fidelity reconstruction, and efficient transmission of facial expressions can be integrated into a unified framework. By dynamically scheduling cloud-edge computing and data streams based on facial expression dynamics and network status, end-to-end low-latency generation of high-precision 3D facial expression mesh sequences can be achieved with limited network bandwidth and terminal computing power.

[0010] Furthermore, the data acquisition and preprocessing module includes a perception and evaluation unit and a basic modeling unit; The perception and evaluation unit is used to perform feature point tracking and optical flow calculation on the acquired face image sequence, and generate the region complexity information that characterizes the intensity of movement and geometric change requirements of different parts of the face. The basic modeling unit stores a neutral expression reference mesh template and drives the neutral expression reference mesh template in real time according to the face image sequence to generate the basic expression mesh with consistent topology.

[0011] By deploying perception and evaluation units on the terminal, the complexity of facial expression changes in different regions can be quantified in real time, providing accurate decision-making basis for subsequent adaptive processing. By establishing and driving a unified neutral expression baseline mesh template, it is possible to ensure that the real-time generated base mesh has a stable topological structure, laying a geometric foundation for streaming coding and high-quality fusion.

[0012] Furthermore, the modeling and enhancement module includes a layered modeling engine and a high-fidelity refinement unit; The hierarchical modeling engine is used to receive regional complexity information and real-time network bandwidth information from the perception and evaluation unit, make decisions, and send image data marked as high detail requirement areas to the high-fidelity retouching unit. The high-fidelity refinement unit includes a model based on physical simulation or neural network super-resolution, used to calculate regions with high detail requirements and generate high-precision geometric enhancement data containing local geometric details or displacement information.

[0013] Among these features, the layered modeling engine deployed in the cloud can intelligently determine data processing and transmission strategies for high-detail areas based on the complexity reported by the terminal and network information. By using high-fidelity refinement units to process designated areas in the cloud, intensive detailed computation can be offloaded from the terminal, thereby generating local geometric details in the cloud that far exceed the computing power limits of the terminal.

[0014] Furthermore, the decision logic of the hierarchical modeling engine includes: determining facial regions with regional complexity information below a preset threshold as to be fully processed by the basic modeling unit; determining facial regions with regional complexity information above the preset threshold as high detail requirement regions, and, based on real-time network bandwidth information, selecting to send the original image patch of the high detail requirement region or the intermediate neural features extracted from the original image patch to the high-fidelity retouching unit.

[0015] By setting a regional complexity threshold to divide the processing path, precise allocation of computing resources can be achieved, avoiding unnecessary redundant calculations and transmissions in low-motion areas. By selecting to upload the original image or intermediate features based on real-time network bandwidth, the system can adaptively balance the input quality of cloud-reconstructed data with uplink bandwidth overhead under different network conditions.

[0016] Furthermore, the high-precision geometric enhancement data is differential information relative to a neutral expression reference mesh template. This differential information is defined for regions with high detail requirements and is in the format of local vertex displacement fields, detail normal map indexes, or mesh patch containing subdivision instructions.

[0017] By defining cloud-based augmented data as differential information relative to a baseline template, the amount of data transmitted can be significantly reduced, containing only the necessary local variation information. Encapsulating the differential information using formats such as displacement fields, texture indexes, or mesh patches enables flexible and efficient digital representations of different types and levels of geometric detail.

[0018] Furthermore, the adaptive streaming module defines and encapsulates streaming incremental data packets; The streaming incremental data packet includes at least global table case-varying parameters, sparse vertex offsets, and high-precision geometric augmentation data; The global expression variation parameters are used to drive the neutral expression reference mesh template to generate basic deformation; The sparse vertex offset is used to calibrate the position of key feature points.

[0019] By defining a streaming incremental data packet structure that includes global parameters, sparse offsets, and local enhancement data, a highly efficient 3D mesh video coding format can be constructed. By transforming the transmission of the entire mesh into the transmission of incremental instructions, each frame can be rapidly reconstructed at the receiving end using a reference template, significantly reducing the continuously high demand for transmission bandwidth.

[0020] Furthermore, the user terminal also includes a real-time fusion rendering unit; The real-time fusion rendering unit is used to receive streaming incremental data packets, sequentially apply global surface-varying parameters and sparse vertex offsets to the locally stored neutral expression reference mesh template to obtain an intermediate mesh, and then apply the mesh patch or displacement field corresponding to the high-precision geometric enhancement data to the corresponding area of ​​the intermediate mesh to complete the reconstruction and rendering of the final three-dimensional expression mesh.

[0021] By setting up a real-time fusion rendering unit on the terminal, the received compact incremental data packets can be decoded and gradually applied to the local reference template, quickly completing the reconstruction of the final high-fidelity mesh. By applying global deformation, sparse calibration, and local enhancement step by step, it is possible to ensure that the reconstructed mesh maintains the real-time performance of the overall expression while incorporating the high-precision details from cloud computing.

[0022] Furthermore, the neutral expression reference mesh template is a pre-generated, personalized 3D mesh model bound to the user's identity, serving as a unified geometric reference established for each user during the system initialization phase.

[0023] By employing a personalized baseline mesh template bound to the user's identity, all expression meshes output by the system can naturally fit the user's specific facial anatomy. Using a personalized template as the geometric benchmark for the entire streaming processing workflow ensures high consistency throughout the entire process from encoding and transmission to reconstruction, avoiding distortions introduced by insufficient template universality.

[0024] Furthermore, the data acquisition and preprocessing module, the modeling and enhancement module, and the adaptive streaming module constitute a closed-loop collaborative mechanism; The regional complexity information and real-time network bandwidth information output by the perception and evaluation unit are used together as inputs to the hierarchical modeling engine to dynamically determine the data processing allocation strategy between the user terminal and the cloud computing node, as well as the generation and transmission content of high-precision geometric augmentation data.

[0025] By constructing the perception, decision-making, and transmission modules into a closed-loop collaborative mechanism, the system can achieve real-time response and global optimization to dynamic input factors such as facial expression content and network environment. By using regional complexity and network status as core decision inputs, the system can break free from the limitations of fixed processing modes and achieve dynamic and intelligent adjustment of resource allocation and data processing strategies.

[0026] Furthermore, when applying mesh patching, the real-time fusion rendering unit performs real-time subdivision of the local mesh topology or vertex replacement operations to achieve seamless geometric fusion between high-precision geometric enhancement data and the basic facial expression mesh.

[0027] By performing real-time subdivision or vertex replacement operations on local meshes during the blending rendering process, high-precision patches from the cloud, which may have different topologies, can be seamlessly integrated into the underlying base mesh. Through this seamless geometric blending mechanism, ultra-high-detail geometric expressions can be inserted into specific frames and regions without disrupting the overall smoothness of the animation, achieving dynamic and content-adaptive rendering quality.

[0028] Compared with existing technologies, this real-time modeling system for 3D mesh data based on dynamic capture of facial expressions has the following advantages: I. This invention employs a distributed architecture with terminal-cloud collaboration to construct a closed-loop collaborative mechanism for data acquisition and preprocessing, modeling and enhancement, and adaptive streaming transmission modules. Based on the complexity of facial regions and real-time network status, it makes dynamic decisions to optimize the terminal-cloud allocation strategy and transmission content for data processing. Under dynamic network environments and limited terminal computing power constraints, it can accurately focus on high-detail areas for high-precision enhancement, while ensuring low latency and efficient transmission of end-to-end data processing. This effectively solves the problems of high-fidelity facial expression reconstruction versus portability, real-time performance, and low-bandwidth transmission in existing technologies. It provides a high-quality 3D facial expression mesh real-time modeling solution for distributed application scenarios such as virtual reality and augmented reality, ensuring the accuracy and fluency of facial expression reproduction in remote real-time interaction.

[0029] Second, this invention defines streaming incremental data packets, transmitting only incremental information relative to the neutral expression reference mesh template. Combined with dynamic transmission scheduling and data format optimization strategies, it significantly reduces data transmission redundancy. At the same time, it deploys high-precision augmented computing with high computing power consumption on cloud computing nodes, with the terminal only undertaking basic data processing and fusion rendering tasks. This allows it to adapt to different network bandwidths and terminal hardware conditions, maintaining stable feature point tracking and reconstruction accuracy in complex scenarios such as lighting changes and posture fluctuations. This improves the system's scene adaptability and robustness, and reduces the hardware threshold and network cost for application deployment.

[0030] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0032] Figure 1 This is a diagram of the overall system architecture of the present invention; Figure 2 This is a flowchart of the data acquisition and preprocessing module of the present invention; Figure 3 This is a decision logic diagram for the modeling and enhancement module of the present invention; Figure 4 This is a schematic diagram of the real-time fusion rendering process of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0034] It should be understood that, when used in this specification, the terms “comprising” and “including” indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0035] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” are intended to include the plural forms.

[0036] It should also be further understood that the term "and / or" as used in this specification refers to any combination of one or more of the associated listed items, as well as all possible combinations, and includes such combinations.

[0037] The present invention discloses a real-time modeling system for three-dimensional mesh data based on dynamic capture of facial expressions. It is mainly applied to consumer-grade and distributed scenarios such as virtual reality remote communication, augmented reality interaction, and real-time driving of digital humans. It aims to solve the contradiction between high-fidelity facial expression reconstruction and portability, real-time performance, and low-bandwidth transmission in the existing technology.

[0038] In this embodiment, the system adopts a distributed architecture of terminal-cloud collaboration. It constructs a closed-loop collaborative technical system through a data acquisition and preprocessing module deployed on the user terminal, a modeling and enhancement module deployed on the cloud computing node, and an adaptive streaming transmission module connecting the two. This enables integrated processing of the entire process, from facial image sequence acquisition to 3D expression mesh sequence output. The core design idea of ​​the system is to optimize the terminal-cloud allocation strategy and data transmission content based on dynamic decision-making according to the complexity of the facial region and the real-time network status, balancing reconstruction accuracy and real-time performance within the constraints of limited terminal computing power and network bandwidth.

[0039] like Figure 1 The system architecture diagram shown clearly presents the connection relationships and data flow paths between the modules. The user terminal and the cloud computing node achieve bidirectional data interaction through the adaptive streaming transmission module. The functional division and collaborative logic of each module can be intuitively understood through the architecture diagram.

[0040] The following section provides a detailed explanation of each module and its technical features, including specific technical details, algorithm implementation, parameter settings, and actual operation procedures.

[0041] In this embodiment, the data acquisition and preprocessing module is implemented as follows: The data acquisition and preprocessing module is the system's front-end data processing unit. Its core function is to acquire facial image sequences in real time, and through feature point tracking, optical flow calculation, and basic mesh generation, output basic expression meshes and region complexity information, providing a data foundation for subsequent modeling and transmission. This module includes a perception and evaluation unit and a basic modeling unit, which work together to complete the front-end data processing.

[0042] Implementation of the perception and assessment unit: The core task of the perception and evaluation unit is to extract features and dynamically analyze the acquired facial image sequences, generating region complexity information that characterizes the intensity of movement and geometric changes in different parts of the face, while simultaneously providing feature point data support to the basic modeling unit. The specific implementation process is as follows: Face image sequence acquisition: The system supports image acquisition via built-in cameras on user terminals, such as smartphone front-facing cameras, tablet cameras, or external USB cameras. The acquisition resolution supports 1080P (1920×1080 pixels) or 720P (1280×720 pixels), with a frame rate set at 30fps to meet the time requirements of real-time interactive scenarios. During acquisition, the terminal device automatically performs white balance calibration and exposure adjustment to ensure the clarity and brightness stability of facial images under different lighting conditions. The light intensity adaptation range is 50 lux to 1000 lux, covering common scenarios such as ordinary indoor lighting and outdoor overcast days.

[0043] Feature point tracking: A modified Lucas-Kanade optical flow method is used to achieve real-time tracking of facial feature points. This method is based on the assumption of image grayscale invariance and calculates the motion trajectory of feature points through the correlation of pixel grayscale within a local window, thus achieving both real-time performance and tracking accuracy.

[0044] First, during the system initialization phase, a lightweight CNN architecture is used based on a pre-trained facial feature point detection model. The input is a 640×480 pixel cropped face image, and the output is the coordinates of 68 key feature points. Feature point detection is performed on the first frame of the acquired image to determine the feature point set of key facial regions, including feature points of the brow bone, eyelids, eyeballs, nose wings, nostrils, lips, cheeks, and jawline. Among them, there are 12 feature points in the lip region, 22 feature points in the eye region, 10 feature points in the nose region, and 24 feature points in the facial contour. These 68 feature points constitute the initial reference point set for tracking.

[0045] Subsequently, inter-frame feature point tracking is performed based on the LK optical flow method, and its core constraint equation is as follows: in, express Image at pixel coordinates grayscale value at that location and These represent the feature points at... direction and The displacement in the direction. To improve tracking robustness, a pyramid hierarchical tracking strategy is introduced, constructing a 4-layer image pyramid, with each layer's image size being 1 / 2 of the previous layer. By tracking step by step from the top layer to the bottom layer, the problem of tracking failure of large displacement feature points is solved.

[0046] During tracking, the window size is set to 15×15 pixels, the number of iterations is 10, and the convergence threshold is 0.01. When the L2 norm of the displacement increment is less than the convergence threshold during iteration, the iteration stops and the feature point displacement result is output. Simultaneously, a reverse verification mechanism is used to remove falsely tracked feature points. For feature points with a tracking confidence level below 0.8, their coordinates are re-acquired through a re-detection mechanism to ensure the continuity and accuracy of feature point tracking.

[0047] Optical flow calculation: Based on the displacement data of the tracked feature points, the optical flow field distribution of each region of the face is calculated to characterize the motion intensity and direction of facial pixels. The optical flow calculation employs a sparse optical flow interpolation method. First, a sparse optical flow field is obtained from the displacement vectors of the feature points. Then, based on a bilinear interpolation algorithm, the sparse optical flow field is expanded into a dense optical flow field to obtain the motion vector of each pixel on the face. .

[0048] To quantify motion intensity, pixel-level motion amplitude is defined. : This amplitude directly reflects the intensity of motion of the corresponding pixel, providing basic data for subsequent region complexity assessment.

[0049] Region complexity information generation: The intensity of facial facial expressions and the required geometric changes vary significantly across different regions. For example, the lips and eyes exhibit vigorous movement and rich geometric detail during facial expressions, while the cheeks and forehead show relatively gentler movements. Based on this characteristic, the face is divided into eight independent evaluation regions: the left eye region, right eye region, upper lip region, lower lip region, nose region, left cheek region, right cheek region, and forehead region. The boundaries of each region are defined by the coordinate range of an initial feature point set.

[0050] Region complexity information is a quantitative indicator characterizing the intensity of motion and geometric change requirements of each facial region. Its calculation is based on the statistical characteristics of pixel motion amplitude and the weight of geometric change requirements within the region. The specific calculation equation is as follows: in, Indicates the first The complexity value of each facial region Corresponding to the above 8 facial regions; Indicates the first The total number of pixels in each region; Indicates the first A set of pixels in each region; and The weights are exercise intensity and geometric change requirement, respectively, both set to 0.5, to ensure a balanced impact of the two indicators; For the first The geometric change requirement coefficients for each region are preset based on the physiological structure and facial expression characteristics of the facial regions, including the eye and lip regions. The value is 1.0 for the nose region, 0.7 for the cheek region, and 0.3 for the forehead region. This coefficient reflects the degree of need for geometric detail reconstruction in different regions.

[0051] The complexity values ​​for each region were calculated. Then, compare it with a preset threshold. The comparison determines whether the area requires high detail. A preset threshold is used. Calibrated through extensive experimental data, a value of 8.0 pixels per displacement unit was determined. This threshold was based on testing 300 sets of image sequences with different facial expressions, including 10 common expressions such as smiling, frowning, surprise, and sadness. It can accurately distinguish between regions of intense movement and regions of smooth movement. The output format of the region complexity information is an 8-dimensional vector. Each of the eight facial regions corresponds to one of them, providing a basis for decisions in subsequent modeling and enhancement modules.

[0052] Implementation of basic modeling units: The core task of the basic modeling unit is to store a neutral facial expression baseline mesh template bound to the user's identity. Based on the feature point tracking data output by the perception and evaluation unit, it drives this template to generate a topologically consistent basic facial expression mesh in real time, providing a basic geometric framework for subsequent high-precision geometric enhancement. The specific implementation process is as follows: Neutral expression baseline grid template generation: The neutral expression baseline mesh template is a pre-generated, personalized 3D mesh model bound to the user's identity. It serves as a unified geometric baseline established for each user during the system initialization phase, ensuring consistent mesh topology across different expression frames. Its generation process includes three steps: user facial data acquisition, 3D reconstruction, and template optimization. User facial data collection: When the system is used for the first time, the user is guided to take 5 images with neutral expressions in different postures, including a frontal image, a left 45° side image, a right 45° side image, an image with the head up, and an image with the head down. The image resolution is set to 1080P. During the shooting process, the user is required to keep no obvious facial expression, and the range of head posture changes is controlled within ±45° horizontal direction and ±30° vertical direction.

[0053] 3D Reconstruction: An image-based 3D reconstruction algorithm is employed to construct a 3D point cloud model of the user's face through feature matching and dense point cloud generation from multi-view images. First, feature extraction and matching are performed on five acquired images. The SIFT algorithm is used to extract image feature points, and the RANSAC algorithm is used to remove mismatched feature points, obtaining the correspondence between multiple viewpoints. Then, the intrinsic and extrinsic parameters of the camera are calculated based on the Structure for Motion Recovery (SfM) algorithm. Finally, a dense point cloud of the face is generated using a dense reconstruction algorithm, with a point cloud density of 10 points per square millimeter to ensure a complete representation of the facial geometry.

[0054] Template Optimization: Dense point clouds undergo mesh reconstruction and topology optimization to generate a 3D mesh model. The Poisson surface reconstruction algorithm is used to convert the point cloud into an initial mesh with approximately 10,000 vertices and 20,000 faces. Subsequently, topology simplification and smoothing are applied to optimize the mesh structure, ensuring consistent topology and eliminating folds and intersections. Finally, the optimized mesh model is aligned with neutral facial expression features to obtain a neutral facial expression baseline mesh template. The vertex coordinates are established with the geometric center of the user's face as the origin, using a local coordinate system. The x-axis follows the horizontal direction of the face, the y-axis follows the vertical direction, and the z-axis follows the depth direction, with the coordinate unit being millimeters.

[0055] After the neutral expression baseline mesh template is generated, it is bound to the user account and stored locally on the user's terminal, and backed up on a cloud computing node to avoid system failure due to terminal data loss. The template data format is OBJ, which supports reading and editing by mainstream 3D modeling software.

[0056] Basic facial expression mesh generation: The base expression mesh is a 3D mesh with basic expression deformation generated in real time from a neutral expression reference mesh template. Its topology is completely consistent with the neutral expression reference mesh template, ensuring compatibility with subsequent high-precision geometric enhancements. The driving process is based on a feature point-constrained mesh deformation algorithm, implemented as follows: First, a correspondence is established between the vertices of the neutral expression reference mesh template and the feature points output by the perception and evaluation unit. Each feature point corresponds to a vertex on the mesh template, forming 68 constrained vertex pairs. Then, based on the thin-plate spline interpolation TPS algorithm, the displacements of all vertices of the mesh template are calculated using the displacements of the constrained vertex pairs as input, realizing the mesh appearance variation. The core equation is as follows: in, The neutral expression baseline grid template is shown below. The original coordinates of the vertices, This represents the coordinates of the vertex after deformation; Indicates the first The original coordinates of the grid vertices corresponding to each feature point; For interpolation weights; This is the TPS kernel function; These are global affine transformation parameters used to ensure the overall rigidity of the mesh.

[0057] Weight The global parameters are obtained by solving the following system of linear equations: in, Indicates the first The displacement of the mesh vertex corresponding to each feature point is obtained from the feature point tracking data output by the perception and evaluation unit.

[0058] like Figure 2 The flowchart shown in the diagram illustrates the complete process from image acquisition to the output of basic facial expression grid and region complexity information. The collaborative logic of the perception and evaluation unit and the basic modeling unit, as well as the execution order of key algorithms, can be clearly understood through the flowchart.

[0059] Using the algorithm described above, the displacement of the mesh vertices for each frame is calculated in real time to generate a basic facial expression mesh. The generation frame rate is kept consistent with the image acquisition frame rate of 30fps to ensure the continuity of facial expression dynamics. The basic facial expression mesh has the same number of vertices as the neutral expression baseline mesh template, approximately 10,000, which can accurately represent the overall deformation of facial expressions and provide a stable basic framework for subsequent high-precision geometric enhancement.

[0060] In this embodiment, the modeling and enhancement module is implemented as follows: The modeling and enhancement module is deployed on cloud computing nodes. Its core function is to receive regional complexity information and related image data output by the data acquisition and preprocessing module. Based on the decisions of the hierarchical modeling engine, it performs high-precision geometric enhancement processing on areas with high detail requirements, generating high-precision geometric enhancement data to compensate for the lack of detail in the basic facial expression mesh. This module includes a hierarchical modeling engine and a high-fidelity refinement unit, which work together to calculate and generate high-precision geometric details.

[0061] Implementation of the layered modeling engine: The hierarchical modeling engine is the decision-making core of the modeling and enhancement module. Its core task is to receive regional complexity information from the perception and evaluation unit and real-time network bandwidth information from the adaptive streaming module, dynamically determine the data processing allocation strategy between the user terminal and cloud computing nodes, and the data content sent to the high-fidelity refinement unit, thereby optimizing resource allocation. Its specific implementation process is as follows: Information input and preprocessing: The hierarchical modeling engine receives two parts of input information: one is the region complexity information, i.e., an 8-dimensional vector. Secondly, real-time network bandwidth information is obtained by the adaptive streaming module through network status monitoring at a frequency of 10 Hz, outputting the average value of the current uplink and downlink bandwidth in megabits per second.

[0062] The input region complexity information is normalized, and the complexity values ​​of each region are normalized. Mapping to the [0,1] interval, the normalization equation is: in, The minimum complexity value for all regions is preset to 0. The maximum value of complexity for all regions is preset to 20.0 to ensure that complexity values ​​are comparable across different scenarios.

[0063] Real-time network bandwidth information is smoothed using a sliding window averaging algorithm with a window size of 5 monitoring points to eliminate instantaneous interference caused by network bandwidth fluctuations, resulting in a smoothed bandwidth value. .

[0064] Decision logic implementation: The decision logic of the hierarchical modeling engine is based on the normalized value of the region complexity. With preset threshold The comparison uses a value of 0.4, combined with the smoothed network bandwidth value. The data processing allocation strategy and transmission content are determined, and the specific decision-making process is as follows: Region processing method determination: For each facial region, if If the region is determined to be a low-details region, it will be fully processed by the basic modeling unit of the user terminal, and the cloud computing node will not perform additional high-precision enhancement; The region was determined to be a high-details area, requiring high-fidelity refinement units on cloud computing nodes to perform high-precision geometric enhancement processing.

[0065] Content selection: For areas deemed to have high detail requirements, the content is selected based on the smoothed network bandwidth value. Select the content to transfer: when At megabits per second, the original image blocks of the area are transmitted. The resolution of the image blocks is consistent with the resolution of the acquired image, and the format is RGB888. This ensures that the high-fidelity retouching unit can obtain the most complete image information and generate the highest precision geometric enhancement data. when megabits per second At megabits per second, intermediate neural features extracted from the original image patch are transmitted. These features are extracted through a lightweight feature extraction network based on the MobileNetV2 architecture. The input is a 256×256 pixel image patch, and the output is a 512-dimensional feature vector. The amount of feature data is only 1 / 10 of that of the original image patch, which reduces the transmission bandwidth requirement while ensuring enhanced accuracy. when At megabits per second, the compressed intermediate neural features are transmitted. The 512-dimensional feature vector is compressed to 128 dimensions using the PCA dimensionality reduction algorithm, further reducing the amount of data transmitted and ensuring the normal operation of the system in low-bandwidth scenarios.

[0066] The decision results are output as region processing mode identifiers (low detail / high detail) and transmission content identifiers (original image patch / intermediate neural feature / compressed intermediate neural feature), which are sent to the high-fidelity refining unit and the adaptive streaming module, respectively, to guide subsequent high-precision enhancement calculations and data transmission.

[0067] Implementation of the high-fidelity retouching unit: The core task of the high-fidelity refining unit is to receive raw image patches or intermediate neural features of high-detail regions from the hierarchical modeling engine, and generate high-precision geometric enhancement data containing local geometric details or displacement information through model calculations based on physical simulation or neural network super-resolution, thus providing detailed supplementation to the base expression mesh. Its specific implementation process is as follows: Model selection and configuration: The high-fidelity enhancement unit offers two high-precision enhancement models, which can be selected according to the application scenario requirements. The output results of the two models are in the same format, ensuring compatibility with the basic facial expression mesh: The physics-based model: This model constructs an elastic deformation model of facial soft tissue based on the physical laws governing facial muscle movement and skin elastic deformation. It is suitable for scenarios requiring high geometric realism. The core equation of the model is Hooke's Law from elasticity mechanics, specifically in the following form: in, For stress tensor, The elastic modulus is taken as 1.5 × 10^5 Pa, calibrated based on the mechanical properties of human facial skin tissue. The strain tensor is obtained from the facial motion gradient calculated from the image data.

[0068] The above equations were solved using the finite element method. The mesh of the region requiring high detail was divided into 1000 finite element elements, each containing 4 vertices. The displacement of each vertex was obtained through iterative calculation, with 20 iterations and a convergence threshold of 1×10^-6m, to ensure the accuracy and stability of the calculation results.

[0069] The neural network-based super-resolution model: This model utilizes deep learning-based image super-resolution and geometric reconstruction techniques, suitable for scenarios with high real-time requirements. The model employs a lightweight CNN with an encoder-decoder architecture. The encoder contains six convolutional layers, each with a 3×3 kernel size, a stride of 1, and the same padding method, using the ReLU activation function. The decoder contains four deconvolutional layers, each with a 4×4 kernel size, a stride of 2, and the same padding method, using the ReLU activation function. The output layer uses the Sigmoid activation function to output the vertex displacement field for regions requiring high detail.

[0070] The model's training dataset contains 100,000 sets of facial expression images and corresponding high-precision 3D scan data. The image resolution is 1080P, and the number of vertices in the 3D scan data is 50,000. The training process uses the Adam optimizer with a learning rate of 1×10^-4, a batch size of 32, and 100 epochs. The loss function is the mean squared error (MSE), and the loss function equation is as follows: in, The first output of the model The displacement of each vertex, The third corresponding to the 3D scan data The actual displacement of each vertex The number of vertices in regions requiring high detail.

[0071] High-precision geometric augmentation data generation: High-precision geometric augmentation data is differential information relative to a neutral expression baseline mesh template. It is defined only for areas requiring high detail to avoid data redundancy. It supports three formats, which can be selected according to the actual application scenario: Local vertex displacement field: This format stores the displacement of each vertex in a region requiring high detail as a vector field. The displacement vector contains components in the x, y, and z directions, with units in millimeters and a precision of 0.01 millimeters. This format is suitable for scenarios with the highest requirements for detail precision. The data size is proportional to the number of vertices in the region requiring high detail. Each vertex's displacement vector occupies 12 bytes, with each component occupying 4 bytes, and is stored in float32 format.

[0072] Detail Normal Map Index: High-detail geometric details are encoded into normal maps. The normal map resolution is 512×512 pixels, and each pixel stores the x, y, and z components of the normal vector, with values ​​ranging from [0, 255]. These are mapped to the actual normal direction through an index. Simultaneously, the texture coordinate mapping relationship between the normal map and the underlying facial mesh is stored to ensure that the normal map accurately fits the underlying facial mesh. This format has a fixed data size of 768KB (512×512×3 bytes), suitable for scenarios with strict data constraints.

[0073] Mesh patch with subdivision instructions: For mesh patches in regions requiring high detail, patch data containing subdivision instructions is generated. The subdivision instructions use the Catmull-Clark subdivision algorithm with a subdivision level of 2, subdividing each original patch into 4 new patches. Simultaneously, the coordinate offsets of the subdivided patch vertices are stored to ensure that the subdivided mesh accurately represents the high-detail geometry. The data size of this format is proportional to the number of patches in the region requiring high detail; each patch data occupies 32 bytes, making it suitable for scenarios requiring consistent mesh topology.

[0074] like Figure 3 The modeling and enhancement module decision logic diagram shown clearly illustrates the decision process of the hierarchical modeling engine, the model selection and data processing logic of the high-fidelity refinement unit, and the input-output relationship of each link is presented intuitively through the diagram.

[0075] After high-precision geometric enhancement data is generated, it is bound to the facial region to which the data belongs by the corresponding region identification information, and sent to the adaptive streaming module for subsequent dynamic transmission and fusion rendering.

[0076] In this embodiment, the adaptive streaming module is implemented as follows: The adaptive streaming module is the core data transmission unit connecting user terminals and cloud computing nodes. Its core function is to dynamically schedule the transmission of image data and high-precision geometric augmentation data based on real-time network conditions. By defining and encapsulating streaming incremental data packets, it achieves efficient data transmission and real-time fusion, ensuring low-latency end-to-end transmission. Its specific implementation process is as follows: Real-time network status monitoring: The adaptive streaming module has a built-in network status monitoring unit that collects network transmission parameters between user terminals and cloud computing nodes in real time, including uplink bandwidth, downlink bandwidth, network latency, packet loss rate, etc. The monitoring frequency is 10 Hz to ensure timely detection of network status changes.

[0077] Network bandwidth monitoring uses the packet probing method, which involves periodically sending fixed-size probe packets (1KB each) and recording the sending and receiving acknowledgment times to calculate the transmission rate, i.e., the bandwidth value. Network latency monitoring uses the round-trip time (RTT) measurement method, which records the total time from sending the probe packets to receiving acknowledgments and takes the average of five measurements as the current network latency. Packet loss rate monitoring uses a statistical method, which calculates the difference between the total number of packets sent and the total number of received acknowledgments per second within a certain time period.

[0078] The monitored network parameters are fed back to the hierarchical modeling engine and transmission scheduling unit in real time, providing a basis for data processing decisions and transmission strategy adjustments.

[0079] Definition and encapsulation of streaming incremental data packets: To achieve efficient data transmission and real-time fusion, the adaptive streaming module defines a streaming incremental data packet. This packet contains only incremental information relative to the neutral expression reference grid template, significantly reducing data transmission volume. The structure of the streaming incremental data packet consists of a header and a data body, as defined below: Packet header: Occupies 16 bytes, including a 4-byte packet sequence number (an incrementing sequence used for frame synchronization), a 2-byte data type identifier (identifying the incremental data type contained in the packet), a 4-byte data length (identifying the total number of bytes in the data body), a 4-byte checksum using the CRC32 algorithm (used for data integrity verification), and a 2-byte reserved field (used for future function expansion).

[0080] Data body: Contains global table-variable parameters, sparse vertex offsets, and high-precision geometric augmentation data. The storage order and length of each part of the data are as follows: Global face-changing parameters: occupying 24 bytes, including 3 translation parameters (x, y, z directions, each 8 bytes, stored in double format) and 3 rotation parameters (rotation angles around the x, y, z axes, each 8 bytes, stored in double format). These parameters are used to drive the neutral expression baseline mesh template to generate basic deformations and adapt to different head pose changes.

[0081] Sparse vertex offsets: occupying 68 × 12 bytes = 816 bytes, containing the offsets of 68 key feature points in the x, y, and z directions. Each offset is 4 bytes and stored in float32 format. It is used to calibrate the positions of key feature points of the basic facial expression mesh and improve the pose accuracy of the basic facial expression mesh.

[0082] High-precision geometric enhancement data: The length is not fixed and is dynamically adjusted according to the data format and the number of areas requiring high detail. The storage format is consistent with the output format of the high-fidelity retouching unit, and includes area identification information and corresponding difference information to ensure that the data can be accurately mapped to the corresponding areas of the basic expression mesh.

[0083] The encapsulation process of streaming incremental data packets uses binary encoding to ensure data compactness and transmission efficiency. The encapsulated data packets are transmitted via the UDP protocol, balancing real-time performance and transmission reliability. For scenarios with a packet loss rate higher than 5%, a selective retransmission mechanism is automatically enabled, retransmitting only the lost critical data packets containing high-precision geometric augmentation data to avoid increased latency caused by full retransmission.

[0084] Dynamic scheduling of data transmission: The adaptive streaming module uses bandwidth values ​​obtained from real-time network status monitoring. ,Delay With packet loss rate The priority and rate of data transmission are dynamically adjusted to ensure the priority transmission and low-latency arrival of critical data. The specific scheduling strategy is as follows: Transmission priority sorting: The data in the streaming incremental data packets are sorted according to importance, with priority from high to low as follows: global table case-variable parameters > sparse vertex offsets > high-precision geometric augmentation data for the eye and lip regions > high-precision geometric augmentation data for the nose region > high-precision geometric augmentation data for the cheek and forehead regions, ensuring that core data that affects the overall facial expression and posture are transmitted first.

[0085] Transmission rate adjustment: based on the current bandwidth value Dynamically adjust the transmission rate when At megabits per second, the transmission rate is set to 30 megabits per second to transmit all incremental data, ensuring the integrity of details; when megabits per second When the transmission rate is set to 10 megabits per second, high-priority data is prioritized for transmission, and the transmission quality of low-priority data is appropriately compressed, such as reducing the resolution of high-precision geometrically enhanced data; when At megabits per second, the transmission rate is set to 2 megabits per second, transmitting only global facial variation parameters, sparse vertex offsets, and high-precision geometric enhancement data for the eye and lip regions to ensure the normal transmission of core facial expression details.

[0086] Delay control strategy: When network latency In milliseconds, the resolution of data transmission is automatically reduced, such as reducing the resolution of image patches in high-precision geometric augmentation data from 1080P to 720P, thereby reducing data processing and transmission time; when the delay... At a certain time (ms), high-precision geometric enhancement data transmission for non-critical areas such as the cheeks and forehead is temporarily disabled, while only enhancement data for core areas is retained to ensure real-time facial expression dynamics.

[0087] Through the above dynamic scheduling strategy, the adaptive streaming module can flexibly adjust the transmission strategy in different network environments, achieve a balance between data transmission efficiency and real-time performance, and ensure that user terminals can receive and process transmitted data in a timely manner.

[0088] In this embodiment, the real-time fusion rendering unit is implemented as follows: The user terminal also includes a real-time fusion rendering unit, whose core function is to receive streaming incremental data packets sent by the adaptive streaming module, and then, by sequentially applying global surface-varying parameters, sparse vertex offsets, and high-precision geometric enhancement data, complete the reconstruction and rendering of the final 3D facial expression mesh, outputting a visualized 3D facial expression sequence. The specific implementation process is as follows: Data packet parsing and verification: After receiving the streaming incremental data packets, the real-time fusion rendering unit first performs parsing and integrity verification: Parse the packet header information: Extract the packet sequence number and compare it with the sequence number of the previous frame packet in the local cache to ensure the consistency of the packet order. If the sequence number changes, wait for the arrival of subsequent packets or trigger a retransmission request; Extract the checksum and compare it with the locally calculated packet checksum. If the checksum fails, discard the packet and request a retransmission.

[0089] Parse data body information: Based on the data type identifier, extract global table case-variable parameters, sparse vertex offsets and high-precision geometric augmentation data, and store them in the corresponding cache areas. The cache areas adopt a circular queue structure with a maximum cache capacity of 10 frames of data to avoid processing anomalies caused by data overflow.

[0090] Reconstruction of 3D facial expression mesh: The reconstruction process of the 3D facial expression mesh consists of three steps, which sequentially apply incremental information from streaming incremental data packets to achieve gradual optimization from a neutral facial expression baseline mesh template to the final 3D facial expression mesh: Basic Deformation Application: Global facial transformation parameters are applied to a locally stored neutral facial reference mesh template. Through translation and rotation transformations, an initial mesh adapted to the current head pose is generated. The translation transformation equation is: The rotational transformation equation is: in, This is the translation parameter in the global table's variable parameters. These are the rotation matrices about the x, y, and z axes, respectively, calculated from the rotation angles.

[0091] Key Vertex Calibration: Sparse vertex offsets are applied to 68 key feature points in the initial mesh to calibrate the positions of these feature points. The calibration equation is as follows: in, For the first grid in the initial grid The coordinates of the key feature points This represents the offset value corresponding to the sparse vertex offset. After calibration, the calibration results of key feature points are diffused to the entire initial mesh using a thin-plate spline interpolation algorithm to obtain an intermediate mesh, ensuring the accuracy of the overall mesh pose.

[0092] High-precision detail fusion: High-precision geometric augmentation data is applied to the corresponding regions of the intermediate mesh. Different fusion methods are used depending on the data format. Local vertex displacement field: The displacement vectors in the displacement field are directly superimposed onto the vertex coordinates of the corresponding region in the intermediate mesh. The fusion equation is: in, For the middle grid, the first The coordinates of the vertices, This is the displacement vector corresponding to the local vertex displacement field.

[0093] Detailed Normal Map Index: Based on the texture coordinate mapping relationship, the normal vectors in the normal map are assigned to the vertices of the corresponding region of the middle mesh, the normal information of the vertices is updated, and then the visual performance of facial details is enhanced through lighting calculations. The lighting model adopts the Phong model, which includes the superposition calculation of ambient light, diffuse light and specular light.

[0094] Mesh patching with subdivision instructions: Real-time subdivision of the local mesh topology is performed on the patches corresponding to the intermediate mesh area. The Catmull-Clark subdivision algorithm is used to subdivide each original patch into four new patches. The vertex offsets from the patch data are then applied to the subdivided vertices to achieve high-precision geometric detail fusion. During the subdivision process, it is ensured that the subdivided mesh and the original intermediate mesh topology are seamlessly connected, without gaps or overlaps.

[0095] Rendering of 3D facial expression meshes: After the 3D facial expression mesh is reconstructed, the real-time fusion rendering unit uses GPU-accelerated rendering technology to render the mesh in real time and output a visual 3D facial expression sequence. The specific rendering process is as follows: Texture mapping: The facial texture image stored locally on the user terminal is bound to a neutral expression reference mesh template and generated. The resolution is 2048×2048 pixels. The texture coordinates are mapped to the reconstructed 3D expression mesh, and the texture coordinates correspond one-to-one with the mesh vertices to ensure the accuracy of texture matching.

[0096] Lighting rendering: A real-time dynamic lighting model is used to simulate the illumination effect of ambient light and directional light sources. The ambient light intensity is set to a relative value of 0.3, and the direction and intensity of the directional light source can be dynamically adjusted according to the user terminal's posture sensor data to adapt to different usage scenarios.

[0097] Output display: The rendered 3D expression sequence is output to the user terminal's display screen at a frame rate of 30fps, supporting full-screen display or window display. It can also be output to third-party applications such as video call software and AR interactive applications through an interface to realize real-time interactive functions.

[0098] like Figure 4The diagram showing the real-time fusion rendering process fully presents the entire process from data packet parsing to 3D facial expression mesh rendering output. The execution logic and data processing methods of each step can be clearly understood through the diagram, providing an intuitive reference for technical personnel to implement.

[0099] In this embodiment, the closed-loop coordination mechanism is implemented as follows: The data acquisition and preprocessing module, the modeling and enhancement module, and the adaptive streaming module form a closed-loop collaborative mechanism. Through real-time information feedback and dynamic adjustment, this mechanism enables efficient collaboration among the system's modules, ensuring performance optimization in different scenarios. The specific implementation process is as follows: Information feedback path: The perception and evaluation unit of the data acquisition and preprocessing module outputs regional complexity information, which is sent to the basic modeling unit for the generation of basic facial expression meshes, and to the hierarchical modeling engine of the cloud computing node as the basis for data processing allocation strategy decisions.

[0100] The network status monitoring unit of the adaptive streaming module outputs real-time network bandwidth information, latency, and packet loss rate. On the one hand, it sends the data to the hierarchical modeling engine as a basis for decision-making on content selection; on the other hand, it feeds back to the data acquisition and preprocessing module to adjust the resolution of image acquisition. When the bandwidth is consistently below 1 megabit per second, the acquisition resolution is automatically reduced from 1080P to 720P.

[0101] The high-fidelity refinement unit of the modeling and enhancement module outputs high-precision geometric enhancement data generation status information, such as generation time and data volume, which is fed back to the hierarchical modeling engine to adjust the model's computational accuracy. When the generation time exceeds 30ms, the number of model iterations or network layers is appropriately reduced.

[0102] Dynamic adjustment process: The closed-loop coordination mechanism has a dynamic adjustment cycle of 100ms. Within each cycle, a decision and adjustment are made based on the feedback information. The specific process is as follows: Information collection: Collect information on regional complexity, network status, and generation status, and perform preprocessing and normalization; Decision Update: Based on the updated information, the hierarchical modeling engine re-determines the regional processing method and transmitted content; Strategy execution: The data acquisition and preprocessing module adjusts the acquisition parameters, the adaptive streaming module adjusts the transmission strategy, and the modeling and enhancement module adjusts the calculation accuracy; Results feedback: Real-time fusion of the rendering unit outputs reconstruction accuracy and rendering frame rate information to evaluate the effectiveness of the adjustment strategy. If the reconstruction accuracy is lower than the preset threshold, such as vertex error greater than 0.5 mm or rendering frame rate lower than 25 fps, further optimization decisions will be made in the next adjustment cycle.

[0103] Through the aforementioned closed-loop collaborative mechanism, the system can adapt to changes in facial expressions, network state fluctuations, and computational resource constraints in real time, dynamically optimize the working parameters of each module, and ensure that high-fidelity, low-latency real-time modeling and transmission of 3D facial expression meshes can be achieved in different scenarios.

[0104] In this embodiment, the system implementation effect and verification are as follows: To verify the feasibility and effectiveness of the technical solution of this invention, an experimental prototype system was constructed based on the above specific embodiments, and performance testing and effect verification were conducted. The test environment and results are as follows: Test environment: User terminal: It uses a smartphone Snapdragon 888 processor, 8GB of RAM, and a 1080P front-facing camera; Cloud computing nodes: Alibaba Cloud ECS instance with 8-core CPU, 16GB memory, and NVIDIA Tesla T4 GPU; Network environment: Simulate different bandwidth scenarios (1 Mbps, 3 Mbps, and 10 Mbps) using a network simulator, with latency ranging from 50ms to 300ms and packet loss rate ranging from 0% to 10%. Test data: Ten test subjects were selected, and each person was photographed with image sequences of 10 common facial expressions, for a total of 100 sets of test data.

[0105] Test results: Real-time performance: In a scenario with 10 Mbps bandwidth, 50 ms latency, and 0% packet loss, the system's end-to-end processing latency is 85 ms, and the rendering frame rate is stable at 30 fps; in a scenario with 1 Mbps bandwidth, 200 ms latency, and 5% packet loss, the end-to-end processing latency is 180 ms, and the rendering frame rate is stable at 25 fps, meeting the requirements for real-time interaction.

[0106] Reconstruction accuracy: The average vertex error for high-detail areas such as the eyes and lips is 0.35 mm, and the average vertex error for low-detail areas such as the cheeks and forehead is 0.8 mm, which is better than the reconstruction accuracy of existing consumer-grade facial expression capture systems, which typically have a vertex error greater than 1 mm.

[0107] Bandwidth usage: In a 3 Mbps bandwidth scenario, the average transmission rate of streaming incremental data packets is 1.2 Mbps, which is only 1.5% of the original image sequence transmission rate of about 80 Mbps, significantly reducing network bandwidth requirements.

[0108] Robustness: In scenarios with illumination intensity of 100 lux to 800 lux and head posture variation range of ±30°, the system's feature point tracking accuracy remains above 95%, and the reconstructed 3D facial expression mesh can accurately reflect the test subject's facial expression changes without obvious distortion or drift.

[0109] The test results above show that the technical solution of the present invention can achieve real-time modeling and transmission of high-fidelity three-dimensional facial expression meshes under the constraints of limited terminal computing power and network bandwidth, taking into account real-time performance, accuracy and robustness, and meeting the application needs of consumer-grade and distributed scenarios.

[0110] In this embodiment, the contradiction between high-fidelity facial expression reconstruction and portability, real-time performance, and low-bandwidth transmission in existing technologies is resolved through a distributed architecture of terminal-cloud collaboration, a dynamic decision-making mechanism based on regional complexity and network status, and streaming incremental transmission and real-time fusion rendering technology. Compared with existing technologies, this embodiment has the following advantages: Balancing high fidelity and real-time performance: Accurate enhancement of high-detail areas is achieved through regional complexity assessment, while optimizing computing power allocation through terminal-cloud collaborative processing. While ensuring the fidelity of geometric details in key areas such as eyes and lips, the system ensures that the end-to-end latency is less than 200ms, meeting the requirements of real-time interaction.

[0111] Reduced bandwidth and computing power requirements: By using streaming incremental data packet transmission and dynamic scheduling strategies, the amount of data transmission is significantly reduced, and stable transmission can be achieved with a bandwidth of 3 megabits per second; at the same time, high-precision enhanced computing with high computing power consumption is deployed in the cloud, reducing the computing power burden on user terminals, making it suitable for ordinary consumer-grade mobile devices.

[0112] Highly robust: It adapts to changes in facial expressions, network fluctuations, and lighting conditions in real time through a closed-loop collaborative mechanism, ensuring stable operation in different scenarios. The accuracy of feature point tracking and reconstruction precision remain at a high level, making it adaptable to a wide range of application scenarios.

[0113] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A real-time modeling system for 3D mesh data based on dynamic capture of facial expressions, characterized in that, The system comprises: a data acquisition and preprocessing module deployed on the user terminal, a modeling and enhancement module deployed on the cloud computing node, and an adaptive streaming transmission module connecting the user terminal and the cloud computing node; The data acquisition and preprocessing module is used to acquire and analyze facial image sequences in real time, and output basic expression grids and region complexity information based on expression dynamics. The modeling and enhancement module is used to receive image data of the high-detail region specified by the region complexity information and generate high-precision geometric enhancement data of the corresponding region. The adaptive streaming module is used to dynamically schedule the transmission of the image data and the high-precision geometric enhancement data between the user terminal and the cloud computing node according to the real-time network status, and to fuse the high-precision geometric enhancement data and the basic expression mesh in real time at the user terminal to output the final three-dimensional expression mesh sequence.

2. The real-time modeling system for three-dimensional meshed data based on dynamic capture of facial expressions according to claim 1, characterized in that, The data acquisition and preprocessing module includes a perception and evaluation unit and a basic modeling unit; The perception and evaluation unit is used to perform feature point tracking and optical flow calculation on the acquired face image sequence, and generate the region complexity information that characterizes the intensity of movement and geometric change requirements of different parts of the face. The basic modeling unit stores a neutral expression reference mesh template and drives the neutral expression reference mesh template in real time according to the face image sequence to generate the basic expression mesh with consistent topology.

3. The real-time modeling system for three-dimensional meshed data based on dynamic capture of facial expressions according to claim 1, characterized in that, The modeling and enhancement module includes a layered modeling engine and a high-fidelity refinement unit; The hierarchical modeling engine is used to receive regional complexity information and real-time network bandwidth information from the perception and evaluation unit, make decisions, and send image data marked as high detail requirement areas to the high-fidelity retouching unit. The high-fidelity refinement unit includes a model based on physical simulation or neural network super-resolution, used to calculate regions with high detail requirements and generate high-precision geometric enhancement data containing local geometric details or displacement information.

4. The real-time modeling system for three-dimensional meshed data based on dynamic capture of facial expressions according to claim 3, characterized in that, The decision logic of the hierarchical modeling engine includes: determining facial regions with regional complexity information below a preset threshold as to be fully processed by the basic modeling unit; determining facial regions with regional complexity information above the preset threshold as high detail requirement regions, and selecting to send the original image block of the high detail requirement region or the intermediate neural features extracted from the original image block to the high-fidelity retouching unit based on real-time network bandwidth information.

5. A real-time modeling system for three-dimensional meshed data based on dynamic capture of facial expressions according to claim 1, characterized in that, The high-precision geometric enhancement data is differential information relative to a neutral expression reference mesh template. This differential information is defined for regions with high detail requirements and is in the format of local vertex displacement fields, detail normal map indexes, or mesh patch containing subdivision instructions.

6. A real-time modeling system for three-dimensional meshed data based on dynamic capture of facial expressions according to claim 1, characterized in that, The adaptive streaming module defines and encapsulates streaming incremental data packets; The streaming incremental data packet includes at least global table case-varying parameters, sparse vertex offsets, and high-precision geometric augmentation data; The global expression variation parameters are used to drive the neutral expression reference mesh template to generate basic deformation; The sparse vertex offset is used to calibrate the position of key feature points.

7. A real-time modeling system for three-dimensional meshed data based on dynamic capture of facial expressions according to claim 1, characterized in that, The user terminal also includes a real-time fusion rendering unit; The real-time fusion rendering unit is used to receive streaming incremental data packets, sequentially apply global surface-varying parameters and sparse vertex offsets to the locally stored neutral expression reference mesh template to obtain an intermediate mesh, and then apply the mesh patch or displacement field corresponding to the high-precision geometric enhancement data to the corresponding area of ​​the intermediate mesh to complete the reconstruction and rendering of the final three-dimensional expression mesh.

8. A real-time modeling system for three-dimensional meshed data based on dynamic capture of facial expressions according to claim 2, characterized in that, The neutral expression reference mesh template is a pre-generated, personalized 3D mesh model bound to the user's identity, serving as a unified geometric reference established for each user during the system initialization phase.

9. A real-time modeling system for three-dimensional meshed data based on dynamic capture of facial expressions according to claim 2, characterized in that, The data acquisition and preprocessing module, the modeling and enhancement module, and the adaptive streaming module constitute a closed-loop collaborative mechanism. The regional complexity information and real-time network bandwidth information output by the perception and evaluation unit are used together as inputs to the hierarchical modeling engine to dynamically determine the data processing allocation strategy between the user terminal and the cloud computing node, as well as the generation and transmission content of high-precision geometric augmentation data.

10. A real-time modeling system for three-dimensional meshed data based on dynamic capture of facial expressions according to claim 7, characterized in that, When applying mesh patch, the real-time fusion rendering unit performs real-time subdivision of the local mesh topology or vertex replacement operation to achieve seamless geometric fusion between high-precision geometric enhancement data and the basic facial expression mesh.

Images