Geometric constraint-based face mouth shape coordination generation method

Abstract

The invention discloses a human face mouth shape coordination generation method based on geometric constraints, and relates to the technical field of image generation, and the method comprises the steps: 1, synchronously generating face vertex index tables corresponding to mouth opening and closing spatial form matrixes one by one; 2, based on the facial vertex index table, mapping the lip and gum combined boundary constraint set to a unified coordinate system to form a geometric prior constraint tensor; 3, performing multi-modal topological cascade reversible mixed transformation on the geometric prior constraint tensor to obtain a high-dimensional tooth texture candidate tensor and constraint consistency confidence spectrum; and step 4, using the high-dimensional tooth texture candidate tensor and the constraint consistency confidence spectrum as input, completing anti-aliasing, illumination compensation and gamma correction on a unified rendering pipeline, and finally obtaining a mouth shape coordination synthetic frame sequence. According to the method, the sense of reality, geometric consistency and time sequence coherence of tooth textures in the mouth shape synthesis process are effectively improved, and the problem that lips and teeth are not coordinated in an existing method is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image generation, and in particular to a method for generating coordinated face and mouth shapes based on geometric constraints. Background Art

[0002] Against the backdrop of the rapid development of facial modeling and expression-driven technologies, key issues surrounding lip shape generation, especially the modeling and rendering accuracy of the internal geometry and texture representation of the oral cavity, have become common challenges faced in many application scenarios. With the growing demand for virtual digital humans, voice-synchronized animation, video reconstruction, and lip shape replacement in film and television post-production, the authenticity and coordination of facial lip shape have directly affected the usability and visual consistency of the entire facial drive system. However, existing technical solutions still have a series of key technical shortcomings in the modeling and synthesis of the oral region, which are particularly prominent in terms of multimodal alignment, anatomical structure constraints, and texture generation consistency.

[0003] Current mainstream face modeling methods typically rely on 3D morphable models (such as 3DMM or FLAME) or deep learning-driven implicit shape representations for 3D fitting. These models can effectively restore the overall head shape, expression, and posture parameters, and ensure surface structure accuracy through keypoint alignment. However, they often suffer from structural simplification when handling the oral region. Specifically, to ensure uniformity and convergence speed during model training, many publicly available models adopt a unified topology, abstracting the interior of the mouth as a simple opening and closing parameter or mask layer. While this modeling approach can roughly control the opening and closing of the mouth, it cannot accurately represent the true spatial distribution of complex structures such as teeth, gums, and alveoli. In particular, it lacks the ability to resolve the lip-gum interface during intense facial expressions such as speaking, smiling, or grinning. This limitation directly leads to frequent problems such as missing tooth texture, blurred lip-tooth boundaries, and "hollow" areas in the mouth when generating lip-shaped images. Summary of the Invention

[0004] In order to solve the above technical problems, a method for generating coordinated facial lip shapes based entirely on geometric constraints is provided. By constructing the spatial morphological matrix of the mouth opening and closing, the alveolar posture reference system and the geometric prior constraint tensor, the fine modeling and dynamic control of the lip-tooth boundary and the internal structure of the oral cavity are achieved. Combined with multimodal feature transformation and confidence-weighted texture fusion, the realism, geometric consistency and temporal coherence of the tooth texture in the lip shape synthesis process are effectively improved, and the problems of lip-tooth incoordination, texture jump and rendering distortion in the existing methods are solved. It has significant advantages such as high fidelity, stable structure and strong adaptability.

[0005] In order to achieve the above objects, the technical solution adopted by the present invention is:

[0006] A method for generating coordinated facial and lip shapes based on geometric constraints, the method comprising:

[0007] Step 1: Use the 3D deformation model to initialize the 3D morphology of the face image to be processed, obtain the mouth opening and closing spatial morphology matrix, and simultaneously generate a facial vertex index table that corresponds one-to-one to the mouth opening and closing spatial morphology matrix;

[0008] Step 2: Based on the facial vertex index table, extract the lip contour, gingival boundary and alveolar pose, construct the lip-gingival joint boundary constraint set, and map the lip-gingival joint boundary constraint set to a unified coordinate system to form a geometric prior constraint tensor;

[0009] Step 3: Perform a multimodal topological cascade reversible hybrid transformation on the geometric prior constraint tensor to obtain a high-dimensional tooth texture candidate tensor and a constraint consistency confidence spectrum;

[0010] Step 4: Use the high-dimensional tooth texture candidate tensor and the constrained consistency confidence spectrum as input, map it according to the mouth opening and closing spatial morphology matrix, and complete anti-aliasing, lighting compensation, and gamma correction in a unified rendering pipeline to finally obtain a lip-coordinated synthesis frame sequence.

[0011] Furthermore, step 1 specifically includes: obtaining a face image to be processed, which is a static face image or a dynamic frame image sequence containing at least one face from a frontal or near-frontal perspective; performing face detection and facial key point positioning operations on the face image to be processed, extracting coordinates of several two-dimensional facial feature points including eyebrows, eyes, nose wings, and lips, and forming a two-dimensional facial feature point set; based on the two-dimensional facial feature point set, constructing an initialization corresponding mapping relationship that matches the three-dimensional deformation model, and calling the three-dimensional deformation model to perform three-dimensional fitting on the face image to be processed, and outputting a three-dimensional morphological parameter set, including shape parameters, expression parameters, and posture parameters; and recovering the face image to be processed according to the three-dimensional morphological parameter set. Process a three-dimensional deformable model with consistent resolution of facial images; the three-dimensional deformable model consists of several facial vertices, where each vertex corresponds to a three-dimensional coordinate value; in the three-dimensional deformable model, for a subset of vertices related to the upper and lower boundaries of the lips and the opening and closing areas of the mouth, extract their corresponding three-dimensional coordinate changes in each frame image, establish a continuous description function for the degree of lip closure, and generate a mouth opening and closing spatial morphology matrix; the mouth opening and closing spatial morphology matrix is ​​used to represent the spatial configuration of the oral cavity at different time points or expression states; all vertices in the three-dimensional deformable model are numbered according to their original order in the model initialization stage, and their spatial coordinate indexes and topological connection relationships are recorded to form a facial vertex index table.

[0012] Furthermore, the 3D morphable model uses a fixed-topology facial triangular mesh consisting of 53,215 facial vertices and 105,054 edges, with each vertex remaining numbered unchanged throughout the life cycle of the 3D morphable model. The fixed topology covers the entire visible area from the hairline to the mandible, and reserves deformable vertices on the inner edge of the lips to ensure the accuracy and consistency of the subsequent spatial morphological matrix of mouth opening and closing.

[0013] Furthermore, in step 2, in the facial vertex index table, the predefined upper lip outer edge vertex number range and the lower lip outer edge vertex number range are retrieved to obtain the upper lip outer edge vertex subset and the lower lip outer edge vertex subset respectively; according to the topological connection relationship of the three-dimensional deformation model, the upper lip outer edge vertex subset and the lower lip outer edge vertex subset are traversed ring by ring to obtain an ordered sequence of vertices that monotonically increases along the lip contour direction, forming a lip contour polygon chain; in the facial vertex index table, the gingival inner edge vertex number range is retrieved to obtain the gingival inner edge vertex subset; based on the vertex normal vector direction and the local curvature threshold, the redundant vertices connected to the hard palate are eliminated, and only the vertices located in the gingival-alveolar junction area are retained to generate a purified gingival boundary sequence.

[0014] Furthermore, in step 2, the centroid of each vertex in the purified gingival boundary sequence is calculated and recorded as the alveolar center point; the centroid of each vertex in the purified gingival boundary sequence is calculated and recorded as the alveolar center point; the right-handed coordinate system composed of the alveolar center point, the alveolar long axis direction, and the alveolar vertical direction is defined as the alveolar pose reference system; the lip contour polygon chain, the purified gingival boundary sequence, and the alveolar pose reference system are encapsulated together as a lip-gingival joint boundary constraint set; in the lip-gingival joint boundary constraint set, the following are recorded for each vertex: the original three-dimensional coordinate value, the normal vector, the curvature scalar, and the boundary type mark.

[0015] Furthermore, in step 2, the center point of the alveolar is taken as the origin, and the three-axis vectors of the alveolar posture reference system are used as basis vectors to define a unified coordinate system of the oral cavity; a rigid coordinate transformation is performed on all vertices in the lip-gingival joint boundary constraint set, and the three-dimensional coordinates of the vertices are mapped from the global coordinate system of the three-dimensional deformation model to the unified coordinate system of the oral cavity; a rigid coordinate transformation is performed on all vertices in the lip-gingival joint boundary constraint set, and the three-dimensional coordinates of the vertices are mapped from the global coordinate system of the three-dimensional deformation model to the unified coordinate system of the oral cavity; the mapped three-dimensional coordinates of the vertex, normal vector, curvature scalar and the boundary type mark are tensor-stored to form a geometric prior constraint tensor; a normalization operation is applied to the geometric prior constraint tensor so that the three-dimensional coordinates are distributed in the range of -1 to 1, the normal vector is distributed in the range of -1 to 1, and the curvature scalar is distributed in the range of 0 to 1; a normalization operation is applied to the geometric prior constraint tensor so that the three-dimensional coordinates are distributed in the range of -1 to 1, the normal vector is distributed in the range of -1 to 1, and the curvature scalar is distributed in the range of 0 to 1.

[0016] Furthermore, in step 3, the geometric prior constraint tensor is separated according to the four attribute dimensions of vertex three-dimensional coordinates, normal vectors, curvature scalars and boundary type labels to obtain four single-mode tensor branches; for the three-dimensional coordinates in each single-mode tensor branch, the Euclidean space point is embedded in the hypersphere space with a radius of 1 using stereoscopic projection to form a hypersphere embedding tensor; linear dimension expansion is performed on the normal vector, curvature scalar and boundary type label to make their dimensions consistent with the hypersphere embedding tensor; the topological reentrant kernel function is called in batches on the hypersphere embedding tensor, which converts the tensor into a hypersphere through a reversible torus mapping. The radial amplitude on the surface is compressed into hyperbolic amplitude, and the absolute phase information is retained at each reentry point; the cascaded multiple-address sampling function is called to perform multiple hop sampling with an exponentially decaying step size in the reentry domain to obtain cross-scale feature representation; the topological reentry kernel function and the cascaded multiple-address sampling function alternate to form a complete iteration; after each iteration, the global topological energy index is calculated. If the index decreases by less than one thousandth compared with the previous iteration, early convergence is triggered; otherwise, the next iteration is continued; if the number of iterations reaches one hundred and still has not converged, the iteration result with the lowest energy is taken as the provisional output tensor.

[0017] Furthermore, in step 3, the tentative output tensor is input into the greedy minimum topological energy path searcher; the searcher uses the global topological energy index as the cost function, performs a depth-first traversal on the high-dimensional energy graph of the tensor, and evaluates the geometric consistency cost of each candidate path in real time; when the geometric consistency cost is lower than the preset threshold of 50% for the first time, the current path is marked as the optimal topological path and the search is terminated; a reversible hybrid reverse mapping is performed on the tensor on the optimal topological path, and the transformations of the cascaded multi-access sampling function and the topological reentry kernel function are undone in turn to obtain the reconstructed modal tensor back to the Euclidean space; the four reconstructed modal tensors are arranged according to The original modal order is reassembled to obtain a three-dimensional array of tooth texture candidates; the first 128 elements of the three-dimensional array of tooth texture candidates are taken and stacked according to the original sequence to form a high-dimensional tooth texture candidate tensor; for each candidate instance in the high-dimensional tooth texture candidate tensor, its Euclidean consistency distance with the geometric prior constraint tensor in the three attribute dimensions of vertex three-dimensional coordinates, normal vector and curvature scalar is calculated; the Euclidean consistency distance is mapped to a confidence level in the interval 0 to 1 using a soft maximum operation to obtain the corresponding single instance confidence value; the 128 single instance confidence values ​​are arranged in sequence to form a constraint consistency confidence spectrum with a length of 128.

[0018] Furthermore, step 4 specifically includes: one-to-one correspondence between the high-dimensional tooth texture candidate tensor and the constrained consistency confidence spectrum according to the candidate serial number dimension; applying the corresponding single instance confidence value in the constrained consistency confidence spectrum to each candidate texture instance in the high-dimensional tooth texture candidate tensor as a fusion weight to obtain a confidence-weighted tooth texture fusion tensor; performing weighted summation on the confidence-weighted tooth texture fusion tensor along the candidate serial number dimension to obtain a single fused tooth texture representation vector, and the fused tooth texture representation vector is used to represent the optimal tooth texture state corresponding to the current input sample; expanding the fused tooth texture matrix into a one-dimensional vector by column, and rearranging it into a two-dimensional mapping grid according to the unified UV layout table of the tooth area; performing bidirectional Laplace interpolation filling on the missing pixels in the two-dimensional mapping grid, and filling the edge pixels. Perform Gaussian weighted smoothing to form an initial tooth texture map; apply local histogram equalization to the initial tooth texture map to enhance the enamel highlights and tooth neck shadow details, and output an enhanced tooth texture map; obtain the mouth opening and closing spatial morphology matrix corresponding to the enhanced tooth texture map; for each time frame in the mouth opening and closing spatial morphology matrix, locate the corresponding oral region vertex subset in the facial vertex index table; map the two-dimensional texture coordinates of the enhanced tooth texture map to the oral region vertex subset in the three-dimensional deformation model according to the mouth opening and closing spatial morphology matrix of the current frame, and generate a frame-level tooth texture vertex map dataset; perform anti-aliasing, lighting compensation and gamma correction on each frame in the frame-level tooth texture vertex map dataset on a unified rendering pipeline, and finally obtain a lip-coordinated synthesis frame sequence.

[0019] Compared with existing technologies, the present invention has the advantage of being able to achieve high-precision, temporally continuous, and visually natural lip synthesis while maintaining consistency between the real facial structure and the inside and outside of the mouth. By introducing a three-dimensional deformation model and constructing a spatial morphological matrix for mouth opening and closing, the present invention achieves quantitative modeling of the dynamic geometric behavior of the mouth, effectively solving the problems of discontinuous lip and tooth structure and the inability to accurately control the mouth opening and closing state in traditional methods. Furthermore, the alveolar posture reference system and unified oral coordinate system defined in the present invention ensure that the boundary expression of the lip margin and gum remains consistent across multiple frames, significantly improving the accuracy of spatial mapping across time steps. In terms of texture generation, the present invention guides the generation and screening of candidate textures through geometric prior constraint tensors and uses a confidence-weighted fusion strategy to avoid the texture jumps and distortion caused by traditional hard matching, further enhancing the realism and stability of tooth texture details. Furthermore, the anti-aliasing, lighting compensation, and gamma correction mechanisms introduced in the rendering process ensure that the final synthesized frame sequence maintains a natural color and clear contrast visual effect under different lighting and display environments. Overall, the present invention can take into account anatomical consistency, texture continuity and optical naturalness, and has broad application prospects in the fields of facial animation, voice-driven synthesis, virtual digital humans and high-quality lip replacement. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a schematic diagram of the method flow of the method for generating coordinated face and mouth shape based on geometric constraints proposed by the present invention;

[0021] Figure 2 A schematic diagram of generating a three-dimensional deformable model and a mouth opening and closing spatial morphology matrix in an embodiment of the present invention;

[0022] Figure 3 Schematic diagram of lip contour and gum boundary extraction in an embodiment of the present invention;

[0023] Figure 4 Schematic diagram of hypersphere embedding in an embodiment of the present invention;

[0024] Figure 5 Schematic diagram of a topological reentry kernel function and a cascaded multiple access sampling function in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are merely examples, and those skilled in the art may conceive of other obvious variations.

[0026] Reference Figure 1 As shown, a method for generating coordinated face and mouth shapes based on geometric constraints includes:

[0027] Step 1: Use the 3D deformation model to initialize the 3D morphology of the face image to be processed, obtain the mouth opening and closing spatial morphology matrix, and simultaneously generate a facial vertex index table that corresponds one-to-one to the mouth opening and closing spatial morphology matrix;

[0028] Step 2: Based on the facial vertex index table, extract the lip contour, gingival boundary and alveolar pose, construct the lip-gingival joint boundary constraint set, and map the lip-gingival joint boundary constraint set to a unified coordinate system to form a geometric prior constraint tensor;

[0029] Step 3: Perform a multimodal topological cascade reversible hybrid transformation on the geometric prior constraint tensor to obtain a high-dimensional tooth texture candidate tensor and a constraint consistency confidence spectrum;

[0030] Step 4: Use the high-dimensional tooth texture candidate tensor and the constrained consistency confidence spectrum as input, map it according to the mouth opening and closing spatial morphology matrix, and complete anti-aliasing, lighting compensation, and gamma correction in a unified rendering pipeline to finally obtain a lip-coordinated synthesis frame sequence.

[0031] This method considers the oral cavity, a highly variable and anatomically constrained region, as driven by two coupled continuous curves: "spatial form" and "texture state." The former defines the temporal evolution of the dentition and lip margin in 3D space, while the latter describes the optical distribution of enamel, gingiva, and even oral soft tissues in 2D space. To simultaneously recover these two curves from any input facial image sequence and maintain their exact match across frames, the algorithm first performs morphological initialization using a 3D morphable model. This step goes beyond simply fitting 2D pixels to a 3D mesh. Instead, it simultaneously maps shape, expression, and posture parameters onto a unified anatomical reference surface. This is further abstracted into a spatial morphological matrix of the mouth's opening and closing, capturing geometric metrics such as the time-varying opening and closing angle at the oral entrance, the distance between the upper and lower lips, and the curvature of the border. Furthermore, a simultaneously generated facial vertex index table creates an immutable global index for all vertices, ensuring that subsequent frame differences are reflected only in coordinate values ​​rather than topological relationships, ultimately ensuring temporal consistency.

[0032] Next, the algorithm extracts three geometric entities based on this index table: the lip contour, the gingival boundary, and the alveolar posture. These entities, located at the interfaces between soft tissue and air, the mucosa and alveolar bone, and the alveolar bone and the overall oral posture, respectively, physically constitute the closed boundary of the oral entrance. By mapping these three entities into a unified coordinate system centered at the alveolar center, a geometric prior constraint tensor with consistent scale and orientation across individuals and expressions is obtained. This tensor is essentially a set of high-dimensional vector fields, whose components correspond to the three-dimensional coordinates of vertices, normal vectors, curvature scalars, and boundary type markers. Because these components are derived from the shared geometric features of soft and hard tissues, their temporal variations are much more stable than those of pixel-level textures, thus providing reliable morphological anchors for subsequent texture generation. The third step introduces a multimodal topological cascade reversible hybrid transformation. This transformation aims not at simple compression or dimensionality reduction, but rather at uniformly stretching the inherent geodesic distances of each mode in Euclidean space into topological energy by repeatedly switching between hypersphere and hyperbolic space. Multiscale focused search is then achieved using the exponential decay strategy of the cascaded multiple access sampling function. The reversibility of the entire process means that no information is lost at any stage; the algorithm simply rearranges information in different geometric domains to expose potential cross-modal alignment relationships. When the global topological energy index reaches stability or an iteration limit, the searcher selects the path with the lowest cost and optimal geometric consistency on the energy graph, reverse-mapping it back to Euclidean space to generate a high-dimensional tooth texture candidate tensor. Simultaneously, the Euclidean consistency distance between the candidate instance and the geometric prior constraint tensor in three dimensions—vertex coordinates, normal vectors, and curvature scalars—is calculated and normalized using a softmax function to obtain a constraint consistency confidence spectrum.

[0033] This confidence spectrum can be regarded as a statistical evaluation of the degree of match between the candidate texture and the real geometry. The higher the value, the more the candidate texture conforms to the morphological constraints in key details such as the crown edge and the gingival sulcus line. In the last step, the high-dimensional tooth texture candidate tensor and the constraint consistency confidence spectrum are weighted and fused one by one according to the serial number to obtain a single fused tooth texture representation vector, which is then flattened into a two-dimensional mapping grid according to the unified UV layout. In order to eliminate sampling holes and smooth the boundaries, bidirectional Laplace interpolation is used to fill the missing pixels, and then the peripheral color level is corrected with a Gaussian weighted kernel. Subsequently, local histogram equalization is performed to enhance the contrast of enamel highlights and tooth neck shadows. At this point, a one-to-one mapping has been established between the fused texture and the spatial morphological matrix of the mouth opening and closing. The system traverses each time frame in the matrix and projects the mapping coordinates back to the oral vertex subset of the three-dimensional deformation model to obtain a frame-level tooth texture vertex map dataset. The unified rendering pipeline performs anti-aliasing on each frame during the output phase to avoid pixel jagged edges where the lip edge meets the gums. A lighting compensation algorithm balances energy on the oral cavity's sensitive inner walls to ensure consistent brightness across frames. Gamma correction further eliminates differences in color gradation responses across different display devices. The resulting lip-coordinated synthetic frame sequence not only visually maintains a smooth transition between upper and lower lip closure and opening, but also maintains high consistency with the real image in terms of dental occlusion, gum exposure, and even alveolar parallax details, achieving dynamic restoration driven by the dual constraints of oral geometry and texture. This progressive design, from morphology to texture to optical rendering, enables the algorithm to output stable and reliable results even when dealing with rapid speech, exaggerated expressions, and even scenes with drastic lighting changes. This establishes a complete and rigorous technical pipeline for applications such as real-time expression driving, facial video redirection, and live broadcasting of virtual digital humans.

[0034] Furthermore, in the geometrically constrained face-mouth coordination generation method, the core principle of step 1 can be understood as transforming facial observations, which are discrete and susceptible to illumination, pose, and occlusion in the 2D pixel domain, into a dynamic morphological description with complete topological structure and physical interpretability in 3D Euclidean space, thereby providing deterministic boundary conditions for the subsequent geometric constraint calculations. The system first requires a set of 2D facial landmark locators covering key anatomical landmarks such as eyebrows, eyes, nose wings, and lips. These locators output precise pixel coordinates in the image coordinate system through a convolutional neural network. These coordinates contain only local grayscale gradient information and lack depth and true metric properties, making them incapable of directly inferring oral volume or lip curvature. To compensate for the lack of depth, the algorithm incorporates a 3D morphable model, a linear subspace model whose shape parameters statistically encode facial differences based on race, gender, and age. Expression parameters control the nonrigid deformation of soft tissue in the surface coordinate system. Pose parameters ensure that the global position and orientation of the head can freely vary in the world coordinate system. To align the external 2D observations with the internal 3D basis, the system predefines an initial mapping relationship. This mapping is generated using a combined average face shape and thin plate spline alignment based on a homologous training set, ensuring that each 2D facial feature point has a specific candidate vertex on the 3D morphable model. During the fitting phase, the optimizer minimizes the projection error and regularization term while preserving the topology of the 3D morphable model. This ensures that shape parameters do not overfit local noise, expression parameters do not overbend the patch, and pose parameters maintain depth consistency for monocular reconstruction. The projection function uses perspective geometry to project 3D vertices back onto the 2D image plane and backpropagates gradients via partial derivatives, allowing shape, expression, and pose parameters to converge simultaneously in a single end-to-end backpropagation. This joint estimation based on differentiable projections ensures a natural decoupling of the coupled parameters: shape parameters account for long-term geometric features that remain invariant across frames, expression parameters respond to snapshot-level muscle motion, and pose parameters account for only global rigid body transformations. After convergence, the recovered 3D morphable model maintains a vertex density consistent with the input image resolution, ensuring that the model accurately fits the original contours while avoiding accumulation of resampling errors during subsequent texture mapping.

[0035] To measure the dynamic state of the oral region, the algorithm extracts a subset of vertices related to the upper and lower lip boundaries and the mouth's opening and closing regions from the vertex set of the 3D deformable model. This selection criterion is not a simple filter based on spatial location, but rather based on pairs of curvature extrema corresponding to the lip edges, ensuring a topologically consistent vertex sequence across different face shapes. For each frame, the system records the relative changes in the 3D coordinates of these vertices and combines continuous metrics such as the shortest Euclidean distance between the upper and lower lips, the angle between the local normal vectors, and the cross-sectional area of ​​the mouth into a continuous function describing the degree of mouth closure. This function is discretely sampled across all frames and stacked in time to generate a spatial morphology matrix for mouth opening and closing. The matrix's row indices map to frame timestamps, while its column indices correspond to specific geometric metrics. This fully describes the evolution of the oral cavity's instantaneous state during speech production, emotional expression, or spontaneous movement. Because the matrix elements are derived directly from the 3D coordinates and do not contain any texture or color information, they are inherently immune to external factors such as lighting, occlusion, and camera exposure, providing a high-confidence reference for the subsequent construction of the geometric prior constraint tensor.

[0036] The facial vertex index table is designed to ensure consistent addressing across frames, stages, and even modules. During model initialization, the system assigns globally unique numbers to all vertices in native vertex array order and stores each vertex's in-memory offset, patch index, and a linked list of neighboring vertices. The static nature of this index table means that as long as the 3D morphable model maintains topology throughout its lifetime, any algorithmic step can use the same number to access the same anatomical location, eliminating the need for repeated construction or spatial search inference. The topological connections recorded in the index table can be directly accessed later when calculating local tangent planes, discrete curvature, or performing loop traversals, significantly reducing time complexity. More importantly, the index table provides constant-time retrieval of a subset of lip boundary vertices. When the mouth opening and closing spatial morphology matrix is ​​updated at runtime, the corresponding rows and columns are refreshed synchronously simply by vertex index, eliminating the need for hashing or depth-first search of the entire mesh. This design transforms the unordered problem of continuous frame deformation into the problem of evaluating an ordered sequence of vertex indices, fundamentally ensuring the robustness and interpretability of lip coordination generation in the temporal dimension.

[0037] Furthermore, a fixed-topology facial triangle mesh serves as the "geometric skeleton" for the geometrically constrained face-mouth coordination generation method. Its vertices, edges, and patches remain intact throughout the algorithm's lifecycle and are immutable with input samples, ensuring that all geometric processing is based on the same topological baseline. This mesh consists of 53,215 facial vertices and 105,054 edges. The number of patches corresponds one-to-one with the number of vertices and edges via an Euler relationship, and each patch is a triangle. This ensures that the mesh can be directly accepted by the hardware rasterization pipeline during transmission over the GPU data pipeline, eliminating the need for dynamic retriangulation. Vertex numbers are written once to the model metadata file during data preprocessing and remain resident in memory as sequential integers. This facilitates fast addressing via offsets in downstream modules and ensures that vertex index mapping remains identical across frames, completely eliminating potential topological ambiguity between frames. To ensure that the fixed topology balances macroscopic integrity with local plasticity, the mesh covers the entire visible area from the hairline to the mandible, with a deformable vertex strip reserved at the inner edge of the lips. Specifically, a ring of vertices with slightly higher density is inserted at the junction of the lip and the oral cavity. These vertices are marked as "deformation freedom enhancement" type during the mesh generation stage, and their first-order and second-order neighbor weights are assigned lower Laplace smoothing coefficients in the constraint matrix. This allows for a higher deformability than other facial areas when driven by local expressions or when the mouth opens and closes. Since the curvature change of the inner edge of the lip directly determines the accuracy of the spatial morphological matrix of the mouth opening and closing, insufficient vertex density or overly rigid topological connections can cause the lip edge surface to be overstretched or folding artifacts to appear. This deformable vertex strip can locally enhance the geometric freedom while maintaining the global topology unchanged, making the closure degree description function show smoother and more refined changes in the continuous time domain.

[0038] To avoid redundant storage caused by localized high vertex density, the mesh uses a multi-level subdivision template in other areas: medium density is set for the cheeks and forehead, and low density is set for the hairline and mandible transition zone. A four-level level of detail flag is also recorded in the mesh vertex attribute table. During rendering, vertex sets of different resolutions can be dynamically selected based on viewing distance or actual requirements, thus achieving a balance between performance and accuracy. At the data structure level, all vertex coordinates, normals, and their topological adjacency indices are laid out as structure arrays. The vertex index table is stored as a separate read-only constant in a read-only buffer in the graphics memory, ensuring write conflicts are avoided when accessed by multiple threads in parallel. When constructing the spatial morphology matrix for mouth opening and closing, the algorithm only queries the subset of vertices related to the upper and lower lip boundaries and the oral opening and closing area. The numbering of these vertex segments is included in the metadata manifest during model packaging, allowing queries to be completed in constant time. This approach maintains the semantic consistency provided by the global numbering while ensuring that the sampling resolution of local deformations is sufficient to capture even the most subtle lip edge dynamics.

[0039] Furthermore, in the geometrically constrained method for generating coordinated facial lip shapes, the core task of step 2 is to transform the discrete and unordered set of lip and gingival vertices into a chain of lip contour polygons and a sequence of purified gingival boundaries that can be continuously indexed by arc length. This process relies on the "number-topology-geometry" triple mapping provided by the facial vertex index table: the number determines the search scope, the topology defines adjacency relationships, and the geometric properties distinguish soft and hard tissue within a local area. First, the system directly locates two lip edge subsets by searching the predefined upper and lower lip outer edge vertex number ranges. The predefined numbering is based on the topological rings at the time of model generation and is unaffected by individual shape and expression parameters, ensuring consistency across users and frames. This numbering-based search essentially reduces the complex spatial search to an integer interval query, significantly reducing initialization time. The algorithm then performs a ring-by-ring traversal of the upper and lower lip outer edge vertex subsets based on the topological connectivity of the 3D deformable model. Unlike a simple adjacency list crawler, loop-by-loop traversal requires consistent curvature sign and edge orientation throughout the traversal process to achieve a monotonically increasing vertex order along the lip contour at each step. After the traversal is complete, the system uses topological clipping to patch closed loops at the junction of two ordered sequences, ultimately outputting a self-intersecting, end-to-end connected lip contour polygon chain with a monotonic arc length. The significance of this chain lies in discretizing the original two-dimensional curve into a one-dimensional parameter domain. Subsequent texture mapping and curvature analysis require only linear interpolation within the chain index space, eliminating the need to return to the three-dimensional mesh for a costly path search.

[0040] The extraction process for the gingival border is similar to that for the lip border, but the challenge lies in the lack of a clear geometric boundary between the gingiva and hard palate. Instead, the border is defined by a transition between normal vectors and a gradient of curvature. The algorithm first retrieves the vertex number range of the inner gingival border from the facial vertex index table, obtaining a subset of inner gingival border vertices that includes redundant hard palate vertices. To eliminate redundancy, the system calculates the normal direction of each vertex and compares the angle with the normal reference of the alveolar pose reference system. Hard palate vertices face the oral dome, with normal vectors mostly pointing toward the interior of the cranial cavity, while gingival vertices face the alveolar bone, with normal vectors tending toward the front of the mouth. A threshold is set to distinguish between these two orientations, allowing for the initial removal of the hard palate region. Local curvature is then estimated for the remaining vertices. This curvature exhibits a distinct inflection at the hard palate-gingival transition. Using an increasing curvature window, the system identifies the inflection point and discards the hard palate vertices following it. Ultimately, only vertices located at the gingival-alveolar junction are retained, resulting in a purified gingival border sequence. The purified sequence ensures surface continuity while aligning with the polar coordinate parameterized framework of the alveolar center point, facilitating subsequent rigid coordinate transformations and tensor storage. Through this multi-stage screening process of "number retrieval - topology traversal - normal filtering - curvature removal," Step 2 compresses the critical soft and hard tissue boundaries into two highly controllable one-dimensional ordered vertex sequences without destroying the global topology. This provides a precise spatial anchor for the geometric prior constraint tensor and establishes a morphological benchmark for subsequent tooth texture candidate search and constraint consistency assessment.

[0041] Furthermore, in the geometric prior construction process of the geometrically constrained face-mouth coordination generation method, the purified gingival boundary sequence serves as a geometric bridge connecting soft and hard tissues. The system first takes the arithmetic average of the three-dimensional coordinates of all vertices in the sequence in the world coordinate system to obtain a unique center of mass, which is designated as the alveolar center. The center of mass is chosen as the origin because it simultaneously minimizes the squared distance to all gingival vertices, making the translation component in subsequent coordinate transformations statistically stable. The algorithm then performs a principal axis analysis on the purified gingival boundary sequence, extracting the eigenvector with the largest variation in the longitudinal direction relative to the boundary, denoted as the alveolar long axis. The normal vector is then estimated on the tangent plane of the local fit of the purified gingival boundary sequence and orthogonalized to the alveolar long axis to obtain the alveolar perpendicular direction. Combining the alveolar center, alveolar long axis, and alveolar perpendicular direction according to the right-hand rule forms the alveolar pose reference system. This local coordinate system not only conforms to the individual alveolar bone orientation in a three-dimensional geometric sense but is also decoupled from global head pose parameters, thus providing a unified and stable pose reference across multiple frames or samples. The algorithm then maps the lip contour polygon chain and the purified gingival boundary sequence into the alveolar pose reference frame, normalizing their 3D coordinates, normal vectors, and curvature scalars to a standardized representation independent of the individual head pose. The relative positions of the 3D point sets are no longer affected by the camera's perspective and rely solely on the true geometric relationship between the lip edge and gingiva.

[0042] To facilitate subsequent reversible topological recursion in high-dimensional feature space, the system encapsulates the lip contour polygon chain, the purified gingival boundary sequence, and the alveolar pose reference frame into a lip-gingival joint boundary constraint set. This encapsulation is more than a simple structural combination; it also establishes explicit coupling across boundary types at the data semantic level: for each vertex in the joint boundary constraint set, the algorithm simultaneously records the original 3D coordinates, normal vector, curvature scalar, and boundary type tag. The original 3D coordinates retain the true spatial position, the normal vector characterizes the local surface orientation, the curvature scalar measures the degree of surface curvature, and the boundary type tag uses integers to semantically group vertices. This ensures that the multimodal topological cascade reversible hybrid transformation can be aligned dually along the attribute dimension and the boundary semantic dimension during subsequent processing. The same vertex has the same number across different time frames, but its coordinates, normal vector, and curvature change as the expression evolves. This design makes the joint boundary constraint set a composite data structure that combines static numbering with dynamic geometric signals. Because the alveolar pose reference system provides a local orthogonal basis, the components of the normal vector under this basis directly reflect the actual inclination angle of the vertex relative to the alveolar bone. The curvature scalar can be obtained through ellipsoid fitting or discrete Laplacian operator, and the numerical distribution is constrained to the range of zero to one to perform dimensionless comparisons between different vertices. Boundary type labeling strictly distinguishes the lip contour from the gingival boundary, allowing the subsequent soft maximum consistency mapping to adaptively assign weights according to boundary type, preventing lip edge details from being overwhelmed by the high curvature peaks of the gingiva.

[0043] Through the above steps, the algorithm not only transforms the two originally independent boundaries into collaborative entities living together in the alveolar posture reference frame, but also prepares a complete triple eigenvector of shape, direction and curvature, as well as a clear category label for each vertex. The joint boundary constraint set thus becomes the direct upstream input of the geometric prior constraint tensor, and has complete geometric context information before entering the multimodal topological cascade reversible hybrid transformation. The entire process self-consistently solves the three key problems of local coordinate unification, boundary combination and attribute recording, providing a reversible, interpretable, and high-quality morphological anchor that conforms to individual anatomical characteristics for the subsequent generation of high-dimensional tooth texture candidate tensors.

[0044] Furthermore, in step 2, the center point of the alveolar is taken as the origin, and the three-axis vectors of the alveolar posture reference system are used as basis vectors to define a unified coordinate system of the oral cavity; a rigid coordinate transformation is performed on all vertices in the lip-gingival joint boundary constraint set, and the three-dimensional coordinates of the vertices are mapped from the global coordinate system of the three-dimensional deformation model to the unified coordinate system of the oral cavity; a rigid coordinate transformation is performed on all vertices in the lip-gingival joint boundary constraint set, and the three-dimensional coordinates of the vertices are mapped from the global coordinate system of the three-dimensional deformation model to the unified coordinate system of the oral cavity; the mapped three-dimensional coordinates of the vertex, normal vector, curvature scalar and the boundary type mark are tensor-stored to form a geometric prior constraint tensor; a normalization operation is applied to the geometric prior constraint tensor so that the three-dimensional coordinates are distributed in the range of -1 to 1, the normal vector is distributed in the range of -1 to 1, and the curvature scalar is distributed in the range of 0 to 1; a normalization operation is applied to the geometric prior constraint tensor so that the three-dimensional coordinates are distributed in the range of -1 to 1, the normal vector is distributed in the range of -1 to 1, and the curvature scalar is distributed in the range of 0 to 1.

[0045] For the sake of numerical stability and cross-sample comparability, the system applies normalization operations to the geometric prior constraint tensor. First, the extreme values ​​of the three-dimensional coordinates after mapping in the entire joint boundary constraint set are counted, and all coordinates are compressed to the range of -1 to 1 with the maximum absolute value as the denominator; the normal vector itself is already a unit vector, but in order to maintain consistency with the coordinate distribution, a linear stretch is also performed on each component so that its value falls between -1 and 1. The curvature scalar is usually non-negative, and the numerical range may differ by an order of magnitude depending on the individual face shape. The algorithm performs an inverse scaling on its global maximum value, so that the curvature distribution is mapped to the range of 0 to 1, while retaining the contribution of high curvature peaks to the sensitivity of subsequent features. The boundary type label is not normalized because this field is a discrete semantic label and should not participate in continuous space scaling. After normalization, the tensor becomes a geometrically constrained tensor with a controlled numerical domain, clear attribute dimensions, and a consistent coordinate system. It maintains static topological consistency between vertices at the storage level and eliminates scale and posture differences at the numerical level, providing an input format with minimum redundancy and maximum information gain for subsequent multimodal topological cascade reversible hybrid transformations.

[0046] Furthermore, during the high-dimensional feature modeling phase of the geometric constraint-based face-mouth coordination generation method, the geometric prior constraint tensor, after entering step 3, is first separated into four attribute dimensions: vertex 3D coordinates, normal vectors, curvature scalars, and boundary type labels. This allows the inherently different geometric information to reside in statistically homogeneous subspaces, thereby reducing coupling interference during subsequent transformations. The 3D coordinate branch directly characterizes the spatial distribution of vertices in the unified oral coordinate system and is a dimension with a significant Euclidean distance. The normal vector branch reflects the orientation of the local tangent plane, with a constant vector norm but a complex directional distribution. The curvature scalar branch reflects second-order morphological changes and is highly sensitive to local concavity. The boundary type label branch is a discrete semantic label that only logically distinguishes between the lip edge and the gingiva. To handle these four modalities within the same reversible geometric framework, the algorithm embeds the 3D coordinate subtensors into a hypersphere of radius 1 via stereographic projection, thereby transitioning all points from linear Euclidean space to a Riemannian manifold with constant curvature. This compresses the scale differences between the different coordinate dimensions, transforming the distance metric into an angle metric on the sphere.

[0047] To maintain consistent tensor dimensionality, the system linearly expands the normal vector, curvature scalar, and boundary type marker to the same vector length as the hypersphere embedding tensor. The normal vector branch is directly replicated and normalized, the curvature scalar branch is interpolated to fill the expanded dimension with continuous values, and the boundary type marker is mapped to the same dimension using a one-hot encoding. The four equal-length unimodal tensors are then fed into a topological reentry kernel function, which performs a reversible torus mapping on the hypersphere embedding tensor, compressing the radial amplitude into the hyperbolic amplitude domain. The constant curvature transition to negative curvature significantly improves resolution in highly convex or concave regions, while recording absolute phase information at each reentry to ensure reversibility. A cascaded multi-access sampling function is then called, performing multiple hop sampling within the reentry domain with an exponentially decaying step size. This hopping strategy is equivalent to gradually downsampling regions away from the reentry center while preserving local high-frequency details, automatically balancing detail preservation with global context. The topological reentrant kernel function and the cascaded multiple access sampling function are executed alternately to form a complete iteration. After each iteration, the global topological energy index is calculated. This index combines the Boolman energy of the tensor in the hyperbolic amplitude domain, the reentrant phase consistency, and the cross-modal alignment error. The lower the value, the better the geometric consistency. When the energy of the current iteration decreases by less than one thousandth compared to the previous round, it means that the marginal benefit of continuing to iterate is extremely low, and the algorithm immediately triggers early convergence to save computing resources; if the energy still decreases significantly, the iteration will continue, and it will be run up to one hundred times to avoid falling into an infinite loop. If the convergence condition is still not met before the iteration reaches the upper limit, the system selects the iteration result with the lowest energy as the provisional output tensor, because this result represents the optimal compromise between geometric consistency and cross-modal balance in the searchable energy landscape. The entire process uses a reversible mapping from the hypersphere to the hyperbolic space to pull the information of the four modes into the same topological domain, and then adaptively screens the most geometrically representative features through multi-scale sampling and energy monitoring, so that the search space of the subsequent high-dimensional tooth texture candidate tensor can fully accommodate the local deformation of the lip edge and gums without being overwhelmed by high-dimensional noise, thereby ensuring that the constrained consistency confidence spectrum can respond sensitively and stably to the real geometric conditions.

[0048] Furthermore, in the geometrically constrained face-mouth coordination generation method, the provisional output tensor is merely a set of candidate states found in the high-dimensional energy landscape by a multimodal topological cascade reversible mixing transformation, which have not yet been refined for geometric consistency. The locations that truly represent the optimal coupling of tooth texture and lip-tooth geometry must be further screened using a greedy minimum topological energy path searcher. This searcher first maps the provisional output tensor into a high-dimensional energy graph. Each node in the energy graph corresponds to a combination of reentrant phase, hyperbolic amplitude, and cross-modal alignment error, while edge weights are given by a global topological energy metric. This global topological energy metric aggregates the mutual information loss between the four modalities and the topological distortion within the tensor, so a lower value indicates a higher degree of geometric matching. The searcher uses a depth-first traversal rather than a breadth-first traversal because the high-dimensional energy graph exhibits a distinct hierarchical gradient structure, and connected branches along decreasing energy directions tend to reach local minima more quickly. During the traversal process, each time the recursion goes deeper into a layer, the geometric consistency cost of the current path is calculated immediately; this cost is measured by the cumulative Euclidean difference between the three-dimensional coordinates of the vertex, the normal vector, and the curvature scalar in the provisional output tensor and the original geometric prior constraint tensor. The greedy strategy is reflected in the fact that as soon as the geometric consistency cost of a path is found to be lower than the preset threshold of 50% for the first time, the path is immediately locked as the optimal topological path and the traversal is terminated, avoiding the exponential computational redundancy caused by continuing the search when the quality requirements have been met. This early stopping mechanism uses the correlation between the geometric consistency cost and the energy index to converge quickly, while ensuring that the optimal path still maintains low energy characteristics.

[0049] After determining the optimal topological path, the system performs a reversible hybrid inverse mapping on the tensors along the path, strictly undoing the transformations of the cascaded multi-access sampling function and the topological reentrant kernel function in their order. When undoing the cascaded multi-access sampling function, the low-scale samples are inserted back to their corresponding high-scale locations according to the jump log. The torus map is then reversed based on the reentrant phase information, and the hyperbolic amplitudes are re-expanded back to the hyperspherical radial amplitudes, ultimately returning to Euclidean space. This reversible step ensures that any optimal path obtained within the high-dimensional energy graph can be losslessly restored as a reconstructed modal tensor that corresponds one-to-one with the original four modes. The algorithm then reconstructs the four reconstructed modal tensors in the original modal order (vertex 3D coordinates, normal vectors, curvature scalars, and boundary type labels) to obtain a 3D array of tooth texture candidates. The first 128 elements of the array are stacked in their original order to form the high-dimensional tooth texture candidate tensor. This upper limit of 128 controls the complexity of subsequent distance calculations while ensuring sufficient diversity in the candidate set to capture local extrema.

[0050] Next, the system processes each candidate instance in the high-dimensional tooth texture candidate tensor one by one, and calculates the Euclidean consistency distance with the geometric prior constraint tensor in the three attribute dimensions of vertex three-dimensional coordinates, normal vectors, and curvature scalars. The smaller the distance, the more consistent the candidate instance is with the actual position of the lips and teeth in the geometric space. However, in order to avoid the sharp selection caused by the simple minimum value and sacrifice the overall stability, the algorithm performs a soft maximum operation on the distance and maps it to a confidence interval from zero to one. The soft maximum operation is essentially an exponential weighting of the negative distances of all candidate instances, and then normalized to a probability distribution. Therefore, the confidence value of each single instance reflects both its own matching degree and the relative advantages and disadvantages with other candidates. When all one hundred and twenty-eight single instance confidence values ​​are calculated and arranged according to the element number, the system obtains a constraint consistency confidence spectrum with a length of one hundred and twenty-eight. The curve shape of the confidence spectrum not only intuitively shows which groups of tooth textures best meet the geometric constraints, but also provides continuous weights for subsequent confidence-weighted fusion, ensuring that the final fusion result maintains a smooth transition to low-confidence textures when selecting high-confidence textures, preventing texture jumps caused by the dominance of a single candidate. Through this continuous chain of greedy search, reversible inverse mapping, and soft maximum normalization, step three transforms the abstract energy minimum solution extracted from the multimodal reentry domain into a high-dimensional tooth texture candidate tensor and its reliable measurement that simultaneously meets geometric consistency in physical space, direction space, and curvature space. This lays the data and statistical foundation for subsequent steps to complete precise mapping and optical correction in a unified rendering pipeline.

[0051] Furthermore, the system maps high-dimensional tooth texture candidate tensors to constraint-consistency confidence spectra along the candidate sequence number dimension, treating each single-instance confidence value in the confidence spectrum as the fusion weight for that candidate texture instance. Unlike traditional hard-selection of the best texture, a soft fusion strategy is employed: weights are directly multiplied by the high-dimensional tooth texture candidate tensor to produce a confidence-weighted tooth texture fusion tensor. A weighted summation is performed along the candidate sequence number dimension, equivalent to a probability density integral over the candidate space, outputting a single fused tooth texture representation vector. Because the confidence is derived from a soft maximum mapping of geometric consistency distances, this approach numerically ensures that texture regions most sensitive to geometric constraints receive higher weights, while the contributions of candidates with larger geometric errors are automatically attenuated, thus avoiding texture jumps or local gaps caused by hard thresholding. The system then expands the fused tooth texture matrix column-wise into a one-dimensional vector and rearranges it into a two-dimensional mapping grid based on a unified UV layout table for the tooth region. This unified UV layout table is pre-expanded for tooth patches of a fixed-topology 3D morphable model to ensure consistent positional correspondence within the texture domain across different frames or individuals. Since the candidate fusion process may cause some pixels to remain null values, the algorithm performs bidirectional Laplacian interpolation to fill the missing pixels within the two-dimensional mapping grid. Bidirectional Laplacian interpolation takes into account the second-order smoothness constraints in the horizontal and vertical directions, not only eliminating holes but also maintaining gradient continuity at the boundaries. After interpolation, Gaussian weighted smoothing is performed on the grid edge pixels to reduce high-frequency noise and avoid aliasing artifacts during subsequent rendering. The initial tooth texture map obtained at this time already has complete spatial coverage, but the tonal contrast is still limited by the average characteristics of the candidate texture. To improve the visual texture, the system applies local histogram equalization to the map. The method is to calculate the grayscale distribution in each small area and equalize it, and then eliminate the transition band at the area boundary through bilinear interpolation, thereby enhancing the enamel highlight and tooth neck shadow details, and outputting an enhanced tooth texture map. Compared with global processing, local equalization can more accurately preserve the microscopic texture of the tooth surface while avoiding overall exposure shift.

[0052] At this point, the system has obtained a high-quality texture map that matches the UV layout of the tooth region and boasts enhanced color dynamic range. To accurately match the texture to the dynamic mouth shape, time-dependent spatial pose information is required. Therefore, the system obtains the mouth opening and closing spatial morphology matrix corresponding to the enhanced tooth texture map. For each time frame in the matrix, the system locates a subset of oral region vertices in the facial vertex index table. Because the vertex numbers in the index table remain constant throughout the model's lifecycle, the corresponding set of triangles within the mouth also remains consistent, making frame-level retrieval very cost-effective. The algorithm then maps the 2D texture coordinates of the enhanced tooth texture map to a subset of oral region vertices in the 3D morphable model, based on the current frame's mouth opening and closing spatial morphology matrix. This mapping process first performs polar expansion of the 3D vertex coordinates in the unified oral coordinate system. The corresponding texture coordinates are then retrieved by referencing the index relationship in the UV layout table. Ultimately, precise texture sampling points are generated for each vertex, forming a frame-level tooth texture vertex map dataset. Upon receiving the vertex map dataset for each frame, the unified rendering pipeline performs anti-aliasing, lighting compensation, and gamma correction. The anti-aliasing stage uses a multi-sampling anti-aliasing algorithm to perform intra-pixel subsampling on the lip edge, gum junction, and crown contours to reduce high-frequency aliasing and ensure smooth edges. The lighting compensation stage uses ambient occlusion and radiosity buffers to perform global light energy balance on the inner wall of the mouth to prevent unnatural bright spots or shadows caused by changes in head posture. The final gamma correction converts the color level of the rendered result according to the standard gamma curve of the target display device to ensure that the correct brightness and contrast are maintained on different hardware. As the time frame progresses, the mouth opening and closing spatial shape matrix drives the continuous change of position and orientation of a subset of vertices in the mouth area, while the enhanced tooth texture map is always closely attached to these vertices through UV coordinate projection. The rendering pipeline dynamically adjusts the lighting and color tone in each frame, thereby outputting a sequence of lip-coordinated composite frames. This sequence visually demonstrates natural and coherent pronunciation and expression transitions: the highlights of the tooth crowns shift with changing light, the gum margin details remain seamless during mouth opening and closing, and the upper and lower lip occlusion relationships are updated in real time. All frames strictly adhere to geometric prior constraints, allowing direct use without post-processing in scenarios sensitive to lip consistency, such as voice-driven virtual characters, lip replacement in film and television post-production, or real-time facial driving for live virtual hosts. Step 4, through a layered approach of confidence-weighted fusion, hole interpolation, local histogram equalization, and a unified rendering pipeline, successfully couples the high-dimensional tooth texture candidate tensor with the time-varying geometry. This ensures high consistency between lip movements and tooth details in both spatial and temporal dimensions, bringing a rigorous and high-fidelity conclusion to the entire geometrically constrained method for generating coordinated facial lip shapes.

[0053] The following example shows an offline execution process on a workstation with Intel Core i9-12900K, NVIDIA RTX 4090 and 64GB RAM. The input is a segment containing Frame RGB video clip (approx. Second, fps), the subject's face is The positive angle of view within.

[0054] Call the face detection and key point positioning model (RetinaFace+PIPNet) for each frame and output Individual facial feature points. Frame as an example, the pixel coordinates of the left eye outer corner , pixel coordinates of nose tip , pixel coordinates of the lower lip center The 3D deformable model uses a fixed topology triangular mesh with a vertex count of , the number of edges . Linear subspace dimension: shape parameter , expression parameters and posture parameters ,in is the three Euler angles, .

[0055] First Frame is initialized as the benchmark. Minimize the projection error ;middle is perspective projection, are the shape and expression basis matrices respectively, is the mean vertex. LBFGS optimization After that, it converges to .

[0056] Oral-related vertex subset Contains Vertices. Calculate the distance between the upper and lower lips for each frame .Will , average curvature of lip , oral cross-sectional area Combined into column vectors . Stack Frame-by-frame mouth opening and closing spatial morphology matrix . No. Frame Sample Value . Record facial vertex index table: vertex to Number them in the order of the generator output. Read the predefined number intervals of lip and gingiva: Traverse the ring by ring to get the length of the ordered sequence of lip contours The inner edge of the gum is removed from the apex of the hard palate The remaining vertices.

[0057] Calculate the centroid of the cleansed gingiva ; Principal component analysis to obtain the long axis direction of the alveolar .set up is the second eigenvector, and Cross product The three vectors are orthogonally normalized to form a rotation matrix .

[0058] For all joint boundary constraint sets Rigid transformation of vertices ;Will , normal vector , curvature and type tags As a four-way attribute, the dimension Divide the coordinate and normal components by mm gets the normalized interval The curvature normalization factor is .

[0059] The four single-mode tensor branches are denoted as .right Coordinates of each row Stereographic projection to a hypersphere: Will Linear expansion to , recorded as . Reentrant kernel function : .

[0060] In vector form, the radial component Application. Energy decreases after the first level reentry The cascaded multiple access sampling function uses a step sequence . Energy drops after sampling The return and sampling alternation is considered as one iteration. The global topological energy index after eight iterations is . At fourteen iterations Trigger early convergence and output a tentative tensor.

[0061] Number of nodes in high-dimensional energy graph (Each return dimension File, 3D ). Average branching factor of depth-first traversal .path In the Achieving geometric consistency cost when expanding at a sub-depth Reverse mapping undoes the wrapping and sampling to restore the four-way tensor. Splice it into the candidate three-dimensional array dimension of the tooth texture . Calculate the Euclidean consistency distance, where Candidate No. , No. Candidate No. Confidence .For example , . Get the confidence spectrum .

[0062] Weighted by sequence number ;Will Column vectorization and rearrange to UV grid. Missing pixel ratio Bidirectional Laplace interpolation iteration Subconvergent, bounded Gaussian kernel . Local histogram equalization window , pixel intensity range The average brightness of the crown highlights after enhancement is Promote to For each frame, a subset of mouth vertices Do texture sampling. Frame Map Coordinates Corresponding three-dimensional vertices Rendering stage MSAA , ambient occlusion coefficient , gamma index .generate Frame lip-sync coordination synthesis sequence, average frame time ms, real-time preview is available.

[0063] Figure 2 This is a schematic diagram of the overall process of generating a three-dimensional deformation model and a mouth opening and closing spatial morphology matrix in the method for generating coordinated facial and lip shapes based on geometric constraints of the present invention. This figure shows in detail the specific implementation process of step 1, including the complete technical path from the input of the face image to be processed to the final generation of the mouth opening and closing spatial morphology matrix and the facial vertex index table. Figure 2 As shown on the left, the present invention first acquires a facial image to be processed. This image is a static image or a sequence of dynamic frame images containing at least one frontal or near-frontal view of the face. In this example, the facial image to be processed exhibits standard facial features, including clearly identifiable key facial regions such as eyebrows, eyes, nose wings, and lips. By performing face detection and facial landmark location operations on this input image, the system accurately extracts the coordinates of several two-dimensional facial feature points, including eyebrows, eyes, nose wings, and lips, thereby forming a set of two-dimensional facial feature points. Figure 2The first picture in the middle shows the construction process of the 3D deformable model. Based on the set of 2D facial feature points, the system constructs an initial corresponding mapping relationship that matches the 3D deformable model, and calls the 3D deformable model to perform 3D fitting on the face image to be processed. As shown in the figure, the 3D deformable model uses a set of fixed topology facial triangle meshes, which contains 53,215 facial vertices and 105,054 edges, and the number of each vertex remains unchanged during the life cycle of the 3D deformable model. The fixed topology covers the entire visible area from the hairline to the mandible, and reserves deformable vertices on the inner edge of the lips to ensure the accuracy and consistency of the subsequent spatial morphological matrix of the mouth opening and closing. The black dots in the figure represent vertices distributed in various areas of the face. Each vertex corresponds to a 3D coordinate value. In particular, the vertices in the lip area are more densely distributed to ensure accurate capture of changes in mouth shape. Figure 2 The second picture in the middle shows the process of generating the spatial morphology matrix of the mouth opening and closing. In the three-dimensional deformation model, the system extracts the corresponding three-dimensional coordinate changes in each frame image for the vertex subset related to the upper and lower boundaries of the lips and the opening and closing area of ​​the mouth, and establishes a continuous description function for the degree of lip closure. As shown in the matrix grid structure, the black rectangular elements represent the spatial configuration data of the oral cavity at different time points or expression states. The spatial morphology matrix of the mouth opening and closing is used to represent the spatial configuration of the oral cavity at different time points or expression states, providing a geometric basis for the subsequent coordinated generation of the lip shape. Figure 1 The right side shows the generated facial vertex index table. The system numbers all vertices in the 3D deformable model according to their original order during the model initialization phase, records their spatial coordinate indexes and topological connections, and forms a facial vertex index table. As shown in the figure, this index table stores the 3D coordinate information of each vertex in a structured manner, including the complete coordinate data (x1, y1, z1) to (xN, yN, zN) for vertex 1 to vertex N. This provides a precise vertex positioning foundation for subsequent lip contour extraction and gingival boundary recognition.

[0064] Figure 3The figure shows the detailed process for extracting the lip contour and gingival boundary. During the lip contour extraction phase, the system first searches the facial vertex index table for the predefined upper and lower lip outer edge vertex number ranges to obtain the upper and lower lip outer edge vertex subsets, respectively. The upper lip outer edge vertex subset shown in the figure includes several key vertices distributed along the upper lip edge, while the lower lip outer edge vertex subset corresponds to the corresponding vertices along the lower lip edge. Based on the topological connectivity of the 3D deformable model, a ring-by-ring traversal operation is performed on these two vertex subsets to obtain a monotonically increasing ordered sequence of vertices along the lip contour, ultimately forming a complete lip contour polygon chain. This polygon chain, represented by a dashed line, accurately describes the geometric features of the lip boundary. During the gingival boundary extraction phase, the system searches the facial vertex index table for the inner gingival edge vertex number range to obtain the inner gingival edge vertex subset. The key technical step is to eliminate redundant vertices based on the vertex normal direction and local curvature threshold. The redundant vertices marked with dotted circles in the figure represent irrelevant vertices connected to the hard palate. These vertices are effectively eliminated during the processing, and only the valid vertices located in the gingival-alveolar junction area are retained, thereby generating a purified gingival boundary sequence.

[0065] like Figure 4 As shown, the present invention first separates the geometric prior constraint tensor according to four attribute dimensions: vertex three-dimensional coordinates, normal vectors, curvature scalars, and boundary type markers, to obtain four single-modal tensor branches. For the three-dimensional coordinates in each single-modal tensor branch, the system uses a stereoscopic projection method to embed the Euclidean space point into a hypersphere space with a radius of one to form a hypersphere embedding tensor. As shown in the figure, there are multiple projection points evenly distributed on the hypersphere with a radius of one, and these points establish a corresponding relationship with the vertices in the original Euclidean space through the projection line starting from the center of the sphere. The hypersphere embedding transformation ensures the topological consistency of the three-dimensional coordinate data in the high-dimensional space, laying a geometric foundation for subsequent complex transformations. At the same time, linear dimensional expansion is performed on the normal vector, curvature scalar, and boundary type marker to make their dimensions consistent with the hypersphere embedding tensor.

[0066] Figure 5 The center-left figure illustrates the transformation process of the topological reentry kernel function. This function compresses the radial amplitude of a tensor on a hypersphere into a hyperbolic amplitude through a reversible torus mapping, preserving the absolute phase information at each reentry point. As shown in the torus structure, the outer ring represents the original radial distribution, while the inner ring represents the compressed hyperbolic distribution. The reentry points are connected by phase-preserving lines, ensuring that key geometric information is not lost during the transformation. This transformation process achieves a reversible mapping from a radial coordinate system to a hyperbolic coordinate system, effectively reducing the dimensional complexity of the data without losing topological characteristics. Figure 5The center-right figure illustrates the processing mechanism of the cascaded multi-access sampling function. This function performs multiple hop samplings with an exponentially decaying step size within the reentrant domain to obtain a cross-scale feature representation. As shown in the figure, the size of the sampling points follows an exponentially decaying distribution, gradually decreasing from an initial step size to a tiny step size, and the connecting lines represent the hop sampling paths. This sampling strategy captures the characteristic variations of the tensor at different scales, ensuring that both macroscopic topological structure and local details are preserved. The topological reentrant kernel function and the cascaded multi-access sampling function alternate to form a complete iteration, and the optimal transformation of the tensor is achieved through multiple iterative optimizations.

[0067] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for generating coordinated face and mouth shapes based on geometric constraints, characterized in that: The method comprises: Step 1: Use the 3D deformation model to initialize the 3D morphology of the face image to be processed, obtain the mouth opening and closing spatial morphology matrix, and simultaneously generate a facial vertex index table that corresponds one-to-one to the mouth opening and closing spatial morphology matrix; Step 2: Based on the facial vertex index table, extract the lip contour, gingival boundary and alveolar pose, construct the lip-gingival joint boundary constraint set, and map the lip-gingival joint boundary constraint set to a unified coordinate system to form a geometric prior constraint tensor; Step 3: Perform a multimodal topological cascade reversible hybrid transformation on the geometric prior constraint tensor to obtain a high-dimensional tooth texture candidate tensor and a constraint consistency confidence spectrum; Step 4: Use the high-dimensional tooth texture candidate tensor and the constrained consistency confidence spectrum as input, map it according to the mouth opening and closing spatial morphology matrix, and complete anti-aliasing, lighting compensation, and gamma correction in a unified rendering pipeline to finally obtain a lip-coordinated synthesis frame sequence.

2. The method for generating coordinated face and mouth shapes based on geometric constraints according to claim 1, wherein: Step 1 specifically includes: obtaining a face image to be processed, which is a static face image or a dynamic frame image sequence containing at least one face from a frontal or near-frontal perspective; performing face detection and facial key point positioning operations on the face image to be processed, extracting the coordinates of several two-dimensional facial feature points including eyebrows, eyes, nose wings, and lips, and forming a two-dimensional facial feature point set; based on the two-dimensional facial feature point set, constructing an initialization corresponding mapping relationship that matches the three-dimensional deformation model, and calling the three-dimensional deformation model to perform three-dimensional fitting on the face image to be processed, and outputting a three-dimensional morphological parameter set, including shape parameters, expression parameters, and posture parameters; based on the three-dimensional morphological parameter set, restoring the face image to be processed. A three-dimensional deformable model with consistent resolution of facial images; the three-dimensional deformable model consists of several facial vertices, where each vertex corresponds to a three-dimensional coordinate value; in the three-dimensional deformable model, for a subset of vertices related to the upper and lower boundaries of the lips and the opening and closing areas of the mouth, the corresponding three-dimensional coordinate changes in each frame image are extracted, a continuous description function for the degree of lip closure is established, and a mouth opening and closing spatial morphology matrix is ​​generated; the mouth opening and closing spatial morphology matrix is ​​used to represent the spatial configuration of the oral cavity at different time points or expression states; all vertices in the three-dimensional deformable model are numbered according to their original order in the model initialization stage, and their spatial coordinate indexes and topological connection relationships are recorded to form a facial vertex index table.

3. The method for generating coordinated face and mouth shapes based on geometric constraints according to claim 2, wherein: The 3D morphable model uses a fixed-topology facial triangular mesh consisting of 53,215 facial vertices and 105,054 edges, with each vertex numbered unchanged throughout the lifecycle of the 3D morphable model. The fixed topology covers the entire visible area from the hairline to the mandible, and reserves deformable vertices at the inner edge of the lips to ensure the accuracy and consistency of the subsequent spatial morphological matrix of mouth opening and closing.

4. The method for generating coordinated face and mouth shapes based on geometric constraints according to claim 3, wherein: In step 2, the predefined upper lip outer edge vertex number range and lower lip outer edge vertex number range are retrieved from the facial vertex index table to obtain the upper lip outer edge vertex subset and the lower lip outer edge vertex subset respectively; according to the topological connection relationship of the three-dimensional deformation model, the upper lip outer edge vertex subset and the lower lip outer edge vertex subset are traversed ring by ring to obtain a monotonically increasing ordered sequence of vertices along the lip contour direction to form a lip contour polygon chain; in the facial vertex index table, the gingival inner edge vertex number range is retrieved to obtain the gingival inner edge vertex subset; based on the vertex normal vector direction and the local curvature threshold, the redundant vertices connected to the hard palate are eliminated, and only the vertices located in the gingival-alveolar junction area are retained to generate a purified gingival boundary sequence.

5. The method for generating coordinated face and mouth shapes based on geometric constraints according to claim 4, wherein: In step 2, the centroid of each vertex in the purified gingival boundary sequence is calculated and recorded as the alveolar center point; the centroid of each vertex in the purified gingival boundary sequence is calculated and recorded as the alveolar center point; the right-handed coordinate system composed of the alveolar center point, the alveolar long axis direction, and the alveolar vertical direction is defined as the alveolar pose reference system; the lip contour polygon chain, the purified gingival boundary sequence, and the alveolar pose reference system are encapsulated together as a lip-gingival joint boundary constraint set; in the lip-gingival joint boundary constraint set, the following are recorded for each vertex: the original three-dimensional coordinate value, the normal vector, the curvature scalar, and the boundary type mark.

6. The method for generating coordinated face and mouth shapes based on geometric constraints according to claim 5, wherein: In step 2, the center point of the alveolar is used as the origin, and the three-axis vectors of the alveolar pose reference system are used as basis vectors to define the unified coordinate system of the oral cavity. A rigid coordinate transformation is performed on all vertices in the lip-gingival joint boundary constraint set, and the three-dimensional coordinates of the vertices are mapped from the global coordinate system of the three-dimensional deformable model to the unified coordinate system of the oral cavity. A rigid coordinate transformation is performed on all vertices in the lip-gingival joint boundary constraint set, and the three-dimensional coordinates of the vertices are mapped from the global coordinate system of the three-dimensional deformable model to the unified coordinate system of the oral cavity. The mapped vertex three-dimensional coordinates, normal vector, curvature scalar and boundary type tag are stored in tensor form to form a geometric prior constraint tensor; Apply a normalization operation to the geometric prior constraint tensor so that the three-dimensional coordinates are distributed in the range of -1 to 1, the normal vectors are distributed in the range of -1 to 1, and the curvature scalars are distributed in the range of 0 to 1; apply a normalization operation to the geometric prior constraint tensor so that the three-dimensional coordinates are distributed in the range of -1 to 1, the normal vectors are distributed in the range of -1 to 1, and the curvature scalars are distributed in the range of 0 to 1.

7. The method for generating coordinated face and mouth shapes based on geometric constraints according to claim 6, wherein: In step 3, the geometric prior constraint tensor is separated according to four attribute dimensions: vertex three-dimensional coordinates, normal vector, curvature scalar, and boundary type label, to obtain four single-modal tensor branches; For the three-dimensional coordinates in each single-modal tensor branch, a stereoscopic projection method is used to embed the Euclidean space point into a hypersphere space with a radius of 1 to form a hypersphere embedding tensor. The normal vector, curvature scalar, and boundary type marker are linearly expanded to make their dimensions consistent with the hypersphere embedding tensor. The topological reentry kernel function is called in batches on the hypersphere embedding tensor. This function compresses the radial amplitude of the tensor on the hypersphere into a hyperbolic amplitude through a reversible torus mapping and retains the absolute phase information at each reentry point. The cascaded multi-address sampling function is called to perform multiple hop sampling with an exponentially decaying step size in the reentry domain to obtain a cross-scale feature representation. The topological reentry kernel function and the cascaded multiple-access sampling function alternate to form a complete iteration. After each iteration, the global topological energy index is calculated. If the index decreases by less than one thousandth compared with the previous iteration, early convergence is triggered; otherwise, the next iteration is continued. If the number of iterations reaches 100 and still no convergence occurs, the result of the iteration with the lowest energy is taken as the provisional output tensor.

8. The method for generating coordinated face and mouth shapes based on geometric constraints according to claim 7, wherein: In step 3, the tentative output tensor is input into the greedy minimum topological energy path searcher; the searcher uses the global topological energy index as the cost function, performs a depth-first traversal on the high-dimensional energy graph of the tensor, and evaluates the geometric consistency cost of each candidate path in real time; when the geometric consistency cost is lower than the preset threshold of 50% for the first time, the current path is marked as the optimal topological path and the search is terminated; a reversible hybrid reverse mapping is performed on the tensor on the optimal topological path, and the transformations of the cascaded multiple access sampling function and the topological reentry kernel function are successively undone to obtain the reconstructed modal tensor back to the Euclidean space; the four reconstructed modal tensors are reassembled in the original modal order to obtain a three-dimensional array of tooth texture candidates; the first 128 elements of the three-dimensional array of tooth texture candidates are taken and stacked in the original sequence to form a high-dimensional tooth texture candidate tensor; For each candidate instance in the high-dimensional tooth texture candidate tensor, the Euclidean consistency distance between it and the geometric prior constraint tensor in the three attribute dimensions of vertex three-dimensional coordinates, normal vector and curvature scalar is calculated; the Euclidean consistency distance is mapped to a confidence level in the interval 0 to 1 using a soft maximum operation to obtain the corresponding single-instance confidence value; the 128 single-instance confidence values ​​are arranged in sequence to form a constraint consistency confidence spectrum with a length of 128.

9. The method for generating coordinated face and mouth shapes based on geometric constraints according to claim 8, wherein: Step 4 specifically includes: one-to-one correspondence between the high-dimensional tooth texture candidate tensor and the constrained consistency confidence spectrum according to the candidate serial number dimension; applying the corresponding single instance confidence value in the constrained consistency confidence spectrum to each candidate texture instance in the high-dimensional tooth texture candidate tensor as a fusion weight to obtain a confidence-weighted tooth texture fusion tensor; performing weighted summation on the confidence-weighted tooth texture fusion tensor along the candidate serial number dimension to obtain a single fused tooth texture representation vector, which is used to represent the optimal tooth texture state corresponding to the current input sample; expanding the fused tooth texture matrix into a one-dimensional vector by column, and rearranging it into a two-dimensional mapping grid according to the unified UV layout table of the tooth area; in the two-dimensional mapping grid, performing bidirectional Laplace interpolation filling on the missing pixels and performing high-resolution interpolation on the edge pixels. The method comprises the following steps: performing weighted smoothing processing on the teeth to form an initial tooth texture map; applying local histogram equalization to the initial tooth texture map to enhance the enamel highlights and the tooth neck shadow details, and outputting an enhanced tooth texture map; obtaining the mouth opening and closing spatial morphology matrix corresponding to the enhanced tooth texture map; locating the corresponding oral region vertex subset in the facial vertex index table for each time frame in the mouth opening and closing spatial morphology matrix; mapping the two-dimensional texture coordinates of the enhanced tooth texture map to the oral region vertex subset in the three-dimensional deformation model according to the mouth opening and closing spatial morphology matrix of the current frame, and generating a frame-level tooth texture vertex map dataset; performing anti-aliasing, lighting compensation and gamma correction on each frame in the frame-level tooth texture vertex map dataset on a unified rendering pipeline, and finally obtaining a lip-coordinated synthesis frame sequence.

Images