A face key point generation method based on emotion prior and low-dimensional residual correction

Abstract

The application discloses a face key point generation method based on emotion prior and low-dimensional residual correction, relates to the technical field of image data processing, and generates a base face key point sequence irrelevant to emotion based on input text; a layered emotion prompt tensor is generated based on an emotion identifier, the layered emotion prompt tensor is injected into a decoder layer by layer, a decoder hidden sequence with injected emotion prompts is obtained, an emotion prior feature sequence is extracted from the decoder hidden sequence with injected emotion prompts, an average value of a base expression point subset sequence in a time dimension is calculated according to a predefined expression point index set, a geometric context feature is obtained, and the geometric change is constrained to a specific expression area to maintain the stability of other areas; and a predicted expression residual is calculated based on the emotion prior feature sequence, the geometric context feature and the emotion identifier; the application effectively suppresses geometric drift and maintains the stability of non-expression points while enhancing emotion regulation capability.

Description

Technical Field

[0001] This application relates to the field of image data processing technology, specifically the field of facial landmark generation technology, and more specifically to a facial landmark generation method based on emotion prior and low-dimensional residual calibration. Background Technology

[0002] With the rapid development of virtual digital humans, human-computer interaction, and intelligent content generation technologies, text-driven lip-syncing and facial motion synthesis has gradually become an important research topic in visual language. Text-to-keypoint generation, in particular, can transform language input into a temporally continuous two-dimensional sequence of facial keypoints, providing fundamental support for lip-syncing, expression transfer, and interactive animation in virtual avatars. In recent years, researchers have widely adopted sequence modeling methods to characterize the mapping relationship between text and motion sequences. For example, the Transformer-based encoder-decoder framework learns long-range dependencies and gradually decodes keypoint trajectories through a self-attention mechanism. Simultaneously, to enhance emotional expression, some works fuse emotion labels or emotion embeddings as conditional signals with text features, aiming to control lip expressions and facial movements during the generation process.

[0003] However, existing methods still have significant shortcomings in terms of emotional controllability and geometric stability: when emotional signals and language-driven stable oral motor trunks are directly entangled and modeled, problems such as unstable emotional modulation, temporal jitter, and geometric drift of key points are likely to occur, thereby destroying the temporal consistency of the oral region and the credibility of facial expressions.

[0004] To address the aforementioned issues, this application provides a method for generating facial key points based on emotion priors and low-dimensional residual calibration. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, the present invention provides a method for generating facial key points based on emotion prior and low-dimensional residual calibration.

[0006] To achieve the above objectives, the present invention employs the following technical solution, including: S1, acquiring input text and emotion identifiers, and generating a base facial keypoint sequence unrelated to emotion based on the input text; S2, generating a hierarchical emotion cue tensor based on the emotion identifiers; S3, injecting the hierarchical emotion cue tensor layer by layer into the decoder to obtain the decoder hidden sequence with injected emotion cue; S4, extracting the emotion prior feature sequence from the decoder hidden sequence after the injection of emotion cue; S5, extracting a base expression point subset sequence from the base facial keypoint sequence according to a predefined expression point index set, and calculating the base expression point subset sequence in time... The geometric context features are obtained by averaging the values ​​over the interdimensional dimensions. The expression point index set is a subset of key points related to expression changes, used to constrain geometric changes to specific expression regions to maintain the stability of other regions; S6. Based on the emotion prior feature sequence, geometric context features, and emotion identifiers, the subspace coefficient sequence of the low-dimensional subspace is calculated, and the subspace coefficient sequence is decoded to obtain the predicted expression residual. The predicted expression residual represents the amount of geometric change added frame by frame; S7. The predicted expression residual is written back to the position in the base facial key point sequence related to the expression point index set to obtain an emotion-controllable facial key point sequence.

[0007] Preferably, in step S2, generating a hierarchical emotion cue tensor based on emotion identifiers includes: S21, mapping the emotion identifiers to an emotion embedding vector through an emotion embedding table lookup; S22, inputting the emotion embedding vector into a style mapping network to obtain a style vector. S23. Input the style vector into the layered cue generator head to generate a layered sentiment cue tensor with the same number of layers as the decoder.

[0008] Preferably, in step S3, the layered sentiment cue tensor is injected into the decoder layer by layer, including: S31, mapping the target-side input sequence to the decoder's initial representation, where the target-side input sequence is the input sequence formed by removing the last frame from the original target keypoint sequence; S32, concatenating the additional sequence vector with the initial representation in the temporal dimension to obtain an extended sequence, where the additional sequence vector is initialized as the first-layer sentiment cue vector of the layered sentiment cue tensor; S33, extending the lengths of the decoder's padding mask and causal mask from the target-side input sequence length to the extended sequence length; S34, for layers 1 to L of the decoder, performing the following operations in sequence: replacing the additional sequence vector of the current extended sequence with the sentiment cue vector of the same layer in the layered sentiment cue tensor; inputting the replaced extended sequence together with the extended padding mask and causal mask into the current decoding layer to obtain the updated extended sequence; S35, after completing the decoding of all layers L, removing the additional sequence elements from the updated extended sequence of layer L to obtain the decoder hidden sequence with injected sentiment cue.

[0009] Preferably, in step S4, extracting the emotion prior feature sequence from the decoder hidden sequence with injected emotion cues includes: S41, during the step-by-step generation of the decoder, extracting the hidden state vector of the last position of the decoder hidden layer at each time step as the emotion prior feature of the current time step; S42, stacking the emotion prior features of all time steps in chronological order to form the emotion prior feature sequence.

[0010] Preferably, before step S6, a low-dimensional subspace is pre-established, including: S51, obtaining the real keypoint sequence and the base facial keypoint sequence of each data in the training dataset; S52, extracting the real expression point subset sequence of the training dataset from the real keypoint sequence of each data, and extracting the base expression point subset sequence of the training dataset from the base facial keypoint sequence of each data; S53, calculating the target expression residual by the difference between the real expression point subset sequence and the base expression point subset sequence of the training dataset, and establishing a low-dimensional subspace based on the target expression residual.

[0011] Preferably, in step S6, based on the emotion prior feature sequence, geometric context features, and emotion identifier, a subspace coefficient sequence in the low-dimensional subspace is calculated, and the subspace coefficient sequence is decoded to obtain the predicted expression residual, including: S61, mapping and projecting the geometric context features and emotion prior feature sequence to the calibration hidden space; S62, querying the embedding table according to the emotion identifier to obtain the emotion embedding vector, and expanding the emotion embedding vector along the time dimension to obtain a frame-by-frame emotion category embedding sequence; S63, fusing the mapped geometric context features, emotion prior feature sequence, and frame-by-frame emotion category embedding sequence to obtain fused features; S64, obtaining the subspace coefficient sequence in the low-dimensional subspace based on the fused features and the coefficient head; S65, decoding the subspace coefficient sequence into the predicted expression residual.

[0012] Preferably, in step S7, the write-back is achieved by: initializing the output sequence as a base facial key point sequence, and writing the predicted expression residual back to the key point coordinates related to the expression point index set in the output sequence in a weighted superposition manner.

[0013] Preferably, the method also includes a training step S8, which involves: weighting the residual regression loss, temporal smoothing loss, coefficient supervision loss, sentiment separability loss, inner lip protection loss, and geometric surrogate loss to construct a joint objective function, wherein: Residual regression loss: , in, To predict facial residuals, Let E be the residual of the target facial expression, and E be the average. It is a norm; Temporal smoothing loss: , in, , Let be the predicted facial expression residual at time t. The predicted facial expression residual at time t-1; Coefficient supervised loss: , in, For the subspace coefficient sequence and For the true coefficient sequence , The square of the L2 norm; Loss of separability: , in, For cosine similarity, This is the interval threshold; Loss of inner lip protection: , in, For the inner lip point set; Geometric proxy loss: , in, For real relative change, To predict relative changes.

[0014] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements a method for generating facial key points based on emotion priors and low-dimensional residual calibration.

[0015] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for generating facial key points based on emotion priors and low-dimensional residual calibration.

[0016] The advantages of this invention are: (1) Through the structured link of “layer-by-layer emotion injection in decoder - prior feature export - low-dimensional residual calibration - expression subset writing back”, emotion controllability is achieved while maintaining the stability of base generation, and the risk of emotion modulation instability and geometric drift is reduced.

[0017] (2) This application proposes a layered injection strategy for emotion cues within the decoder: emotion cues are placed in the form of emotion cues vectors before the target sequence, and the attention mask is aligned and expanded; the vector representations at predetermined positions are replaced layer by layer in each decoding layer, and the cues vectors are removed before the output, thereby achieving continuous emotion modulation without changing the length of the output sequence, which is convenient for adapting to existing models.

[0018] (3) This application derives the prior features of emotion from the hidden sequence of the decoder after injection, and projects and fuses the geometric context and emotion embedding frame by frame in the residual calibration module to output the low-dimensional subspace coefficient sequence and decode it into the predicted expression residual, so as to realize the structured representation of the expression situation and suppress instability.

[0019] (4) This application further proposes a multi-objective training constraint system: supervision is carried out at two levels: expression residual and low-dimensional coefficient, and constraints such as emotion separability, temporal smoothing, inner lip protection and geometric semantic proxy are introduced to improve emotion controllability, geometric stability and temporal coherence.

[0020] (5) This application adopts a formula for writing back expression points subsets, which writes back expression residuals only for a predetermined set of expression key points. The writing back can be achieved through equivalent methods such as indexing, masking or scattering, so that non-expression key points remain unchanged or have limited changes. Furthermore, it can be combined with inference strategies such as writing back scaling and low-dimensional coefficient pruning to achieve expression intensity control and drift suppression in different application scenarios. Attached Figure Description

[0021] Figure 1 A flowchart illustrating a method for generating facial key points based on emotion prior and low-dimensional residual calibration, provided in an embodiment of this application; Figure 2 A visual schematic diagram illustrating emotion-controllable facial generation according to an embodiment of this application; Figure 3 A schematic diagram illustrating the controllable emotion generation results under different identity samples provided in an embodiment of this application; Figure 4 This is a schematic diagram illustrating the continuous control results of the same identity sample under different emotional and intensity conditions, as provided in an embodiment of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] To address the aforementioned technical challenges, we propose a structured "stable backbone-controllable residual" generation mechanism: First, we construct an emotion-independent text-to-keypoint basic generation path to learn stable language-driven lip-sync dynamics. Then, we introduce layer-by-layer emotion cues into the decoder by mapping emotion embeddings to hierarchical cue vectors and performing layer-by-layer replacement and attention mask expansion at preset positions to achieve continuous emotion injection without changing the output length. Furthermore, we derive explicit emotion prior feature sequences from the decoder's hidden state after emotion injection and fuse them with the geometric context calculated from the basic keypoints. Finally, we design a low-dimensional residual calibration mechanism that predicts subspace coefficients in the expression residual subspace obtained from training residual statistical estimation and decodes them into predicted expression residuals, writing them back only to a predefined subset of expression keypoints. This enhances emotion regulation capabilities while effectively suppressing geometric drift and maintaining the stability of non-expression points.

[0024] The present invention provides a method for generating facial key points based on emotion prior and low-dimensional residual calibration, the specific steps of which are as follows: S1. Obtain the input text and emotion markers, and generate an emotion-independent base facial keypoint sequence based on the input text. .

[0025] S11. Obtain the input text and perform lexicalization to obtain a lexical sequence: , in, For the first Each word element, The number of lexical units. Construct a word embedding matrix. : , Among them, word embedding matrix An embedding table is an embedding table representing a two-dimensional real matrix with V rows (representing the vocabulary size) and D columns (representing the word vector dimension). It maps discrete lexical units to continuous vector representations, enabling subsequent neural networks to process the semantic information of the text. For each lexical unit... Perform a lookup table mapping to obtain the corresponding word vectors. and word vectors Arrange word vectors into a sequence based on word order; combine the word vectors into a single sample word vector sequence based on word order; for batch input... For each sample, the word vector sequences corresponding to each sample are stacked along the batch dimension to obtain the word tensor. : , in, Batch size. Word tensor. It inherits the word embedding matrix For discrete lexical units The mapping effect is used as input to the subsequent text sequence encoding subnetwork.

[0026] S12, Word Tensors The text sequence encoding subnetwork of the input base model performs context modeling on the word vector sequence to obtain the text feature sequence. : , Among them, text feature sequences The textual conditional information, already incorporating contextual semantic relationships, serves as the semantic driver for subsequent keypoint generation. The text sequence encoding sub-network employs a Transformer encoder. This represents the Transformer encoder, where F is a three-dimensional real tensor and D′ represents the dimension of the encoded features.

[0027] S13, Text feature sequence The decoded facial feature sequence is obtained by inputting it into the decoder of the pedestal model. : , in, It serves as an intermediate representation for the subsequent generation of the base facial keypoint sequence. To output the sequence length, For decoder dimensions.

[0028] S14. Facial feature sequence Frame by frame, the keypoint coordinates are mapped to vectors and reshaped to obtain the keypoint sequence of the base face. .

[0029] Single frame coordinate vector dimension for: , in The number of key points per frame. The coordinate dimensions of each key point.

[0030] For the Frame, will Coordinate vectors are obtained through linear layer mapping. : , in, Representing facial feature sequences The t-th frame in For linear layer weights, The linear layer is used to project the facial motion representation in the hidden feature space onto the keypoint coordinate space, thereby obtaining the predicted facial keypoint coordinates for each frame.

[0031] All frames Resulting from stacking by time: , Press again Reshaping into keypoint coordinate tensors to obtain the base facial keypoint sequence. : , Given a four-dimensional real tensor, the reshaping operation is used to adjust the tensor shape without changing the total number of elements. The pedestal facial keypoint sequence is used to represent stable text-driven facial motion trajectories and serves as the baseline sequence for subsequent emotion residual calibration and expression subset write-back.

[0032] To ensure consistency in subsequent residual learning, , The resulting prior emotion feature sequences are then aligned to the same length in the time dimension. When the length of the original sequence is At that time, linear interpolation resampling is performed along the time dimension to obtain a length of The sequence. Linear interpolation, an existing time alignment method, is used to estimate the value at an intermediate time point based on known values ​​at adjacent time points. In this invention, its function is to unify the time length of each sequence to facilitate subsequent frame-by-frame residual modeling and loss calculation. The alignment length... The physical meaning of this is: it represents the time series length uniformly adopted in the residual calibration stage.

[0033] S2, based on emotion labels Generate hierarchical sentiment cue tensors .

[0034] This step converts discrete emotion markers into continuous, hierarchical cue representations, allowing emotion conditions to be injected layer by layer during subsequent decoding. The hierarchical emotion cue tensor does not directly correspond to the final keypoint coordinates; instead, it serves as a modulation signal during the decoding phase, ensuring that the emotion information remains effective throughout the multi-layer decoding process.

[0035] S21. Obtaining Emotion Labels , , used to indicate the target emotion category for each sample. Let the total number of emotion categories be . The emotion embedding dimension is The emotion embedding vector is obtained by looking up the emotion embedding table. : , in, For size Embedded table. This belongs to the existing discrete category representation method, and its function is to map discrete target emotion categories into continuous feature vectors. The emotion embedding vector is the semantic representation of the target emotion category in the continuous feature space, which is used to generate stable and controllable emotion cues later.

[0036] S22. Embedding emotions into vectors Input Style Mapping Network To obtain style vectors : , in, It consists of two linear layers and GELU activation. For style vector dimensions. This belongs to a common multilayer perceptron architecture and is used to perform nonlinear feature mapping; GELU is a smooth activation function used to improve nonlinear expressive power. Style mapping network Its function is to transform the original emotional embedding into a style representation more suitable for generating hierarchical cues. Style vector It is a high-level style representation extracted from the target emotion category, which is used to generate emotion cues for each decoding layer in the subsequent process.

[0037] Preferably, in one embodiment of this application, to enhance generalization ability and maintain consistency with implementation, in the style mapping network... After output and before generating the layered hints head, apply random deactivation to the style vector: , in, The style vector after applying random deactivation, It is a random deactivation regularization operation that sets some feature components to zero or scales them according to corresponding rules with a preset probability during training, so as to weaken feature co-adaptation and improve generalization ability. This operation is used to enhance the robustness of sentiment style representation on different samples and avoid the prompt generation head from relying too much on a few fixed feature components.

[0038] S23, Style Vector Input layered prompts to generate a layered sentiment prompt tensor. This is so that corresponding emotional cues can be injected into different decoding layers in step S3. The number of decoder layers is... The decoder has a hidden dimension of 1. Layered prompts generate head output intermediate prompt vectors. , ; And reshape it into a hierarchical sentiment cue tensor. : , in, This represents a reshaping operation, used to adjust the dimensional organization of a tensor without changing the total number of elements. Let L represent a two-dimensional real tensor, where the first dimension is the batch dimension, the second dimension is the total dimension L×H of the flattened hierarchical cue, and H represents the intermediate cue vector output by the hierarchical cue generator head. This represents the tensor after reshaping into a hierarchical sentiment cue tensor. Previously, the result of concatenating the prompt vectors from each layer.

[0039] Layered sentiment cue tensor This is a set of emotion modulation vectors prepared for each layer of the decoder, where each layer slice corresponds to an emotion cue vector. To constrain the cue amplitude, [the following is omitted as the text is incomplete and requires further context]. Apply hyperbolic tangent compression and multiply by the suggested scaling factor. : , in, It is a hyperbolic tangent function used to compress the elements of the cue tensor into a bounded interval to avoid the cue amplitude being too large and impacting the decoding stability; This is a scaling factor used to adjust the overall intensity of the emotion cue injection into the decoder. Typically, When the value is small, the emotion modulation is milder and more stable; Larger values ​​result in more pronounced emotional responses, but excessively large values ​​may cause localized disturbances. In implementation, It can be a positive real number, and the preferred choice is... .

[0040] S3. Layered sentiment cue tensor The signal is injected layer by layer into the decoder to obtain the decoder hidden sequence with injected emotion cues. In steps S31 to S35, the decoder processes the target-side input sequence; wherein, the target-side input sequence represents the input sequence used by the decoder on the target side. The target-side input sequence after removing the last frame is used to form an initial representation. After prepending the first-layer emotion cue vector to this initial representation, an extended sequence is formed. After removing the first-layer emotion cue vector, the corresponding decoder hidden sequence is restored.

[0041] S31. The target input sequence Q is linearly mapped to obtain the initial representation. : , This is a linear mapping operation, where the target-side input sequence Q represents the input sequence used by the decoder on the target side. The target-side input sequence Q is formed by removing the last frame from the original target keypoint sequence. The linear mapping operation projects the target-side input sequence Q onto the same feature dimension as the emotion cue vector, facilitating subsequent pre-assembly and layer-by-layer replacement injection.

[0042] S32. Layered sentiment cue tensor The first layer sentiment cue vector obtained by slicing the first layer As an additional sequence vector: , in, This indicates taking the first slice along the layer dimension, while keeping the length of that dimension at 1. Here, the dimension... The corresponding batch dimension is 1, and the prompt corresponds to the sequence position dimension. This represents the first-level sentiment cue vector extracted from the hierarchical sentiment cue tensor, used to inject a sentiment cue position into the input sequence Q before decoding begins.

[0043] First-level emotion cue vector With initial representation The extended sequence is obtained by pre-concatenating the time dimension. : , in, This represents a splicing operation along the time dimension. The spliced ​​extended sequence provides the initial state for subsequent layer-by-layer replacement injection.

[0044] S33. Extend the lengths of the decoder's padding mask and causal mask from the target-side input sequence length to the length of the extended sequence.

[0045] Let the original padding mask be: Expand it to: The padding mask is used to identify which positions in the input sequence Q are valid positions and which positions are alignment padding positions, so as to match the time length of the extended sequence.

[0046] Simultaneously construct the causal (subsequent) mask as follows: , in, Causal masking is used before attention weight normalization, so that each position only focuses on itself and the positions before it; The padding mask is used to expand the sequence and, after normalization, to reset the attention weights at invalid locations to zero. Both can be expanded in multi-head dimensions via broadcasting. Softmax is a normalization function used to convert attention scores into weight distributions; broadcasting means expanding tensors of smaller shapes to larger shapes according to rules, while maintaining dimensionality compatibility, in order to perform element-wise operations.

[0047] S34, regarding the first One decoding layer ( Take the layered sentiment cue tensor The The sentiment cue vector for the layer is: , in, This represents the number of decoder layers.

[0048] Replace the first element of the extended sequence: , in, Indicates the input number The extended sequence representation of each decoding layer; symbols This represents taking the hidden representation corresponding to the first position along the sequence dimension. The above replacement operation replaces the vector at the first time position of the extended sequence with the sentiment cue vector corresponding to the current layer. The sequence representation at the remaining positions remains unchanged. The first-position substitution allows the emotional cues of the current layer to be explicitly injected into the decoding layer, thus enabling emotional information to continue to participate in subsequent self-attention computations and nonlinear feature transformations.

[0049] Replaced Aligned mask , Enter the first one together A decoding layer is obtained.

[0050] This allows emotional cues to remain effective across all layers of self-attention and subsequent nonlinear transformations, thus achieving layer-by-layer, continuous emotional modulation, rather than a one-time fusion at the input.

[0051] S35, after completion After layer decoding, before the output head calculation, the additional sequence vector is removed to obtain the decoder hidden sequence after injecting sentiment cues. : , in, This means removing the first cue position from the L-th layer decoded output along the sequence dimension, retaining only the subsequent T hidden states corresponding to the original target sequence. Decoder hidden sequence Used for subsequent extraction of emotion prior feature sequences And it serves as an intermediate hidden representation in the residual calibration path. It retains both text-driven temporal motion information and the modulation results after emotion injection.

[0052] S4. Extract the prior emotion feature sequence from the decoder hidden sequence after injecting emotion cues; S41. During the decoding process, from completing the decoder hidden sequence The system collects prior emotion features at each time step. Let the first time step be... Decoder hidden sequence at each time step Hidden state sequence for: , Where t+1 represents the length of the effective sequence formed at the current t-th time step; Take the hidden state sequence The hidden state vector at the last position is used as the prior feature for that time step: , Among them, symbols The hidden state sequence represents the sequence of hidden state vectors corresponding to each position in the sequence at the current time step, taken along the last position in the time dimension; the hidden state vector represents the hidden state sequence. The H-dimensional feature representation corresponding to a certain sequence position is used to characterize the current decoding state at that position. It represents the latest hidden representation of the current time step after considering the comprehensive text conditions, historical generation status, and sentiment cues. It can be regarded as the dynamic sentiment prior of the current moment.

[0053] S42. Stack the prior features of all time steps in chronological order to obtain the emotion prior feature sequence. : , in, Used as input for subsequent residual calibration modules, it represents the sequential expression of frame-by-frame dynamic emotional information.

[0054] Preferably, when the emotion adaptation module is not enabled or the corresponding parameters are not loaded, the system can revert to the baseline generation mode without emotion modulation; at this time, the corresponding hidden state sequence and emotion prior feature sequence can still be obtained in the above manner, but the emotion prior feature sequence is not used as a valid emotion prior to participate in subsequent residual calibration.

[0055] S5. Based on the predefined set of facial expression point indices, extract the base facial expression point subset sequence from the base facial key point sequence, calculate the average value of the base facial expression point subset sequence in the time dimension, and obtain the geometric context features. Determine the set of expression point indices ,in The total number of key points in each frame; A subset of key points related to facial expressions is used to identify these points, and this subset remains consistent throughout the training and inference phases. Let... ,in, Let the number of emoji points be denoted as , and the flattening dimension of the emoji subset be defined as follows: , in, The coordinate dimensions of each key point. The emotional residual is limited to the expression-related areas such as eyebrows, eyes, and mouth, thereby avoiding unnecessary disturbance to non-expression key points.

[0056] Facial key point sequence of the base according to Extracting the expression point subset yields the base expression point subset sequence. : , in, This represents the number of keypoints in the expression point subset, where d represents the coordinate dimension of each keypoint. Extracting by index means retaining only the set of expression point indices. The corresponding key point coordinates. This represents the trajectory of key points in the facial expression region based on the generated result.

[0057] Subsequence of base expression points Calculate the mean in the time dimension : , in, This represents averaging over the time dimension, used to extract the overall geometric state. The averaging operation extracts a relatively stable overall geometric reference from a sequence of facial expression points that changes frame by frame.

[0058] Flattening yields the geometric context feature vector of the base. : , in, , representing the geometric context dimension of the flattened set of emoji dots.

[0059] Preferred, It can be determined by a pre-defined index table or key point semantic tags, and the inner lip point set can be further defined. This is used for lip shape protection constraints. The inner lip point set represents the set of key point indices corresponding to the internal oral cavity region, defining the core lip shape region that requires focused protection, so that subsequent inner lip protection loss can constrain the residual magnitude of that region.

[0060] S6. Based on the prior emotion feature sequence, geometric context features, and emotion identifier, calculate the subspace coefficient sequence of the low-dimensional subspace, and decode the subspace coefficient sequence to obtain the predicted expression residual.

[0061] S61. Map and align the sequences of geometric context features and sentiment prior features to the calibration hidden space. Let the calibration hidden dimension be... Geometric context features The vector is subjected to linear mapping and nonlinear activation to obtain the geometric hidden representation b obtained by the base geometric context mapping: , in, Indicates the calibration dimension. It is a combination of linear layers and modified linear units, and the linear layers satisfy... .

[0062] Will Extending along the time dimension yields the frame-by-frame aligned geometric context sequence. This aligns the static geometric context with the emotion prior feature sequence in the time dimension: the geometric context sequence is derived from the geometric context representation after the pedestal expression point subset sequence is averaged and mapped in the time dimension.

[0063] , Prior feature sequences of emotions Frame-by-frame emotion hiding representation is obtained by performing linear mapping and non-linear activation on each frame. : , in, It is a combination of linear layers and modified linear units, and the linear layers satisfy... . Used to reflect emotion modulation information at different time steps.

[0064] S62, Emotional Labels ( The emotion embedding vector is obtained by looking up the embedding table. : , The total number of emotion categories is The emotion embedding dimension is , For size The embedding table is an existing discrete category representation method used to map discrete emotion categories to continuous vector representations. In this invention, its role is to provide stable category-level emotion conditions for the residual calibration module.

[0065] embedding emotion vectors Extending along the time dimension to a frame-by-frame emotion category embedding sequence Provide the same emotion category conditions for each time frame: .

[0066] S63. Hide the frame-by-frame emotion representation Geometric context sequence With frame-by-frame emotion category embedding sequence The fused input is obtained by concatenating the features: , in, This indicates a splicing operation; the spliced ​​result is... It simultaneously carries dynamic emotional information, basic geometric information, and target emotional category information, serving as a composite input for subsequent fusion networks.

[0067] Will Input fusion network Obtain the fused frame-by-frame hidden representation : , in, It consists of two linear layers and a modified linear unit, with random deactivation in the middle layer. Random deactivation is a regularization operation that sets some feature components to zero with a certain probability during training to reduce feature co-adaptation, improve generalization ability, and enhance robustness to different samples and different emotional conditions.

[0068] S64. Subspace coefficient sequence obtained through coefficient head : , in, For linear mapping , This represents the frame-by-frame projection intensity of the model-predicted facial expression residuals in each basis direction of the low-dimensional subspace.

[0069] S65. Decode the subspace coefficient sequence into the predicted expression residual.

[0070] Decoding the subspace coefficients yields the residual vector sequence. : , Where μ represents the mean vector of the low-dimensional subspace, U represents the basis matrix of the low-dimensional subspace, and the decoding operation yields a sequence of residual vectors that can directly correspond to changes in geometric coordinates.

[0071] The residual vector sequence Reconstruction for predicting facial residuals : , Predicting facial residuals This represents the amount of additional geometric change predicted on a subset of facial expression points, which is used for subsequent write-back updates performed only on the subset of facial expression points.

[0072] The establishment of the low-dimensional subspace includes: S51. Obtain the true key point sequences of each data point in the training data set. and base facial key point sequence Real key point sequence This represents the trajectory of real facial landmarks corresponding to the training samples, used as a subsequent supervision signal. .

[0073] Preferably, the publicly available MEAD dataset is used, and the real keypoint sequence is extracted using dlib68.

[0074] Preferably, to ensure consistency in subsequent residual learning, , And align the prior feature sequences of emotion to the same length in the time dimension. When the length of the original sequence is At that time, linear interpolation resampling is performed along the time dimension to obtain a length of The sequence. Among them, linear interpolation is an existing time alignment method used to estimate the value of the intermediate time point based on the known values ​​of adjacent time points; in this invention, its role is to unify the time length of each sequence so as to facilitate subsequent frame-by-frame residual modeling and loss calculation.

[0075] S52, from the base facial key point sequence of each data point Extract the base expression point subset sequence from the training dataset The extraction process is the same as S5 above, and will not be repeated here.

[0076] Real key point sequence according to Extracting a subset of facial expressions yields a sequence of true facial expression subsets. : , in, This represents the trajectory of key points in the facial expression region of a real sample.

[0077] S53, using a subset sequence of real facial expression points from the training dataset. With base expression dot set sequence Calculate the target expression residual by the difference between them. : , And flatten it into tensor form: , in, The expression represents the additional change in the true expression relative to the base backbone result on the subset of expression key points, and is the direct supervised target for residual calibration learning.

[0078] Let the dimension of the low-dimensional subspace be . Tensor form of the target facial expression residuals in the training dataset Statistical analysis yields the mean vector. With basis matrix : , in, This represents the mean vector obtained from the training residual statistics, used to center the residuals; The basis matrix represents the low-dimensional subspace, and its column vectors correspond to the basis directions in the low-dimensional representation space. and A low-dimensional representational subspace for the target facial expression residual was jointly defined. This low-dimensional subspace is used to confine the facial expression residual within a structured, low-degree-of-freedom representational space, thus suppressing overly arbitrary global geometric drift. Parameters The physical meaning is: to determine the degree of freedom of expression of facial residuals; when When the value is smaller, the constraint is stronger, which is beneficial to stability; when When the value is larger, the expression is more flexible, but the risk of drift increases. In this example, 16 is a good choice.

[0079] The target residual vector is encoded into a low-dimensional subspace to obtain the true coefficient sequence. : .

[0080] Mean vector Time dimension broadcast method and target residual tensor Alignment. Among them, Let be a frame-by-frame sequence of projection coefficients of the target residual vector in the low-dimensional subspace, which is a three-dimensional real tensor, where Indicates batch size, Indicates the number of time frames. Indicates the dimension of the subspace; During the training phase, it also serves as a coefficient supervision target, used to supervise the predicted subspace coefficient sequence. Mean vector Time dimension broadcast method and target residual tensor Alignment, where broadcasting means aligning the mean vector. Extending along the time dimension to The same time length is used to perform frame-by-frame subtraction.

[0081] S7. Predict facial residuals Write back to the base facial key point sequence and expression point index By combining relevant locations, a sequence of facial key points with controllable emotions can be obtained.

[0082] Let the initial output sequence be: ; set of facial expression points The corresponding position performs addition and writes back: , in, The write-back scaling factor is used to adjust the actual write-back strength of the predicted expression residual to the base keypoint sequence, controlling the strength of the effect of the emotion residual in the final output, thereby controlling the amplitude of the final emotional expression. Generally, When they are younger, their emotions are less intense but more stable; When the value is large, the amplification of emotions is more pronounced, but excessively large values ​​may increase the risk of local geometric drift.

[0083] "Write back" only applies to the index set. The corresponding key points are updated without modifying other non-expression area key points, thus achieving structured control that "applies emotional residuals only to expression-related areas".

[0084] Output a sequence of facial key points with controllable emotions: .

[0085] Because only for The corresponding location performs a write-back update, index set. The coordinates of key points outside the facial expression remain unchanged before and after the write-back, thus ensuring that non-expression areas are not modified. Therefore, While maintaining the stability of the basic language-driven facial movement backbone, the final output result is achieved by superimposing controlled emotional changes on key points related to facial expressions.

[0086] S8. Training steps: Construct a joint objective function to train the residual calibration network.

[0087] S81. Let the joint loss L be: , in, These are the weight coefficients for each loss term. This joint objective function is used to jointly optimize the residual calibration network and its associated trainable modules during the training phase.

[0088] More specifically, in the training phase, first obtain , , , , and The intermediate values ​​and output results are then used, and the relevant parameters are updated through backpropagation based on the various losses in S8. During the inference phase, these losses are no longer calculated; instead, the trained parameters are used directly for forward generation.

[0089] S82, Residual Regression Loss Based on predicted facial expression residuals residual expression with the target The residual regression loss is constructed as follows: , Where E represents the average value. The loss function uses a norm of 1, meaning it takes the absolute value of the element-wise differences. This loss function directly constrains the difference between the predicted expression residual and the true expression residual, and its effect is on the predicted expression residual. Therefore, it is used to improve the geometric accuracy of facial expression residual prediction.

[0090] in, Represents the valid frame mask, which is a binary indicator or its equivalent real-valued weight: In one type of data construction, It can be all 1s, thus indicating that all frames are considered valid frames in the loss calculation.

[0091] S83, Temporal Smoothing Loss : For predicting facial expression residuals Construct adjacent frame difference constraints as temporal smoothing loss: , , in, Similarly, it can be combined Weighted. This represents the change in the predicted facial expression residual between adjacent time frames. This loss is used to constrain the predicted facial expression residual. The continuity in the temporal dimension suppresses abrupt changes between adjacent frames, thereby reducing facial shaking.

[0092] S84, Coefficient Supervision Loss : Subspace coefficient sequence With the true coefficient sequence Construction coefficient monitoring loss: , in, The squared L2 norm represents the loss function, which directly affects the subspace coefficients and constrains the subspace coefficient sequence predicted in step 6.4. With the obtained true coefficient sequence The consistency between them. Therefore, the technical feature constrained by the loss is the "projection representation of the expression residual in the low-dimensional expression subspace", rather than directly constraining the final keypoint coordinates.

[0093] S85, Loss of Separability (Cosine interval): Subspace coefficient sequence The mean value obtained by taking the time dimension is: , And normalize it: , , in, It is a 2-norm. It is a numerically stable term. The sample-level average coefficient representation is a two-dimensional real tensor, where represents the overall sentiment expression obtained by aggregating the frame-by-frame coefficient sequence along the time dimension. Numerical stability term. This is used to prevent numerical instability caused by a normalized denominator that is zero or too small.

[0094] Samples with different emotion labels in the same batch Calculate cosine similarity and apply an interval threshold. : , in, The threshold value is used to constrain the degree of separation between different emotion representations in the low-dimensional subspace, i.e., to control the minimum discriminative strength required between different emotion representations. This loss is applied to the subspace coefficient sequence. The temporally aggregated representation is mainly used to enhance the separability of different emotion categories in the low-dimensional representation space and reduce the aliasing between different emotions.

[0095] S86, Inner lip protection loss This is used to limit the prediction residual amplitude of a subset of inner lip keypoints related to mouth shape, suppressing excessive perturbation of key articulation structures by emotional injection, thereby reducing mouth shape drift and inner lip instability. Define the inner lip point set. Take the corresponding predicted facial expression residual. : , Its amplitude is suppressed to limit the residual amplitude in the lip-shape region; in one implementation, the L2 norm is calculated for each point and the expectation is obtained. And can be combined Weighted. This represents the portion of the predicted facial expression residual corresponding to the inner lip region. This loss is applied to the inner lip subset of the predicted facial expression residual to limit the residual amplitude in the oral cavity region, preventing emotional enhancement from causing excessive perturbation to the core lip shape region.

[0096] S87, Geometric Proxy Loss It is used to constrain the changing trends of geometric semantic quantities such as mouth corner displacement, lip opening and closing, and eye opening and closing, so that the prediction results are consistent with the actual emotional changes in the geometric patterns of key expressions.

[0097] Calculate several geometric surrogate functions Examples include the relative height of the corners of the mouth, the distance between the opening and closing of the upper and lower lips, the distance between the opening and closing of the eyelids, and left-right asymmetry. Calculate the actual relative change and the predicted relative change separately: , ; And define: , In one implementation, weights can be assigned to different proxy quantities based on emotion categories to achieve differentiated constraints. The geometric proxy loss does not directly constrain the coordinates of individual keypoints, but rather uses geometric semantic quantities as constraints to ensure the final output sequence. It aligns with the target's facial expression changes at the geometric semantic level, including changes in the corners of the mouth, lip opening and closing, and eyelids.

[0098] Therefore, this loss primarily constrains the geometric semantic rationality of the S7 output.

[0099] S88 employs an adaptive optimizer to iteratively update the parameters of the residual calibration network and can prune the gradient norm to improve training stability until the joint loss converges.

[0100] In this embodiment, to verify the effectiveness of the method of the present invention, experiments were conducted using the GRID dataset and the MEAD dataset, respectively. The results are shown in Tables 1 to 4. Table 1 verifies the generation quality and semantic consistency of the present invention in the basic content generation task; Table 2 verifies the overall performance of the present invention in the emotion injection task; Table 3 verifies the contribution of each component module of the present invention to the technical effect; and Table 4 verifies the protection capability of the present invention for the oral region, especially the key structures of the inner lip, during the emotion injection process. Table 1 uses BLEU-1, BLEU-2, BLEU-3, BLEU-4, and WER as evaluation metrics. BLEU is the bilingual evaluation substitute metric, used to measure the degree of matching between the generated sequence and the real sequence; WER is the word error rate, used to measure the semantic recognition deviation of the generated result. PT-GN is the method disclosed in the prior art "Progressive Transformers for End-to-end Sign Language Production", GEN-OBT is the method disclosed in the prior art "Gloss Semantic-Enhanced Network with Online Back-Translation for Sign Language Production", and LVMCN is the method disclosed in the prior art "Linguistics-vision monotonic consistent network for sign language production". EVP and EAT in Table 2 are emotion-driven face generation methods disclosed in the prior art "Audio-Driven Emotional Video Portraits" and "Efficient Emotional Adaptation for Audio-Driven Talking-Head Generation", respectively. w / oSep, w / oAUProxy, and w / oInner-Lip in Table 3 represent ablation variants after removing emotion separation constraints, geometric semantic proxy constraints, and inner lip protection constraints, respectively. "Residual calibration only" in Table 4 indicates a simplified scheme that uses only the residual calibration module without using the complete structured constraints of this invention. As shown in Tables 1 to 4, the method of the present invention can not only improve the consistency between the generated result and the real sequence, but also effectively improve the separability of emotions during the emotion injection process, and reduce the abnormal displacement and unstable fluctuations in the mouth region, especially the inner lip region. This shows that the present invention can achieve effective emotion modulation while ensuring the stability of the basic content movement, and has a comprehensive technical effect that is superior to the prior art.

[0101] Table 1. Metric values ​​of each method on the GRID dataset.

[0102] Table 2. Metric values ​​of each method on the MEAD dataset.

[0103] Table 3 Ablation results of various methods on MEAD

[0104] Table 4 Oral stability analysis

[0105] like Figure 2 As shown, this embodiment also presents a visual comparison case of emotion-controlled facial generation to demonstrate the generation effect of the present invention. The output text is “bin white in k three now”. Comparing the generation result of the present invention with the standard neutral pose (Ground Truth) demonstrates that the technical solution of this application can achieve emotion control while ensuring correct lip movements.

[0106] like Figure 3 As shown, this embodiment demonstrates the controllable generation results of emotions under different identity samples to illustrate the cross-identity applicability and consistency of emotion expression of the present invention. Here, "Right now it may not be the best time for business mergers" represents the language output text for different identities, and "M007, M019, M034, W011" represent the speakers. Figure 3 As can be seen, even when the input identity changes, this application can still generate key expression patterns consistent with the target emotion on different facial samples. This indicates that the proposed structured emotion injection method does not depend on a specific identity appearance, but can stably act on expression-related geometric regions under different identity conditions. Specifically, different samples show consistent emotional response trends in key expression cues such as eyebrow lifting, eyelid opening and closing, and mouth opening and closing, while no obvious abnormal displacement or overall drift occurs in non-expression areas. This shows that the present invention, through the link of "emotion prior derivation—low-dimensional residual calibration—expression point subset writing back," can achieve good cross-identity emotion control effects while maintaining the basic facial geometric stability.

[0107] like Figure 4 As shown, this embodiment also demonstrates the continuous control results of the same identity sample under different emotional and intensity conditions, illustrating the adjustable emotion intensity capability of the present invention. From Figure 4As can be seen, under fixed identity conditions, this invention can continuously regulate the target emotion; along the direction of the arrow in the figure, the intensity of emotional expression gradually increases, and the geometric changes of key expression areas such as eyebrows, eyes, and mouth show a continuous, smooth, and identifiable increasing trend. At the same time, the mouth opening and closing structure maintains good coherence, without obvious inner lip drift, local misalignment, or abrupt tremors. This result demonstrates that this invention can not only achieve discriminative control of different emotion categories but also achieve fine-grained regulation of emotion intensity, maintaining good geometric stability and temporal consistency during intensity changes.

[0108] comprehensive Figure 2 , Figure 3 and Figure 4 Furthermore, this invention not only enables controllable emotion generation while maintaining the stability of basic pronunciation geometry and mouth shape, but also maintains a stable emotional response pattern under different identity conditions and supports continuous adjustment of emotional intensity, thus demonstrating good emotional controllability, geometric stability, cross-identity generalization ability, and practical application adaptability.

[0109] The facial recognition technology involved in this patent application strictly complies with the provisions of the *Personal Information Protection Law of the People's Republic of China*, the *Cybersecurity Law of the People's Republic of China*, the *Data Security Law of the People's Republic of China*, and relevant national standards. All technical implementations are based on the principles of legality, legitimacy, and necessity. In the data collection stage, we obtain facial information through explicit notification and user authorization, ensuring that collection does not exceed the permitted scope. In the data processing stage, we employ de-identification and encrypted storage technologies to protect the original biometric data, ensuring data security during transmission, storage, and use. We promise not to use the relevant data for any purpose unrelated to identifying natural persons, nor will we share or publicly disclose it to any third party, except as otherwise provided by law. Furthermore, the system design follows the principle of minimum necessity to avoid undue infringement on public privacy rights. All algorithms have undergone ethical review, eliminating discriminatory judgments based on race, religion, gender, or other characteristics. We have established a comprehensive data security management system, including access control, operation log auditing, and data leakage emergency plans, to fully protect user rights and social morality.

[0110] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating facial key points based on emotion priors and low-dimensional residual calibration, characterized in that, Includes the following steps: S1. Obtain the input text and emotion identifier, and generate a base facial key point sequence that is unrelated to emotion based on the input text; S2. Generate a hierarchical emotion cue tensor based on the emotion identifier; S3. Inject the layered emotion cues tensor into the decoder layer by layer to obtain the decoder hidden sequence with injected emotion cues; S4. Extract the prior emotion feature sequence from the decoder hidden sequence after the emotion cues are injected; S5. Based on the predefined expression point index set, extract the base expression point subset sequence from the base facial key point sequence, calculate the average value of the base expression point subset sequence in the time dimension, and obtain the geometric context features. The expression point index set is a subset of key points related to expression changes, which is used to constrain geometric changes to a specific expression region in order to maintain the stability of other regions. S6. Based on the emotion prior feature sequence, the geometric context features, and the emotion identifier, calculate the subspace coefficient sequence of the low-dimensional subspace, and decode the subspace coefficient sequence to obtain the predicted expression residual, wherein the predicted expression residual represents the amount of geometric change added frame by frame. S7. Write the predicted expression residual back to the position in the base facial key point sequence that is related to the expression point index set to obtain an emotion-controllable facial key point sequence.

2. The facial landmark generation method based on emotion prior and low-dimensional residual calibration according to claim 1, characterized in that, In step S2, generating the hierarchical emotion cue tensor based on the emotion identifier includes: S21. Obtain the emotion embedding vector by looking up the emotion embedding table through the emotion embedding table. S22. Input the emotion embedding vector into the style mapping network to obtain the style vector; S23. Input the style vector into the hierarchical cue generation head to generate the hierarchical sentiment cue tensor with the same number of layers as the decoder.

3. The facial landmark generation method based on emotion prior and low-dimensional residual calibration according to claim 2, characterized in that, In step S3, the step of injecting the layered sentiment cue tensor into the decoder layer by layer includes: S31. Map the target-side input sequence to the initial representation of the decoder, wherein the target-side input sequence is the input sequence formed by removing the last frame from the original target key point sequence; S32. The additional sequence vector is concatenated with the initial representation in the time dimension to obtain the extended sequence. The additional sequence vector is initialized as the first-level emotion prompt vector of the hierarchical emotion prompt tensor. S33. Extend the lengths of the decoder's padding mask and causal mask from the target-side input sequence length to the length of the extended sequence; S34. For layers 1 to L of the decoder, perform the following operations in sequence: replace the additional sequence vector of the current extended sequence with the sentiment cue vector of the same layer in the hierarchical sentiment cue tensor; input the replaced extended sequence together with the extended padding mask and causal mask into the current decoding layer to obtain the updated extended sequence; S35. After completing the decoding of all layers, remove the additional sequence vector in the extended sequence after the update of the Lth layer to obtain the decoder hidden sequence with injected sentiment cues.

4. The facial landmark generation method based on emotion prior and low-dimensional residual calibration according to claim 3, characterized in that, In step S4, the prior emotion feature sequence is extracted from the decoder hidden sequence containing the injected emotion cue, including: S41. During the stepwise generation of the decoder, at each time step, the hidden state vector of the last position of the hidden layer of the decoder at that time step is extracted as the sentiment prior feature of the current time step. S42. Stack the prior emotion features of all time steps in chronological order to form the prior emotion feature sequence.

5. The facial landmark generation method based on emotion prior and low-dimensional residual calibration according to claim 1, characterized in that, Before step S6, a low-dimensional subspace is pre-established, including: S51. Obtain the true key point sequence of each data point in the training dataset and the base facial key point sequence. S52. Extract the real facial expression point subset sequence of the training dataset from the real key point sequence of each data, and extract the base facial expression point subset sequence of the training dataset from the base facial key point sequence of each data. S53. Calculate the target expression residual using the difference between the sequence of real expression points in the training dataset and the sequence of base expression points, and establish a low-dimensional subspace based on the target expression residual.

6. The facial landmark generation method based on emotion prior and low-dimensional residual calibration according to claim 5, characterized in that, In step S6, the step of calculating the subspace coefficient sequence of the low-dimensional subspace based on the emotion prior feature sequence, the geometric context features, and the emotion identifier, and decoding the subspace coefficient sequence to obtain the predicted expression residual, includes: S61. Map and project the geometric context features and the emotion prior feature sequence to the calibration hidden space; S62. Query the emotion embedding table according to the emotion identifier, obtain the emotion embedding vector, and extend the emotion embedding vector along the time dimension to obtain a frame-by-frame emotion category embedding sequence. S63. The mapped geometric context features, the emotion prior feature sequence, and the frame-by-frame emotion category embedding sequence are fused to obtain fused features; S64. Based on the fusion features, obtain the subspace coefficient sequence in the low-dimensional subspace through the coefficient head; S65. Decode the subspace coefficient sequence into the predicted expression residual.

7. The facial landmark generation method based on emotion prior and low-dimensional residual calibration as described in claim 1, characterized in that, In step S7, the write-back is achieved by: initializing the output sequence as the base facial key point sequence, and writing the predicted expression residual back to the key point coordinates related to the expression point index set in the output sequence in a weighted superposition manner.

8. The facial landmark generation method based on emotion prior and low-dimensional residual calibration according to any one of claims 1-7, characterized in that, It also includes training step S8, which includes: The residual regression loss, temporal smoothing loss, coefficient supervision loss, sentiment separability loss, inner lip protection loss, and geometric surrogate loss are weighted and calculated to construct a joint objective function, where: Residual regression loss: ， in, To predict facial residuals, Let E be the residual of the target facial expression, and E be the average. It is a norm; Temporal smoothing loss: ， in, , Let be the predicted facial expression residual at time t. Let be the predicted facial expression residual at time t-1; Coefficient supervised loss: ， in, For the predicted subspace coefficient sequence, For the true subspace coefficient sequence , It is the square norm 2; Loss of separability: ， in, For cosine similarity, This is the interval threshold. This represents the average coefficient after normalization for sample i. The average coefficients are normalized over sample j. Loss of inner lip protection: ， in, For the inner lip point set; Geometric proxy loss: ， in, For real relative change, To predict relative changes.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 8.

Images