Three-dimensional face data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a three-dimensional face data processing method and device, electronic equipment and a storage medium, the method is applied to an encoder, and the method comprises the following steps: firstly, obtaining three-dimensional face sequence data, and generating position coding data corresponding to the three-dimensional face sequence data; then, carrying out superposition processing on the position coding data and the three-dimensional face sequence data to obtain face position sequence data; thirdly, inputting the face position sequence data into a first series feature extraction module for data feature processing to obtain feature state sequence data; and finally, performing low-dimensional compression processing based on the feature state sequence data to obtain a hidden variable sequence corresponding to the three-dimensional face sequence data, and sending the hidden variable sequence to a decoder for decoding processing, thereby greatly improving the naturalness and generation quality of the three-dimensional face animation.

Description

Technical Field

[0001] This application relates to the field of visual data processing technology, and in particular to three-dimensional face data processing methods, devices, electronic devices, and storage media. Background Technology

[0002] In the field of 3D digital human animation, lip-sync animation generation generally relies on BlendShape (BS) coefficient sequences. To automate the production of animation content, machine learning methods, such as autoencoder architectures, are commonly used to generate these BS coefficient sequences. In this method, an encoder typically compresses high-dimensional BS sequence data into a lower-dimensional latent variable sequence, and a decoder then reconstructs the final animation sequence from this latent variable sequence.

[0003] However, in the generation of such animation sequences, the autoencoder typically processes a single frame of data from the animation sequence, or a small window based on the target frame and a few neighboring frames. But when an actual person speaks, their lip movements are a continuous and interconnected process; current muscle movements are influenced by past actions (lag) and prepare for the syllable to be uttered (foresight). Because existing technologies only process time-fragmented data frames, they struggle to effectively capture and model the long-term data dependencies when processing sequence data composed of Black-Scholes coefficients. This results in poor quality animation sequences generated by the decoder, often exhibiting discontinuous disturbances and jitter, lacking the necessary stability and smoothness. Consequently, existing 3D face data processing methods produce animation sequences with low naturalness. Summary of the Invention

[0004] This application provides a three-dimensional face data processing method, apparatus, electronic device, and storage medium that can improve the naturalness of generated animation sequences.

[0005] To achieve the above objectives, a first aspect of this application proposes a three-dimensional face data processing method, the method being applied to an encoder, the method comprising: Acquire three-dimensional face sequence data and generate location-coded data corresponding to the three-dimensional face sequence data; The location-encoded data and the three-dimensional face sequence data are superimposed to obtain face location sequence data; The face location sequence data is input into the first concatenated feature extraction module for data feature processing to obtain feature state sequence data. Based on the feature state sequence data, low-dimensional compression processing is performed to obtain the latent variable sequence corresponding to the three-dimensional face sequence data, and the latent variable sequence is sent to the decoder for decoding processing.

[0006] To achieve the above objectives, a second aspect of this application proposes a three-dimensional face data processing method, which is applied to a decoder and includes: Receive the sequence of hidden variables generated by the encoder; The latent variable sequence is then subjected to high-dimensional expansion processing to obtain expanded sequence data; The extended sequence data is input into the second tandem feature extraction module for data feature processing to obtain the restored feature sequence data. The second tandem feature extraction module is structurally symmetrical with the first tandem feature extraction module in the encoder. Based on the restored feature sequence data, mapping processing is performed to obtain restored 3D face sequence data.

[0007] To achieve the above objectives, a third aspect of the present application provides a three-dimensional face data processing apparatus, the apparatus comprising at least one of an encoder and a decoder; The encoder is used to acquire three-dimensional face sequence data and generate positional encoding data corresponding to the three-dimensional face sequence data; the positional encoding data and the three-dimensional face sequence data are superimposed to obtain face positional sequence data; the face positional sequence data is input into a first concatenated feature extraction module for data feature processing to obtain feature state sequence data; low-dimensional compression processing is performed on the feature state sequence data to obtain the latent variable sequence corresponding to the three-dimensional face sequence data, and the latent variable sequence is sent to the decoder for decoding processing; The decoder is used to receive the latent variable sequence generated by the encoder; perform high-dimensional expansion processing on the latent variable sequence to obtain expanded sequence data; input the expanded sequence data into the second concatenated feature extraction module for data feature processing to obtain restored feature sequence data, wherein the second concatenated feature extraction module is structurally symmetrical with the first concatenated feature extraction module in the encoder; and perform mapping processing based on the restored feature sequence data to obtain restored three-dimensional face sequence data.

[0008] To achieve the above objectives, a fourth aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the three-dimensional face data processing method as described in the first aspect or the three-dimensional face data processing method as described in the second aspect.

[0009] To achieve the above objectives, a fifth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the three-dimensional face data processing method as described in the first aspect or the three-dimensional face data processing method as described in the second aspect.

[0010] The three-dimensional face data processing method, apparatus, electronic device, and storage medium proposed in this application are applied to an encoder. The method includes: first, acquiring three-dimensional face sequence data and generating positional encoding data corresponding to the three-dimensional face sequence data; then, superimposing the positional encoding data and the three-dimensional face sequence data to obtain face positional sequence data; next, inputting the face positional sequence data into a first concatenated feature extraction module for data feature processing to obtain feature state sequence data; finally, performing low-dimensional compression processing based on the feature state sequence data to obtain a latent variable sequence corresponding to the three-dimensional face sequence data, and sending the latent variable sequence to a decoder for decoding processing. This application embodiment inputs the entire face position sequence data superimposed with position encoding into the first concatenated feature extraction module for data feature processing at once. This allows the model to comprehensively analyze the global temporal context of the animation sequence during the encoding stage. The feature extraction module can effectively capture and model the long-range dependencies between any data frames in the sequence, so that each frame in the generated feature state sequence contains the preceding and following correlation information of the entire sequence. This solves the problem of information loss caused by processing time-fragmented data frames in the prior art. Finally, the latent variable sequence generated based on this feature state sequence rich in global context information allows the decoder to reconstruct a more stable, smooth, and natural animation sequence, significantly reducing discontinuous disturbances and jitter, thereby greatly improving the naturalness and generation quality of 3D face animation.

[0011] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0012] Figure 1 This is a flowchart of a three-dimensional face data processing method provided in an embodiment of this application.

[0013] Figure 2 yes Figure 1 The flowchart for step 101.

[0014] Figure 3 yes Figure 1 The flowchart for step 103.

[0015] Figure 4 yes Figure 3 The flowchart for step 301.

[0016] Figure 5 yes Figure 4 The flowchart for step 401.

[0017] Figure 6 yes Figure 4 The flowchart for step 402.

[0018] Figure 7 yes Figure 7 The flowchart for step 702.

[0019] Figure 8 yes Figure 4 The flowchart for step 404.

[0020] Figure 9 This is a schematic diagram of a data processing flow for an MLP layer provided in another embodiment of this application.

[0021] Figure 10 yes Figure 1 The flowchart for step 104.

[0022] Figure 11 This is a flowchart of another three-dimensional face data processing method provided in another embodiment of this application.

[0023] Figure 12 This is a schematic diagram of a three-dimensional face data processing procedure provided in another embodiment of this application.

[0024] Figure 13 This is a schematic diagram illustrating the process of facial image changes in three-dimensional facial data processing according to an embodiment of this application.

[0025] Figure 14 This is a schematic diagram of the structure of a three-dimensional face data processing device provided in an embodiment of this application.

[0026] Figure 15 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0029] In the field of 3D digital human animation, lip-sync animation generation generally relies on BlendShape (BS) coefficient sequences. To automate the production of animation content, machine learning methods, such as autoencoder architectures, are commonly used to generate these BS coefficient sequences. In this method, an encoder typically compresses high-dimensional BS sequence data into a lower-dimensional latent variable sequence, and a decoder then reconstructs the final animation sequence from this latent variable sequence.

[0030] However, in the generation of such animation sequences, the autoencoder typically processes a single frame of data from the animation sequence, or a small window based on the target frame and a few neighboring frames. But when an actual person speaks, their lip movements are a continuous and interconnected process; current muscle movements are influenced by past actions (lag) and prepare for the syllable to be uttered (foresight). Because existing technologies only process time-fragmented data frames, they struggle to effectively capture and model the long-term data dependencies when processing sequence data composed of Black-Scholes coefficients. This results in poor quality animation sequences generated by the decoder, often exhibiting discontinuous disturbances and jitter, lacking the necessary stability and smoothness. Consequently, existing 3D face data processing methods produce animation sequences with low naturalness.

[0031] The following will describe in detail the three-dimensional face data processing method, apparatus, electronic device, and storage medium in the embodiments of this application. First, the three-dimensional face data processing method in the embodiments of this application will be described. (Refer to...) Figure 1 This is an optional flowchart of the three-dimensional face data processing method provided in the embodiments of this application. Figure 1 The method described may include, but is not limited to, steps 101 to 104. It is also understood that this embodiment... Figure 1 The order of steps 101 to 104 is not specifically limited; the order of steps can be adjusted or certain steps can be added or removed according to actual needs. The three-dimensional face data processing method provided in this application can be applied to the encoder in any control system (such as a server, computer, etc.) with computing resources.

[0032] Step 101: Obtain 3D face sequence data and generate location encoding data corresponding to the 3D face sequence data.

[0033] Step 101 will be described in detail below.

[0034] In some embodiments, in response to a request to generate facial animation, the first step is data acquisition and preprocessing. Here, "3D facial sequence data" refers to a series of structured data used to drive a 3D digital human model to generate lip-sync animation. Specifically, it can be a time-varying sequence of blend shape (BS) coefficients, where each time point in the sequence corresponds to a set of coefficients that define the specific expression or lip shape of the face at that moment.

[0035] Below is an example of a blending deformation coefficient sequence corresponding to a simple five-frame lip-sync animation from opening to closing the mouth. This sequence consists of five vectors, each representing the lip-sync state at a given point in time, and is driven by changing the values ​​of key coefficients such as "JawOpen" (jaw open), "MouthFunnel" (lips pursed), and "MouthSmile" (smile). The sequence begins with a near-closed, natural state, achieves a fully open mouth by increasing the "JawOpen" coefficient to its maximum value (1.0) in intermediate frames, and then gradually reduces the coefficient value back to near zero, thus simulating a complete, coherent, and smooth dynamic lip-sync change.

[0036] Since the subsequent feature extraction module itself lacks the ability to perceive data order, and the BS sequence does not contain location information, it is necessary to explicitly introduce temporal information into the sequence. Therefore, this task is accomplished by generating "location-encoded data," which involves calculating a unique mathematical vector representing the absolute or relative position of each time position (or data frame) in the sequence. This location-encoded data (PE) ensures that the model can distinguish face data from different times, laying the foundation for subsequent capture of temporal dependencies.

[0037] The following section will further describe how to generate location-coded data corresponding to the 3D face sequence data.

[0038] Reference Figure 2 The process of generating location-coded data corresponding to the three-dimensional face sequence data includes the following steps 201 to 205.

[0039] Step 201: Based on the dimension index parameters, obtain the odd-numbered dimension index parameters and the even-numbered dimension index parameters.

[0040] Step 202: Based on the product of the odd-dimensional index parameter and the preset value, divide by the feature dimension of the 3D face sequence data, and then perform exponential processing to obtain the odd-dimensional index.

[0041] Step 203: Based on the product of the even-numbered dimension index parameter and the preset value, divide by the feature dimension, and then perform exponential processing to obtain the even-numbered exponent.

[0042] Step 204: Based on the product of the odd exponent and the position parameter, perform cosine processing to obtain the odd position data, and based on the product of the even exponent and the position parameter, perform sine processing to obtain the even position data.

[0043] Step 205: Obtain position-coded data based on odd-numbered position data and even-numbered position data.

[0044] Steps 201 to 205 are described in detail below.

[0045] In some embodiments, when generating position-encoded data, if the tensor shape of the BS sequence is (B, L, 51), where B is the batch size, representing the number of independent animation sequences simultaneously input to the model in one processing step; L is the sequence length, representing the time step or number of frames contained in each animation sequence; and 51 is the feature dimension, representing the number of blend shape coefficients used to define specific lip shapes or facial expressions at each time point (each frame) of the animation sequence.

[0046] To compute a specific value for each dimension of a location-coded data vector, these dimensions must first be identified and grouped. This step is based on the "dimensional index parameter". That is, the numerical designation of each dimension (e.g., dimension 0, dimension 1, dimension 2, etc.) is used to divide it into "odd-number dimension index parameters". And "Even-numbered dimension index parameter" Two groups, of which ,and This division is a fundamental preparatory step, because different trigonometric functions (cosine and sine) will be used to calculate the dimensions of odd-numbered positions and even-numbered positions in the subsequent calculations, and the final positional code will be constructed through this alternating method.

[0047] Next, frequency-related baseline values ​​are calculated for the odd-numbered dimensions. This step involves calculating the "odd-numbered dimension index parameter". With a "preset value" (usually a fixed hyperparameter, i.e.) The product operation is performed, and the result is then divided by the "feature dimension of the 3D face sequence data" (i.e., the total number of dimensions of the input vector, for example, 51), thereby generating a normalized, unique scalar for each odd-numbered dimension. Subsequently, the final "odd exponent" is obtained through "exponential processing" (i.e., exponentiation of e). The core purpose of this series of operations is to set different change frequencies for different dimensions, which is key to the subsequent generation of periodic position signals.

[0048] Following a symmetrical approach to the previous step, frequency-related baseline values ​​are calculated for even-numbered dimensions. This step is based on the "even-numbered dimension index parameter". The product is multiplied by the same "preset value", divided by the same "feature dimension", and then subjected to "exponential processing" to obtain an "even exponent". By performing calculations in parallel with those in odd-numbered dimensions for even-numbered dimensions, the frequency distribution of the entire positional encoding vector across all dimensions is ensured to be regular and consistent, providing mathematical consistency for subsequent models to understand relative positional relationships.

[0049] Next, the core operation combines time and frequency information to generate the final position code value. This step introduces the "position parameter". This parameter represents the specific time position or sequential index of the data in the sequence (e.g., frame 0, frame 1, etc.). By multiplying the "odd exponent" by this "position parameter" and then performing "cosine processing," the specific values ​​for all odd-numbered dimensions at that time position are obtained, i.e., the "odd-numbered position data." As shown in the formula below.

[0050]

[0051] Similarly, multiplying the "even exponent" by the "position parameter" and then performing a "sine transformation" yields the "even position data". As shown in the formula below.

[0052]

[0053] This step utilizes the periodicity of sine and cosine functions, so that each time position corresponds to a unique vector composed of trigonometric function values ​​of different frequencies.

[0054] Finally, the calculated "odd-number position data" and "even-number position data" are merged to form a complete vector, which is the final "location encoding data" corresponding to a specific time position. This merging is typically done by interleaving and padding according to the original dimensional index order, thus creating a vector with dimensions completely consistent with the 3D face sequence data. Repeating the above steps for each position in the sequence generates a complete location encoding matrix.

[0055] Through steps 201 to 205 above, a deterministic position encoding generation method based on sine and cosine functions is constructed. This method can generate a unique, high-dimensional vector representation for each time position in the sequence. This encoding method not only explicitly injects information about the absolute position of the data frame into the model, but more importantly, its periodic mathematical characteristics enable the relative relationship between any two time positions to be represented linearly. This greatly helps the subsequent feature extraction module to understand and learn the order and temporal dependencies of the sequence data, and is a key technical prerequisite for effectively processing long sequence data.

[0056] Step 102: Overlay the location-coded data and the 3D face sequence data to obtain the face location sequence data.

[0057] Step 102 is described in detail below.

[0058] In some embodiments, after obtaining the 3D face sequence data and location-encoded data, the acquired raw "3D face sequence data" and the generated "location-encoded data" are overlaid. After information fusion, the data is then processed through an MLP layer for multi-layer perceptual processing to obtain "face location sequence data." This "overlay processing" is typically a vectorized addition operation, aiming to organically combine the feature information representing the lip shape with the temporal information representing the time position of that shape. After this step, each data unit in the resulting new sequence simultaneously contains two key pieces of information: "what shape" and "at what time," thus constituting complete input data suitable for deep temporal feature analysis.

[0059] Step 103: Input the face location sequence data into the first concatenated feature extraction module for data feature processing to obtain feature state sequence data.

[0060] Step 103 will be described in detail below.

[0061] Furthermore, the core deep feature extraction step of this technical solution is performed. This step treats the face location sequence data, which includes location information, as a whole and inputs it into the "first concatenated feature extraction module" for data feature processing. This module is typically composed of multiple concatenated sub-modules with self-attention mechanisms. It can perform global analysis on the entire input sequence (i.e., face location sequence data), calculating the interdependencies between data frames at any two time points in the sequence, rather than processing isolated or local data as in existing technologies. In this way, the module can effectively model and capture long-range dependencies in the animation sequence. The final output "feature state sequence data" is a highly refined feature representation, where the feature state at each time point contains the global context information of the entire sequence.

[0062] The following describes how data feature processing is performed in the first concatenated feature extraction module.

[0063] Reference Figure 3 The first cascaded feature extraction module includes multiple cascaded feature extraction modules. The face position sequence data is input into the first cascaded feature extraction module for data feature processing to obtain feature state sequence data, including the following steps 301 to 302.

[0064] Step 301: Based on the cascaded order of multiple feature extraction modules, the input data is processed in each feature extraction module one by one to obtain the output feature sequence data; wherein, the input data of the first feature extraction module is the face position sequence data, and the input data of the other feature extraction modules is the output feature sequence data of the previous feature extraction module.

[0065] Step 301 will be described in detail below.

[0066] In some embodiments, a progressively deeper, iterative feature extraction process is performed. The first cascaded feature extraction module in this application includes multiple cascaded feature extraction modules, where each feature extraction module can be a Transformer Block, that is, the first cascaded feature extraction module can be four cascaded Transformer Blocks.

[0067] Therefore, based on a pre-defined "serialization order," the data sequence passes through each feature extraction module sequentially. Specifically, the first feature extraction module receives initial "input data" consisting of "face location sequence data" containing temporal information. After processing by this module, a preliminary refined "output feature sequence data" is generated. This output then becomes the "input data" for the next feature extraction module, and is processed again, in a continuous cycle. This process ensures that feature extraction at each level is based on the results of the previous level, thereby achieving gradual abstraction and deepening of the features.

[0068] The data processing flow in each feature extraction module will be described in further detail below.

[0069] Reference Figure 4 The feature extraction module includes a first normalization layer, a multi-head attention layer, a second normalization layer, and a multi-head perception layer. In each feature extraction module, the input data is processed to obtain the output feature sequence data, including the following steps 401 to 404.

[0070] Step 401: Input the input data into the first normalization layer for the first normalization process to obtain the first normalized data.

[0071] Step 401 will be described in detail below.

[0072] In some embodiments, the first step in each feature extraction module is data preprocessing and stabilization. This step processes the "input data" received by the current feature extraction module. First, the data is fed into the "first normalization layer" (LayerNorm 1) for "first normalization processing," where b is the batch parameter, l is the length parameter, and c is the feature dimension parameter. This "first normalization layer" is a component used to stabilize the data distribution. It typically calculates statistics (such as mean and variance) of the input data along the feature dimensions and standardizes the data accordingly. This is done to mitigate the vanishing or exploding gradient problems that may occur during model depth stacking and training, thereby improving training stability and efficiency. After this processing, the resulting data distribution is more normalized, which is called "first normalized data." This provides a solid data foundation for subsequent attention calculations.

[0073] The data processing flow in the first normalization layer will be described in more detail below.

[0074] Reference Figure 5 The input data is input into the first normalization layer for the first normalization process to obtain the first normalized data, including the following steps 501 to 502.

[0075] Step 501: In the first normalization layer, calculate the mean and variance of the input data respectively.

[0076] Step 502: Based on the difference between the input data and the data mean, divide by the data variance and combine with the training parameters to obtain the first normalized data.

[0077] Steps 501 to 502 are described in detail below.

[0078] In some embodiments, the first normalization layer performs statistical analysis of the data before normalization. Within the "first normalization layer," the received "input data" is first processed. The distribution characteristics are quantified. This step involves calculating two key statistical indicators: the "data mean". And "data variance" As shown in the formula below.

[0079]

[0080] Here, C represents the feature dimension; the "data mean" refers to the average value of the input data across a specific dimension (usually the feature dimension), reflecting the central tendency of the data; while the "data variance" measures the dispersion of the data around the mean. The calculation of these two statistics forms the mathematical basis for subsequent standardization operations, providing a necessary benchmark for eliminating differences in the magnitude and distribution of the data.

[0081] Next, the core standardization and rescaling transformation is performed. This step is based on the statistics calculated in the previous step (including the "data mean"). And "data variance" The input data is transformed. First, the difference between the input data and the mean data is calculated. The data center is shifted to zero; then, this difference is divided by the "data variance". (or its square root, i.e., standard deviation), scaling the data to a standard range. After this standardization, the results are then combined with the "training parameters." The "training parameters" are a set of parameters (typically including a gain parameter) that can be learned and optimized through the model training process. and a bias parameter These mathematical operations allow the network to adaptively scale and translate standardized data to recover or learn a data distribution more beneficial to the task. The final data obtained after this series of mathematical operations is called "first-canonical data." As shown in the formula below.

[0082]

[0083] Through steps 501 to 502 above, a complete layer normalization processing method is constituted. Its core beneficial effect lies in the fact that by independently calculating the mean and variance of each input data and performing standardization accordingly, it effectively solves the problem of inconsistent internal data distribution (i.e., internal covariate shift) caused by changes in parameters of each layer during the training of deep neural networks. This processing method not only keeps the data flow stable across the network layers, but also enhances the model's generalization ability and makes the training process more robust to the setting of hyperparameters such as initial values ​​and learning rates, thereby significantly improving the training speed and final performance of the entire model.

[0084] Step 402: Input the first standardized data into the multi-head attention layer for attention processing to obtain attention feature data.

[0085] Step 402 will be described in detail below.

[0086] After obtaining the first normalized data, the core modeling of intra-sequence dependencies is performed. This step inputs the "first normalized data" obtained in the previous step into the "multi-head attention layer" (i.e., the MHA layer) for attention processing. This MHA layer is the key computational unit of this feature extraction module, and its core function is to dynamically calculate the association weights between each data point in the sequence and all other data points. Through this mechanism, the model can autonomously learn the long-range and short-range dependencies in the sequence data. The term "multi-head" refers to the fact that this attention calculation process is performed in parallel in multiple independent subspaces, enabling the model to capture various complex feature associations from different perspectives. After processing, the output "attention feature data" is a new sequence in which the features at each position are integrated with the contextual information of the entire sequence.

[0087] The data processing flow in the multi-head attention layer will be described in more detail below.

[0088] Reference Figure 6 The first standardized data is input into the multi-head attention layer for attention processing to obtain attention feature data, including the following steps 601 to 604.

[0089] Step 601: In the multi-head attention layer, the first normalized data is projected into an attention parameter sequence, and the attention parameter sequence is evenly divided in the last dimension to obtain multiple sets of attention parameter subsequences.

[0090] Step 602: Rotate and encode the key parameter vector and value parameter vector in each attention parameter subsequence to obtain the encoded attention parameter subsequence.

[0091] Steps 601 to 602 are described in detail below.

[0092] In some embodiments, the first step in the multi-head attention layer is parameter preparation and multi-head partitioning before attention computation. Inside the multi-head attention layer, the received "first-canonical data" of shape (B, L, C) is first transformed into a set of "attention parameter sequences" q, k, v specifically for attention computation through a linear "projection" operation. This sequence typically contains three different role vectors: query, key, and value, each with a shape of (B, L, C) (all n×192 dimensions). Subsequently, to achieve multi-angle feature capture, the "attention parameter sequence" is "equally divided in its last dimension," that is, the feature dimension of each role vector is divided into multiple equal parts (i.e., the three vectors are equally divided into 12 groups in the last dimension (each group has a shape of n×16 dimensions), and the q, k, v in each group have a shape of (B, L, C / 12)). After this processing, the original single parameter sequence is decomposed into "multiple sets of attention parameter subsequences," and each set of subsequences will act as an independent "head," performing attention computation in parallel in subsequent steps.

[0093] Next, the crucial operation of integrating positional information into the attention parameters is performed. This step processes each "attention parameter subsequence" obtained in the previous step, and performs "Rotation Position Encoding" (ROPE) on the "key parameter vector q" and "value parameter vector k" within each subsequence. ROPE is an advanced positional information embedding technique that directly encodes relative positional information into the vector representation by rotating the vector in relation to its position within the sequence. After this processing, the key and value vectors in the resulting "encoded attention parameter subsequence" not only carry their own feature information but also contain precise temporal positional relationships, laying the foundation for subsequent position-aware attention computation.

[0094] The following section will further describe how to perform rotational position encoding.

[0095] Reference Figure 7 The key parameter vector and value parameter vector in each attention parameter subsequence are rotated and encoded to obtain the encoded attention parameter subsequence, including the following steps 701 to 707.

[0096] Step 701: Use the key parameter vector and value parameter vector as rotation processing parameter vectors respectively.

[0097] Step 702: Based on the feature dimensions of the 3D face sequence data, calculate the frequency vector, and obtain the rotation angle based on the product of the current position parameter corresponding to the rotation processing parameter and the frequency vector.

[0098] Step 703: Based on the positional order of the vector elements in the rotation processing parameter vector, divide the odd and even positional elements in the rotation processing parameter vector to obtain odd rotation vectors and even rotation vectors.

[0099] Step 704: Multiply the even-numbered rotation vector with the cosine of the rotation angle element by element to obtain an even-numbered cosine value; multiply the even-numbered rotation vector with the sine of the rotation angle element by element to obtain an even-numbered sine value; multiply the odd-numbered rotation vector with the cosine of the rotation angle element by element to obtain an odd-numbered cosine value; multiply the odd-numbered rotation vector with the sine of the rotation angle element by element to obtain an odd-numbered sine value.

[0100] Step 705: Based on the difference between even cosine values ​​and odd sine values, obtain an even rotation encoding vector, and based on the sum of even sine values ​​and odd cosine values, obtain an odd rotation encoding vector.

[0101] Step 706: Based on the merging process of even-numbered and odd-numbered rotation encoding vectors, the rotation position encoding vector is obtained.

[0102] Step 707: Update the attention parameter subsequence based on the rotation position encoding vector to obtain the encoded attention parameter subsequence.

[0103] Steps 701 to 707 are described in detail below.

[0104] In this application, the feature dimension C must be a multiple of 12. Taking C=192 as an example, the calculation process of ROPE is as follows.

[0105] In some embodiments, the first step is to specify the objects to be encoded. By uniformly specifying the two types of vectors, "key parameter vector" q and "value parameter vector" k, as "rotation processing parameter vector", the subsequent encoding process can be described and executed in a general way, that is, applying the exact same rotation encoding procedure to the key vector and the value vector, thus ensuring consistency in processing.

[0106] Next, the core rotation parameters are calculated. This step first calculates a frequency vector based on the feature dimensions of the 3D face sequence data. This vector specifies different rotation fundamental frequencies for different dimensions or pairs of dimensions, and is the basis for achieving positional discrimination. Subsequently, this "frequency vector" is coupled with the "current position parameter," which represents the current data frame's order in the sequence. Performing the product operation yields the "rotation angle". This "rotation angle" is specific to each time position. It determines how much the parameter vector at that position needs to be rotated and is the core basis for subsequent rotation transformations.

[0107] Further, a structural decomposition of the "rotation processing parameter vector" is performed. To perform a rotation operation similar to that on a two-dimensional plane, this step divides the vector into odd-dimensional and even-dimensional parts based on the "positional order of elements" (i.e., dimension indices), resulting in "odd-dimensional rotation vectors." And "even-numbered rotation vectors" As shown in the formula below.

[0108]

[0109] This partitioning treats a high-dimensional vector as a pair of multiple two-dimensional sub-vectors, laying a structural foundation for subsequent transformations using the mathematical principles of two-dimensional rotation matrices.

[0110] Next, the pre-calculation of the components required for the rotation transformation is performed. This step involves using the "even-numbered rotation vector" obtained in the previous step. And "odd-numbered rotation vectors" Respectively related to "rotation angle" The sine and cosine values ​​are multiplied element-wise. Specifically, these multiplication operations yield the even-numbered cosine values. "Even-numbered sine values" "Odd cosine values" And "odd sine values" These four intermediate results are essentially the various intersection terms generated when a two-dimensional rotation matrix is ​​applied to a vector, and are necessary components of the final rotated vector.

[0111] Next, the synthesis of the rotated vector components is performed. This step combines the four intermediate results obtained in the previous step according to the standard two-dimensional rotation formula. By calculating the difference between the even-numbered cosine value and the odd-numbered sine value, the even-numbered dimension component of the rotated vector, i.e., the "even-numbered rotated encoded vector," is obtained. Simultaneously, by calculating the "cumulative value" of the "even-number sine value" and the "odd-number cosine value," the odd-number dimension portion of the rotated vector, i.e., the "odd-number rotated encoded vector," is obtained. This step directly reflects the mathematical principles of rotational transformation.

[0112] Next, the final rotation-coded vector is reassembled. This step merges the "even-number rotation-coded vectors" and "odd-number rotation-coded vectors" calculated in the previous step. The "merging process" here typically refers to re-intertwining the two vectors according to their original odd-even dimension order, thereby constructing a complete "rotation position encoding vector" with the same dimensions as the original "rotation processing parameter vector". This new vector is the final form of the original vector after rotation encoding of position information.

[0113] Finally, the original attention parameters are updated. This step uses the "rotation position encoding vector" obtained in the previous step to update the corresponding original key vector q or value vector k in the "attention parameter subsequence". After the update, the final "encoded attention parameter subsequence" is obtained. The key and value vectors in this subsequence now inherently carry precise temporal position information through their own vector representations and can be directly used for subsequent attention calculations.

[0114] Through steps 701 to 707 above, a precise Rotation Position Encoding (RoPE) method is constructed. Unlike traditional methods that add position information as an independent vector to the content vector, this method integrates position information losslessly into the vector representation through rotation operations. This encoding method allows the dot product between any two encoded vectors (i.e., the basis for calculating attention scores) to naturally reflect their relative positional relationship. This enables the attention mechanism to perceive and utilize the relative distance between data frames in the sequence more naturally and accurately, thereby greatly improving the model's ability and accuracy in modeling temporal dependencies.

[0115] Step 603: Perform attention calculations on each encoded attention parameter subsequence one by one to obtain the attention result subsequence.

[0116] Step 604: Concatenate all attention result subsequences to obtain attention feature data.

[0117] Steps 603 to 604 are described in detail below.

[0118] After obtaining the encoded attention parameter subsequence, parallel, multi-faceted dependency computation is performed. This step executes the core attention computation for each encoded attention parameter subsequence one by one, as shown in the following formula.

[0119]

[0120] In this computational process, the model performs similarity matching between the query vector in the subsequence and the position-encoded key vector to generate a set of attention weights. These weights reflect the mutual importance between different positions in the sequence. Then, the model uses these weights to perform a weighted summation of the position-encoded value vectors. This process occurs independently in each "head," ultimately generating an "attention result subsequence" for each subsequence. This subsequence is the result of re-aggregating and weighting the original sequence information from the perspective of a specific feature subspace.

[0121] Finally, the multi-dimensional information is integrated and output. After all the independent "heads" have completed their respective attention calculations, this step involves "concatenating" all the resulting "attention result subsequences" (that is, concatenating the 12 groups of calculation results in their original order and then aggregating them through a fully connected layer). This "concatenation" process typically refers to reconnecting these parallel, low-dimensional subsequences along the feature dimension, restoring them to a single sequence consistent with the input data dimension. In this way, the different dependencies and feature information captured by the different attention "heads" are effectively integrated. The final complete sequence is the final output of the "multi-head attention layer," referred to as "attention feature data."

[0122] Through steps 601 to 604 above, a highly efficient and powerful multi-head attention mechanism is constructed. By decomposing attention computation into multiple parallel "heads", this mechanism can simultaneously learn and capture diverse dependencies within the data sequence from different representation subspaces. At the same time, by applying rotational position encoding to the key vector and value vector, the attention computation itself can accurately perceive and utilize the relative positional information between data points. Finally, the results of the multi-head process are spliced ​​and integrated to form a deep feature representation that contains rich contextual associations and has accurate temporal awareness, namely "attention feature data", which greatly enhances the model's ability to model complex sequence patterns.

[0123] Step 403: Input the attention feature data into the second normalization layer for second normalization processing to obtain the second normalized data.

[0124] Step 404: Input the second standardized data into the multi-head perception layer for perception processing to obtain the output feature sequence data.

[0125] Steps 403 to 404 are described in detail below.

[0126] After obtaining the attention feature data, a second data distribution stabilization operation is performed. This step inputs the "attention feature data" generated by the multi-head attention layer into the "second normalization layer" (LayerNorm 2) for a second normalization process. Similar to the first normalization layer, this layer standardizes the attention-processed feature data to ensure that the data distribution remains within a stable and controllable range before entering the subsequent multi-head perception layer. This step is crucial for maintaining the stability of the entire deep network's information flow and is a key link connecting attention computation and subsequent nonlinear transformations. The output obtained after this processing is called "second normalized data."

[0127] After obtaining the second-normalized data, nonlinear transformations and in-depth processing of the features are performed. This step inputs the "second-normalized data" into the "multi-head perceptron layer" (i.e., the MLP layer) for perceptual processing. Technically, this MLP layer is typically a feed-forward network that independently applies the same set of nonlinear transformations to the feature vectors at each time point in the sequence. The purpose of this processing is to further enhance the model's expressive power, performing deeper feature extraction and mapping on the information integrated through the attention mechanism. Finally, the data output by this layer is the "output feature sequence data" after the entire feature extraction module has completed its processing.

[0128] The following section will describe in more detail the data processing flow in the multi-head perception layer.

[0129] Reference Figure 8 The second standardized data is input into the multi-head perception layer for perception processing to obtain output feature sequence data, including the following steps 801 to 804.

[0130] Step 801: In the multi-head perception layer, the second standardized data is expanded in the last dimension to obtain expanded standardized data.

[0131] Step 802: Generate a gating signal based on the extended canonical data using an activation function.

[0132] Step 803: Perform element-wise multiplication based on the gating signal and the extended specification data to obtain the gating processing data.

[0133] Step 804: Compress the gating data in the last dimension to obtain the output feature sequence data. The output feature sequence data has the same data dimension as the second normalized data.

[0134] Steps 801 to 804 are described in detail below.

[0135] In some embodiments, the first step in the multi-head perception layer is a nonlinear mapping preparation of the feature space. Inside the multi-head perception layer, the input second-canonical data of shape (B, L, C) is first subjected to a linear transformation, resulting in a "quantum expansion in its last dimension." This means increasing the dimension of the feature vector to a larger size, such as several times the original dimension (e.g., four times). The purpose of this is to project the features into a higher-dimensional space, thereby providing greater capacity for learning and representing more complex and abstract feature relationships. After this step, the resulting data is "expanded canonical data" of shape (B, L, 4C).

[0136] Next, the generation of dynamic information flow control signals is performed. This step uses a non-linear activation function (such as SiLU, or Sigmoid weighted linear unit) to process the extended canonical data obtained in the previous step, thereby generating a gating signal. This gating signal is a vector with the same dimension as the extended canonical data, and the value of each element is typically constrained to a specific range (such as between 0 and 1) by the activation function. This signal acts like an adjustable valve or filter, and its value determines which features should be preserved and which should be suppressed in the next step.

[0137] Next, the core gating adjustment operation is performed. This step involves element-wise multiplication of the "gating signal" generated in the previous step with the "extended canonical data" (or another parallel linear transformation result) generated in the first step. Through this multiplication operation, the values ​​in the gating signal are adjusted proportionally to the corresponding feature values ​​in the extended canonical data. When the value of a certain element in the gating signal is close to 0, the corresponding feature information flow is shut off; conversely, when its value is close to 1, information flows smoothly. The data obtained after this step is called "gating processed data".

[0138] Finally, the feature space is restored and output. This step takes the gated data of shape (B, L, 4C) and performs another linear transformation to compress its last dimension, remapping its dimensions from the expanded high-dimensional space back to the original dimensions, resulting in output feature sequence data of shape (B, L, C). This step ensures that the final output feature sequence data is consistent with the input second-canonical data in terms of data dimension. This is crucial for maintaining the size uniformity of the data flow in the entire cascaded feature extraction module architecture, allowing the output of this module to be directly used as the input of the next module.

[0139] Reference Figure 9This is a schematic diagram of a data processing flow for an MLP layer provided in an embodiment of this application. Figure 9 The diagram shows a schematic representation of the multi-head perceptual layer (MLP layer) within the feature extraction module. This structure employs a gated linear unit (GLU) design paradigm. Figure 9 As shown, the input data (of shape B, L, C) enters two independent fully connected layers in parallel. Figure 9 The green vertical bars 1 and 2 are used to expand the last dimension of the data by a factor of four in both branches (changing the shape to B, L, 4C). The output of one branch is passed through a SiLU activation function to generate a gating signal that acts as a dynamic information flow controller. This gating signal is then multiplied element-wise with the output of the other branch. Figure 9 The purple circle X in the middle allows for selective enhancement or suppression of features. Finally, the gated data passes through a fully connected layer (…). Figure 9 The green vertical bar 3 in the middle compresses its dimensions back to (B, L, C) consistent with the input, as the final output of this multi-head perception layer.

[0140] Through steps 801 to 804 above, an advanced gated linear unit network structure is constructed. Compared with the traditional feedforward network that only uses a single activation function, this structure introduces a data-driven "gating signal" to dynamically control the information flow, which greatly enhances the nonlinear expressive power and flexibility of the "multi-head perception layer". This gating mechanism allows the model to learn to select and combine features autonomously based on the input content, thereby more effectively processing and refining the information integrated by the attention layer, and ultimately significantly improving the performance of the entire feature extraction module.

[0141] Steps 401 to 404 constitute the internal processing flow of a complete and efficient feature extraction module. This flow ensures data stability and training feasibility during complex transformations through two standardization operations: a "first normalization layer" and a "second normalization layer." The "multi-head attention layer" captures the dynamic dependencies within the input sequence from a global perspective, while the "multi-head perception layer" performs independent deep nonlinear processing on the features at each location. This design, combining "global correlation" and "local transformation," allows each feature extraction module to effectively map the input sequence to a higher-level, more expressive feature space, providing a solid modular foundation for ultimately generating high-quality "feature state sequence data."

[0142] Step 302: Based on the output feature sequence data of the last feature extraction module, obtain the feature state sequence data.

[0143] Step 302 will be described in detail below.

[0144] After the data sequence has undergone layer-by-layer processing by all the cascaded "feature extraction modules" (i.e., Transformer Blocks), the "output feature sequence data" generated by the last module is taken as the final processing result. This is a high-level feature representation obtained after thorough and in-depth processing, which can comprehensively characterize the intrinsic state and complex dependencies of the original input sequence. This "feature state sequence data" will then be used for subsequent tasks such as low-dimensional compression.

[0145] Through steps 301 to 302 above, a hierarchical processing architecture is constructed by stacking multiple feature extraction modules in series. This architecture allows the model to learn features from shallow to deep, progressively abstracting features from the input face position sequence data. Shallow modules may capture local and simple temporal patterns, while deep modules can integrate the outputs of all previous modules, thereby identifying and encoding more complex, global, and time-spanning dependencies. This iterative refining process ensures that the final "feature state sequence data" is a feature representation with extremely high information density and strong expressive power, providing a solid foundation for accurately capturing the subtle dynamics and internal logic of 3D face animation.

[0146] Step 104: Perform low-dimensional compression processing on the feature state sequence data to obtain the latent variable sequence corresponding to the 3D face sequence data, and send the latent variable sequence to the decoder for decoding processing.

[0147] Step 104 is described in detail below.

[0148] In some embodiments, after obtaining the feature state sequence, feature compression and output are performed. This step performs low-dimensional compression on the "feature state sequence data" output from the previous step, which is rich in global context information. This process typically involves mapping the high-dimensional feature state sequence to a predefined low-dimensional probability distribution using a predictive model (such as a fully connected layer) (e.g., predicting a mean and variance for each time point in the sequence), and then sampling from this distribution to obtain the final "latent variable sequence." This latent variable sequence is a highly condensed, information-density, compact representation of the original 3D face sequence data, significantly reducing data dimensionality while preserving key animation dynamic features. Finally, this latent variable sequence is sent to a subsequent decoder for final animation sequence reconstruction.

[0149] The following section will further describe how to perform low-dimensional compression on characteristic state sequence data.

[0150] Reference Figure 10The latent variable sequence corresponding to the three-dimensional face sequence data is obtained by performing low-dimensional compression processing on the feature state sequence data, including the following steps 1001 to 1003.

[0151] Step 1001: Input the feature state sequence data into the fully connected layer for compression and projection to obtain the feature mean sequence and the feature variance log sequence with consistent data dimensions. The data dimension of the feature mean sequence is smaller than that of the feature state sequence data.

[0152] Step 1002: Based on the characteristic mean sequence and the characteristic variance logarithm sequence, obtain the characteristic normal distribution.

[0153] Step 1003: Sample the characteristic normal distribution to obtain the latent variable sequence.

[0154] Steps 1001 to 1003 are described in detail below.

[0155] In some embodiments, the first step is a mapping from a high-dimensional feature space to a low-dimensional probability space. This step involves feeding the deeply processed "feature state sequence data" into a "normalization layer" (LayerNorm) and a "fully connected layer" for "compressed projection." Unlike directly outputting a compressed vector, the fully connected layer predicts a set of statistical parameters for each time point in the sequence, specifically a "feature mean sequence" and a "feature variance logarithm sequence," both of which are vectors of shape (B, L, C / 4). Here, the "feature mean sequence" represents the expected center position of the compressed features in the low-dimensional space, while the "feature variance logarithm sequence" describes the uncertainty or distribution range around that center position. The logarithmic form is used to ensure that the variance is positive and to increase the numerical stability of the training. A key feature of this step is that the data dimension of the output "feature mean sequence" (e.g., a vector of shape (B, L, C / 4)) is explicitly "smaller than the data dimension of the feature state sequence data" (e.g., a vector of shape (B, L, C)), thus achieving dimensionality reduction of the data.

[0156] Next, the probability distribution is constructed. Logically, this step builds upon the output of the previous step, utilizing the reparameterization technique in Variational Autoencoders (VAEs). By combining the resulting "feature mean sequence" and "feature variance logarithm sequence," a specific probability distribution model, namely the "feature normal distribution," is defined for each time point in the sequence. A normal distribution (or Gaussian distribution) is entirely determined by its mean and variance. Therefore, through this step, the original, deterministic feature state sequence is successfully transformed into a probabilistic representation—a sequence of continuous normal distributions, where each distribution describes the possible location of the compressed feature in the latent space at the corresponding time point.

[0157] Finally, the process of generating concrete instances from the probability distribution is performed. To obtain a definite vector that can be used by the subsequent decoder, this step requires "sampling" the "feature normal distribution" constructed in the previous step. "Sampling" here refers to randomly selecting a point from the normal distribution as the final representation at that time point. This process typically employs reparameterization techniques to ensure that the entire encoding process is differentiable and trainable. After sampling once for each distribution in the feature normal distribution sequence, the resulting sequence of points constitutes the final "latent variable sequence." This sequence is a highly condensed and probabilistically meaningful compact representation of the original 3D face sequence data.

[0158] Through steps 1001 to 1003 above, these steps collectively constitute a low-dimensional compression method based on the concept of variational autoencoder (VAE). Its core benefit lies in the fact that it does not rigidly map the input data to a fixed point in the latent space, but rather encodes it as a probability distribution region. This probabilistic encoding method, by introducing a sampling process, brings a good regularization effect to the model, forcing the latent space to become smoother and more structured. This not only effectively prevents the model from overfitting the training data and enhances its generalization ability, but also ensures that the generated "latent variable sequence" is a more robust representation that better captures the essential changes in the data, providing a foundation for the subsequent generation of high-quality, diverse animation sequences.

[0159] In addition, refer to Figure 11 This application also provides an optional flowchart of another three-dimensional face data processing method. Figure 11 The method may include, but is not limited to, steps 1101 to 1104. It is also understood that this embodiment... Figure 11 The order of steps 1101 to 1104 is not specifically limited; the order of steps can be adjusted or certain steps can be added or removed according to actual needs. The three-dimensional face data processing method provided in this application can be applied to the decoder in any control system (such as a server, computer, etc.) with computing resources.

[0160] Step 1101: Receive the sequence of latent variables generated by the encoder.

[0161] Step 1102: Perform high-dimensional expansion processing on the latent variable sequence to obtain expanded sequence data.

[0162] Step 1103: Input the extended sequence data into the second tandem feature extraction module for data feature processing to obtain the restored feature sequence data. The second tandem feature extraction module is structurally symmetrical with the first tandem feature extraction module in the encoder.

[0163] Step 1104: Perform mapping processing based on the restored feature sequence data to obtain restored 3D face sequence data.

[0164] Steps 1101 to 1104 are described in detail below.

[0165] In some embodiments, the decoder first initializes the decoding process. This step establishes an information channel between the decoder and encoder by "receiving the latent variable sequence generated by the encoder." Here, the "latent variable sequence" is a low-dimensional, high-information-density compact representation obtained by the encoder after deep compression of the original 3D face sequence data. This sequence contains the core dynamic features and temporal logic of the original animation and is the sole source of information for the decoder to perform all subsequent reconstruction work, marking the transition from data compression to data reconstruction.

[0166] Next, the feature dimension preparation and restoration are performed. The decoder takes the received "latent variable sequence" of shape (B, L, C / 4) and passes it through a fully connected layer for "high-dimensional expansion processing" (that is, expanding the last dimension of the latent variables), resulting in "expanded sequence data" of shape (B, L, C). Since the dimension of the latent variable sequence is much lower than the working dimension of the core processing module inside the decoder, it is necessary to expand its feature dimension through a dimensionality increase operation (usually a fully connected layer). This step is a necessary preparatory step in the decoding process, and its purpose is to map the compact latent variable information into a richer and more expressive high-dimensional feature space so that the subsequent feature extraction module can effectively process and restore details.

[0167] Next, the core feature reconstruction and detail generation are performed. This step inputs the "extended sequence data" of shape (B, L, C) into the "second cascaded feature extraction module" for "data feature processing." The "second cascaded feature extraction module" here maintains structural symmetry with the "first cascaded feature extraction module" in the encoder; that is, the second cascaded feature extraction module also includes multiple cascaded feature extraction modules (i.e., the second cascaded feature extraction module can be four cascaded Transformer Blocks). This means it also has the ability to capture global context and long-range dependencies. However, its function is opposite: it uses this ability to interpret and "decode" the temporal information encoded in the extended sequence data, gradually and hierarchically reconstructing specific, high-dimensional animation details from abstract features. The final output, the "restored feature sequence data," is a high-level feature representation that has essentially restored the complex structure of the original sequence.

[0168] Finally, the final output is formatted and generated. This step performs a "mapping process" on the "reconstructed feature sequence data" obtained in the previous step to obtain the final "reconstructed 3D face sequence data". Since the reconstructed feature sequence data is still a feature representation within the decoder, its dimensions and numerical range may not match the original data format. Therefore, it needs to be transformed into a standard 3D face sequence data format (such as a fused deformation coefficient sequence) through a final mapping layer (e.g., a fully connected layer). After this step, the output "reconstructed 3D face sequence data" is the final animation sequence generated by this method, which is highly similar to the original input in content and structure.

[0169] Through steps 1101 to 1104 above, a complete and efficient 3D face sequence data decoding process is formed. By adopting a "second serial feature extraction module" that is "structurally symmetrical" with the encoder, the decoder can accurately understand and reverse restore the global temporal information and long-range dependencies compressed by the encoder in a "mirror" manner. This symmetrical design ensures that in the process of reconstructing high-dimensional animation from low-dimensional "latent variable sequences", the dynamic coherence and internal logic of the original sequence can be preserved and restored to the maximum extent. This ensures that the final "reconstructed 3D face sequence data" is not only accurate in a single frame, but also exhibits high stability, smoothness and naturalness in the entire time series, achieving high-quality animation sequence reconstruction.

[0170] Reference Figure 12 This is a schematic diagram illustrating a three-dimensional face data processing workflow provided in an embodiment of this application. Figure 12 The diagram shown illustrates the overall system architecture of the proposed 3D face data processing method, which is a symmetric variational autoencoder (VAE). Figure 12 As shown, the architecture mainly consists of an encoder and a decoder. In the encoder, the input fused deformation (BS) coefficient sequence is first superimposed with positional encoding (PE), and then sequentially passed through a multilayer perceptron (MLP) and four cascaded Transformer Blocks for deep feature extraction. Each Transformer Block contains a multi-head attention (MHA) module, and rotational positional encoding (ROPE) is applied to the query (q) and key (k) vectors before attention calculation. The processed feature sequence is then passed through a fully connected layer to generate the mean and logvariance of the latent variable sequence, and the latent variables are sampled. In the decoder, this latent variable sequence is expanded in dimension by a fully connected layer, and then similarly input into four structurally symmetrical Transformer Blocks for feature reconstruction. Finally, it is mapped through another fully connected layer to obtain the reconstructed 3D face (BS) sequence data.

[0171] Reference Figure 13 This is a schematic diagram illustrating the process of facial image changes in three-dimensional facial data processing, provided in an embodiment of this application. Figure 13 The figure shown is a schematic diagram illustrating the generation principle of the 3D face sequence data processed in this application. This figure demonstrates how the BlendShape (BS) technique generates rich facial expressions through linear combination. As shown in each row of the figure, a final facial animation frame (e.g., "Facial Animation Frame 1") is obtained by weighted summing of multiple basic, independent BS deformations (e.g., "BS Name: JawOpen", "BS Name: JawRight", etc.) according to their respective corresponding "BS coefficients" (e.g., 0.5, 0.7, etc.). Figure 13 The paper demonstrates, by showing multiple rows of facial animation frames with different combinations of BS coefficients, that a continuous lip-sync animation sequence (i.e., the "three-dimensional face sequence data" in this invention) is essentially generated by a time-varying BS coefficient vector sequence.

[0172] In one example, the entire Variational Autoencoder (VAE) process begins with a BlendShape (BS) sequence as input. This sequence is represented digitally as a tensor, its shape in this example being (1, L, 51), representing a sequence containing L frames of animation, each frame defined by 51 coefficient values. For instance, the input sequence starts at (0.5, 0.7, 0.3, ...), and this vector generates the specific lip shape for the first frame by weighted combination of multiple basic facial deformities (such as JawOpen, JawRight, etc.). Subsequent frames in the sequence, such as (0.5, 0.6, 0.2, ...), define the lip shapes that change over time, collectively forming a complete dynamic expression.

[0173] Next, the input sequence is fed into the encoder for processing. The cascaded TransformerBlocks within the encoder perform a global analysis of the entire sequence, capturing dependencies that span long time intervals and deeply compressing the original 51-dimensional features. The encoder's final output is a latent variable sequence of the same length as the input sequence (L frames), but with significantly reduced dimensionality. For example, the original first frame input (0.5, 0.7, 0.3, ...) is encoded and mapped to a 16-dimensional vector in the latent space, such as (0.2314, 0.3074, ..., 0.7892). Similarly, other frames in the sequence are compressed accordingly, forming vectors such as (0.2275, 0.3122, ..., 0.7886), which together constitute the complete latent variable sequence. This sequence is a highly condensed, compact representation of the original animation that retains the core dynamic information.

[0174] Finally, the low-dimensional latent variable sequence is fed into the decoder for reconstruction. The decoder, utilizing its symmetrical structure with the encoder, reads and "decompresses" the information contained in the latent variable sequence, gradually reconstructing the high-dimensional animation details. The final output of the decoder is a sequence of Black-Scholes coefficients with the exact same format and dimensions as the original input. For example, based on the latent variables (0.2314, 0.3074, ..., 0.7892), the decoder will reconstruct the BS coefficient values ​​for the first frame as (0.494, 0.703, 0.311, ...). By decoding the entire latent variable sequence, a complete and restored animation sequence is generated. Although the output values ​​differ slightly from the original input, it visually highly reproduces the original animation, and because the model learns a smooth latent space, the reconstructed animation sequence is usually more stable and natural, effectively reducing disjointed perturbations.

[0175] This application also provides a three-dimensional face data processing device, which can implement the above-described three-dimensional face data processing method, as described above. Figure 14 The device 1400 includes: Encoder 1410 is used to acquire 3D face sequence data and generate positional encoding data corresponding to the 3D face sequence data; the positional encoding data and the 3D face sequence data are superimposed to obtain face position sequence data; the face position sequence data is input into the first concatenated feature extraction module for data feature processing to obtain feature state sequence data; low-dimensional compression processing is performed on the feature state sequence data to obtain the latent variable sequence corresponding to the 3D face sequence data, and the latent variable sequence is sent to the decoder for decoding processing; Decoder 1420 is used to receive the latent variable sequence generated by encoder; perform high-dimensional expansion processing on the latent variable sequence to obtain expanded sequence data; input the expanded sequence data into the second concatenated feature extraction module for data feature processing to obtain restored feature sequence data. The second concatenated feature extraction module is structurally symmetrical with the first concatenated feature extraction module in encoder; perform mapping processing based on restored feature sequence data to obtain restored three-dimensional face sequence data.

[0176] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, the specific implementation of the three-dimensional face data processing device is basically the same as the specific implementation of the three-dimensional face data processing method described above, and will not be repeated here.

[0177] This application also provides an electronic device, including: At least one memory; At least one processor; At least one program; The program is stored in memory, and the processor executes at least one program to implement the three-dimensional face data processing method described above in this application. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0178] Please see Figure 15 , Figure 15 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1502 can be implemented in the form of ROM (Read-Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 1502 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1502 and is called and executed by the processor 1501 using the three-dimensional face data processing method of the embodiments of this application. The input / output interface 1503 is used to implement information input and output; The communication interface 1504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1505 transmits information between various components of the device (e.g., processor 1501, memory 1502, input / output interface 1503, and communication interface 1504); The processor 1501, memory 1502, input / output interface 1503 and communication interface 1504 are connected to each other within the device via bus 1505.

[0179] This application embodiment also provides a storage medium, which is a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the aforementioned three-dimensional face data processing method. The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

Claims

1. A three-dimensional face data processing method, characterized by, The method is applied to an encoder, and the method comprises: obtaining three-dimensional face sequence data, and generating position encoding data corresponding to the three-dimensional face sequence data; superimposing the position encoding data and the three-dimensional face sequence data to obtain face position sequence data; inputting the face position sequence data into a first serial feature extraction module for data feature processing to obtain feature state sequence data; performing low-dimensional compression processing based on the feature state sequence data to obtain hidden variable sequence corresponding to the three-dimensional face sequence data, and sending the hidden variable sequence to a decoder for decoding processing.

2. The three-dimensional face data processing method of claim 1, wherein, The generation of the position encoding data corresponding to the three-dimensional face sequence data comprises: obtaining odd-dimensional index parameters and even-dimensional index parameters based on dimension index parameters; obtaining odd exponents based on the product of the odd-dimensional index parameters and a preset numerical value, dividing by the feature dimension of the three-dimensional face sequence data, and then performing exponential processing; obtaining even exponents based on the product of the even-dimensional index parameters and the preset numerical value, dividing by the feature dimension, and then performing exponential processing; obtaining odd position data based on the product of the odd exponents and position parameters, and then performing cosine processing, and obtaining even position data based on the product of the even exponents and the position parameters, and then performing sine processing; obtaining the position encoding data based on the odd position data and the even position data.

3. The three-dimensional face data processing method of claim 1, wherein, The first serial feature extraction module comprises a plurality of serial feature extraction modules, and the inputting of the face position sequence data into the first serial feature extraction module for data feature processing to obtain feature state sequence data comprises: sequentially performing data processing on input data in each feature extraction module based on the serial order of a plurality of feature extraction modules to obtain output feature sequence data; wherein the input data of the first feature extraction module is the face position sequence data, and the input data of other feature extraction modules is the output feature sequence data of the previous feature extraction module; obtaining the feature state sequence data based on the output feature sequence data of the last feature extraction module.

4. The three-dimensional face data processing method of claim 3, wherein, The feature extraction module comprises a first normalization layer, a multi-head attention layer, a second normalization layer, and a multi-head perception layer, and the data processing on input data in each feature extraction module to obtain output feature sequence data comprises: inputting the input data into the first normalization layer for first normalization processing to obtain first standard data; inputting the first standard data into the multi-head attention layer for attention processing to obtain attention feature data; inputting the attention feature data into the second normalization layer for second normalization processing to obtain second standard data; inputting the second standard data into the multi-head perception layer for perception processing to obtain the output feature sequence data.

5. The three-dimensional face data processing method of claim 4, wherein, The inputting of the first standard data into the multi-head attention layer for attention processing to obtain attention feature data comprises: In the multi-head attention layer, the first canonical data is projected into an attention parameter sequence, and the attention parameter sequence is evenly divided in the last dimension to obtain a plurality of groups of attention parameter subsequences; The key parameter vector and the value parameter vector in each of the attention parameter subsequences are respectively subjected to a rotating position encoding process to obtain an encoded attention parameter subsequence; Each of the encoded attention parameter subsequences is subjected to attention calculation to obtain an attention result subsequence; All the attention result subsequences are spliced to obtain the attention feature data.

6. The three-dimensional face data processing method of claim 5, wherein, The rotating position encoding process of the key parameter vector and the value parameter vector in each of the attention parameter subsequences respectively to obtain an encoded attention parameter subsequence comprises: The key parameter vector and the value parameter vector are respectively taken as a rotating processing parameter vector; Based on the feature dimension of the three-dimensional face sequence data, a frequency vector is calculated, and a rotation angle is obtained based on the product of the current position parameter corresponding to the rotating processing parameter and the frequency vector; Based on the position order of the vector elements in the rotating processing parameter vector, the odd and even position elements in the rotating processing parameter vector are divided to obtain an odd rotating vector and an even rotating vector; Based on the element-by-element multiplication of the even rotating vector and the cosine value of the rotation angle, an even cosine value is obtained, based on the element-by-element multiplication of the even rotating vector and the sine value of the rotation angle, an even sine value is obtained, based on the element-by-element multiplication of the odd rotating vector and the cosine value of the rotation angle, an odd cosine value is obtained, and based on the element-by-element multiplication of the odd rotating vector and the sine value of the rotation angle, an odd sine value is obtained; Based on the difference between the even cosine value and the odd sine value, an even rotating encoding vector is obtained, and based on the accumulation of the even sine value and the odd cosine value, an odd rotating encoding vector is obtained; Based on the merging of the even rotating encoding vector and the odd rotating encoding vector, a rotating position encoding vector is obtained; Based on the rotating position encoding vector, the attention parameter subsequence is updated to obtain the encoded attention parameter subsequence.

7. The three-dimensional face data processing method of claim 4, wherein, The perception processing of the second canonical data input into the multi-head perception layer to obtain the output feature sequence data comprises: In the multi-head perception layer, the second canonical data is expanded in number in the last dimension to obtain expanded canonical data; Using an activation function, a gating signal is generated based on the expanded canonical data; Based on the gating signal and the expanded canonical data, element-by-element multiplication is performed to obtain gated processing data; The gated processing data is compressed in number in the last dimension to obtain the output feature sequence data, and the output feature sequence data is consistent with the data dimension of the second canonical data.

8. A three-dimensional face data processing method, characterized by, The method is applied to a decoder, and the method comprises: receiving a latent variable sequence generated by an encoder; performing high-dimensional expansion processing on the latent variable sequence to obtain expanded sequence data; The extended sequence data is input into a second serial feature extraction module for data feature processing to obtain restored feature sequence data, and the second serial feature extraction module is structurally symmetrical with the first serial feature extraction module in the encoder; Mapping processing is performed based on the restored feature sequence data to obtain restored three-dimensional face sequence data.

9. An electronic device, comprising: The three-dimensional face data processing method comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the three-dimensional face data processing method in any one of claims 1 to 7 or the three-dimensional face data processing method in claim 8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the three-dimensional face data processing method in any one of claims 1 to 7 or the three-dimensional face data processing method in claim 8.

Images