A three-dimensional face reconstruction method, device and equipment based on facial electromyographic signals

Abstract

The application discloses a three-dimensional face reconstruction method, device and equipment based on facial electromyographic signals, comprising the following steps: preprocessing original signals including facial electromyographic signals and video frame data collected to obtain electromyographic signal sequences and video frame sequences; based on the video frame sequences, facial key points are extracted and identity information is removed to obtain standardized facial feature point sequences; a pre-network including a convolutional neural network module and a Transformer encoder module is self-supervised pre-trained using a facial electromyographic signal dataset to obtain a facial feature extraction model; the facial feature extraction model is used as a fixed-parameter feature extractor and is connected with a Transformer decoder module to train a cross-modal mapping model from the electromyographic signal sequences to the standardized facial feature point sequences; the facial electromyographic signals to be reconstructed are input into the cross-modal mapping model to obtain facial feature point sequences, and the facial feature point sequences are input into a preset statistical three-dimensional head model to generate corresponding three-dimensional face reconstruction results.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional face reconstruction technology, and in particular to a three-dimensional face reconstruction method, apparatus and equipment based on facial electromyography signals. Background Technology

[0002] Facial expressions, as a core carrier of emotion transmission and social interaction, have always been an important research direction in the fields of computer vision and graphics for accurate modeling. Facial expression analysis based on physiological signals, especially using facial electromyography (EMG) signals to directly reflect muscle activity, provides a more fundamental physiological basis for expression modeling. However, existing EMG-based expression reconstruction methods still face significant challenges: On the one hand, there is a highly complex nonlinear relationship between EMG signals and the final 3D facial morphology, making it extremely challenging to establish an accurate and robust mapping model; traditional methods often rely on pre-defined feature engineering or limited data fitting, making it difficult to capture the deep and dynamic correlation. On the other hand, acquiring large-scale, high-quality, and accurately synchronized "EMG signal-3D expression" labeled data is extremely costly, and EMG signals themselves are easily affected by multiple factors such as individual physiological differences, electrode positions, and environmental noise. This results in poor generalization ability of models trained under limited data conditions, making them prone to overfitting to training data and unable to adapt to new users or environments, thus hindering the practical application and popularization of this technology. Summary of the Invention

[0003] In view of this, the purpose of this invention is to propose a method, apparatus and device for three-dimensional face reconstruction based on facial electromyography signals, which aims to solve the problems of difficulty in modeling mapping relationships and the tendency of models to overfit and have weak generalization ability when reconstructing three-dimensional facial expressions based on facial electromyography signals in the prior art.

[0004] To achieve the above objectives, the present invention provides a three-dimensional face reconstruction method based on facial electromyography signals, the method comprising:

[0005] The raw signals, including facial electromyography (EMG) signals and video frame data, are preprocessed to obtain EMG signal sequences and video frame sequences. Based on the video frame sequence, facial key points are extracted and identity information is removed to obtain a standardized facial feature point sequence; A facial feature extraction model is obtained by using a facial electromyography signal dataset to perform self-supervised pre-training on a front-end network including a convolutional neural network module and a Transformer encoder module. The facial feature extraction model is used as a feature extractor with fixed parameters and connected to the Transformer decoder module to train a cross-modal mapping model from electromyographic signal sequences to standardized facial feature point sequences. The electromyographic signal of the face to be reconstructed is input into the cross-modal mapping model to obtain a sequence of facial feature points. The sequence of facial feature points is then input into a preset statistical three-dimensional head model to generate the corresponding three-dimensional face reconstruction result.

[0006] Preferably, the preprocessing of the acquired raw signals, including facial electromyography (EMG) signals and video frame data, to obtain EMG signal sequences and video frame sequences includes: The facial electromyography signals and the video frame data are time-synchronized. A face alignment network is used to perform face detection and cropping on the synchronized video frame data, and a notch filter is used to filter and denoise the synchronized facial electromyography (EMG) signal to obtain the corresponding video frame sequence and the EMG signal sequence.

[0007] Preferably, the step of extracting facial key points and removing identity information based on the video frame sequence to obtain a standardized facial feature point sequence includes: A face alignment network is used to extract facial key points from each video frame in the video frame sequence to obtain a facial key point sequence corresponding to the video frame sequence. The facial key point sequence is aligned with the average face shape using facial deformation technology to obtain the standardized facial feature point sequence with identity information removed.

[0008] Preferably, the step of aligning the facial key point sequence with the average face shape based on facial deformation technology to obtain the standardized facial feature point sequence with identity information removed includes: Calculate the affine transformation matrix between the key points of the first frame of the facial key point sequence and the predefined average face shape; The affine transformation matrix is ​​used to transform each frame of the facial key point sequence to the coordinate system of the average face shape, and the resulting key point coordinate sequence is used as the standardized facial feature point sequence.

[0009] Preferably, the step of using a facial electromyography (EMG) signal dataset to perform self-supervised pre-training on a pre-network including a convolutional neural network module and a Transformer encoder module to obtain a facial feature extraction model includes: The pre-network is trained through two phases: a mask reconstruction training phase and a noise-invariant contrastive learning training phase, to obtain the facial feature extraction model. The mask reconstruction training includes inputting signal values ​​of an electromyographic signal sequence through a random mask, reconstructing the signal through a pre-network and based on reconstruction loss; The noise-invariant contrastive learning training includes adding noise to a portion of the electromyographic signal sequence to generate a perturbed version, and extracting noise-robust features from the pre-network based on contrastive learning loss constraints.

[0010] Preferably, the reconstruction loss is the mean square error loss L at the mask location. Recon ,in, In the formula, M represents the set of all mask positions in the batch. and Let represent the original electromyographic signal value at the i-th position and and the predicted value reconstructed by the preceding network, respectively. The contrastive learning loss is the InfoNCE loss. ,in, In the formula, This represents the feature representation extracted from the original electromyography signal sample by the aforementioned preprocessor network. This represents the feature representation of the perturbed sample after adding noise to the original electromyography signal sample. Represents cosine similarity. The value represents the temperature hyperparameter, N represents the batch size, and the denominator is the sum of all 2N samples in the batch.

[0011] Preferably, the step of using the facial feature extraction model as a feature extractor with fixed parameters and connecting it to the Transformer decoder module to train a cross-modal mapping model from electromyographic signal sequences to standardized facial feature point sequences includes: Freeze the parameters of the facial feature extraction model, and input the electromyographic feature representation sequence extracted by the facial feature extraction model into the randomly initialized Transformer decoder module; The electromyographic feature representation sequence is aligned with the facial key point sequence to be predicted using the cross-attention mechanism in the Transformer decoder module. Using the standardized facial feature point sequence as a supervision signal, the mean squared error is used to optimize the difference between the predicted key points and the true key points, where the mean squared error is expressed as... In the formula, T represents the total number of time steps. and Let represent the actual keypoint coordinates and the predicted keypoint coordinates at step t, respectively; The cross-modal mapping model is obtained by training the Transformer decoder module and the output layer by minimizing the mean squared error loss function.

[0012] Preferably, the statistical 3D head model is a FLAME model; the step of inputting the facial feature point sequence into the preset statistical 3D head model to generate the corresponding 3D face reconstruction result includes: Using the facial feature point sequence as a two-dimensional constraint target, the expression parameters of the FLAME model are optimized based on the mean square error between the two-dimensional projection key points generated by the FLAME model and the facial feature point sequence by minimizing the mean square error. The optimized facial expression parameters drive the FLAME model to generate corresponding 3D facial expression animations as the 3D face reconstruction result.

[0013] To achieve the above objectives, the present invention also provides a three-dimensional face reconstruction device based on facial electromyography signals, the device comprising: The signal acquisition unit is used to preprocess the acquired raw signals, including facial electromyography signals and video frame data, to obtain electromyography signal sequences and video frame sequences. The video frame processing unit is used to extract facial key points and remove identity information based on the video frame sequence to obtain a standardized facial feature point sequence. The first training unit is used to perform self-supervised pre-training on the front-end network, which includes a convolutional neural network module and a Transformer encoder module, using a facial electromyography signal dataset to obtain a facial feature extraction model. The second training unit is used to use the facial feature extraction model as a feature extractor with fixed parameters and connect it to the Transformer decoder module to train a cross-modal mapping model from electromyographic signal sequences to standardized facial feature point sequences. The 3D face reconstruction unit is used to input the electromyographic signals of the face to be reconstructed into the cross-modal mapping model to obtain a sequence of facial feature points, and input the sequence of facial feature points into a preset statistical three-dimensional head model to generate the corresponding three-dimensional face reconstruction result.

[0014] To achieve the above objectives, the present invention also proposes a three-dimensional face reconstruction device based on facial electromyography signals, including a processor, a memory, and a computer program stored in the memory. The computer program is executed by the processor to implement the steps of a three-dimensional face reconstruction method based on facial electromyography signals as described in the above embodiments.

[0015] To achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a computer program that is executed by a processor to implement the steps of a three-dimensional face reconstruction method based on facial electromyography signals as described in the above embodiments.

[0016] Beneficial effects: The above solution constructs an end-to-end solution, achieving a robust mapping from raw multimodal data to high-fidelity 3D face reconstruction. By introducing a self-supervised pre-training mechanism, it significantly reduces the dependence on large-scale labeled data, improving the model's applicability in data-scarce scenarios. Furthermore, the design of the cross-modal mapping model effectively captures the complex nonlinear relationship between electromyographic signals and facial movements, enhancing the accuracy of expression reconstruction. The overall process achieves a coherent and automatic conversion from low-level bioelectrical signals to high-level visual expressions, providing a feasible path for high-fidelity facial expression capture and reconstruction.

[0017] Precise time synchronization ensured the temporal alignment between electromyographic signals and video frames, laying an accurate data foundation for subsequent cross-modal learning. Meanwhile, face alignment networks were used for face localization and cropping, and notch filters were used for signal filtering. This effectively removed non-expression-related components such as video background interference, power frequency noise, and baseline drift in the signal, improving the purity and consistency of the input data. This, in turn, helped the model learn more essential expression-related features.

[0018] Geometric feature points for each frame are precisely extracted using a face alignment network to form a temporal keypoint sequence. Then, a face deformation technique based on affine transformation is used to align all sequences to a unified average shape space. This process effectively eliminates geometric feature variations caused by differences in facial identity (such as skeletal structure and inherent shape), ensuring that the final standardized facial feature point sequence retains only dynamic information related to facial expression movements. This eliminates the interference of identity bias on subsequent expression mapping model training, improving the model's versatility and robustness.

[0019] By designing a training mechanism that combines mask reconstruction with noise-invariant contrastive learning, the model can learn noise-robust and semantically rich feature representations from unlabeled data. Mask reconstruction forces the network to deeply understand the local contextual structure of EMG signals and learn their internal patterns, while noise-contrast learning enhances the network's robustness to non-semantic noise in the signal, focusing on extracting expression-related semantic features. This training method not only reduces the dependence on data labeling, utilizing only the characteristics of EMG signals themselves to alleviate the data scarcity problem, but also significantly improves the stability of the feature extractor in the face of signal fluctuations and interference, and endows the feature extraction network with powerful generalization capabilities, laying a solid feature foundation for cross-modal mapping.

[0020] By freezing the pre-trained feature extractor and utilizing the cross-attention mechanism of the Transformer decoder, efficient alignment of EMG feature sequences with facial keypoint sequences was achieved. Freezing the pre-trained parameters prevents overfitting on small-scale supervised data while preserving strong feature representation capabilities. The Transformer decoder, through the cross-attention mechanism, effectively models the complex temporal alignment and mapping relationship between EMG feature sequences and target keypoint sequences. This design ensures that the model can accurately learn a stable transformation from the EMG signal space to the facial geometric expression space, thereby achieving high-precision keypoint sequence prediction.

[0021] By using the predicted 2D facial feature point sequence as the optimization objective, and minimizing the error between this sequence and the projection of the 3D vertices of the FLAME model onto the 2D plane, the expression parameters of the FLAME model are optimized in reverse. This approach decomposes the difficult-to-solve "signal-to-3D" problem into "signal-to-2D" and "2D-to-3D" methods, fully utilizing the prior knowledge of existing high-precision parameterized 3D facial models to achieve a precise and efficient conversion from 2D information to 3D geometry. Ultimately, by driving the expression parameters of the FLAME model, high-quality 3D facial animations with rich texture and geometric details and natural and coherent facial expressions can be generated. This achieves the final reconstruction from electromyographic signals to realistic 3D expressions, ensuring that the generated 3D face has good visual plausibility and geometric accuracy. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a three-dimensional face reconstruction method based on facial electromyography signals, provided as an embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram illustrating the specific process of model processing provided in an embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram of the structure of a reconstructed three-dimensional face model example provided in an embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram of a three-dimensional face reconstruction device based on facial electromyography signals, provided as an embodiment of the present invention.

[0027] The realization of the invention's objective, its functional characteristics, and advantages will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] The present invention will be described in detail below with reference to the embodiments.

[0030] Reference Figure 1 The diagram shows a flowchart of a three-dimensional face reconstruction method based on facial electromyography signals according to an embodiment of the present invention. In this embodiment, the method includes: S11, preprocess the raw signals, including facial electromyography signals and video frame data, to obtain electromyography signal sequences and video frame sequences.

[0031] Furthermore, in step S11, the preprocessing of the acquired raw signals, including facial electromyography (EMG) signals and video frame data, to obtain EMG signal sequences and video frame sequences includes: S11-1, Time synchronization processing is performed on the facial electromyography signal and the video frame data; S11-2, a face alignment network is used to perform face detection and cropping on the synchronized video frame data, and a notch filter is used to filter and denoise the synchronized facial electromyography (EMG) signal to obtain the corresponding video frame sequence and the EMG signal sequence.

[0032] In this embodiment, the preprocessing of the raw signal includes: First, to ensure precise time alignment between the facial electromyography (EMG) signal and the video stream, a pre-experiment phase is set before the formal experiment begins. At the start of the experiment, the EMG signal recording software automatically marks the timestamp and synchronously activates the camera to record the video stream, thereby establishing a synchronization relationship between the two modalities. Subsequently, the video stream is processed: the video sampling rate is set to 25 frames per second (fps) to ensure clear and effective video output, and a deep learning-based Face Alignment Network (FAN) classifier is used for accurate face detection to extract segmented video frames. After detecting a face, to eliminate the influence of irrelevant information such as background, the face region is centered and uniformly cropped into 256-pixel × 256-pixel square images per frame, forming a standardized video frame sequence. Simultaneously, the synchronously acquired facial EMG signals are processed: this embodiment uses two channels to acquire signals from the target muscles (such as the bilateral zygomaticus major muscles), that is, electrodes for acquiring EMG signals are attached to the bilateral zygomaticus major muscles of the face, and reference electrodes are attached to the forehead. To obtain clean facial expression-related signals, a notch filter with a center frequency of 50Hz was first used to filter the raw electromyography (EMG) signals to eliminate power frequency interference. Next, a high-frequency filter was used to remove baseline drift components introduced by equipment and other factors. After the above synchronization, cropping, and filtering processes, a time-aligned and content-clean EMG signal sequence and video frame sequence were finally obtained, providing high-quality input data for subsequent model training and inference.

[0033] S12, based on the video frame sequence, extract facial key points and remove identity information to obtain a standardized facial feature point sequence.

[0034] Furthermore, in step S12, the step of extracting facial key points and removing identity information based on the video frame sequence to obtain a standardized facial feature point sequence includes: S12-1, Use a face alignment network to extract the facial key points of each video frame in the video frame sequence to obtain a facial key point sequence corresponding to the video frame sequence; S12-2, Based on face deformation technology, the facial key point sequence is aligned with the average face shape to obtain the standardized facial feature point sequence with identity information removed.

[0035] Further, in step S12-2, aligning the facial key point sequence with the average face shape based on facial deformation technology to obtain the standardized facial feature point sequence with identity information removed includes: S12-2-1, Calculate the affine transformation matrix between the first frame key points of the facial key point sequence and the predefined average facial shape; S12-2-2, the affine transformation matrix is ​​used to transform each frame of the facial key point sequence to the coordinate system of the average face shape, and the resulting key point coordinate sequence is used as the standardized facial feature point sequence.

[0036] In this embodiment, to extract facial geometric features with high accuracy from the preprocessed video frame sequence, a Face Alignment Network (FAN) is used. This FAN uses a stacked structure of four Hourglass networks as its backbone and introduces hierarchical, parallel, and multi-scale blocks to replace the traditional Hourglass backbone blocks, resulting in superior performance in human pose estimation and facial landmark alignment tasks. The FAN processes each frame in the input video frame sequence and outputs the coordinates of 51 predefined two-dimensional facial keypoints in each frame, thus obtaining a sequence of facial keypoints that corresponds to the video frame sequence frame by frame in time.

[0037] To eliminate the interference of individual differences (such as inherent features like face width and mouth shape) on facial expression modeling, identity information needs to be extracted from the aforementioned facial landmark sequences. This embodiment employs facial deformation technology to align the facial shapes of all individuals to a unified average shape space. Specifically, the average shape of the first frame of all facial landmark sequences in the training set is first calculated as a standardized reference benchmark; this average shape L... avg Through formula The calculations were performed to obtain the result, where, (m = 2 × 51 = 102, corresponding to the horizontal and vertical dimensions of 51 two-dimensional key points), representing the average facial shape of all samples. This is the first frame of the k-th facial feature point sequence in the training set (i.e., the basic facial shape of the sample, containing identity information), where n is the total number of facial feature point sequences in the training set. Next, for any facial keypoint sequence, calculate its first frame I. k With average shape L avg The affine transformation matrix A is used to establish the mapping relationship from the individual basic shape to the average shape. This affine transformation matrix A is obtained through the formula... To solve the problem, A is an m×n matrix, where m = 2×51 = 102 (where 51 corresponds to the number of facial key points and 2 represents the two-dimensional coordinates of each key point), n represents the number of video frames, and T represents the transpose operation.

[0038] Then, the obtained affine transformation matrix A is used to process the entire sequence, specifically including: calculating the relationship between each frame in the facial keypoint sequence and its own first frame I. k The difference is calculated by combining the affine transformation with a scaling factor λ, where the difference... scaling factor Ultimately, by using the average shape L... avg Combined with the scaled difference information, a standardized facial feature point sequence, result, is obtained, completely stripped of identity information. The calculation formula is as follows: The resulting standardized facial feature point sequence exhibits shape changes solely due to facial expression movements, providing a clean and uniform supervisory signal for the subsequent training of cross-modal mapping models.

[0039] S13. A facial feature extraction model is obtained by self-supervised pre-training of a pre-network including a convolutional neural network module and a Transformer encoder module using a facial electromyography signal dataset.

[0040] Furthermore, in step S13, the pre-training of the facial feature extraction model using the facial electromyography signal dataset, which includes a convolutional neural network module and a Transformer encoder module, to obtain the facial feature extraction model includes: The pre-network is trained through two phases: a mask reconstruction training phase and a noise-invariant contrastive learning training phase, to obtain the facial feature extraction model. The mask reconstruction training includes inputting signal values ​​of an electromyographic signal sequence through a random mask, reconstructing the signal through a pre-network and based on reconstruction loss; The noise-invariant contrastive learning training includes adding noise to a portion of the electromyographic signal sequence to generate a perturbed version, and extracting noise-robust features from the pre-network based on contrastive learning loss constraints.

[0041] Furthermore, the reconstruction loss is the mean square error loss L at the mask location. Recon ,in, In the formula, M represents the set of all mask positions in the batch. and Let represent the original electromyographic signal value at the i-th position and and the predicted value reconstructed by the preceding network, respectively. The contrastive learning loss is the InfoNCE loss. ,in, In the formula, This represents the feature representation extracted from the original electromyography signal sample by the aforementioned preprocessor network. This represents the feature representation of the perturbed sample after adding noise to the original electromyography signal sample. Represents cosine similarity. The value represents the temperature hyperparameter, N represents the batch size, and the denominator is the sum of all 2N samples in the batch (including N original samples and N noisy samples).

[0042] In this embodiment, firstly, a pre-training network is constructed, which includes a one-dimensional convolutional neural network (1D-CNN) module and a Transformer encoder module, and a shared-weight feature extractor F. θ (·). Specifically, the one-dimensional convolutional neural network module contains four convolutional layers, each with a kernel size of 4, a stride of 2, and channel numbers ranging from shallow to deep: 64, 128, 256, and 512, respectively, and employs the ReLU activation function. The Transformer encoder module contains two encoder layers, employing an 8-head self-attention mechanism. Its feedforward network has a hidden layer dimension of 1024, using sine and cosine positional encoding and the GeLU activation function. The input dimension of the feedforward network is 960×2, representing a two-channel facial electromyography signal sequence with a length of 1 second and a sampling rate of 960Hz; its output is a high-level feature representation.

[0043] Reference Figure 2 As shown in (b), the pre-training employs a two-stage self-supervised strategy, requiring no manually labeled data, aiming to learn general and robust expression-related feature representations from electromyography (EMG) signals. This embodiment uses a large-scale open-source facial EMG dataset (e.g., Facial EMG and Subjective Liking Data from 70 New Zealand Participants) for pre-training. This dataset is a dataset about facial EMG and subjective preferences, containing facial muscle activity records and subjective preference ratings from 70 New Zealand participants while viewing food images and tasting chocolate samples. Before using this open-source dataset, its EMG signal data was downsampled to 960Hz to maintain a consistent sampling rate, and the dataset was normalized. The first stage is mask signal reconstruction training: given a batch of raw facial electromyography (EMG) signal sequences, 20% of the signal values ​​are randomly selected and replaced with the same learnable mask label (e.g., [MASK]) to generate a mask signal; the mask signal is then input into the pre-processor network F. θ (·), to obtain its feature representation, and then predict the original signal value at the masked location using a linear projection head, outputting the reconstructed signal. The optimization objective at this stage is to minimize the mean square error between the original signal and the reconstructed signal at the masked location, forcing the network to learn the local structure and contextual information of the signal. The reconstruction loss L Recon The formula is: In the formula, M represents the set of all masked positions in the current batch. and and represent the original electromyographic signal value at the i-th position and the predicted value reconstructed by the pre-network, respectively.

[0044] The second stage is noise-invariant contrastive learning training: This stage aims to enhance the model's robustness to non-semantic noise. For EMG signal samples in a batch, 20% of the sample segments are randomly selected, and Gaussian noise is added to them to create corresponding noisy perturbation versions. The remaining 80% of the samples remain unchanged. The original samples and their corresponding noisy versions form positive sample pairs, and all other samples (including other original samples and other noisy samples) form negative sample pairs. The mixed batch (containing N original samples and N noisy samples, a total of 2N) is input into the pre-processor network F. θ (·) is used to extract feature representations. Subsequently, InfoNCE loss is used as the contrastive learning loss L. Contrast The optimization objective is to maximize the similarity between features of positive sample pairs while minimizing the similarity with negative sample pairs. The formula for the loss function is: In the formula, This represents the feature representation extracted from the original electromyography (EMG) signal sample by the preprocessor network. This represents the feature representation of the noisy, perturbed version of the original electromyography signal sample; the two together constitute a positive sample pair. This is the cosine similarity calculation function. The temperature hyperparameter is used to adjust the attention given to difficult negative samples. The denominator represents the summation of features over all 2N samples in the batch (i.e., N original samples + N noisy samples), excluding the anchor point itself. The total loss at this stage is the sum of the contrastive losses of all valid positive sample pairs in the batch.

[0045] The entire pre-training process uses the AdamW optimizer with a learning rate of 0.0001 and a batch size of 32. Through the above two-stage training, the pre-trained network F... θ (·) is endowed with powerful feature representation capabilities, enabling it to extract noise-resistant and expression-semantic-rich general features from raw facial electromyography signals, thus becoming a high-quality facial feature extraction model that can be used for downstream tasks.

[0046] S14, the facial feature extraction model is used as a feature extractor with fixed parameters and connected to the Transformer decoder module to train a cross-modal mapping model from electromyographic signal sequence to standardized facial feature point sequence.

[0047] Furthermore, in step S14, the step of using the facial feature extraction model as a feature extractor with fixed parameters and connecting it to the Transformer decoder module to train a cross-modal mapping model from electromyographic signal sequences to standardized facial feature point sequences includes: S14-1, Freeze the parameters of the facial feature extraction model, and input the electromyographic feature representation sequence extracted by the facial feature extraction model into the randomly initialized Transformer decoder module; S14-2, The electromyographic feature representation sequence is aligned with the facial key point sequence to be predicted through the cross-attention mechanism in the Transformer decoder module; S14-3, using the standardized facial feature point sequence as a supervision signal, the mean squared error is used to optimize the difference between the predicted key points and the true key points, wherein the mean squared error is expressed as... In the formula, T represents the total number of time steps. and Let represent the actual keypoint coordinates and the predicted keypoint coordinates at step t, respectively; S14-4, The Transformer decoder module and the output layer are trained by minimizing the mean squared error loss function to obtain the cross-modal mapping model.

[0048] Reference Figure 2 As shown in (a). In this embodiment, this stage is the supervised training stage, and the goal is to learn the accurate mapping from the preprocessed facial electromyography (EMG) signal sequence to the 3D facial keypoint sequence to drive subsequent 3D expression generation. First, the facial feature extraction model obtained through self-supervised pre-training is loaded. In order to make full use of its learned powerful feature representation capabilities and prevent overfitting on limited supervised data, all parameters of the model are frozen, so that it is used only as a fixed feature extractor in subsequent training. The EMG signal sequence in the supervised training dataset is input into the model to obtain the corresponding EMG feature representation sequence. Then, a cross-modal mapping network is constructed. This network uses the frozen facial feature extraction model as the front end and connects to a randomly initialized Transformer decoder module at the back end. Specifically, the Transformer decoder module in this embodiment contains a 2-layer decoder, and the EMG feature representation sequence extracted from the front end is input into this decoder module.

[0049] Then, by utilizing the internal cross-attention mechanism of the Transformer decoder module, the most relevant parts of the input electromyography feature sequence are dynamically focused on and fused when decoding (predicting) the facial key points at each time step, thereby effectively aligning the electromyography feature representation sequence with the facial key point sequence to be predicted in terms of both time and semantics.

[0050] Subsequently, the standardized facial landmark sequence obtained above is used as the supervision signal (i.e., the ground truth). The features output by the decoder are passed through a fully connected layer and mapped to the predicted facial landmark coordinates. The final output dimension is L×102, corresponding to the (x, y) coordinates of 51 landmarks in each frame of a sequence of length L. The model predicts the facial landmark sequence in an autoregressive manner. The training objective is to minimize the difference between the predicted landmarks and the ground truth landmarks. In this embodiment, mean squared error (MSE) is used as the loss function to measure this difference, and its formula is expressed as: In the formula, L sup This represents the loss during the supervised training phase, and T represents the total number of time steps (total frames) of the facial landmark sequence. and and represent the actual standardized facial feature point coordinates and the model-predicted facial key point coordinates at the t-th time step, respectively.

[0051] Finally, by minimizing the above mean square error loss function L sup The Transformer decoder module and its subsequent fully connected output layer are trained end-to-end. This supervised training phase uses the AdamW optimizer with a learning rate of 0.001 and a batch size of 32. Upon completion of training, a cross-modal mapping model is obtained that can accurately map facial electromyography signal sequences to standardized facial feature point sequences.

[0052] S15, input the electromyographic signal of the face to be reconstructed into the cross-modal mapping model to obtain the facial feature point sequence, input the facial feature point sequence into the preset statistical three-dimensional head model to generate the corresponding three-dimensional face reconstruction result.

[0053] Furthermore, in step S15, the statistical 3D head model is a FLAME model; the step of inputting the facial feature point sequence into the preset statistical 3D head model to generate the corresponding 3D face reconstruction result includes: S15-1, using the facial feature point sequence as a two-dimensional constraint target, by minimizing the mean square error between the two-dimensional projection key points generated by the FLAME model and the facial feature point sequence, the expression parameters of the FLAME model are optimized based on the mean square error. S15-2, Based on the optimized expression parameters, drive the FLAME model to generate corresponding 3D facial expression animations as the 3D facial reconstruction results.

[0054] Reference Figure 2As shown in (c). In this embodiment, the inference and application stage of the modal mapping model is implemented, aiming to generate corresponding, expressive 3D facial animation driven by the facial electromyography (EMG) signals of the user to be reconstructed. First, the preprocessed EMG signal sequence of the user to be reconstructed is obtained and input into the trained cross-modal mapping model. The model processes the input signal in the aforementioned manner and outputs the corresponding facial feature point sequence. This sequence is a standardized two-dimensional keypoint coordinate sequence after removing identity information, describing the facial expression movements reflected by the EMG signals. Subsequently, the predicted two-dimensional facial feature point sequence is used as the driving source and input into a preset statistical 3D head model to complete the 3D reconstruction. In this embodiment, the FLAME model is used as the statistical 3D head model. The FLAME model is a parameterized 3D face and head general model, and its output mesh contains 5023 vertices and 9976 triangular faces. The model controls the generated results through multiple sets of parameters, including: identity shape parameters (describing the inherent static shape features of the face, such as face shape), expression parameters (describing the dynamic facial deformation caused by muscle activity), and pose parameters (controlling the global rotation of the head and the rotation of the neck, jaw, and eyeballs). In specific application scenarios, reconstruction is performed for a specific user; therefore, the identity shape parameters of the FLAME model can be determined and kept fixed based on the target user's 3D scan data or a preset identity. The pose parameters can usually be set to a basic or neutral state, unless head pose changes are required. The core of the 3D reconstruction process lies in using the aforementioned predicted 2D facial feature point sequence as a precise 2D constraint target, and solving for the expression parameter sequence that best explains the 2D motion trajectory through an optimization algorithm. Specifically, the reconstruction process includes parameter optimization and animation generation, namely: (1) Parameter Optimization: Using the facial feature point sequence predicted by the model as the precise two-dimensional constraint target, the three-dimensional face model generated by the FLAME model under the current parameters (mainly initializing expression parameters, fixing or giving identity and basic pose parameters) is projected onto the two-dimensional image plane through a preset camera projection model (such as weak perspective projection) to obtain its two-dimensional projection key points; through an iterative optimization algorithm, the expression parameters of the FLAME model are adjusted to minimize the difference between the two-dimensional projection key points obtained by projection and the input facial feature point sequence. In this embodiment, this difference is measured by mean squared error (MSE), that is, minimizing the mean squared error of the two in the coordinates. At the same time, in order to prevent overfitting or obtaining unnatural expressions, a regularization penalty term is usually applied to the expression parameters during the optimization process. This optimization process transforms the abstract two-dimensional key point sequence into a specific and continuous expression parameter change sequence in the FLAME model space.

[0055] (2) Animation Generation: Based on the expression parameter sequence corresponding to each frame obtained through the above optimization, and combined with the set identity and posture parameters, the FLAME model is driven frame by frame. The FLAME model uses linear blending skinning technology to smooth vertex motion, and finally outputs a three-dimensional mesh sequence in which the vertex position changes over time, i.e., a three-dimensional facial expression animation. This animation accurately reflects the facial muscle activity pattern corresponding to the input electromyographic signal, thus realizing the reconstruction result of high-fidelity, customizable three-dimensional facial dynamic expression from physiological signals.

[0056] To verify the performance of the proposed method on facial reconstruction tasks, it was quantitatively compared with a series of representative benchmark methods. The methods compared included advanced vision-based approaches (HRNet, PFLD, SAN) as well as innovative methods based on other biological signals in recent years (Bioface-3D, EARFace, mm3DFace). Normalized mean error (NME, lower values ​​are better) was used as the key metric for evaluation, and the comparison results are shown in Table 1.

[0057] Table 1. Comparison of Facial Keypoint Prediction Accuracy (NME) of Different Methods

[0058] Since this invention focuses on mapping electromyographic signals to facial geometry, a video-based preprocessing workflow (including inter-frame tracking, denoising, and smoothing) is employed to automatically generate high-precision ground truth facial landmarks to obtain more reliable supervision signals, without relying on manual annotation. Therefore, this comparison is mainly based on quantifiable NME (Not Measured Entities) metrics. As shown in Table 1, in comparison with vision-based models (HRNet, etc.) and other biosignal-based methods (Bioface-3D, etc.), the method of this invention (EMG2Face) achieved the lowest NME value (2.78%), indicating that it has higher prediction accuracy in the task of mapping facial electromyographic signals to facial landmark sequences. The benchmark methods used in the comparison are briefly described below: HRNet improves keypoint detection accuracy by maintaining high-resolution representations; PFLD is a lightweight and practical face keypoint detector; SAN enhances model robustness by aggregating style information; Bioface-3D uses ear biosensors for face capture; EARFace uses in-ear acoustic sensors to measure ear canal deformation to reconstruct facial motion; and mm3DFace uses millimeter-wave radar signals to achieve non-contact 3D face reconstruction. In summary, the experimental results show that the proposed 3D face reconstruction method based on facial electromyography signals outperforms the comparative schemes in key technical indicators, verifying its effectiveness and advancement.

[0059] Reference Figure 4The diagram shown is a structural schematic of a three-dimensional face reconstruction device based on facial electromyography signals according to an embodiment of the present invention.

[0060] In this embodiment, the device 20 includes: The signal acquisition unit 21 is used to preprocess the acquired raw signals, including facial electromyography signals and video frame data, to obtain electromyography signal sequences and video frame sequences. Video frame processing unit 22 is used to extract facial key points and remove identity information based on the video frame sequence to obtain a standardized facial feature point sequence; The first training unit 23 is used to perform self-supervised pre-training on a front-end network including a convolutional neural network module and a Transformer encoder module using a facial electromyography signal dataset to obtain a facial feature extraction model. The second training unit 24 is used to use the facial feature extraction model as a feature extractor with fixed parameters and connect it to the Transformer decoder module to train a cross-modal mapping model from electromyographic signal sequence to standardized facial feature point sequence. The 3D face reconstruction unit 25 is used to input the electromyographic signals of the face to be reconstructed into the cross-modal mapping model to obtain a sequence of facial feature points, and input the sequence of facial feature points into a preset statistical three-dimensional head model to generate the corresponding three-dimensional face reconstruction result.

[0061] Each unit module of the device 20 can execute the corresponding steps in the above method embodiment, so the details of each unit module will not be elaborated here. Please refer to the description of the corresponding steps above for details.

[0062] This invention also provides a three-dimensional face reconstruction device based on facial electromyography (EMG) signals. The device includes the three-dimensional face reconstruction apparatus based on facial EMG signals described above. The three-dimensional face reconstruction apparatus based on facial EMG signals can employ… Figure 4 The structure of the embodiment, correspondingly, can be executed Figure 1 The technical solutions of the method embodiments shown are similar in implementation principle and technical effect. For details, please refer to the relevant records in the above embodiments, which will not be repeated here.

[0063] The device includes: a mobile phone, digital camera, or tablet computer, or other device with a camera function; or a device with an image processing function; or a device with an image display function. The device may include components such as a memory, processor, input unit, display unit, and power supply.

[0064] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory can also include a memory controller to provide access to the memory for the processor and input units.

[0065] The input unit can be used to receive input numerical, character, or image information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, in addition to a camera, the input unit of this embodiment may also include a touch-sensitive surface (e.g., a touch screen) and other input devices.

[0066] The display unit can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. The display unit may include a display panel, optionally configured as an LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar display panel. Furthermore, a touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it transmits the information to the processor to determine the type of touch event. Subsequently, the processor provides corresponding visual output on the display panel based on the type of touch event.

[0067] This invention also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement... Figure 1 The method for three-dimensional face reconstruction based on facial electromyography signals is shown. The computer-readable storage medium can be a read-only memory, a hard disk, or an optical disk, etc.

[0068] This invention also provides a computer program product, including a computer program / instructions, which are loaded and executed by a processor to implement... Figure 1 This paper presents a three-dimensional face reconstruction method based on facial electromyography signals.

[0069] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the device embodiments, equipment embodiments, and storage medium embodiments, since they are basically similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions in the method embodiments.

[0070] Furthermore, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0071] The foregoing description illustrates and describes preferred embodiments of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept by means of the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method for three-dimensional face reconstruction based on facial electromyography signals, characterized in that, The method includes: The raw signals, including facial electromyography (EMG) signals and video frame data, are preprocessed to obtain EMG signal sequences and video frame sequences. Based on the video frame sequence, facial key points are extracted and identity information is removed to obtain a standardized facial feature point sequence; A facial feature extraction model is obtained by using a facial electromyography signal dataset to perform self-supervised pre-training on a front-end network including a convolutional neural network module and a Transformer encoder module. The facial feature extraction model is used as a feature extractor with fixed parameters and connected to the Transformer decoder module to train a cross-modal mapping model from electromyographic signal sequences to standardized facial feature point sequences. The electromyographic signal of the face to be reconstructed is input into the cross-modal mapping model to obtain a sequence of facial feature points. The sequence of facial feature points is then input into a preset statistical three-dimensional head model to generate the corresponding three-dimensional face reconstruction result.

2. The three-dimensional face reconstruction method based on facial electromyography signals according to claim 1, characterized in that, The preprocessing of the acquired raw signals, including facial electromyography (EMG) signals and video frame data, to obtain EMG signal sequences and video frame sequences includes: The facial electromyography signals and the video frame data are time-synchronized. A face alignment network is used to perform face detection and cropping on the synchronized video frame data, and a notch filter is used to filter and denoise the synchronized facial electromyography (EMG) signal to obtain the corresponding video frame sequence and the EMG signal sequence.

3. The three-dimensional face reconstruction method based on facial electromyography signals according to claim 1, characterized in that, The step of extracting facial key points and removing identity information based on the video frame sequence to obtain a standardized facial feature point sequence includes: A face alignment network is used to extract facial key points from each video frame in the video frame sequence to obtain a facial key point sequence corresponding to the video frame sequence. The facial key point sequence is aligned with the average face shape using facial deformation technology to obtain the standardized facial feature point sequence with identity information removed.

4. The three-dimensional face reconstruction method based on facial electromyography signals according to claim 3, characterized in that, The step of aligning the facial key point sequence with the average face shape based on facial deformation technology to obtain the standardized facial feature point sequence with identity information removed includes: Calculate the affine transformation matrix between the key points of the first frame of the facial key point sequence and the predefined average face shape; The affine transformation matrix is ​​used to transform each frame of the facial key point sequence to the coordinate system of the average face shape, and the resulting key point coordinate sequence is used as the standardized facial feature point sequence.

5. The three-dimensional face reconstruction method based on facial electromyography signals according to claim 1, characterized in that, The facial feature extraction model is obtained by using a facial electromyography (EMG) signal dataset to perform self-supervised pre-training on a pre-network including a convolutional neural network module and a Transformer encoder module, resulting in the following: The pre-network is trained through two phases: a mask reconstruction training phase and a noise-invariant contrastive learning training phase, to obtain the facial feature extraction model. The mask reconstruction training includes inputting signal values ​​of an electromyographic signal sequence through a random mask, reconstructing the signal through a pre-network and based on reconstruction loss; The noise-invariant contrastive learning training includes adding noise to a portion of the electromyographic signal sequence to generate a perturbed version, and extracting noise-robust features from the pre-network based on contrastive learning loss constraints.

6. The three-dimensional face reconstruction method based on facial electromyography signals according to claim 5, characterized in that, The reconstruction loss is the mean square error loss L at the mask location. Recon ,in, In the formula, M represents the set of all mask positions in the batch. and Let represent the original electromyographic signal value at the i-th position and and the predicted value reconstructed by the preceding network, respectively. The contrastive learning loss is the InfoNCE loss. ,in, In the formula, This represents the feature representation extracted from the original electromyography signal sample by the aforementioned preprocessor network. This represents the feature representation of the perturbed sample after adding noise to the original electromyography signal sample. Represents cosine similarity. The value represents the temperature hyperparameter, N represents the batch size, and the denominator is the sum of all 2N samples in the batch.

7. The three-dimensional face reconstruction method based on facial electromyography signals according to claim 1, characterized in that, The step of using the facial feature extraction model as a feature extractor with fixed parameters and connecting it to the Transformer decoder module to train a cross-modal mapping model from electromyographic signal sequences to standardized facial feature point sequences includes: Freeze the parameters of the facial feature extraction model, and input the electromyographic feature representation sequence extracted by the facial feature extraction model into the randomly initialized Transformer decoder module; The electromyographic feature representation sequence is aligned with the facial key point sequence to be predicted using the cross-attention mechanism in the Transformer decoder module. Using the standardized facial feature point sequence as a supervision signal, the mean squared error is used to optimize the difference between the predicted key points and the true key points, where the mean squared error is expressed as... In the formula, T represents the total number of time steps. and Let represent the actual keypoint coordinates and the predicted keypoint coordinates at step t, respectively; The cross-modal mapping model is obtained by training the Transformer decoder module and the output layer by minimizing the mean squared error loss function.

8. The three-dimensional face reconstruction method based on facial electromyography signals according to claim 1, characterized in that, The statistical 3D head model is a FLAME model; the step of inputting the facial feature point sequence into the preset statistical 3D head model to generate the corresponding 3D face reconstruction result includes: Using the facial feature point sequence as a two-dimensional constraint target, the expression parameters of the FLAME model are optimized based on the mean square error between the two-dimensional projection key points generated by the FLAME model and the facial feature point sequence by minimizing the mean square error. The optimized facial expression parameters drive the FLAME model to generate corresponding 3D facial expression animations as the 3D face reconstruction result.

9. A three-dimensional face reconstruction device based on facial electromyography signals, characterized in that, The device includes: The signal acquisition unit is used to preprocess the acquired raw signals, including facial electromyography signals and video frame data, to obtain electromyography signal sequences and video frame sequences. The video frame processing unit is used to extract facial key points and remove identity information based on the video frame sequence to obtain a standardized facial feature point sequence. The first training unit is used to perform self-supervised pre-training on the front-end network, which includes a convolutional neural network module and a Transformer encoder module, using a facial electromyography signal dataset to obtain a facial feature extraction model. The second training unit is used to use the facial feature extraction model as a feature extractor with fixed parameters and connect it to the Transformer decoder module to train a cross-modal mapping model from electromyographic signal sequences to standardized facial feature point sequences. The 3D face reconstruction unit is used to input the electromyographic signals of the face to be reconstructed into the cross-modal mapping model to obtain a sequence of facial feature points, and input the sequence of facial feature points into a preset statistical three-dimensional head model to generate the corresponding three-dimensional face reconstruction result.

10. A three-dimensional face reconstruction device based on facial electromyography signals, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory, wherein the computer program, when executed by the processor, implements the steps of a three-dimensional face reconstruction method based on facial electromyography signals as described in any one of claims 1 to 8.

Images