Deep learning neural network oriented video expression natural migration method and system

Abstract

The invention relates to the technical field of video expression natural migration, in particular to a deep learning neural network-oriented video expression natural migration method and system, and the method comprises the steps: collecting multi-subject expression video samples, and carrying out the classification and marking of the multi-subject expression video samples, so as to generate a training data set; establishing a double-branch deep learning architecture comprising a feature extraction branch and a time sequence modeling branch, performing model training on the double-branch deep learning architecture by using the training data set, and iteratively optimizing network parameters through back propagation in the model training by adopting a perception loss and adversarial loss composite function; and performing reasoning deployment on the double-branch deep learning architecture after model training is completed.

Description

Technical Field

[0001] This application relates to the technical field of natural transfer of video expressions, and in particular to a method and system for natural transfer of video expressions guided by deep learning neural networks. Background Technology

[0002] Existing technologies in the same field as this patented technology, namely video expression transfer, can be broadly categorized into three main technical paths, each with distinct technical characteristics and application boundaries: 1. GAN-based face-swapping technology: Represented by Deepfake, this technology employs a core architecture of a "general encoder + target-specific decoder," utilizing the adversarial training mechanism of Generative Adversarial Networks (GANs) to optimize the visual realism of cross-subject facial feature transfer. The technical logic involves first extracting general facial features through an encoder, and then reconstructing a facial image containing the target's identity features using a dedicated decoder trained for a specific target subject, thereby achieving expression transfer. This type of technology requires collecting 600-3000 high-quality frontal image samples for a single target subject, with a training period of 48-72 hours, and is mainly applied to transfer scenarios involving static images or low frame rate videos.

[0003] 2. Keypoint-guided transfer technology: Represented by Kuaishou LivePortrait, this technology employs an implicit 2D keypoint detection framework combined with the ConvNeXt-V2-Tiny lightweight network. It extracts facial keypoints to capture dynamic changes in facial expressions, and uses cascaded perceptual loss and GAN loss functions to improve texture reconstruction quality. Its advantage lies in enhancing the generalization ability of stylized portrait transfer through a video-image hybrid training mode, enabling rapid transfer of basic expressions. However, it relies heavily on the accuracy of explicit keypoints and has relatively weak adaptability to pose changes.

[0004] 3. Single-Image Driven Expression Transfer Techniques: These encompass solutions such as FaceSwapNet and ByteDance's X-Portrait 2. FaceSwapNet achieves single-frame static expression transfer through a "keypoint converter + conditional GAN" architecture, introducing an identity-aware loss function to improve identity feature preservation. X-Portrait 2, on the other hand, constructs an expression encoder that learns implicit motion representations and combines them with a diffusion model to achieve single-image driven video generation. Both techniques focus on identity preservation optimization, but the former only supports static transfer, while the latter, although supporting dynamic generation, suffers from insufficient stability in complex scenes.

[0005] While the aforementioned existing technologies have achieved basic facial expression transfer functionality, they suffer from four major technical bottlenecks in practical applications, making it difficult to meet the actual requirements for high precision and high robustness: 1. Incomplete feature separation leads to identity-expression distortion: Traditional single-branch networks do not have a dedicated feature separation module. The identity information and expression information in the extracted facial features interfere with each other. After transfer, "expression misalignment" (such as a sad expression showing a smiling outline) or "identity ambiguity" often occurs (the similarity between the person and the original subject is less than 60% after transfer), and it is impossible to balance the authenticity of the expression and the recognition of the identity.

[0006] 2. Lack of temporal coherence leads to disjointed actions: Most technologies adopt the "single-frame independent migration + frame splicing" mode, which fails to capture the dynamic changes in facial expressions between video frames. In dynamic scenes such as rapid smiling, frowning, and head turning, the transitions between frames are abrupt, and the smoothness score of the actions is generally below 65 points, which does not meet the requirements of video coherence.

[0007] 3. Poor adaptability to complex scenes: In scenes with 2D key point guidance, the key point detection error accumulates in scenes with side profile (head rotation angle exceeding 30°), partial occlusion (such as wearing glasses or hand occlusion) or exaggerated expressions, resulting in facial contour deformation and loss of eye / mouth details; GAN-based technology is sensitive to changes in lighting, and texture noise is prone to appear after migration in strong or low light environments.

[0008] 4. Imbalance between real-time performance and training cost: High-precision models often use ultra-deep network architectures (such as those with more than 100 convolutional layers), with inference time exceeding 100ms / frame, making them unsuitable for low-latency scenarios such as live streaming and real-time interaction; Lightweight models require dedicated training for a single target subject, with sample collection and training cycles lasting 2-3 days, resulting in extremely poor generalization ability and making them unsuitable for multi-subject transfer requirements. Summary of the Invention

[0009] Therefore, it is necessary to provide a deep learning neural network-guided method and system for natural video expression transfer to address the aforementioned technical problems, which can achieve highly accurate natural video expression transfer.

[0010] In a first aspect, this application provides a deep learning neural network-guided method for natural transfer of video facial expressions, the method comprising: Collect multi-subject facial expression video samples and classify and label the multi-subject facial expression video samples to generate a training dataset; Construct a two-branch deep learning architecture that includes a feature extraction branch and a temporal modeling branch; The dual-branch deep learning architecture is trained using the training dataset, and the network parameters are iteratively optimized through backpropagation during model training using a composite function of perceptual loss and adversarial loss. Deploy inference for the dual-branch deep learning architecture that has completed model training.

[0011] Secondly, this application also provides a deep learning neural network-guided video expression natural transfer system for implementing the deep learning neural network-guided video expression natural transfer method described in any one of the first aspects, comprising: The data acquisition module is used to collect multi-subject facial expression video samples and classify and label the multi-subject facial expression video samples to generate a training dataset. The architecture building module is used to build a two-branch deep learning architecture that includes a feature extraction branch and a temporal modeling branch; The model training module is used to train the dual-branch deep learning architecture using the training dataset, and to iteratively optimize the network parameters through backpropagation using a composite function of perceptual loss and adversarial loss during model training. The inference deployment module is used to perform inference deployment on a two-branch deep learning architecture that has completed model training.

[0012] The beneficial effects of the aforementioned deep learning neural network-guided video facial expression natural transfer method include: 1. Significantly improved identity-expression co-preservation accuracy: Through dual-branch feature separation and attention fusion design, the identity feature similarity has been improved from less than 60% in the existing technology to more than 85%, while the expression naturalness score has reached more than 90 points, solving the pain point of traditional technology "losing some aspects", and improving the identity preservation accuracy by 40% compared with Deepfake.

[0013] 2. Significantly enhanced dynamic facial expression smoothness: The LSTM temporal modeling branch improves the smoothness score of inter-frame action transition from below 65 points in the existing technology to 92 points, with no stuttering in scenes with rapid facial expression changes and head rotation, and the smoothness is improved by 40% compared with single-frame transfer technology.

[0014] 3. Breakthrough in adaptability to complex scenes: Multi-scene adaptive preprocessing combined with dual-branch weight adjustment improves the migration success rate from less than 50% in existing technologies to more than 88% in scenarios such as side face (within 60° head rotation), 30% area occlusion, and strong light and weak light, breaking through the dependence of traditional technologies on simple scenes.

[0015] 4. Dual optimization of real-time performance and generalization ability: The lightweight design reduces the inference speed from more than 100ms / frame of existing high-precision models to 20ms / frame, adapting to real-time interactive scenarios; at the same time, no dedicated training is required, a single model supports multi-subject transfer, the training cycle is shortened from 2-3 days to 8 hours, and the generalization ability and efficiency are greatly improved. Attached Figure Description

[0016] Figure 1 This is a schematic diagram illustrating the steps of a deep learning neural network-guided video facial expression natural transfer method in one embodiment. Figure 2 This is a schematic diagram of the structure of a video facial expression natural transfer system guided by a deep learning neural network in one embodiment; Figure 3 This is a schematic diagram illustrating the principle of a deep learning neural network-guided video facial expression natural transfer system in one embodiment. Figure 4 This is a schematic diagram of the system configuration of a deep learning neural network-guided video facial expression natural transfer system in one embodiment. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] The deep learning neural network-guided video facial expression natural transfer method provided in this application can be applied to, for example... Figures 1-4 In the application environment shown: S1: Collect multi-subject facial expression video samples and classify and label the multi-subject facial expression video samples to generate a training dataset; S4: Build a two-branch deep learning architecture that includes a feature extraction branch and a temporal modeling branch; S3: Use the training dataset to train the dual-branch deep learning architecture, and use a composite function of perceptual loss and adversarial loss to iteratively optimize the network parameters through backpropagation during model training; S4: Deploy inference for the dual-branch deep learning architecture that has completed model training.

[0019] Specifically, in step S1 of the embodiments provided in this application, facial expression videos of multiple subjects of different ages (such as children, youth, middle-aged, and elderly), different genders (male and female), and different postures (front, side, tilt, etc.) are collected through various channels, such as online video platforms and professional video capture equipment. Diverse data can improve the generalization ability of the model. People of different ages and genders have different facial features and expression patterns. Facial expression videos in different postures can enable the model to learn facial expression features from various perspectives. In this way, the trained model can more accurately transfer facial expressions when facing various real-world scenarios.

[0020] More specifically, video processing tools are used to divide each facial expression video into a series of individual video frames at certain time intervals (such as extracting several frames per second). Deep learning models typically process image data. After converting the video into a frame sequence, it is easier to perform operations such as face detection and key point annotation on each video frame. At the same time, the facial expressions in the video change over time, and splitting the video into frames can better capture the dynamic changes in facial expressions.

[0021] More specifically, face detection algorithms (such as deep learning-based face detection models) are used to locate the position and size of faces in each video frame. Within the detected face area, key points of the face, such as the positions of the eyes, nose, and mouth, are marked. Typically, 68 or more key points are marked. Based on the expressions of the people in the video frames, they are classified into different expression categories, such as happy, sad, and angry.

[0022] More specifically, face detection provides accurate face regions for subsequent keypoint annotation and expression analysis, avoiding interference from irrelevant background information. Facial keypoints can reflect the shape and structural information of the face, which helps the model learn the geometric features of the face, thereby enabling more accurate expression transfer. Expression classification and annotation provide the model with clear learning objectives, enabling the model to understand different expression categories, thus generating expressions that meet expectations when transferring expressions.

[0023] More specifically, based on the position of facial key points, the facial images in each video frame are rotated, scaled, and translated to make the positions of the facial key points roughly the same, thus achieving facial image alignment. The pixel values ​​of the images are normalized, usually scaling the pixel values ​​to the range of [0, 1] or [-1, 1]. Semantic interpretation of the image data is performed to obtain pose and lighting information. Angle correction is performed based on the pose information, and brightness correction is performed based on the lighting information. Other data augmentation operations can also be performed, such as random cropping and flipping.

[0024] More specifically, image alignment ensures that faces in different video frames have consistent poses and positions, making it easier for the model to learn the essential features of faces and improving the training effect. Normalization can speed up the training process and avoid gradient instability caused by excessively large pixel value ranges. Data augmentation increases the diversity of training data and improves the model's generalization ability. Angle correction and brightness correction can reduce the impact of pose and lighting on the model, enabling the model to perform well under different lighting and pose conditions. Operations such as random cropping and flipping can generate more training samples, allowing the model to learn more variations.

[0025] Specifically, in step S2 of the embodiments provided in this application, a deep separable convolutional network is typically selected as an improved convolutional neural network. It is an optimized architecture based on the traditional convolutional neural network. The deep separable convolutional network is trained using the previously generated training dataset, and the parameters in the network are continuously adjusted so that it can accurately extract identity features and expression features from the input face image. The trained network is processed using model quantization compression technology to reduce the model's storage space and computational load, thereby optimizing the inference speed.

[0026] More specifically, depthwise separable convolutional networks decompose traditional convolution operations into depthwise convolution and pointwise convolution, significantly reducing the number of parameters and computational cost. While maintaining feature extraction capabilities, they improve computational efficiency, enabling faster extraction of key features from face images. Facial identity features and expression features are crucial information in video expression natural transfer tasks. This branch can extract these features from the input image, providing a foundation for subsequent expression transfer. In practical applications, the model's inference speed is critical. Model quantization compression technology can reduce computational resource consumption without significantly sacrificing model performance, enabling the model to process input data more quickly and meet real-time requirements.

[0027] More specifically, a Long Short-Term Memory (LSTM) network structure is constructed: the number of layers, the number of hidden units, and other hyperparameters of the LSTM network are determined, and a network structure suitable for processing video facial expression sequence data is built. The feature sequence obtained after feature extraction is used as input and is sequentially input into the LSTM network in chronological order. The LSTM network is trained using the same training dataset, and the network parameters are continuously adjusted through the backpropagation algorithm so that the network can accurately capture the dynamic changes of facial expressions over time.

[0028] More specifically, facial expressions in videos are a dynamic process that changes over time. Traditional convolutional neural networks struggle to effectively handle this temporal information. LSTM networks, with their memory units and gating mechanisms, can remember past information and dynamically update the memory based on the current input, making them ideal for processing sequential data. Therefore, they can be used to capture the dynamic changes in facial expressions across different time frames. Understanding these dynamic changes is crucial for achieving natural expression transfer. Through this branch, the model can predict the development trend of facial expressions, thereby generating more natural and coherent expression sequences when transferring expressions.

[0029] More specifically, commonly used attention mechanisms include channel attention and spatial attention. The appropriate attention mechanism is selected according to the task requirements, and its specific structure is designed. The facial identity features and expression features output from the feature extraction branch, as well as the dynamic expression features output from the temporal modeling branch, are input into the fusion layer. The importance of different features is evaluated through the attention mechanism, and their weights are adaptively adjusted. The features with adjusted weights after the attention mechanism are fused to obtain the final fused feature representation, which serves as the input for subsequent processing.

[0030] More specifically, the importance of facial identity features, facial expression features, and facial expression dynamic features varies in different scenarios and tasks. Attention mechanisms can automatically identify which features are more important to the current task based on the characteristics of the input data and adjust their weights accordingly, thereby improving the flexibility and adaptability of the model. By reasonably integrating features from different branches, the advantages of each branch can be fully utilized, enabling the model to handle facial expression transfer tasks more comprehensively and further improve the naturalness and accuracy of expression transfer.

[0031] More specifically, the network architecture of a GAN generator typically consists of multiple convolutional and deconvolutional layers to generate facial images after expression transfer. The fused features output from the fusion layer are input into the GAN generator, which then generates the expression-transferred image. Simultaneously, a discriminator is introduced to determine whether the generated image is realistic or fabricated. Through continuous adversarial training, the generator gradually learns how to generate more realistic and natural expression-transferred images. During training, the generator's parameters are continuously adjusted to optimize the details of the generated images, making them more lifelike in terms of expression richness and texture.

[0032] More specifically, the GAN generator branch can convert fused features into specific face images and continuously improve the quality of generated images through adversarial training with the discriminator. It can achieve facial expression transfer and perform detail optimization on the transferred images to make the generated expressions more natural and realistic, reducing artificial traces. Combined with the previous feature extraction and temporal modeling, the GAN generator branch further enhances the generation capability and expressiveness of the entire model, enabling the model to better meet the needs of practical applications.

[0033] Specifically, in step S3 of the embodiments provided in this application, the previously generated training dataset is randomly divided into training set, validation set and test set according to a certain ratio (common ratios are training set: validation set: test set = 7 : 1.5 : 1.5 or 8 : 1 : 1, etc.) to ensure that each set after division contains diverse data, that is, samples covering different ages, genders, postures and various expression categories.

[0034] More specifically, the training set is used for parameter fitting of the model, allowing the model to learn the mapping relationship between inputs (facial image features, etc.) and outputs (expression transfer results) from a large amount of data; the validation set is used during model training to evaluate the model's performance on unseen data. The validation set allows for model selection (such as choosing models with different architectures or hyperparameter combinations) and hyperparameter tuning, avoiding overfitting of the model to the training set; the test set is used after model training is completed to evaluate the model's generalization ability, that is, the model's performance on completely unknown data, to ensure that the model has good performance in real-world applications.

[0035] More specifically, the data in the training set is input into the two-branch deep learning architecture in batches. The output of the model is calculated through forward propagation. Then, the loss value is calculated based on the difference between the output and the true label (using a composite function of perceptual loss and adversarial loss). Next, the backpropagation algorithm is used to update the model parameters based on the loss value. This process is iterated until the model's performance on the training set reaches a certain requirement. During the model training process, different combinations of hyperparameters (such as learning rate, batch size, number of network layers, etc.) are tried, and the performance of the model under different hyperparameter combinations is evaluated on the validation set. The model and hyperparameter combination that performs best on the validation set is selected.

[0036] More specifically, by continuously adjusting the model's parameters, the model can predict the facial expression transfer results in the training set as accurately as possible, learn the patterns and rules in the data, and different combinations of hyperparameters will have a significant impact on the model's performance. Using a validation set for evaluation can help us find the optimal combination of hyperparameters, avoid overfitting or underfitting the model on the training set, and improve the model's generalization ability and performance.

[0037] More specifically, perceptual loss typically measures the difference in feature space between the generated expression transfer image and the real image. For example, it uses a pre-trained convolutional neural network to extract image features and then calculates the distance between features. Adversarial loss, on the other hand, is based on the principle of generative adversarial networks (GANs). It optimizes the quality of generated images through adversarial training between the generator and the discriminator. These two loss functions are combined into a composite function with certain weights. In each training iteration, the loss value is calculated based on the composite loss function, and then the gradient of the loss function with respect to the model parameters is calculated using the backpropagation algorithm. The model parameters are updated based on the gradient, causing the loss value to gradually decrease. During training, the model is periodically evaluated using a validation set to calculate the naturalness score of expression transfer and the similarity of identity features. Training stops when these two metrics reach a preset standard.

[0038] More specifically, perceptual loss can capture high-level semantic features of images, making the generated expression transfer images more natural and realistic, conforming to human perception. Adversarial loss, through adversarial training between the generator and discriminator, enables the generated images to more closely resemble the distribution of real images, improving image quality and diversity. The composite function combines the advantages of perceptual loss and adversarial loss, enabling more comprehensive optimization of model performance, ensuring that the expression transfer results generated by the model are both natural and retain identity features. Using the evaluation results of the validation set as the basis for stopping training ensures that the model can achieve the expected performance on unseen data, avoiding overfitting.

[0039] More specifically, the data in the test set is input into the trained dual-branch deep learning architecture to obtain the model's prediction results. Various evaluation metrics are calculated between the prediction results and the real labels, such as the naturalness score of facial expression transfer, identity feature similarity, and accuracy. The generalization performance of the model is evaluated based on the evaluation metrics to determine the reliability of the model in practical applications. The test set is data that the model has never seen before. Through evaluation on the test set, the model's generalization ability in real-world scenarios can be truly reflected. If the model performs well on the test set, it indicates that the model can adapt to different data distributions and scenarios and has high reliability and practicality.

[0040] Specifically, in step S4 of the embodiments provided in this application, a suitable model saving format is selected. Common deep learning frameworks (such as PyTorch, TensorFlow, etc.) have their own recommended saving formats. For example, PyTorch usually uses .pth or .pt format, while TensorFlow can use .h5 or SavedModel format. All parameters of the trained two-branch deep learning architecture (including the weights and biases of the feature extraction branch, temporal modeling branch, GAN generator branch, etc.) are saved to the specified file. The model's structural information also needs to be saved so that the model can be correctly reconstructed when loading the model. Training a deep learning model usually requires a lot of computing resources and time. Saving the trained model can avoid repeated training and facilitate subsequent use of the model for inference. The saved model file can be easily transferred and shared between different environments or devices, which is convenient for team collaborative development or applying the model to different projects.

[0041] More specifically, based on the actual application scenario and requirements, choose an appropriate inference environment. Common inference environments include local servers, cloud servers, and mobile devices (such as mobile phones and tablets). For example, if real-time processing of large amounts of data is required, a high-performance cloud server can be chosen; if developing mobile applications, the model needs to be deployed to mobile devices. Based on the characteristics of the inference environment and the model, choose an appropriate inference framework. Common inference frameworks include TensorRT (for NVIDIA GPU-accelerated inference), ONNX Runtime (a cross-platform inference engine), and TFLite (for mobile and embedded devices).

[0042] More specifically, some inference frameworks require models to be input in a specific format. Therefore, it is necessary to convert the previously saved model into a format supported by the inference framework. For example, convert a PyTorch model to ONNX format so that it can be used for inference with ONNXRuntime. Using the selected inference framework, load the converted model into the inference environment, perform the necessary configuration and initialization, and then you can write code to pass the input data to the model for inference and obtain the output results.

[0043] More specifically, different application scenarios have different requirements for the inference environment, such as computing power, storage capacity, and power consumption. Choosing a suitable inference environment can ensure that the model can run efficiently and stably in practical applications. Inference frameworks are specifically optimized for model inference, which can improve the inference speed and efficiency of the model. Different inference frameworks have different performance characteristics on different hardware platforms and model types. Choosing a suitable inference framework can give full play to the advantages of the hardware and improve the inference performance of the model. Through model conversion and deployment steps, the compatibility between the model and the inference environment and inference framework can be ensured, so that the model can work normally in the target environment.

[0044] This invention discloses a deep learning neural network-guided method for natural video expression transfer, which relates to the fields of computer vision and video processing technology. It aims to solve the technical defects of traditional video expression transfer methods, such as expression distortion, discontinuous movements, insufficient preservation of identity features, and poor real-time performance.

[0045] This method employs a customized deep learning neural network as its core execution process: First, data preprocessing is performed, collecting multi-subject facial expression video samples covering different ages, genders, and postures. After extracting video frame sequences, face detection, key point annotation, and expression classification annotation are performed. A training dataset is generated through image alignment, normalization, and data augmentation. Next, a dual-branch deep learning architecture is built. The feature extraction branch uses an improved convolutional neural network (CNN) to extract facial identity features and expression features, while the temporal modeling branch uses a long short-term memory network (LSTM) to capture the dynamic changes in facial expressions. The fusion layer adaptively adjusts the weights of the two types of features through an attention mechanism, and the generator branch uses a generative adversarial network (GAN) to achieve expression transfer and detail optimization. Subsequently, model training is performed, using a composite function of perceptual loss and adversarial loss. The network parameters are iteratively optimized through backpropagation until the expression transfer naturalness score on the validation set reaches above 90 points and the identity feature similarity remains above 85%. Finally, inference deployment is completed. The video to be transferred is input into the trained model after frame sequence extraction, and the model outputs the expression-transferred video after fast inference at 20ms / frame, achieving natural expression restoration, smooth and continuous movements, and accurate preservation of subject identity features.

[0046] This invention breaks through the dependence of traditional methods on a single subject by using feature separation and temporal modeling design. It maintains the naturalness of transfer even in scenarios with side profiles and exaggerated expressions, and takes into account both real-time performance and robustness. According to actual tests, it improves the transfer efficiency by 50% in the field of short video creation and achieves 92% accuracy in expression restoration in the field of virtual digital humans. It is applicable to core fields such as short video production, virtual live streaming, film and television post-production, and human-computer interaction.

[0047] This patented technology differs fundamentally from existing technologies through five key technical aspects, specifically addressing the shortcomings of traditional technologies: 1. Dual-branch feature separation architecture: Abandoning the traditional single-branch design, it innovatively adopts an improved CNN dual-branch structure. One branch focuses on extracting facial identity features (such as facial contours and facial proportions), while the other branch extracts expression features (such as the curvature of the corners of the mouth and the opening and closing of the eyes). Through the feature channel separation design, the interference between the two types of information is avoided from the root. This is the core difference between Deepfake's single-branch hybrid feature extraction.

[0048] 2. Combination of attention fusion and temporal modeling: An attention mechanism is introduced into the feature fusion stage, which can dynamically adjust the weight of identity and facial features according to the scene (such as increasing the weight of identity features in a side profile scene); at the same time, a new LSTM temporal modeling branch is added to capture the temporal correlation of facial expression changes between frames, which is different from technologies such as LivePortrait that only rely on explicit guidance of key points.

[0049] 3. Composite Loss Function Optimization: A composite function of "perceptual loss + adversarial loss" is designed. Perceptual loss ensures the visual rationality of the transferred expression, while adversarial loss improves the realism of the texture. The optimization effect is more comprehensive than that of traditional single loss functions (such as using only MSE loss), and solves the problem of rough technical details in X-Portrait 2 and other technologies.

[0050] 4. Multi-scene adaptive preprocessing: The preprocessing stage adds pose classification and illumination normalization modules to perform adaptive enhancement for complex scenes such as side faces, occlusion, and strong light. For example, it performs angle correction on side face samples, which is more robust than the fixed preprocessing of existing technologies.

[0051] 5. Lightweight Inference Optimization: Depthwise separable convolution is used in the generator branch to replace traditional convolution, reducing the number of parameters by 60%. At the same time, the inference speed is optimized through model quantization compression technology, achieving real-time processing of 20ms / frame, which is different from the shortcomings of traditional high-precision models in terms of insufficient real-time performance.

[0052] In one embodiment, the steps of collecting multi-subject facial expression video samples and classifying and labeling the multi-subject facial expression video samples to generate a training dataset include: S11: Collect facial expression video samples covering multiple subjects of different ages, genders, and postures; S12: Divide each of the aforementioned facial expression video samples into video frames to convert each of the aforementioned facial expression video samples into a video frame sequence; S13: Perform face detection, key point standardization, and expression classification annotation on each of the video frame sequences to obtain preliminary training data; S14: Perform image alignment, normalization, and data augmentation on the prepared training data to generate a training dataset.

[0053] Specifically, video samples can be obtained from multiple channels, such as professional video material websites, social media platforms (e.g., Douyin, Weibo), and publicly available video datasets (e.g., CK+, FER2013). Professional video capture equipment (e.g., high-definition cameras) can also be used to record multi-subject facial expression videos in different scenarios (e.g., classrooms, shopping malls, parks). During the collection process, it's crucial to ensure coverage of subjects of different ages (e.g., children, youth, middle-aged, elderly), genders (male and female), and postures (front, side, tilted, looking up, looking down, etc.). This can be achieved by setting different shooting angles and inviting people with different characteristics to participate in the recording. The real world is diverse; people of different ages, genders, and postures exhibit differences in facial features and facial expressions. By collecting diverse samples, the model can learn a wider range of features, enabling accurate expression transfer in various real-world scenarios. If the training data only contains samples from a specific group of people or postures, the model will overfit to those specific data and perform poorly on other data. Diverse samples can reduce this data bias, making the model more robust.

[0054] More specifically, leveraging the OpenCV library in Python, which offers rich video processing capabilities, one can use the `cv2.VideoCapture` function to open a video file, then read each frame of the video in a loop and save it as a separate image file. The frame rate for extracting frames can be set as needed, such as 10 or 25 frames per second. By controlling the frame rate, the amount of data and the integrity of facial expression information can be balanced. Deep learning models typically use images as input. Converting the video into a frame sequence facilitates subsequent processing and analysis of each frame. Facial expressions in a video change over time. Breaking the video into frames can better capture the dynamic changes in facial expressions, helping the model learn the temporal features of facial expressions.

[0055] More specifically, deep learning-based face detection algorithms, such as MTCNN (Multi-task Cascaded Convolutional Networks) or RetinaFace, can be used. These algorithms can accurately detect the location and size of faces in an image. In Python, the mtcnn or retinaface libraries can be used to implement face detection. After detecting a face, a facial landmark detection algorithm (such as Dlib's 68-point facial landmark detector) is used to annotate the key points of the face, such as the positions of the eyes, nose, and mouth. More specifically, based on the facial expressions of people in video frames, they are classified into different expression categories, such as happiness, sadness, anger, surprise, fear, disgust, and neutrality. This can be achieved through manual annotation, with professional annotators classifying the expressions for each frame. Face detection provides accurate face regions for subsequent keypoint annotation and expression analysis, avoiding interference from irrelevant background information and improving the accuracy of feature extraction. Facial keypoints reflect the shape and structure of the face, helping the model learn the geometric features of the face, thus enabling more accurate expression transfer. For example, during expression transfer, the position and shape of the expression can be adjusted based on changes in keypoints. Expression classification and annotation provide the model with clear learning objectives, enabling the model to understand different expression categories and generate expressions that meet expectations during expression transfer.

[0056] More specifically, based on the location of facial key points, affine or perspective transformations are used to rotate, scale, and translate the facial image to make the key point positions roughly the same, thus aligning the facial image. The pixel values ​​of the image are normalized, typically scaled to the range of [0, 1] or [-1, 1]. Computer vision algorithms are used to perform semantic interpretation of the image data. For example, pose information is obtained through pose estimation algorithms, and illumination information is obtained through illumination analysis algorithms. Based on the pose information, the image is rotated or flipped to make its pose more consistent. Based on the illumination information, the brightness and contrast of the image are adjusted. Random cropping, random scaling, and adding noise can also be performed to increase the diversity of the data.

[0057] More specifically, image alignment ensures consistent pose and position for faces across different video frames, facilitating the model's learning of essential facial features, reducing interference from pose differences, and improving training performance. Normalization accelerates training, avoids gradient instability caused by excessively large pixel value ranges, and ensures comparability between different features, helping the model better learn relationships between features. Data augmentation increases the diversity of training data, improving the model's generalization ability. Angle and brightness correction reduce the impact of pose and lighting on the model, enabling it to perform well under varying lighting and pose conditions.

[0058] In one embodiment, the step of performing data augmentation processing on each of the pre-training data includes: S101: Perform semantic interpretation of the image data on the pre-training data to obtain the pose and illumination information fed back by the pre-training data; S102: Perform adaptive angle correction on the pre-training data based on the posture information, and perform adaptive brightness correction on the pre-training data based on the illumination information.

[0059] In one embodiment, the steps for building a two-branch deep learning architecture that includes a feature extraction branch and a temporal modeling branch include: S21: Construct a feature extraction branch using an improved convolutional neural network to extract facial identity features and expression features; S22: Construct a temporal modeling branch that uses a long short-term memory network to capture the dynamic changes in facial expressions; S23: Based on the feature extraction branch and the temporal modeling branch, build a fusion layer with an attention mechanism for adaptively adjusting the weights of the two types of features.

[0060] Specifically, a deep separable convolutional network (such as the MobileNet series) is selected as the improved convolutional neural network. Model quantization compression technology is used to optimize inference speed. The feature extraction branch is trained and fine-tuned using the previously generated training dataset. A loss function (such as cross-entropy loss) and an optimizer (such as the Adam optimizer) are defined, and then iterative training is performed.

[0061] More specifically, depthwise separable convolutional networks decompose traditional convolution operations into depthwise convolution and pointwise convolution, significantly reducing the number of parameters and computational cost. This allows the model to maintain high feature extraction capabilities while significantly improving computational efficiency, enabling it to extract key features from face images more quickly. Facial identity features and expression features are crucial for the natural transfer of video expressions. Identity features are used to distinguish different individuals, while expression features are used to describe the emotional state of the face. Through the feature extraction branch, these features can be accurately extracted from the input face image, providing a foundation for subsequent expression transfer. In practical applications, the model's inference speed is critical. Model quantization compression technology can reduce the consumption of computational resources without significantly sacrificing model performance, enabling the model to process input data more quickly and meet real-time requirements.

[0062] More specifically, we will build a simple LSTM network using PyTorch. Assuming the input feature dimension is 128, the hidden layer dimension is 64, and the output dimension is 64, we will take the feature sequence obtained after the feature extraction branch as input and feed it into the LSTM network in chronological order. Note that the shape of the input data should be (batch_size, sequence_length, input_size). We will then define the loss function and optimizer and train the network.

[0063] More specifically, facial expressions in videos are a dynamic process that changes over time. Traditional convolutional neural networks struggle to effectively handle this temporal information. LSTM networks, with their memory units and gating mechanisms, can remember past information and dynamically update the memory based on current input, making them ideal for processing sequential data. Through LSTM networks, the dynamic changes in facial expressions across different time frames can be captured. Understanding these dynamic changes is crucial for achieving natural expression transfer. Through this branch, the model can predict the development trend of facial expressions, thereby generating more natural and coherent expression sequences when transferring expressions, avoiding abrupt or unnatural expressions.

[0064] More specifically, an attention mechanism based on attention scores can be used to concatenate the outputs of the feature extraction branch and the temporal modeling branch, calculate the attention score through a fully connected layer, normalize the score using a softmax function, apply the attention weights to the concatenated features, and obtain the weighted fused features. The weighted fused features can then be mapped to the final output dimension through another fully connected layer.

[0065] More specifically, the importance of facial identity features, facial expression features, and facial expression dynamic features varies across different scenarios and tasks. Attention mechanisms can automatically identify which features are more important to the current task based on the characteristics of the input data and adjust their weights accordingly, thereby improving the model's flexibility and adaptability. By reasonably fusing features from different branches, the advantages of each branch can be fully utilized, enabling the model to more comprehensively handle facial expression transfer tasks. Attention mechanisms can help the model better focus on key features, further improving the naturalness and accuracy of expression transfer.

[0066] In one embodiment, the feature extraction branch employs an improved convolutional neural network, specifically a depthwise separable convolutional network, while optimizing inference speed through model quantization compression technology.

[0067] In one embodiment, the dual-branch deep learning architecture further includes a GAN generator branch built on a generative adversarial network, which is used to implement facial expression transfer and detail optimization.

[0068] Specifically, a GAN generator typically consists of multiple convolutional layers, deconvolutional layers (transposed convolutional layers), and activation functions. The features from the feature extraction branch and the temporal modeling branch, after being processed by the fusion layer, are used as the input to the generator. The discriminator is a binary classification network used to determine whether the input image is real or generated. It uses a binary cross-entropy loss function, defines optimizers for the generator and discriminator respectively, and trains the generator and discriminator alternately.

[0069] More specifically, additional constraint terms, such as perceptual loss and feature matching loss, are introduced into the loss function to improve the quality and detail of the generated images. Hyperparameters such as the network structure, learning rate, and batch size of the generator and discriminator are adjusted through experiments to obtain better training results.

[0070] More specifically, GAN generators have powerful generation capabilities, capable of generating facial images with specific expressions based on input features. By taking the outputs of the feature extraction branch and the temporal modeling branch as input, the generator can learn how to transform these features into corresponding expression transfer images. It can generate expression transfer results for different individuals based on different input features, thus achieving personalized expression transfer.

[0071] More specifically, through adversarial training between the discriminator and the generator, the generator can continuously learn how to generate more realistic and natural images, while the discriminator continuously evaluates the generated images, prompting the generated images to be more realistic in details, such as facial textures and muscle movements. Introducing additional loss function constraints can further improve the quality of the generated images. For example, perceptual loss can make the generated images closer to real images in the feature space, thereby improving the visual effect and detail of the images.

[0072] More specifically, GAN generators can generate diverse facial expression transfer results, which helps the model generate high-quality facial expression transfer images when faced with different input data, improving the model's generalization ability. In different scenarios and tasks, GAN generators can adapt to various needs by adjusting input features and training parameters, making the model more flexible and practical.

[0073] In one embodiment, the steps of training the dual-branch deep learning architecture using the training dataset and iteratively optimizing the network parameters through backpropagation during model training using a composite function of perceptual loss and adversarial loss include: S31: Divide the training dataset into a training set, a validation set, and a test set; S32: Fit model parameters to the dual-branch deep learning architecture based on the training set, and perform model selection and hyperparameter tuning on the dual-branch deep learning architecture after model parameter fitting based on the validation set. S33: During the model parameter fitting process, the perceptual loss and adversarial loss composite function is used to backpropagate the dual-branch deep learning architecture to iteratively optimize the network parameters until the expression transfer naturalness score and identity feature similarity of the model verification of the validation set reach the preset standard. S34: Test and evaluate the generalization performance of the dual-branch deep learning architecture that has completed model parameter fitting based on the test set.

[0074] Specifically, the `train_test_split` function from the `sklearn` library in Python is used for random partitioning, dividing the entire training dataset into a training set and a temporary set according to a certain ratio. Then, the temporary set is further divided into a validation set and a test set. If the number of samples of different classes in the dataset is unbalanced, a stratified partitioning method can be adopted to ensure that the proportion of each class in each subset is consistent with that of the original dataset. Stratified partitioning can be achieved by setting `stratify=y` in the `train_test_split` function.

[0075] More specifically, the training set is used for parameter fitting of the model, allowing the model to learn the mapping relationship between inputs (such as facial image features) and outputs (expression transfer results) from a large amount of data. The validation set is used during model training to evaluate the model's performance on unseen data. The validation set allows for model selection (such as selecting models with different architectures or hyperparameter combinations) and hyperparameter tuning, avoiding overfitting of the model to the training set. After the model training is completed, the test set is used to evaluate the model's generalization ability, that is, the model's performance on completely unknown data, to ensure that the model has good performance in real-world applications.

[0076] More specifically, the data in the training set is input into the two-branch deep learning architecture in batches. The output of the model is calculated through forward propagation. Then, the loss value is calculated based on the difference between the output and the true label (using a composite function of perceptual loss and adversarial loss). Next, the backpropagation algorithm is used to update the model parameters based on the loss value. This process is iterated continuously. Different combinations of hyperparameters (such as learning rate, batch size, number of network layers, etc.) can be tried using methods such as grid search and random search. The performance of the model under different hyperparameter combinations is evaluated on the validation set, and the model and hyperparameter combination that performs best on the validation set is selected.

[0077] More specifically, by continuously adjusting the model's parameters, the model can predict the facial expression transfer results in the training set as accurately as possible, learn the patterns and rules in the data, and different combinations of hyperparameters will have a significant impact on the model's performance. Using a validation set for evaluation can help us find the optimal combination of hyperparameters, avoid overfitting or underfitting the model on the training set, and improve the model's generalization ability and performance.

[0078] More specifically, perceptual loss typically uses a pre-trained convolutional neural network (such as the VGG network) to extract image features, calculates the difference between the generated image and the real image in the feature space, and, based on the principle of generative adversarial networks, uses a binary cross-entropy loss function to calculate the loss of the generator and discriminator. The perceptual loss and adversarial loss are combined into a composite function according to certain weights. In each training iteration, the loss value is calculated according to the composite loss function, and then the gradient of the loss function with respect to the model parameters is calculated using the backpropagation algorithm. The model parameters are updated according to the gradient, so that the loss value gradually decreases. During the training process, the model is evaluated periodically using a validation set to calculate the naturalness score of facial expression transfer and the similarity of identity features. When these two indicators reach the preset standard, training stops.

[0079] More specifically, perceptual loss can capture high-level semantic features of images, making the generated expression transfer images more natural and realistic, conforming to human perception. Adversarial loss, through adversarial training between the generator and discriminator, enables the generated images to more closely resemble the distribution of real images, improving image quality and diversity. The composite function combines the advantages of perceptual loss and adversarial loss, enabling more comprehensive optimization of model performance, ensuring that the expression transfer results generated by the model are both natural and retain identity features. Using the evaluation results of the validation set as the basis for stopping training ensures that the model can achieve the expected performance on unseen data, avoiding overfitting.

[0080] More specifically, the data in the test set is input into the trained dual-branch deep learning architecture to obtain the model's prediction results. Various evaluation metrics between the prediction results and the real labels are calculated, such as the naturalness score of facial expression transfer, identity feature similarity, accuracy, and recall. The test set is data that the model has never seen before. Through evaluation on the test set, the model's generalization ability in real-world scenarios can be truly reflected. If the model performs well on the test set, it indicates that the model can adapt to different data distributions and scenarios, and has high reliability and practicality.

[0081] In one embodiment, the steps for inference deployment of a two-branch deep learning architecture that has completed model training include: S41: Perform model saving processing on the dual-branch deep learning architecture after model training is completed; S42: Select a specified inference environment and inference framework for the dual-branch deep learning architecture, so as to deploy the dual-branch deep learning architecture in the inference environment through the inference framework.

[0082] Specifically, training deep learning models typically requires significant computational resources and time. Saving the trained model avoids the cost of repeated training, allows for convenient inference when needed, and enables easy transfer and sharing between different environments or devices, facilitating team collaboration or application of the model to various projects.

[0083] More specifically, depending on the actual application scenario and needs, choose a suitable inference environment. If the data volume is small, the real-time requirements are not high, and data privacy is a concern, you can choose to deploy the model on a local server. The local server can be a high-performance desktop or server with the corresponding operating system (such as Linux) and dependent libraries installed. For applications that need to process large amounts of data and have high real-time requirements, cloud servers are a good choice. Common cloud service providers include Alibaba Cloud, Tencent Cloud, and Amazon Web Services. You can choose the appropriate cloud server instance type and configuration according to your needs. If you want to develop mobile applications, you need to deploy the model to mobile devices (such as mobile phones and tablets). Mobile devices have limited computing resources and storage capacity, so you need to choose a lightweight model and inference framework.

[0084] More specifically, TensorRT is suitable for NVIDIA GPU-accelerated inference and can optimize models trained by frameworks such as TensorFlow and PyTorch, significantly improving inference speed. When using TensorRT, the model needs to be converted to ONNX format before using TensorRT for inference.

[0085] More specifically, ONNX Runtime is a cross-platform inference engine that supports models trained using various deep learning frameworks (such as PyTorch, TensorFlow, MXNet, etc.) and allows direct inference using ONNX format models.

[0086] More specifically, TFLite is a lightweight inference framework designed specifically for mobile and embedded devices that can convert TensorFlow models into the TFLite format for inference on mobile devices.

[0087] More specifically, using the selected inference framework, the transformed model is loaded into the inference environment, and the necessary configuration and initialization are performed. Then, code is written to pass the input data to the model for inference and obtain the output results. Different application scenarios have different requirements for the inference environment, such as computing power, storage capacity, power consumption, etc. Choosing a suitable inference environment can ensure that the model can run efficiently and stably in practical applications.

[0088] More specifically, inference frameworks are optimized for model inference, which can improve the inference speed and efficiency of the model. Different inference frameworks have different performance characteristics on different hardware platforms and model types. Choosing a suitable inference framework can give full play to the advantages of the hardware and improve the inference performance of the model. Through model conversion and deployment steps, the compatibility between the model and the inference environment and inference framework can be ensured, so that the model can work normally in the target environment.

[0089] In one embodiment, such as Figure 2 As shown, a deep learning neural network-guided video facial expression natural transfer system is provided to implement the deep learning neural network-guided video facial expression natural transfer method described in any one of the first aspects, comprising: The data acquisition module is used to collect multi-subject facial expression video samples and classify and label the multi-subject facial expression video samples to generate a training dataset. The architecture building module is used to build a two-branch deep learning architecture that includes a feature extraction branch and a temporal modeling branch; The model training module is used to train the dual-branch deep learning architecture using the training dataset, and to iteratively optimize the network parameters through backpropagation using a composite function of perceptual loss and adversarial loss during model training. The inference deployment module is used to perform inference deployment on a two-branch deep learning architecture that has completed model training.

[0090] In this embodiment, the specific implementation of each module in the above system embodiment is described in the above method embodiment, and will not be repeated here.

[0091] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0092] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0093] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A deep learning neural network-guided method for natural transfer of facial expressions in videos, characterized in that, include: Collect multi-subject facial expression video samples and classify and label the multi-subject facial expression video samples to generate a training dataset; Construct a two-branch deep learning architecture that includes a feature extraction branch and a temporal modeling branch; The dual-branch deep learning architecture is trained using the training dataset, and the network parameters are iteratively optimized through backpropagation during model training using a composite function of perceptual loss and adversarial loss. Deploy inference for the dual-branch deep learning architecture that has completed model training.

2. The deep learning neural network-guided video facial expression natural transfer method as described in claim 1, characterized in that, The steps of collecting multi-subject facial expression video samples and classifying and labeling the multi-subject facial expression video samples to generate a training dataset include: Collect facial expression video samples covering multiple subjects of different ages, genders, and postures; Each of the aforementioned facial expression video samples is divided into video frames to convert each of the aforementioned facial expression video samples into a video frame sequence; Face detection, key point standardization, and expression classification and annotation are performed on each of the video frame sequences to obtain preliminary training data; The prepared training data are subjected to image alignment, normalization, and data augmentation to generate a training dataset.

3. The deep learning neural network-guided video facial expression natural transfer method as described in claim 2, characterized in that, The steps for data augmentation processing of the aforementioned pre-training data include: The image data of the pre-training data is semantically interpreted to obtain the pose and illumination information fed back by the pre-training data; The pre-training data is adaptively angle-corrected based on the posture information, and adaptively brightness-corrected based on the illumination information.

4. The deep learning neural network-guided video facial expression natural transfer method as described in claim 1, characterized in that, The steps to build a two-branch deep learning architecture that includes a feature extraction branch and a temporal modeling branch include: A feature extraction branch using an improved convolutional neural network is constructed to extract facial identity features and facial expression features; A temporal modeling branch employing a long short-term memory network is constructed to capture the dynamic changes in facial expressions; A fusion layer with an attention mechanism for adaptively adjusting the weights of the two types of features is constructed based on the feature extraction branch and the temporal modeling branch.

5. The deep learning neural network-guided video facial expression natural transfer method as described in claim 4, characterized in that, The feature extraction branch uses an improved convolutional neural network, specifically a depthwise separable convolutional network, and optimizes inference speed through model quantization compression technology.

6. The deep learning neural network-guided video facial expression natural transfer method as described in claim 4, characterized in that, The dual-branch deep learning architecture also includes a GAN generator branch built on generative adversarial networks, which is used to implement facial expression transfer and detail optimization.

7. The deep learning neural network-guided video facial expression natural transfer method as described in claim 1, characterized in that, The steps of training the dual-branch deep learning architecture using the training dataset and iteratively optimizing the network parameters through backpropagation during model training using a composite function of perceptual loss and adversarial loss include: The training dataset is divided into a training set, a validation set, and a test set. The model parameters of the dual-branch deep learning architecture are fitted based on the training set, and the model selection and hyperparameter tuning of the fitted dual-branch deep learning architecture are performed based on the validation set. During the model parameter fitting process, a composite function of perceptual loss and adversarial loss is used to backpropagate the dual-branch deep learning architecture to iteratively optimize the network parameters until the expression transfer naturalness score and identity feature similarity of the model verification of the validation set reach the preset standard. The generalization performance of the dual-branch deep learning architecture that has completed model parameter fitting is tested and evaluated based on the test set.

8. The deep learning neural network-guided video facial expression natural transfer method as described in claim 1, characterized in that, The steps for inference deployment of a two-branch deep learning architecture that has completed model training include: Save the model of the completed dual-branch deep learning architecture; Select a specified inference environment and inference framework for the dual-branch deep learning architecture, so as to deploy the dual-branch deep learning architecture in the inference environment through the inference framework.

9. A deep learning neural network-guided video facial expression natural transfer system, characterized in that, A method for implementing a deep learning neural network-guided natural video expression transfer as described in any one of claims 1-8 includes: The data acquisition module is used to collect multi-subject facial expression video samples and classify and label the multi-subject facial expression video samples to generate a training dataset. The architecture building module is used to build a two-branch deep learning architecture that includes a feature extraction branch and a temporal modeling branch; The model training module is used to train the dual-branch deep learning architecture using the training dataset, and to iteratively optimize the network parameters through backpropagation using a composite function of perceptual loss and adversarial loss during model training. The inference deployment module is used to perform inference deployment on a two-branch deep learning architecture that has completed model training.

Images