Facial expression prediction method, device, medium and system

Abstract

The invention provides a face expression prediction method and device, a medium and a system. Compared with an existing scheme, the expression prediction precision and reality can be greatly improved through a stable diffusion model such as a graph generation and graph parameter prediction model and a graph generation and graph model, after multiple frames of expression images are generated, a high-quality prediction image is screened out in a timestamp matching and similarity screening mode, and the expression prediction accuracy is improved. According to the method, the image closest to the real expression change can be automatically selected, excessive dependence on a time sequence relation in a traditional scheme is avoided, the requirement of manual labeling is reduced, and therefore the problem that face expression prediction in an existing scheme cannot be applied to a complex scene due to the fact that the requirement for the quality of labeling data is high is solved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, medium and system for predicting facial expressions. Background Art

[0002] Traditional facial expression prediction methods usually rely on key point detection and parameter prediction. Although such methods can capture basic expression changes, they often lack detail processing and naturalness. In addition, the process of manually annotating a large amount of expression data is time-consuming and laborious, and it is difficult to meet the needs of large-scale training data.

[0003] The current mainstream solutions rely heavily on data sets, especially the high quality requirements of labeled data. The model captures changes in facial expressions by learning the temporal relationship in the labeled data. However, labeled data is often obtained through manual or semi-automatic means, and it is inevitable that there will be noise or inconsistency, which causes the model to be affected by these inaccurate data during training. Especially when the amount of labeled data is limited or the quality is not high, the generalization ability of the model may be significantly limited, making it difficult to accurately handle diverse expressions and complex scenes.

[0004] The current mainstream solutions are highly dependent on the changing relationship between time-series image frames. When encountering faces with expression change patterns inconsistent with those in the training data or faces that have never been seen before, the model's prediction accuracy may drop significantly. This robustness problem is particularly prominent in out-of-set tests (such as new faces or extreme expressions), resulting in the model being unable to adapt to a variety of practical application scenarios.

[0005] The current mainstream solutions also have shortcomings in processing details and naturalness. Although they can capture basic changes in facial expressions based on key points or simple parameterized models, the generated facial expressions often appear stiff and unnatural in detail processing because the models learn fixed temporal change patterns. For example, in scenes such as subtle facial muscle movements and complex expression changes (such as mixed emotional expressions), the results generated by the model may not truly reflect these details, resulting in the lack of naturalness and authenticity in the predicted expressions.

[0006] That is, the existing solutions for facial expression prediction have high requirements on the quality of labeled data and thus cannot be applied to complex scenarios. Summary of the invention

[0007] The main purpose of the present application is to provide a method, device, medium and system for predicting facial expressions, so as to at least solve the problem that the facial expression prediction of existing solutions has high requirements on the quality of labeled data and thus cannot be applied to complex scenes.

[0008] In order to achieve the above object, according to one aspect of the present application, a method for predicting facial expressions is provided, the method comprising:

[0009] Acquire an initial face image set, and preprocess the initial face image set to obtain a preprocessed face image set, wherein the initial face image set includes a plurality of frames of initial face images, and the preprocessed face image set includes a plurality of frames of preprocessed face images, and the preprocessing includes detecting and aligning face regions in the initial face images;

[0010] Using an image-generated image parameter prediction model to process the preprocessed face image set to obtain image-generated image parameters, and using an image-generated image model to process the image-generated image parameters and the preprocessed face image set to generate a predicted image set, wherein the image-generated image parameters include a noise level and a number of diffusion steps;

[0011] The predicted image set and the initial face image set are compared frame by frame for similarity, and a facial expression prediction result is determined to be an image in the predicted image set corresponding to a target image similarity, wherein the target image similarity represents an image similarity greater than or equal to a similarity threshold.

[0012] Optionally, before acquiring the initial face image set, the method further includes:

[0013] Acquire a first training image set, and use the image-generated image model to process a preset part of the first training image set to obtain a second training image set;

[0014] Based on the timestamp, the second training image set is compared frame by frame with the image frames after the preset part in the first training image set to obtain a plurality of training image similarities and training image parameters;

[0015] Determining a construction image similarity to be a training image similarity greater than or equal to the similarity threshold;

[0016] The image-derived image parameter prediction model is constructed by using all target training images and the image-derived image parameters of the training images, wherein the target training images are training images of the second training image set corresponding to the similarity with the construction image.

[0017] Optionally, based on the timestamp, the second training image set is compared frame by frame with the image frames after the preset part in the first training image set to obtain a plurality of training image similarities and training image parameters, including:

[0018] First, the second training image set is timestamped and matched with the image frames after the preset part in the first training image set, and then the second training image set is compared with the image frames after the preset part in the first training image set frame by frame using an image similarity algorithm to obtain multiple training image similarities and training image generation parameters.

[0019] Optionally, all target training images and the training image parameters are used to construct the image parameter prediction model, including:

[0020] Constructing a deep learning model, wherein the input of the deep learning model is the target training image and the training image-generated image parameters, and the output of the deep learning model is the image-generated image parameters;

[0021] Adopting multiple loss functions to optimize the deep learning model, wherein the loss values ​​of the multiple loss functions include image reconstruction loss, perception loss, and similarity loss;

[0022] The deep learning model is trained using all target training images and the training graph parameters, and the parameters of the deep learning model are adjusted using a gradient descent optimization method to obtain the graph parameter prediction model.

[0023] Optionally, the deep learning model includes a convolutional neural network or a Transformer architecture.

[0024] Optionally, comparing the similarity of each image in the predicted image set and the initial face image set frame by frame includes:

[0025] First, the predicted image set and the initial face image set are subjected to time alignment processing, and then an image similarity algorithm is used to compare the similarity of each image in the predicted image set and the initial face image set.

[0026] Optionally, after comparing the similarity of each image in the predicted image set and the initial face image set frame by frame and determining that the facial expression prediction result is an image in the predicted image set corresponding to the similarity with the target image, the method further comprises:

[0027] The images in the facial expression prediction result are sorted in the order of timestamps to generate a final video or a final image sequence, and the final video or the final image sequence is applied to a preset application scenario, wherein the preset application scenario includes animation production and virtual reality.

[0028] According to another aspect of the present application, a facial expression prediction device is provided, comprising:

[0029] A first acquisition unit is used to acquire an initial face image set, and preprocess the initial face image set to obtain a preprocessed face image set, wherein the initial face image set includes a plurality of frames of initial face images, the preprocessed face image set includes a plurality of frames of preprocessed face images, and the preprocessing includes detecting and aligning face regions in the initial face images;

[0030] a first processing unit, configured to process the preprocessed face image set using a graph-generated image parameter prediction model to obtain graph-generated image parameters, and to process the graph-generated image parameters and the preprocessed face image set using a graph-generated image model to generate a predicted image set, wherein the graph-generated image parameters include a noise level and a number of diffusion steps;

[0031] The second processing unit is used to compare the similarity of each image in the predicted image set and the initial face image set frame by frame, and determine that the facial expression prediction result is an image in the predicted image set corresponding to the target image similarity, and the target image similarity represents the image similarity greater than or equal to a similarity threshold.

[0032] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0033] According to another aspect of the present application, a facial expression prediction system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the methods described.

[0034] By applying the technical solution of the present application, the accuracy and realism of expression prediction can be greatly improved compared with existing solutions through stable diffusion models such as image-derived image parameter prediction models and image-derived image models. After generating multiple frames of expression images, it is proposed to screen out high-quality predicted images through timestamp matching and similarity screening, and can automatically select images that are closest to real expression changes, avoiding excessive reliance on temporal relationships in traditional solutions and reducing the need for manual labeling, thereby solving the problem that the existing solutions for facial expression prediction have high requirements on the quality of labeled data and cannot be applied to complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings constituting part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0036] Figure 1 A schematic diagram of a process of predicting a facial expression provided according to an embodiment of the present application is shown;

[0037] Figure 2 A schematic diagram showing a flow chart of another method for predicting facial expressions provided according to an embodiment of the present application;

[0038] Figure 3 A structural block diagram of a facial expression prediction device provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0039] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0040] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0042] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0043] The implementation principle of the image-generated parameter prediction model is based on a framework that combines deep learning with a diffusion model. Specifically, the model extracts a high-dimensional representation vector from the input image through a feature extraction network to capture key expression features and fine-grained semantic information. Subsequently, the representation vector is mapped to the generation parameter space required by the diffusion model (such as noise scheduling coefficients, diffusion time steps, etc.) using a parameter regression module. These prediction parameters are used to guide the denoising dynamics in the generation process through the optimized diffusion model to generate the target expression image. The core of the model is to effectively improve the accuracy of parameter prediction through end-to-end mapping learning between features and parameters, and ensure that the expression details of the generated image have high fidelity and consistency. The training principle of the model is based on supervised learning. By constructing a paired dataset containing input images and corresponding target generation parameters, a deep neural network is used to learn the mapping relationship between image features and target parameters. The model first encodes the input image through a feature extraction network to generate a high-dimensional feature representation; then, the feature is mapped to a target parameter vector through a parameter prediction module. During the training process, the difference between the predicted parameters and the true parameters is calculated by defining a loss function (such as mean square error), and the model weights are updated through back propagation and iterative optimization, so that the prediction results gradually approach the true target, and ultimately achieve accurate parameter prediction capabilities.

[0044] The Transformer architecture is a deep learning model based on the self-attention mechanism. It is mainly composed of an encoder and a decoder, and is used to process sequence-to-sequence tasks. The encoder consists of a multi-layer self-attention mechanism and a feed-forward neural network. The self-attention mechanism is used to capture the global context information in the input sequence, and the residual connection and layer normalization are used to improve the training stability. The decoder structure is similar to the encoder, but an encoder-decoder attention mechanism is additionally introduced to process the interaction between input and output. Transformer realizes multi-view feature extraction through a multi-head attention mechanism, and relies on positional encoding to retain the position information of the sequence.

[0045] SSIM (Structural Similarity Index) is an indicator to measure image similarity, mainly used to evaluate the difference in perceived quality between two images. SSIM is based on the characteristics of the human visual system and calculates similarity by simultaneously comparing the brightness (Luminance), contrast (Contrast) and structure (Structure) information of the image. Its core principle is to divide the image into blocks, calculate the similarity of the above three components for each small block, and then comprehensively weight them to obtain the overall SSIM value, ranging from -1 to 1, where 1 means that the two images are exactly the same. SSIM uses the mean and standard deviation in the calculation of brightness and contrast, and the structural information is measured by the normalized covariance. Compared with traditional pixel differences (such as mean square error), it can better reflect the visual quality of the image and is a commonly used evaluation method in the fields of image compression, denoising and enhancement.

[0046] LPIPS (Learned Perceptual Image Patch Similarity) is a perceptual image similarity algorithm based on deep learning, which is used to measure the visual difference between two images. Its core idea is to use a pre-trained deep neural network (such as VGG or AlexNet) to extract high-dimensional features of the image, calculate the differences in feature maps at multiple network layers, and perform weighted summation of these differences combined with perceptual weights to obtain the final similarity score.

[0047] Stable Diffusion is a generative image generation algorithm based on a diffusion model, focusing on gradually generating high-quality images from random noise. Its principle is to gradually remove noise from the image through a reverse diffusion process, and finally generate a clear image. During the training phase, Stable Diffusion learns how to capture latent semantic and visual features from a large number of image-text pairs, realizes noise prediction through the U-Net architecture combined with time-step embedding, and optimizes computational efficiency through the latent space. Compared with the traditional diffusion model, the algorithm confines the generation process to a more compact latent space, significantly reducing the computational complexity, while supporting conditional control (such as text or reference images) to customize the generated content.

[0048] MTCNN (Multi-task Cascaded Convolutional Neural Network) is a multi-task cascaded convolutional neural network for face detection and key point location. It consists of three stages of networks: P-Net (Proposal Network) generates candidate face frames and makes rough adjustments, R-Net (Refine Network) further filters false detections and refines bounding boxes, and O-Net (Output Network) regresses the final face frame and predicts the key point locations (such as eyes, nose, and mouth). While detecting faces, each network completes multi-task learning of bounding box regression and five key point location, thereby improving detection accuracy and efficiency.

[0049] RetinaFace is a high-performance face detection algorithm based on a single-stage target detection architecture, designed for accurate face detection and key point location. The algorithm is improved on the basis of RetinaNet, combined with a multi-task learning framework, integrating face detection, five key point location (such as eyes, nose, mouth) and facial posture estimation into a single network. RetinaFace enhances the detection capability of small-sized faces by utilizing the deep feature pyramid network (FPN) and context information module, and further improves the positioning accuracy using pixel-wise supervision strategy.

[0050] As introduced in the background technology, traditional facial expression prediction methods usually rely on key point detection and parameter prediction. Although such methods can capture basic expression changes, they often have deficiencies in detail processing and naturalness. In addition, the process of manually annotating a large amount of expression data is time-consuming and laborious, and it is difficult to meet the needs of large-scale training data. The current mainstream solutions rely heavily on data sets, especially high requirements for the quality of annotated data. The model captures changes in facial expressions by learning the temporal relationship in the annotated data. However, the annotated data is often obtained by manual or semi-automatic means, and it is inevitable that there will be noise or inconsistency, which causes the model to be affected by these inaccurate data during training. Especially when the amount of annotated data is limited or the quality is not high, the generalization ability of the model may be significantly limited, and it is difficult to accurately process diverse expressions and complex scenes. In order to solve the problem that the facial expression prediction of the existing solution has high requirements for the quality of the annotated data, and thus cannot be applied to complex scenes, the embodiments of the present application provide a method, device, medium and system for predicting facial expressions.

[0051] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0052] In this embodiment, a method for predicting facial expressions is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0053] Figure 1 FIG. 1 is a flow chart of a method for predicting facial expressions according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0054] Step S101, obtaining an initial face image set, and preprocessing the initial face image set to obtain a preprocessed face image set, wherein the initial face image set includes a plurality of frames of initial face images, and the preprocessed face image set includes a plurality of frames of preprocessed face images, and the preprocessing includes detecting and aligning face regions in the initial face images;

[0055] In one embodiment of the present application, before obtaining the initial face image set in step S101, the method further includes:

[0056] Obtain a first training image set, and use the above-mentioned image-generated image model to process a preset part of the above-mentioned first training image set to obtain a second training image set;

[0057] The first training image set is similar to the initial face image set and also includes multiple frames of face images.

[0058] Based on the timestamp, the second training image set is compared frame by frame with the image frames after the preset part in the first training image set to obtain a plurality of training image similarities and training image generation parameters;

[0059] Specifically, firstly, the second training image set is timestamped and matched with the image frames after the preset part in the first training image set, and then the second training image set is compared frame by frame with the image frames after the preset part in the first training image set using an image similarity algorithm to obtain multiple training image similarities and training image generation parameters.

[0060] Determine that the construction image similarity is greater than or equal to the training image similarity of the similarity threshold;

[0061] The above-mentioned image-derived image parameter prediction model is constructed by using all target training images and the above-mentioned training image-derived image parameters, and the above-mentioned target training images are training images of the above-mentioned second training image set corresponding to the similarity with the above-mentioned construction image.

[0062] Specifically, a deep learning model is constructed, the input of the deep learning model is the target training image and the image parameters of the training image, and the output of the deep learning model is the image parameters of the image; multiple loss functions are used to optimize the deep learning model, and the loss values ​​of the multiple loss functions include image reconstruction loss, perceptual loss and similarity loss; all target training images and the image parameters of the training image are used to train the deep learning model, and the parameters of the deep learning model are adjusted by the gradient descent optimization method to obtain the image parameter prediction model.

[0063] In one embodiment of the present application, the above-mentioned deep learning model includes a convolutional neural network or a Transformer architecture.

[0064] The specific training process is as follows:

[0065] The first step is to collect a large-scale and diverse face video dataset to ensure coverage of different genders, ages and emotional expressions. The dataset contains various expression changes, different lighting conditions, perspective changes and diversity of background complexity. The video data is split into frames, and the face image in each frame is extracted and standardized (such as resizing and removing the background). Timestamp information is added to each frame image for use in subsequent timing matching and similarity calculations.

[0066] In the second step, a reference image is selected from the video sequence (for example, starting from the initial frame of the video or the starting frame of a pre-selected expression change, that is, the first frame of the preset part), and a pre-trained stable diffusion model (that is, a graph-generated image model) is used to conditionally input the reference image to gradually generate subsequent multi-frame expression images. The generated image sequence simulates the expression changes of the reference image in the time dimension, and the corresponding graph-generated image model parameters are recorded during the generation process of each frame. According to the specified rules (such as the rule of extracting one frame every 5 frames), the image frames in the video sequence are extracted as the reference images, and this step is repeatedly performed to generate a large number of multi-frame expression images (that is, the second training image set).

[0067] The third step is to timestamp match the generated multi-frame expression image sequence with the actual subsequent frames in the original video to ensure the consistency of the images in time sequence. Use image similarity algorithms (such as SSIM and LPIPS) to compare the similarity between the generated expression images and the matched actual video frame images. Only those generated images whose similarity with the actual frames is higher than the similarity threshold (for example, 80%) are retained, and those images with low similarity are screened out. These retained high-similarity image frames will be used for subsequent model training.

[0068] The fourth step is to use the retained high-similarity image frames and their corresponding graph-based model parameters to construct a new training dataset, the input of which is the reference image or its features (such as facial key points, facial area features), and the output is the graph-based parameters required to generate subsequent expression images; set up a deep learning model, which inputs the features of the reference image and outputs the predicted graph-based parameters, and can use architectures such as convolutional neural networks (CNN) or Transformer to process image features; use multiple loss functions to optimize the model, including image reconstruction loss (measuring the pixel difference between the generated image and the target image), perceptual loss (comparing the difference between the generated image and the target image in a high-level feature space), and similarity loss (ensuring the similarity between the predicted image and the real frame); use the generated training dataset to train the model, adjust the model parameters through the gradient descent optimization method, so that it can accurately predict the graph-based parameters used to generate subsequent frames, and finally train a graph-based parameter prediction model.

[0069] Step S102, using an image-generated image parameter prediction model to process the pre-processed face image set to obtain image-generated image parameters, and using an image-generated image model to process the image-generated image parameters and the pre-processed face image set to generate a predicted image set, wherein the image-generated image parameters include a noise level and a number of diffusion steps;

[0070] Specifically, the user inputs a reference face image or an initial frame of a video. The reference image will serve as the basis for generating subsequent expression images. The face detection algorithm (such as MTCNN, RetinaFace) is used to detect and align the face area in the input image to ensure that the position and posture of the face in different frames are consistent, reducing the error caused by the change of face position during the reasoning process. The aligned face image will pass through a feature extractor (such as a pre-trained CNN or Transformer model) to extract the key features of the face, which will be used as input for subsequent image parameter prediction.

[0071] The features of the reference image are input into the trained graph-based parameter prediction model. The model generates graph-based parameters for the diffusion model based on the input features. These graph-based parameters include key information required for the diffusion model to generate expression images, such as noise level and number of diffusion steps. Parameter generation is based on the rules learned during training to predict the generation parameters required for the image after a given timestamp.

[0072] The predicted image parameters are fed into the image-generated model, which initializes the noise to a random image. The image will be gradually denoised to generate a more realistic expression image. The diffusion model gradually generates the next few frames of expression images through the reverse diffusion process. Starting from the baseline image, the subsequent multi-frame expression images are derived by frame-by-frame generation. The generation of each frame depends on the image result of the previous frame and the current image parameters, ensuring that the generated expression sequence has temporal coherence and natural expression changes.

[0073] Step S103, comparing the similarity of each image in the above-mentioned predicted image set and the above-mentioned initial face image set frame by frame, determining that the facial expression prediction result is an image in the above-mentioned predicted image set corresponding to the similarity of the target image, and the above-mentioned target image similarity represents the above-mentioned image similarity greater than or equal to the similarity threshold.

[0074] In the above steps, the stable diffusion models such as the graph-generated graph parameter prediction model and the graph-generated graph model can greatly improve the accuracy and realism of expression prediction compared with the existing schemes. After generating multiple frames of expression images, it is proposed to screen out high-quality prediction images through timestamp matching and similarity screening, which can automatically select the image closest to the real expression changes, avoiding the excessive reliance on temporal relationships in traditional schemes and reducing the need for manual labeling, thereby solving the problem that the existing schemes for facial expression prediction have high requirements on the quality of labeled data and cannot be applied to complex scenes.

[0075] In one embodiment of the present application, the similarity of each image in the above-mentioned predicted image set and the above-mentioned initial face image set is compared frame by frame, including: firstly performing time alignment processing on the above-mentioned predicted image set and the above-mentioned initial face image set, and then using an image similarity algorithm to compare the similarity of each image in the above-mentioned predicted image set and the above-mentioned initial face image set.

[0076] Specifically, the generated multi-frame expression images will be assigned timestamp information and time-aligned with the subsequent frames of the original video frames. This process ensures the temporal consistency between the generated images and the actual video frames. The generated expression images are compared with the corresponding actual video frames using image similarity algorithms (such as SSIM and LPIPS). The system will retain images with a similarity higher than a preset threshold and discard images with a lower similarity. These selected high-quality frames ensure that the generated expression sequence not only conforms to the logic of the time series, but is also highly consistent with the actual expression changes.

[0077] Based on the original facial expression image frame as the reference image, by gradually generating images in the high-dimensional data space, it is possible to capture complex expression changes and details and generate high-quality expression images. Compared with the traditional methods based on key points or parameterized models, the diffusion model can generate more natural and delicate expressions, and can handle subtle facial muscle movements and complex expressions (such as mixed emotional expressions), which can greatly improve the accuracy and realism of expression prediction; after generating multiple frames of expression images, it is proposed to screen out high-quality predicted images through timestamp matching and similarity screening. This method can automatically select the image that is closest to the real expression change, avoiding the traditional scheme. The excessive reliance on temporal relationships solves the problem of insufficient model generalization due to low quality or noisy training data. This process can improve the reliability of prediction results and reduce the need for manual annotation. The idea of ​​transferring expression prediction from traditional facial image-related parameters to graph-based parameters is introduced, which means that the model no longer directly predicts specific facial feature points or expression parameters, but indirectly generates expression images by predicting the diffusion model parameters of the generated image. The input of this model is the reference image or its related features (such as facial key points, feature representation of facial regions, etc.), and the output is the graph-based parameters provided to the graph-based model for generating the next few frames of expression. This method can effectively decouple the direct connection between the prediction process and the original image, making the model more robust when facing different faces or expression changes, and able to adapt to more diverse input conditions.

[0078] In one embodiment of the present application, after comparing the similarity of the above-mentioned predicted image set and each image in the above-mentioned initial face image set frame by frame, and determining that the facial expression prediction result is an image in the above-mentioned predicted image set corresponding to the similarity of the target image, the above-mentioned method also includes: sorting each image in the above-mentioned facial expression prediction result in the order of timestamps to generate a final video or a final image sequence, and applying the above-mentioned final video or the above-mentioned final image sequence to a preset application scenario, and the above-mentioned preset application scenario includes animation production and virtual reality.

[0079] Specifically, the screened expression image sequences are arranged in chronological order to generate a complete expression change video or a continuous image frame sequence. The generated video or image sequence can be directly output for application scenarios (such as animation production, virtual reality). In some application scenarios, the generated expression sequence can be superimposed on the original video or saved as an independent output image sequence.

[0080] Existing solutions usually rely on expression prediction methods based on facial key points or parameterized models, which generate images of subsequent frames by learning fixed expression change patterns. Although this method can capture basic expression changes, it is often insufficient when dealing with complex expression details (such as subtle facial muscle movements, mixed emotional expressions, etc.), and the generated images may lack naturalness and realism. However, this solution introduces a stable diffusion model (StableDiffusion) to gradually generate expression images in a high-dimensional data space, which can capture more complex and detailed expression changes. This makes the new solution far superior to existing solutions in terms of generated image quality, especially when dealing with subtle facial muscle movements and complex expressions, it can generate more natural and delicate expression images, thereby greatly improving the accuracy and realism of expression prediction.

[0081] Existing solutions rely on the temporal relationship between video frames. The model predicts the expression of subsequent frames by learning the temporal changes in the labeled data. This method has a high dependence on data quality and timing, and is easily affected by noise or inconsistency in the training data, resulting in insufficient model generalization ability, especially when encountering new faces or unseen complex expressions. The reliability of the prediction results may decrease. This solution automatically selects the image closest to the real expression change through timestamp matching and similarity screening mechanism, reducing the dependence on temporal relationship. This not only improves the accuracy and reliability of the prediction results, but also reduces the need for manually labeled data, solving the defect of insufficient model generalization ability caused by data quality problems.

[0082] Existing solutions directly predict feature points or expression parameters related to face images. This method is highly dependent on specific facial features or expression patterns, which results in insufficient robustness when the model processes new faces or unseen expression changes, and may result in inaccurate predictions. In particular, in out-of-set tests or complex environments, the performance of existing solutions is often not ideal. This solution shifts expression prediction from direct feature points or expression parameters to the prediction of image-generated image parameters, and indirectly generates expression images by predicting the generation parameters of the diffusion model. This method effectively decouples the direct connection between the prediction process and the original image, making the model more robust when facing diverse input conditions (such as different faces, complex expressions), and better able to adapt to diverse scenarios, significantly improving the adaptability and generalization ability of the model.

[0083] Application scenarios of this application:

[0084] Expression enhancement and restoration: In some scenarios, the original video or image may have blurred or distorted expression details due to poor shooting conditions (such as poor lighting and low resolution). This solution can enhance or restore the original image by generating higher quality and more detailed expression image frames, which can be used for video post-processing and restoration of old photos or videos.

[0085] Expression interpolation and expansion: If the expressions in the original video change too quickly, resulting in the key expressions not being captured clearly, this solution can generate intermediate frames through interpolation technology to fill in the gaps in the expression change process and provide a smoother transition, which is especially useful when making slow-motion expression animations or videos.

[0086] Expression frame preprocessing: In processes such as real-time video stream processing, it may be necessary to predict the expression images of the next few frames of the face in advance so that preprocessing can be performed in advance (such as real-time face replacement) to increase the processing speed of the real-time video stream and thus ensure real-time performance.

[0087] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the method for predicting facial expressions of the present application will be described in detail below in conjunction with specific embodiments.

[0088] This embodiment relates to a specific method for predicting facial expressions. Figure 2 As shown, the following steps are included:

[0089] Obtain a first training image set, and use a graph-based graph model to process a preset portion of the first training image set to obtain a second training image set;

[0090] Firstly, the second training image set is matched with the image frames after the preset part in the first training image set by time stamp, and then the image frames after the preset part in the second training image set and the first training image set are compared frame by frame using an image similarity algorithm to obtain a plurality of training image similarities and training image parameters;

[0091] Determine that the construction image similarity is greater than or equal to the training image similarity threshold;

[0092] Build a deep learning model, the input of the deep learning model is the target training image and the training image parameters, and the output of the deep learning model is the image parameters;

[0093] Multiple loss functions are used to optimize the deep learning model. The loss values ​​of the multiple loss functions include image reconstruction loss, perception loss, and similarity loss.

[0094] The deep learning model is trained using all target training images and training graph-derived image parameters, and the parameters of the deep learning model are adjusted using a gradient descent optimization method to obtain a graph-derived image parameter prediction model, wherein the target training image is a training image of a second training image set corresponding to a similarity with the construction image;

[0095] Acquire an initial face image set, and preprocess the initial face image set to obtain a preprocessed face image set, wherein the initial face image set includes multiple frames of initial face images, and the preprocessed face image set includes multiple frames of preprocessed face images, and the preprocessing includes detecting and aligning face regions in the initial face images;

[0096] The pre-processed face image set is processed by using a graph-generated graph parameter prediction model to obtain graph-generated graph parameters, and the graph-generated graph model is used to process the graph-generated graph parameters and the pre-processed face image set to generate a predicted image set, wherein the graph-generated graph parameters include a noise level and a number of diffusion steps;

[0097] First, the predicted image set and the initial face image set are time-aligned, and then the image similarity algorithm is used to compare the similarity of each image in the predicted image set and the initial face image set, and the facial expression prediction result is determined to be an image in the predicted image set corresponding to the target image similarity. The target image similarity represents the image similarity greater than or equal to the similarity threshold.

[0098] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0099] The embodiment of the present application also provides a prediction device for facial expressions. It should be noted that the prediction device for facial expressions in the embodiment of the present application can be used to execute the prediction method for facial expressions provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0100] The following is an introduction to the facial expression prediction device provided in the embodiments of the present application.

[0101] Figure 3 is a structural block diagram of a facial expression prediction device provided according to an embodiment of the present application. Figure 3 As shown, the device comprises:

[0102] A first acquisition unit 31 is used to acquire an initial face image set, and preprocess the initial face image set to obtain a preprocessed face image set, wherein the initial face image set includes a plurality of frames of initial face images, and the preprocessed face image set includes a plurality of frames of preprocessed face images, and the preprocessing includes detecting and aligning face regions in the initial face images;

[0103] A first processing unit 32 is used to process the pre-processed face image set using a graph-generated image parameter prediction model to obtain graph-generated image parameters, and to process the graph-generated image parameters and the pre-processed face image set using a graph-generated image model to generate a predicted image set, wherein the graph-generated image parameters include a noise level and a number of diffusion steps;

[0104] The second processing unit 33 is used to compare the similarity of each image in the above-mentioned predicted image set and the above-mentioned initial face image set frame by frame, and determine that the facial expression prediction result is an image in the above-mentioned predicted image set corresponding to the similarity of the target image, and the above-mentioned target image similarity represents the above-mentioned image similarity greater than or equal to the similarity threshold.

[0105] In the above-mentioned device, the accuracy and realism of expression prediction can be greatly improved compared with the existing schemes through stable diffusion models such as the image-generated image parameter prediction model and the image-generated image model. After generating multiple frames of expression images, it is proposed to screen out high-quality prediction images through timestamp matching and similarity screening, and can automatically select the image closest to the real expression change, avoiding the excessive reliance on temporal relationships in traditional schemes and reducing the need for manual labeling, thereby solving the problem that the existing schemes for facial expression prediction have high requirements on the quality of labeled data and cannot be applied to complex scenes.

[0106] In one embodiment of the present application, the above-mentioned device also includes a second acquisition unit, a third processing unit, a fourth processing unit and a fifth processing unit. The second acquisition unit is used to acquire a first training image set before acquiring an initial face image set, and use the above-mentioned graph-generated image model to process a preset part of the above-mentioned first training image set to obtain a second training image set; the third processing unit is used to compare the above-mentioned second training image set with the image frame after the preset part in the above-mentioned first training image set frame by frame based on the timestamp to obtain multiple training image similarities and training graph-generated image parameters; the fourth processing unit is used to determine that the similarity of the construction image is greater than or equal to the similarity threshold of the above-mentioned training image similarity; the fifth processing unit is used to use all target training images and the above-mentioned training graph-generated image parameters to construct the above-mentioned graph-generated image parameter prediction model, and the above-mentioned target training image is a training image of the above-mentioned second training image set corresponding to the above-mentioned construction image similarity.

[0107] In one embodiment of the present application, the third processing unit includes a first processing module, which is used to timestamp match the second training image set with the image frames after the preset part in the first training image set, and then use the image similarity algorithm to compare the second training image set with the image frames after the preset part in the first training image set frame by frame to obtain multiple training image similarities and the training image generation parameters.

[0108] In one embodiment of the present application, the fifth processing unit includes a construction module, a second processing module and a third processing module, the construction module is used to construct a deep learning model, the input of the above-mentioned deep learning model is the above-mentioned target training image and the above-mentioned training image parameters, and the output of the above-mentioned deep learning model is the image parameters; the second processing module is used to optimize the above-mentioned deep learning model using multiple loss functions, and the loss values ​​of the above-mentioned multiple loss functions include image reconstruction loss, perceptual loss and similarity loss; the third processing module is used to train the above-mentioned deep learning model using all target training images and the above-mentioned training image parameters, and adjust the parameters of the above-mentioned deep learning model using the gradient descent optimization method to obtain the above-mentioned image parameter prediction model.

[0109] In one embodiment of the present application, the above-mentioned deep learning model includes a convolutional neural network or a Transformer architecture.

[0110] In one embodiment of the present application, the second processing unit includes a fourth processing module, which is used to first perform time alignment processing on the above-mentioned predicted image set and the above-mentioned initial face image set, and then use an image similarity algorithm to compare the similarity of each of the above-mentioned images in the above-mentioned predicted image set and the above-mentioned initial face image set.

[0111] In one embodiment of the present application, the above-mentioned device also includes a generation unit, which is used to compare the similarity of the above-mentioned predicted image set and each image in the above-mentioned initial face image set frame by frame, determine that the facial expression prediction result is an image in the above-mentioned predicted image set corresponding to the similarity of the target image, and then sort the images in the above-mentioned facial expression prediction result in the order of timestamps to generate a final video or a final image sequence, and apply the above-mentioned final video or the above-mentioned final image sequence to a preset application scenario, and the above-mentioned preset application scenario includes animation production and virtual reality.

[0112] The above-mentioned facial expression prediction device includes a processor and a memory, and the above-mentioned first acquisition unit, the first processing unit, the second processing unit, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in the form of any combination.

[0113] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to solve the problem that the existing solution of facial expression prediction has high requirements on the quality of labeled data, so it cannot be applied to complex scenes.

[0114] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0115] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for predicting facial expressions.

[0116] An embodiment of the present invention provides a processor, and the processor is used to run a program, wherein the facial expression prediction method is executed when the program is run.

[0117] The embodiment of the present invention provides a device, the device includes a processor, a memory and a program stored in the memory and can be run on the processor, and when the processor executes the program, at least the following steps are implemented: obtaining an initial face image set, and preprocessing the initial face image set to obtain a preprocessed face image set, the initial face image set includes multiple frames of initial face images, the preprocessed face image set includes multiple frames of preprocessed face images, and the preprocessing includes detecting and aligning the face area in the initial face image; using a graph-generated image parameter prediction model to process the preprocessed face image set to obtain graph-generated image parameters, and using a graph-generated image model to process the graph-generated image parameters and the preprocessed face image set to generate a predicted image set, the graph-generated image parameters include a noise level and a number of diffusion steps; comparing the similarity of each image in the predicted image set and the initial face image set frame by frame, determining that the facial expression prediction result is an image in the predicted image set corresponding to the target image similarity, and the target image similarity represents the similarity of the images greater than or equal to the similarity threshold. The device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0118] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with at least the following method steps: obtaining an initial face image set, and preprocessing the initial face image set to obtain a preprocessed face image set, wherein the initial face image set includes multiple frames of initial face images, and the preprocessed face image set includes multiple frames of preprocessed face images, and the preprocessing includes detecting and aligning face areas in the initial face images; using an image-generated image parameter prediction model to process the preprocessed face image set to obtain image-generated image parameters, and using an image-generated image model to process the image-generated image parameters and the preprocessed face image set to generate a predicted image set, wherein the image-generated image parameters include a noise level and a number of diffusion steps; comparing the similarity of the predicted image set and each image in the initial face image set frame by frame, and determining that a facial expression prediction result is an image in the predicted image set corresponding to a target image similarity, and the target image similarity represents the image similarity that is greater than or equal to a similarity threshold.

[0119] The present application also provides a facial expression prediction system, including: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any of the above methods. Compared with existing solutions, the accuracy and realism of expression prediction can be greatly improved by using stable diffusion models such as graph-generated graph parameter prediction models and graph-generated graph models. After generating multiple frames of expression images, it is proposed to screen out high-quality predicted images by timestamp matching and similarity screening, which can automatically select images that are closest to real expression changes, avoid excessive reliance on temporal relationships in traditional solutions, and reduce the need for manual annotation, thereby solving the problem that the existing solutions for facial expression prediction have high requirements for the quality of annotation data and cannot be applied to complex scenes.

[0120] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0121] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0122] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0123] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0125] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0126] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0127] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0128] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0129] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0130] 1) The facial expression prediction method of the present application can greatly improve the accuracy and realism of expression prediction compared with existing solutions through stable diffusion models such as image-generated image parameter prediction models and image-generated image models. After generating multiple frames of expression images, it is proposed to screen out high-quality predicted images through timestamp matching and similarity screening, and can automatically select images that are closest to real expression changes, avoiding excessive reliance on temporal relationships in traditional solutions and reducing the need for manual labeling, thereby solving the problem that the existing solutions for facial expression prediction have high requirements on the quality of labeled data and cannot be applied to complex scenes.

[0131] 2) The facial expression prediction device of the present application can greatly improve the accuracy and realism of expression prediction compared with existing solutions through stable diffusion models such as image-generated image parameter prediction models and image-generated image models. After generating multiple frames of expression images, it is proposed to screen out high-quality predicted images through timestamp matching and similarity screening, and can automatically select images that are closest to real expression changes, avoiding excessive reliance on temporal relationships in traditional solutions and reducing the need for manual labeling, thereby solving the problem that the existing solutions for facial expression prediction have high requirements on the quality of labeled data and cannot be applied to complex scenes.

[0132] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for predicting facial expressions, characterized in that: include: Acquire an initial face image set, and preprocess the initial face image set to obtain a preprocessed face image set, wherein the initial face image set includes a plurality of frames of initial face images, and the preprocessed face image set includes a plurality of frames of preprocessed face images, and the preprocessing includes detecting and aligning face regions in the initial face images; Using an image-generated image parameter prediction model to process the preprocessed face image set to obtain image-generated image parameters, and using an image-generated image model to process the image-generated image parameters and the preprocessed face image set to generate a predicted image set, wherein the image-generated image parameters include a noise level and a number of diffusion steps; The predicted image set and the initial face image set are compared frame by frame for similarity, and a facial expression prediction result is determined to be an image in the predicted image set corresponding to a target image similarity, wherein the target image similarity represents an image similarity greater than or equal to a similarity threshold.

2. The method according to claim 1, characterized in that Before acquiring the initial face image set, the method further includes: Acquire a first training image set, and use the image-generated image model to process a preset part of the first training image set to obtain a second training image set; Based on the timestamp, the second training image set is compared frame by frame with the image frames after the preset part in the first training image set to obtain a plurality of training image similarities and training image parameters; Determining a construction image similarity to be a training image similarity greater than or equal to the similarity threshold; The image-derived image parameter prediction model is constructed by using all target training images and the image-derived image parameters of the training images, wherein the target training images are training images of the second training image set corresponding to the similarity with the construction image.

3. The method according to claim 2, characterized in that Based on the timestamp, the second training image set is compared frame by frame with the image frames after the preset part in the first training image set to obtain multiple training image similarities and training image parameters, including: First, the second training image set is timestamped and matched with the image frames after the preset part in the first training image set, and then the second training image set is compared with the image frames after the preset part in the first training image set frame by frame using an image similarity algorithm to obtain multiple training image similarities and training image generation parameters.

4. The method according to claim 2, characterized in that: Using all target training images and the training image parameters, constructing the image parameter prediction model, including: Constructing a deep learning model, wherein the input of the deep learning model is the target training image and the training image-generated image parameters, and the output of the deep learning model is the image-generated image parameters; Adopting multiple loss functions to optimize the deep learning model, wherein the loss values ​​of the multiple loss functions include image reconstruction loss, perception loss, and similarity loss; The deep learning model is trained using all target training images and the training graph parameters, and the parameters of the deep learning model are adjusted using a gradient descent optimization method to obtain the graph parameter prediction model.

5. The method according to claim 4, characterized in that The deep learning model includes a convolutional neural network or a Transformer architecture.

6. The method according to claim 1, characterized in that Comparing the similarity of each image in the predicted image set and the initial face image set frame by frame, comprising: First, the predicted image set and the initial face image set are subjected to time alignment processing, and then an image similarity algorithm is used to compare the similarity of each image in the predicted image set and the initial face image set.

7. The method according to any one of claims 1 to 6, characterized in that After comparing the similarity of each image in the predicted image set and the initial face image set frame by frame and determining that the facial expression prediction result is an image in the predicted image set corresponding to the similarity of the target image, the method further includes: The images in the facial expression prediction result are sorted in the order of timestamps to generate a final video or a final image sequence, and the final video or the final image sequence is applied to a preset application scenario, wherein the preset application scenario includes animation production and virtual reality.

8. A facial expression prediction device, characterized in that: include: A first acquisition unit is used to acquire an initial face image set, and preprocess the initial face image set to obtain a preprocessed face image set, wherein the initial face image set includes a plurality of frames of initial face images, the preprocessed face image set includes a plurality of frames of preprocessed face images, and the preprocessing includes detecting and aligning face regions in the initial face images; a first processing unit, configured to process the preprocessed face image set using a graph-generated image parameter prediction model to obtain graph-generated image parameters, and to process the graph-generated image parameters and the preprocessed face image set using a graph-generated image model to generate a predicted image set, wherein the graph-generated image parameters include a noise level and a number of diffusion steps; The second processing unit is used to compare the similarity of each image in the predicted image set and the initial face image set frame by frame, and determine that the facial expression prediction result is an image in the predicted image set corresponding to the target image similarity, and the target image similarity represents the image similarity greater than or equal to a similarity threshold.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A facial expression prediction system, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 7.

Images