Image processing model training, processing method, system, device and medium thereof
By training image processing models using graph convolutional neural networks and multi-level convolutional neural networks, and combining autoencoders and facial expression features, the problem of insufficient accuracy and robustness of existing voice-driven face models is solved, achieving higher accuracy and wider applicability of facial image generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUA DATA TECH (SHANGHAI) CO LTD
- Filing Date
- 2023-05-18
- Publication Date
- 2026-07-03
AI Technical Summary
Existing voice-driven face models have poor accuracy, robustness, and versatility, and cannot effectively cope with the effects of factors such as lighting and background.
By acquiring raw 3D facial data and sample audio data, facial and audio features are extracted. Image processing models are trained using graph convolutional neural networks and multi-level convolutional neural networks. By combining autoencoders and facial expression features, the model training process is optimized to improve accuracy and robustness.
It improves the accuracy, versatility, and robustness of image processing models, enabling them to better cope with the effects of lighting and background, and generate more accurate and vivid facial images.
Smart Images
Figure CN116580263B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing model training, processing method, system, device and medium thereof. Background Technology
[0002] With the widespread use of virtual avatars, making them more vivid and animated is a crucial task, in which voice-driven facial and image generation technologies play a key role.
[0003] Voice-driven face and image generation technologies currently primarily utilize deep learning methods. Based on different outputs of deep learning, voice-driven face technologies can be divided into two categories: one involves feature modeling of the facial surface mesh to generate high-dimensional features, and then using a deep model to generate corresponding high-dimensional feature weights from speech, thereby driving changes in the facial surface mesh. However, this feature modeling method mainly focuses on changes in the shape of the facial surface and cannot cope with the influence of other factors (such as lighting, background, etc.) on the face, resulting in models with low robustness and accuracy.
[0004] Another approach is to use a deep model to drive the positional changes of all facial surface grids through speech. However, there is currently no mature deep model, and existing deep models have poor accuracy, robustness, and versatility. Summary of the Invention
[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of poor accuracy, robustness and versatility of existing voice-driven face models, and to provide an image processing model training, processing method, system, device and medium.
[0006] The present invention solves the above-mentioned technical problems through the following technical solution:
[0007] The first aspect provides a training method for an image processing model, the training method comprising:
[0008] Acquire raw 3D facial data and several sample audio data;
[0009] Extract sample facial features from the original 3D facial data;
[0010] The sample facial features are used to characterize the node feature information of nodes in several facial regions and the relationship feature information of the topological relationship between different nodes;
[0011] Obtain the sample audio features of the sample audio data;
[0012] The facial features of the sample are adjusted based on the audio features of the sample to obtain the corresponding three-dimensional facial data of the sample;
[0013] The image processing model is obtained by training a preset network using the facial features and audio features of each sample as input and the corresponding three-dimensional facial data of the sample as output.
[0014] Preferably, the step of obtaining the audio features of the sample audio data includes:
[0015] The expressive features in the sample audio data are identified;
[0016] The expressive features are used to characterize the human voice features in the sample audio data;
[0017] The sample audio features are extracted from the expression features.
[0018] Preferably, the step of identifying the expressive features of the sample audio data includes:
[0019] The sample audio data is input into a preset audio recognition model, and the first output feature is output.
[0020] The first output feature includes spectral features and / or semantic features, and the first output feature is used as the expression feature;
[0021] And / or,
[0022] The step of extracting the sample audio features from the expression features includes:
[0023] The expressed features are input into a preset audio feature extraction model, and a second output feature is output.
[0024] The second output feature includes at least one of frequency, amplitude, and resonance peak;
[0025] The second output feature is used as the sample audio feature.
[0026] Preferably, the training method further includes:
[0027] Set source attribute labels for the sample audio features based on the sample audio data;
[0028] The sample audio features are updated based on the sound source attribute labels, and a new image processing model is obtained based on the updated sample audio features.
[0029] Preferably, the step of training a preset network to obtain the image processing model by using the facial features and audio features of each group of samples as input and the corresponding three-dimensional facial data of the samples as output includes:
[0030] The first concatenated feature is obtained by concatenating the facial features of the sample with the audio features of the sample.
[0031] The first spliced feature is input into a first preset convolutional neural network to output fused features;
[0032] Calculate the first loss value of the fused feature and / or the second loss value of the sample three-dimensional facial data, and use the first loss value and / or the second loss value as the target loss value;
[0033] If the target loss value does not meet the preset convergence condition, then return to the step of obtaining the sample facial features and the sample audio features for iterative training;
[0034] If the target loss value meets the preset convergence condition, the latest obtained model is saved as the final image processing model.
[0035] Preferably, the training method further includes:
[0036] Acquire real three-dimensional facial data that is in the same time frame as the sample audio data;
[0037] The step of calculating the first loss value of the fused features and / or the second loss value of the sample 3D facial data includes:
[0038] The real 3D facial data is input into a preset autoencoder to obtain verification feature data;
[0039] The first loss value is calculated based on the fusion features and the verification feature data; and / or,
[0040] The second loss value is calculated based on the real 3D facial data and the sample 3D facial data.
[0041] Preferably, the step of extracting the sample audio features from the expression features includes:
[0042] The expression features are used as input, and the association weights between several first output features and the facial features of the sample are used as output to train a second preset convolutional neural network.
[0043] The first preset convolutional neural network is iteratively trained based on the first loss value;
[0044] The first output feature with an association weight greater than a preset weight threshold is selected as the sample audio feature.
[0045] Preferably, the training method further includes:
[0046] Obtain sample facial expression data;
[0047] The sample facial expression data includes real three-dimensional facial data with set expression labels, and the real three-dimensional facial data represents three-dimensional facial data that is in the same time frame as the sample audio data;
[0048] Obtain facial expression features from the sample facial expression data;
[0049] The facial expression features include the node information of a preset facial region;
[0050] Using the second concatenated features of the sample facial features, the sample audio features, and the expression features as input, and the sample three-dimensional facial data as output, a preset convolutional neural network is trained to obtain an image processing model.
[0051] A second aspect provides a method for processing facial images, the method comprising:
[0052] Acquire the target audio data and the facial image to be processed;
[0053] Obtain the target audio features of the target audio data and the target facial features of the face image to be processed;
[0054] The target audio features and the target image features are input into an image processing model to obtain the corresponding target three-dimensional facial data;
[0055] The image processing model is obtained based on the training method of the image processing model described above.
[0056] The third aspect provides a training system for an image processing model, the training system including a sample acquisition module, an image processing module, an audio processing module, and a model training module;
[0057] The sample acquisition module is used to acquire raw three-dimensional facial data and several sample audio data.
[0058] The image processing module is used to extract sample facial features from the original three-dimensional facial data; wherein, the sample facial features are used to characterize the node feature information of nodes in several facial regions and the relationship feature information of the topological relationship between different nodes;
[0059] The audio processing module is used to acquire sample audio features of the sample audio data; wherein, based on the sample audio features, the sample facial features are adjusted to obtain the corresponding sample three-dimensional facial data;
[0060] The model training module is used to train a preset network with the facial features and audio features of each sample as input and the corresponding three-dimensional facial data of the sample as output, so as to obtain the image processing model.
[0061] Preferably, the audio processing module includes an audio recognition unit and a sample audio feature extraction unit;
[0062] The audio recognition unit is used to identify the expressive features in the sample audio data;
[0063] The expressive features are used to characterize the human voice features in the sample audio data;
[0064] The sample audio feature extraction unit is used to extract the sample audio features from the expression features.
[0065] Preferably, the sample audio feature extraction unit is further configured to input the sample audio data into a preset audio recognition model and output a first output feature;
[0066] The first output feature includes spectral features and / or semantic features, and the first output feature is used as the expression feature;
[0067] And / or,
[0068] The sample audio feature extraction unit is also used to input the expression features into a preset audio feature extraction model and output a second output feature;
[0069] The second output feature includes at least one of frequency, amplitude, and resonance peak;
[0070] The second output feature is used as the sample audio feature.
[0071] Preferably, the audio processing module further includes a tag unit;
[0072] The labeling unit is used to set sound source attribute labels based on the sample audio data for the sample audio features;
[0073] The sample audio feature extraction unit is also used to update the sample audio features based on the sound source attribute labels, and to update the image processing model based on the updated sample audio features.
[0074] Preferably, the model training module 104 includes a splicing unit, a fusion unit, a loss calculation unit, and an iterative processing unit;
[0075] The splicing unit is used to splice the sample facial features and the sample audio features to obtain a first spliced feature.
[0076] The fusion unit is used to input the first spliced feature into a first preset convolutional neural network to output fused features;
[0077] The calculation loss unit is used to calculate the first loss value of the fused feature and / or the second loss value of the sample three-dimensional facial data, and use the first loss value and / or the second loss value as the target loss value;
[0078] The iterative processing unit is used to return to the step of obtaining the sample facial features and the sample audio features for iterative training if the target loss value does not meet the preset convergence condition;
[0079] If the target loss value meets the preset convergence condition, the latest obtained model is saved as the final image processing model.
[0080] Preferably, the sample acquisition module is further configured to acquire real three-dimensional facial data that is in the same time frame as the sample audio data;
[0081] The loss calculation unit is also used to input the real three-dimensional facial data into a preset autoencoder to obtain verification feature data;
[0082] The first loss value is calculated based on the fusion features and the verification feature data; and / or,
[0083] The second loss value is calculated based on the real 3D facial data and the sample 3D facial data.
[0084] Preferably, the audio processing module further includes a weight processing unit;
[0085] The weight processing unit is used to train a second preset convolutional neural network by taking the expression features as input and the association weights between a plurality of first output features in the expression features and the sample facial features as output.
[0086] The second preset convolutional neural network is trained iteratively based on the first loss value;
[0087] The features to be processed that have an association weight greater than a preset weight threshold are selected as the sample audio features.
[0088] Preferably, the training system further includes an expression feature module;
[0089] The sample acquisition module 101 is also used to acquire sample facial expression data;
[0090] The sample facial expression data includes real three-dimensional facial data with set expression labels, and the real three-dimensional facial data represents three-dimensional facial data that is in the same time frame as the sample audio data;
[0091] The facial expression feature module is used to acquire facial expression features from the sample facial expression data;
[0092] The model training module 104 is also used to train a preset convolutional neural network to obtain an image processing model by taking the sample facial features, the sample audio features and the second spliced features of the expression features as input and the sample three-dimensional facial data as output.
[0093] The fourth aspect provides a facial image processing system, the processing system including a data acquisition module, a feature extraction module, a data processing module and an image driving module;
[0094] The data acquisition module is used to acquire target audio data and facial images to be processed;
[0095] The feature extraction module is used to obtain the target audio features of the target audio data and the target facial features of the facial image to be processed.
[0096] The data processing module is used to input the target audio features and the target facial features into the image processing model to obtain the corresponding target three-dimensional facial data;
[0097] The image processing model is obtained based on the training system of the aforementioned image processing model;
[0098] The image driving module is used to adjust the face image to be processed based on the target three-dimensional face data to obtain the target face image.
[0099] A fifth aspect provides an electronic device, including a memory, a processor, and a computer program stored in the memory and for running on the processor, wherein the processor executes the computer program to implement the training method of the image processing model as described above; or to implement the facial image processing method as described above.
[0100] A sixth aspect provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method for the image processing model as described above; or implements the facial image processing method as described above.
[0101] Based on common knowledge in the field, the above-mentioned preferred conditions can be combined arbitrarily to obtain various preferred embodiments of the present invention.
[0102] The significant advantages of this invention are as follows: Facial features are acquired through graph convolutional neural networks, and audio features are obtained through multi-level processing. Using the concatenated features of facial and audio features as input and sample 3D facial data as output, the resulting image processing model, trained through the convolutional neural network, exhibits higher accuracy, versatility, and robustness. Furthermore, by calculating the fusion loss of the fused features and the generation loss of the sample 3D facial data, the model training can be refined, improving both the efficiency and accuracy of the training process. Attached Figure Description
[0103] Figure 1 This is a schematic diagram of the first process of the training method for the image processing model in Embodiment 1 of the present invention;
[0104] Figure 2 This is a schematic diagram of the second process of the training method for the image processing model in Embodiment 1 of the present invention;
[0105] Figure 3 This is a schematic diagram of the third process of the training method for the image processing model in Embodiment 1 of the present invention;
[0106] Figure 4 This is a flowchart illustrating the deep convolutional neural network in Embodiment 1 of the present invention;
[0107] Figure 5 This is a schematic flowchart of the facial image processing method according to Embodiment 2 of the present invention;
[0108] Figure 6 This is a schematic diagram of the training system of the image processing model in Embodiment 3 of the present invention;
[0109] Figure 7 This is a schematic diagram of the facial image processing system according to Embodiment 4 of the present invention;
[0110] Figure 8 This is a schematic diagram of the hardware structure of the electronic device according to Embodiment 5 of the present invention. Detailed Implementation
[0111] The present invention will be further illustrated by way of embodiments below, but the present invention is not limited to the scope of the embodiments described herein.
[0112] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.
[0113] Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.
[0114] Example 1
[0115] This embodiment provides a training method for an image processing model, such as... Figure 1 As shown, the training method includes:
[0116] S101. Obtain raw 3D facial data and sample audio data;
[0117] S102. Extract sample facial features from the original three-dimensional facial data;
[0118] The sample facial features are used to characterize the node feature information of nodes in several facial regions and the relationship feature information of the topological relationship between different nodes;
[0119] S103. Obtain the sample audio features of the sample audio data;
[0120] The facial features of the sample are adjusted based on the audio features of the sample to obtain the corresponding three-dimensional facial data of the sample;
[0121] S104. Using the facial features and audio features of each sample as input and the corresponding three-dimensional facial data of the sample as output, train the preset network to obtain the image processing model.
[0122] This scheme employs graph convolutional neural networks and multiple convolutional neural networks to extract sample facial and audio features, concatenate and fuse these features, and output a generated 3D facial image. Using the original 3D facial data as the basis for feature extraction avoids image interference caused by lighting or background factors in real 3D facial data. In one embodiment, the original 3D facial data can use a different 3D facial mesh template than the real 3D facial data, improving the model's versatility and robustness.
[0123] In this scheme, a graph convolutional neural network is also used to extract facial features of the samples. The graph convolutional neural network has lower requirements for the input facial data, does not require downsampling of the facial data, can retain more information, and is suitable for processing irregular facial shapes. It can better capture the shape and details of the face, and consider facial features from a global perspective to obtain more refined and accurate facial features of the samples.
[0124] Facial features are obtained by using a graph convolutional neural network, including the nodes and topological relationship information of the 3D face mesh. The preferred method is to extract facial features using vertices in the 3D mesh nodes. This allows the facial features trained on the input model to utilize the local and global relationships between the 3D face mesh nodes, resulting in more accurate and refined features for complex structures like the 3D face mesh.
[0125] In one embodiment, a graph convolutional neural network is used to process the raw 3D facial data to obtain 89 sample facial features representing information such as facial shape and texture. These features include the shape of the facial contour curve, the geometry of the eyes, mouth, and nose, skin tone, and texture. This local facial feature information, combined with the global facial information, can be used for audio-driven face recognition tasks.
[0126] By training facial and audio features through a convolutional neural network, the resulting image processing model exhibits good versatility, robustness, and accuracy.
[0127] As one possible approach, the original 3D facial data is a neutral state of real 3D facial data.
[0128] In this scheme, the original 3D facial data adopts a neutral state of the 3D surface mesh of the face corresponding to the sampled audio data, which can improve the accuracy of the trained model, reduce the loss during the model training process, and improve the efficiency of model training.
[0129] As a feasible approach, such as Figure 2 As shown, step S103 includes:
[0130] S1031. Identify and obtain the expressive features in the sample audio data;
[0131] The expressive features are used to characterize the human voice features in the sample audio data;
[0132] S1032. Extract the sample audio features from the expression features.
[0133] In this scheme, to avoid the impact of poor voice quality or interference on the acquired sample audio features and the generated high-dimensional spatial features, the acquired sample audio data is preprocessed to remove noise and identify expressive features in the sample audio. Based on the preprocessed expressive features, sample audio features associated with facial features are obtained, thereby improving the accuracy of the sample audio features.
[0134] As one possible approach, step S1031 includes:
[0135] The sample audio data is input into a preset audio recognition model, and the first output feature is output.
[0136] The first output feature includes spectral features and / or semantic features, and the first output feature is used as the expression feature.
[0137] In this scheme, a pre-trained basic model is used as a preset audio recognition model to output expressive features that fully characterize the sample audio data. The main expressive features include the spectral and / or semantic information in the audio data. The spectral and semantic information can be converted and complemented each other, and provide accurate audio data for subsequent extraction of sample audio features related to sample facial features, thereby improving the accuracy of model training.
[0138] As one possible approach, the preset audio recognition model includes a Deepspeech model (a speech recognition model) and / or a Wav2vec model. Using deep models of Deepspeech and / or Wav2vec (speech recognition models) to recognize sample audio data can expand the range of languages and speech types in the sample audio data, making it suitable for recognizing different dialects and accents. It exhibits high robustness and stability, and can be applied to noisy environments and low SNR (Signal-Noise Ratio). ) This audio recognition capability provides data support for subsequent audio feature extraction.
[0139] As one possible approach, step S1032 includes:
[0140] The expressed features are input into a preset audio feature extraction model, and a second output feature is output.
[0141] The second output feature includes at least one of frequency, amplitude, and resonance peak;
[0142] The second output feature is used as the sample audio feature.
[0143] In this scheme, expressive features are input into a trained audio feature extraction model, which outputs sample audio features highly correlated with the driving face. Frequency features characterize different pitches and volumes of the human voice, and sound information at different frequencies is extracted based on time-frequency representation or Mel frequency, serving as input to the driving facial 3D mesh. Intensity or amplitude features characterize increases or decreases in volume, thereby further controlling the facial expressions and lip movements of the 3D facial mesh. Formant features characterize the duration and formants of the sound, thus controlling the emotion-related expressions and lip movements of the 3D facial mesh. By using sample audio features to correlate sample audio with sample facial 3D data, the accuracy and versatility of audio-driven 3D face images are improved.
[0144] As one possible approach, the training method further includes:
[0145] S1033. Set source attribute labels for the sample audio features based on the sample audio data.
[0146] In this approach, by setting source attribute labels for sample audio features, the accuracy of the feature data input to the model for training is improved, helping the model to better parse the feature information of the audio. By improving the interpretability of the sample audio feature data, the training efficiency of the model is accelerated.
[0147] In one possible way, the sound source attribute tag represents at least one of the following attributes of the sound source: age, gender, and dialect.
[0148] In this scheme, to improve the accuracy of the feature data input to the model training, age and gender labels are set on the sample audio features. For facial 3D meshes of different age groups and genders, facial 3D meshes associated with age and gender are generated during audio-driven training, so that the trained model has higher accuracy when driving facial 3D meshes of corresponding age and gender. Setting dialect labels on the sample audio features can improve the interpretability of audio during model training, so that the trained model has higher accuracy.
[0149] As a feasible approach, such as Figure 3 As shown, step S104 includes:
[0150] S1041. The sample facial features and the sample audio features are concatenated to obtain the first concatenated feature;
[0151] S1042. Input the first splicing feature into the first preset convolutional neural network to output the fused feature;
[0152] S1043. Calculate the first loss value of the fusion feature and / or the second loss value of the sample three-dimensional facial data, and use the first loss value and / or the second loss value as the target loss value;
[0153] S1044. If the target loss value does not meet the preset convergence condition, return to the step of obtaining the sample facial features and the sample audio features for iterative training.
[0154] S1045. If the target loss value meets the preset convergence condition, then the latest obtained model is saved as the final image processing model.
[0155] In this scheme, the sample facial features and sample audio features are first vectorized, and the corresponding sample facial feature vectors and sample audio feature vectors are concatenated. The concatenated features are then fused using a first pre-defined convolutional neural network (CNN). This first CNN consists of convolutional layers, skip connections, regularization layers, and deconvolutional layers; the number of convolutional layers is adjusted based on actual needs. A first loss calculation is performed on the fused features output from the first CNN. The fused features are then input into a third CNN to generate sample 3D facial data. A second loss calculation is performed on the sample 3D facial data generated based on the fused features. By adding a loss calculation for the fused features in the loss calculation, the accuracy of model training is improved.
[0156] As one possible approach, the training method further includes:
[0157] Acquire real three-dimensional facial data, wherein the real three-dimensional facial data is consistent with the time frame of the sample audio data;
[0158] The steps of calculating the first loss value of the fused features and / or the second loss value of the sample 3D facial data include:
[0159] The real 3D facial data is input into a preset autoencoder to obtain verification feature data;
[0160] The first loss value is calculated based on the fusion features and the verification feature data; and / or,
[0161] The second loss value is calculated based on the real 3D facial data and the sample 3D facial data.
[0162] In this scheme, the verification features are the encoded features generated by a trained autoencoder from a real 3D surface mesh of a human face. The autoencoder consists of two parts: encoding and decoding. Its purpose is to better represent the input information. Introducing a first loss value through the verification features can effectively improve the training efficiency and performance of the network. Furthermore, the first and second loss values are combined and reused in memory for iterative model training to improve model performance.
[0163] As one possible approach, step S1032 includes:
[0164] The expression features are used as input, and the association weights between several first output features and the facial features of the sample are used as output to train a second preset convolutional neural network.
[0165] The second preset convolutional neural network is trained iteratively based on the first loss value;
[0166] The first output feature with an association weight greater than a preset weight threshold is selected as the sample audio feature.
[0167] As one possible implementation, the second preset convolutional neural network includes a one-dimensional convolutional layer, a ReLU layer, and a fully connected layer.
[0168] In this scheme, the expressive features are input into the second preset convolutional neural network in the deep convolutional network. The second preset convolutional neural network is composed of a one-dimensional convolutional layer, a ReLU layer (rectified layer unit), and a fully connected layer. It can classify and regress the expressive features of audio. In the machine learning process of model iteration, audio features with high correlation weights to the driving facial three-dimensional mesh are extracted from several unprocessed features of expressive features. The training requires fewer parameters, the training speed is faster, and it is suitable for processing high-dimensional data such as audio data.
[0169] The following example illustrates the working principle of the image processing model training method in this embodiment:
[0170] like Figure 4 As shown, the process involves inputting a 3D face mesh into a convolutional neural network to extract facial features; inputting audio data into a second pre-defined convolutional neural network to extract audio features; concatenating facial and audio features and inputting them into a first pre-defined convolutional neural network to obtain a fused feature; calculating the L1 loss value using the fused feature and the verification feature output by the autoencoder; inputting the fused feature into a third pre-defined convolutional neural network to generate a target 3D face mesh; calculating the L2 loss value using the target 3D face mesh and the real 3D face mesh; returning to the facial and audio feature extraction steps; and reusing the L1 and L2 loss values in memory for the next iteration of the deep convolutional neural network until the loss value of the deep convolutional neural network is less than a preset loss threshold.
[0171] The training process of an autoencoder is as follows: Assuming the input data is X, input it into the autoencoder, and its output value is represented as F(X). The training objective is to minimize ||XF(X)||, that is, for input data X, we want the output F(X) of the autoencoder to be X. Inputting a 3D facial surface mesh, a trained autoencoder can obtain the corresponding features. The encoding part of the autoencoder consists of several sets of convolutional layers, regularization layers, and skip layers, while the decoding part consists of corresponding sets of deconvolutional layers, regularization layers, and skip layers.
[0172] Meanwhile, a second loss is introduced using real 3D facial data. The first and second losses are combined to obtain the overall loss of the model in the current training iteration. Based on the overall loss, the parameters are adjusted in the next iteration to improve the efficiency and accuracy of model training.
[0173] As an feasible approach, the entire model is trained using an end-to-end process, with the loss calculation steps as follows:
[0174] Assuming the model input is 3D facial data X and audio, the output is the vertex position information of the 3D surface mesh corresponding to the audio. The vertex position information of the real three-dimensional surface mesh corresponding to the audio is (y1, y2, ..., y...). n Then the L1 loss for that part is:
[0175]
[0176] The input X is fed into the encoding module of the autoencoder to obtain the features (z1, z2, ..., z). m The feature fusion network outputs the input X and audio. Then the L2 loss for this part is:
[0177]
[0178] As one possible approach, the training method further includes:
[0179] Obtain sample facial expression data;
[0180] The sample facial expression data includes real three-dimensional facial data with set expression labels, and the real three-dimensional facial data represents three-dimensional facial data that is in the same time frame as the sample audio data;
[0181] Obtain facial expression features from the sample facial expression data;
[0182] The facial expression features include the node information of a preset facial region;
[0183] Using the second concatenated features of the sample facial features, the sample audio features, and the expression features as input, and the sample three-dimensional facial data as output, a preset convolutional neural network is trained to obtain an image processing model.
[0184] In this scheme, the acquisition of facial expression features is further enhanced. Input based on facial expression features can improve the accuracy and vividness of the target facial 3D mesh data output by the model.
[0185] This embodiment provides a training method for an image processing model. It acquires facial features through a graph convolutional neural network and obtains audio features through multi-level processing. Using the concatenated features of facial and audio features as input and sample 3D facial data as output, the resulting image processing model, trained through the convolutional neural network, exhibits higher accuracy, versatility, and robustness. Furthermore, by calculating the fusion loss of the fused features and the generation loss of the sample 3D facial data, the model training is refined, improving its efficiency and accuracy.
[0186] Example 2
[0187] This embodiment provides a method for processing facial images, such as... Figure 5 As shown, the processing method includes:
[0188] S201. Acquire target audio data and facial image to be processed;
[0189] S202. Obtain the target audio features of the target audio data and the target facial features of the face image to be processed;
[0190] S203. Input the target audio features and the target image features into the image processing model to obtain the corresponding target three-dimensional facial data;
[0191] The image processing model is obtained based on the training method of the image processing model described in Example 1.
[0192] In this scheme, the audio features and facial features of the target audio data and the face image to be processed are extracted and input into the image processing model to obtain the target three-dimensional face image data of the face image to be processed based on the audio features, thereby realizing the driving force of the face image to be processed by the audio data.
[0193] The facial image processing method provided in this embodiment is based on the image processing model trained in Embodiment 1 and can be applied to different facial images. It uses facial images to represent audio content, accurately drives the lip movements and expressions of facial images, and achieves precise and vivid driving of audio to the facial images to be processed.
[0194] Example 3
[0195] This embodiment provides a training system 100 for an image processing model, such as... Figure 6 As shown, the training system 100 includes a sample acquisition module 101, an image processing module 102, an audio processing module 103, and a model training module 104.
[0196] The sample acquisition module 101 is used to acquire raw three-dimensional facial data and several sample audio data.
[0197] The image processing module 102 is used to extract sample facial features from the original three-dimensional facial data; wherein, the sample facial features are used to characterize the node feature information of nodes in several facial regions and the relationship feature information of the topological relationship between different nodes;
[0198] The audio processing module 103 is used to acquire sample audio features of the sample audio data; wherein, the sample facial features are adjusted based on the sample audio features to obtain corresponding sample three-dimensional facial data;
[0199] The model training module 104 is used to train a preset network with the sample facial features and sample audio features of each group as input and the corresponding sample three-dimensional facial data as output to obtain the image processing model.
[0200] In one possible implementation, the audio processing module 103 includes an audio recognition unit and a sample audio feature extraction unit;
[0201] The audio recognition unit is used to identify the expressive features in the sample audio data;
[0202] The expressive features are used to characterize the human voice features in the sample audio data;
[0203] The sample audio feature extraction unit is used to extract the sample audio features from the expression features.
[0204] As one possible approach, the sample audio feature extraction unit is also used to input the sample audio data into a preset audio recognition model and output a first output feature;
[0205] The first output feature includes spectral features and / or semantic features, and the first output feature is used as the expression feature;
[0206] And / or,
[0207] The sample audio feature extraction unit is also used to input the expression features into a preset audio feature extraction model and output a second output feature;
[0208] The second output feature includes at least one of frequency, amplitude, and resonance peak;
[0209] The second output feature is used as the sample audio feature.
[0210] As one possible implementation, the audio processing module 103 also includes a tag unit;
[0211] The labeling unit is used to set sound source attribute labels based on the sample audio data for the sample audio features;
[0212] The sample audio feature extraction unit is also used to update the sample audio features based on the sound source attribute labels, and to update the image processing model based on the updated sample audio features.
[0213] In one possible implementation, the model training module 104 includes a splicing unit, a fusion unit, a loss calculation unit, and an iterative processing unit;
[0214] The splicing unit is used to splice the sample facial features and the sample audio features to obtain a first spliced feature.
[0215] The fusion unit is used to input the first spliced feature into a first preset convolutional neural network to output fused features;
[0216] The calculation loss unit is used to calculate the first loss value of the fused feature and / or the second loss value of the sample three-dimensional facial data, and use the first loss value and / or the second loss value as the target loss value;
[0217] The iterative processing unit is used to return to the step of obtaining the sample facial features and the sample audio features for iterative training if the target loss value does not meet the preset convergence condition;
[0218] If the target loss value meets the preset convergence condition, the latest obtained model is saved as the final image processing model.
[0219] As one possible approach, the sample acquisition module 101 is also used to acquire real three-dimensional facial data that is in the same time frame as the sample audio data;
[0220] The loss calculation unit is also used to input the real three-dimensional facial data into a preset autoencoder to obtain verification feature data;
[0221] The first loss value is calculated based on the fusion features and the verification feature data; and / or,
[0222] The second loss value is calculated based on the real 3D facial data and the sample 3D facial data.
[0223] As one possible implementation, the audio processing module further includes a weight processing unit;
[0224] The weight processing unit is used to train a second preset convolutional neural network by taking the expression features as input and the association weights between a plurality of first output features in the expression features and the sample facial features as output.
[0225] The second preset convolutional neural network is trained iteratively based on the first loss value;
[0226] The features to be processed that have an association weight greater than a preset weight threshold are selected as the sample audio features.
[0227] As one possible approach, the training system also includes an expression feature module;
[0228] The sample acquisition module 101 is also used to acquire sample facial expression data;
[0229] The sample facial expression data includes real three-dimensional facial data with set expression labels, and the real three-dimensional facial data represents three-dimensional facial data that is in the same time frame as the sample audio data;
[0230] The facial expression feature module is used to acquire facial expression features from the sample facial expression data;
[0231] The model training module 104 is also used to train a preset convolutional neural network to obtain an image processing model by taking the sample facial features, the sample audio features and the second spliced features of the expression features as input and the sample three-dimensional facial data as output.
[0232] It should be noted that the working principle of the image processing model training system 100 in this embodiment is the same as that of the image processing model training method in Embodiment 1, so it will not be described again here.
[0233] The image processing model training system provided in this embodiment acquires facial features through graph convolutional neural networks and obtains audio features through multi-level processing. Using the concatenated features of facial and audio features as input and sample 3D facial data as output, the resulting image processing model, trained through the convolutional neural network, exhibits higher accuracy, versatility, and robustness. Furthermore, by calculating the fusion loss of the fused features and the generation loss of the sample 3D facial data, the model training is refined, improving the efficiency and accuracy of model training.
[0234] Example 4
[0235] This embodiment provides a facial image processing system 200, such as... Figure 7 As shown, the processing system includes a data acquisition module 201, a feature extraction module 202, a data processing module 203, and an image driving module 204;
[0236] The data acquisition module 201 is used to acquire target audio data and facial images to be processed.
[0237] The feature extraction module 202 is used to obtain the target audio features of the target audio data and the target facial features of the facial image to be processed.
[0238] The data processing module 203 is used to input the target audio features and the target facial features into the image processing model to obtain the corresponding target three-dimensional facial data.
[0239] The image processing model is obtained based on the training system of the image processing model described in Example 3;
[0240] The image driving module 204 is used to adjust the face image to be processed based on the target three-dimensional face data to obtain the target face image.
[0241] It should be noted that the working principle of the facial image processing system 200 in this embodiment is the same as that of the facial image processing method in Embodiment 2, so it will not be described again here.
[0242] The facial image processing system provided in this embodiment is based on the image processing model trained in Embodiment 3 and can be applied to different facial images. It expresses audio content through facial images, accurately drives the lip movements and expressions of facial images, and realizes precise and vivid driving of audio to the facial images to be processed.
[0243] Example 5
[0244] like Figure 8 The diagram shown is a structural schematic of an electronic device provided in Embodiment 5 of the present invention. It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the training method of the image processing model of Embodiment 1 described above; or, the facial image processing method of Embodiment 2. Figure 8 The electronic device 30 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.
[0245] The electronic device 30 may be in the form of a general-purpose computing device, such as a server device. The components of the electronic device 30 may include, but are not limited to: at least one processor 31, at least one memory 32, and a bus 33 connecting different system components (including memory 32 and processor 31).
[0246] Bus 33 includes a data bus, an address bus, and a control bus.
[0247] The memory 32 may include volatile memory, such as random access memory (RAM) 321 and / or cache memory 322, and may further include read-only memory (ROM) 323.
[0248] The memory 32 may also include a program / utility 325 having a set (at least one) of program modules 324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.
[0249] The processor 31 executes various functional applications and data processing by running computer programs stored in the memory 32, such as the image processing model training method of Embodiment 1 of the present invention; or the facial image processing method of Embodiment 2.
[0250] Electronic device 30 can also communicate with one or more external devices 34 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 35. Furthermore, the model-generated device 30 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 36. As shown, network adapter 36 communicates with other modules of the model-generated device 30 via bus 33. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the model-generated device 30, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.
[0251] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.
[0252] Example 6
[0253] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method of the image processing model of Embodiment 1; or the facial image processing method of Embodiment 2.
[0254] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.
[0255] In possible implementations, the present invention can also be implemented as a program product comprising program code, which, when the program product is run on a terminal device, causes the terminal device to execute the training method for the image processing model of Embodiment 1; or the facial image processing method of Embodiment 2.
[0256] The program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.
[0257] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.
Claims
1. A method for training an image processing model, characterized in that, The training method includes: Acquire raw 3D facial data and several sample audio data; Extract sample facial features from the original 3D facial data; The facial features of the samples were extracted using a graph convolutional neural network. The sample facial features are used to characterize the node feature information of nodes in several facial regions and the relationship feature information of the topological relationship between different nodes; Obtain the sample audio features of the sample audio data; The facial features of the sample are adjusted based on the audio features of the sample to obtain the corresponding three-dimensional facial data of the sample; The image processing model is obtained by training a preset network using the facial features and audio features of each sample as input and the corresponding three-dimensional facial data of the sample as output. The step of training a preset network to obtain the image processing model by taking the facial features and audio features of each group of samples as input and the corresponding three-dimensional facial data of the samples as output includes: The first concatenated feature is obtained by concatenating the facial features of the sample with the audio features of the sample. The first spliced feature is input into a first preset convolutional neural network to output fused features; Calculate the first loss value of the fused feature and / or the second loss value of the sample three-dimensional facial data, and use the first loss value and / or the second loss value as the target loss value; If the target loss value does not meet the preset convergence condition, then return to the step of obtaining the sample facial features and the sample audio features for iterative training; If the target loss value meets the preset convergence condition, then the latest obtained model is saved as the final image processing model. The sample's three-dimensional facial data includes the position information of the three-dimensional surface mesh vertices; Acquire real three-dimensional facial data that is in the same time frame as the sample audio data; The step of calculating the first loss value of the fused features and / or the second loss value of the sample 3D facial data includes: The real 3D facial data is input into a preset autoencoder to obtain verification feature data; The first loss value is calculated based on the fusion features and the verification feature data; and / or, The second loss value is calculated based on the real 3D facial data and the sample 3D facial data.
2. The method of claim 1, wherein, The step of obtaining the audio features of the sample audio data includes: The expressive features in the sample audio data are identified; The expressive features are used to characterize the human voice features in the sample audio data; The sample audio features are extracted from the expression features.
3. The training method for the image processing model according to claim 2, characterized in that, The step of identifying the expressive features of the sample audio data includes: The sample audio data is input into a preset audio recognition model, and the first output feature is output. The first output feature includes spectral features and / or semantic features, and the first output feature is used as the expression feature; And / or, The step of extracting the sample audio features from the expression features includes: The expressed features are input into a preset audio feature extraction model, and a second output feature is output. The second output feature includes at least one of frequency, amplitude, and resonance peak; The second output feature is used as the sample audio feature.
4. The training method for the image processing model according to claim 2, characterized in that, The training method also includes: Set source attribute labels for the sample audio features based on the sample audio data; The sample audio features are updated based on the sound source attribute labels, and a new image processing model is obtained based on the updated sample audio features.
5. The training method for the image processing model according to claim 2, characterized in that, The step of extracting the sample audio features from the expression features includes: The expression features are used as input, and the association weights between several first output features and the facial features of the sample are used as output to train a second preset convolutional neural network. The second preset convolutional neural network is trained iteratively based on the first loss value; The first output feature with an association weight greater than a preset weight threshold is selected as the sample audio feature.
6. The training method for the image processing model according to claim 1, characterized in that, The training method also includes: Obtain sample facial expression data; The sample facial expression data includes real three-dimensional facial data with set expression labels, and the real three-dimensional facial data represents three-dimensional facial data that is in the same time frame as the sample audio data; Obtain facial expression features from the sample facial expression data; The facial expression features include node information of a preset facial region; The image processing model is obtained by training a preset convolutional neural network with the sample facial features, sample audio features, and expression features as input and the sample three-dimensional facial data as output.
7. A method for processing facial images, characterized in that, The processing method includes: Acquire the target audio data and the facial image to be processed; Obtain the target audio features of the target audio data and the target facial features of the face image to be processed; The target audio features and the target facial features are input into an image processing model to obtain the corresponding target three-dimensional facial data. The image processing model is obtained based on the training method of the image processing model according to any one of claims 1-6.
8. A training system for an image processing model, characterized in that, The training system includes a sample acquisition module, an image processing module, an audio processing module, and a model training module; The sample acquisition module is used to acquire raw three-dimensional facial data and several sample audio data; the image processing module is used to extract sample facial features from the raw three-dimensional facial data. The facial features of the samples were extracted using a graph convolutional neural network. The sample facial features are used to characterize the node feature information of nodes in several facial regions and the relationship feature information of the topological relationship between different nodes; The audio processing module is used to acquire the sample audio features of the sample audio data; Specifically, the facial features of the sample are adjusted based on the audio features of the sample to obtain the corresponding three-dimensional facial data of the sample; The model training module is used to train a preset network with the facial features and audio features of each sample as input and the corresponding three-dimensional facial data of the sample as output, so as to obtain the image processing model. The model training module includes a splicing unit, a fusion unit, a loss calculation unit, and an iterative processing unit; The splicing unit is used to splice the sample facial features and the sample audio features to obtain a first spliced feature; The fusion unit is used to input the first spliced feature into a first preset convolutional neural network to output fused features; The calculation loss unit is used to calculate the first loss value of the fused feature and / or the second loss value of the sample three-dimensional facial data, and use the first loss value and / or the second loss value as the target loss value; The iterative processing unit is used to return to the step of obtaining the sample facial features and the sample audio features for iterative training if the target loss value does not meet the preset convergence condition; If the target loss value meets the preset convergence condition, then the latest obtained model is saved as the final image processing model. The sample's three-dimensional facial data includes the position information of the three-dimensional surface mesh vertices; The sample acquisition module is also used to acquire real three-dimensional facial data that is in the same time frame as the sample audio data; The loss calculation unit is also used to input the real three-dimensional facial data into a preset autoencoder to obtain verification feature data; The first loss value is calculated based on the fusion features and the verification feature data; and / or, The second loss value is calculated based on the real 3D facial data and the sample 3D facial data.
9. A facial image processing system, characterized in that, The processing system includes a data acquisition module, a feature extraction module, a data processing module, and an image driving module; The data acquisition module is used to acquire target audio data and facial images to be processed; The feature extraction module is used to obtain the target audio features of the target audio data and the target facial features of the facial image to be processed. The data processing module is used to input the target audio features and the target facial features into the image processing model to obtain the corresponding target three-dimensional facial data; The image processing model is obtained based on the training system of the image processing model described in claim 8; The image driving module is used to adjust the face image to be processed based on the target three-dimensional face data to obtain the target face image.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and for running on the processor, characterized in that, When the processor executes a computer program, it implements the training method of the image processing model as described in any one of claims 1-6; or, it implements the facial image processing method as described in claim 7.
11. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the image processing model as described in any one of claims 1-6; or, implements the facial image processing method as described in claim 7.