A face feature generation method, system and electronic device

Abstract

The application provides a face feature generation method and system and an electronic device, and relates to the technical field of face recognition. When a target face feature generation model is pre-trained, a set of loss functions is added to the training data of the model. The functions use a gradient inversion method to intentionally make the model unable to distinguish between different speakers, that is, to eliminate the personal characteristics of different speakers. Meanwhile, the training data is randomly enhanced by randomly scaling the spectrum of the audio in the training data to change the timbre of the audio, so that the model can still output the original face feature parameters under the condition of input timbre change, thereby improving the stability of the model to input changes. The target recognition model obtained by pre-training according to the application has better generalization and stability.

Description

Technical Field

[0001] This application relates to the field of facial recognition technology, and in particular to a method, system, and electronic device for generating facial features. Background Technology

[0002] A facial feature generation model must be able to generate coefficients that stably adapt to a "facial rendering model" based on the speech of any person. It must possess universality / generalization, generating stable facial feature coefficients regardless of whose speech is input. Therefore, to improve the generalization ability of a facial feature generation model, it is typically trained jointly using multiple datasets of different speakers, allowing the model to encounter as many varied speech patterns as possible during the training phase.

[0003] However, in pursuit of generalization, existing "facial feature generation models," while using multiple datasets of different speakers for joint training, have not specifically addressed the differences between individuals to improve their generalization. The actual generalization is insufficient; the recognition performance fluctuates significantly when inputting speech from different speakers. Recognition is better when inputting speech similar to the training data, but worse when inputting speech significantly different from the training data.

[0004] Therefore, there is an urgent need for a new method for generating facial features. Summary of the Invention

[0005] In view of the above problems, embodiments of this application provide a method, system, and electronic device for generating facial features, so as to overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect, this application provides a method for generating facial features, the method comprising:

[0007] Acquire the speech data to be recognized;

[0008] The speech data to be recognized is input into a pre-trained target face feature generation model to obtain a target face feature parameter frame sequence. The target face feature generation model is trained through the following steps:

[0009] Obtain the original speech signal feature frame sequence and the original face feature parameter frame sequence, which are aligned in chronological order.

[0010] The original speech signal feature frame sequence is spectrally scaled, and the spectrally scaled original speech signal feature frame sequence, the original face feature parameter frame sequence, and the actual speaker labels of the original video file are combined into a set of training data.

[0011] A set of training data is input into the original face feature generation model to obtain the face feature loss function value. After inputting a set of training data into the original face feature generation model, the output of the preset intermediate layer of the original face feature generation model is input into the speaker adversarial classification module to obtain the target classification loss function value.

[0012] The target classification loss function value is added to the face feature loss function value to obtain the target loss function value. The model parameters of the original face feature generation model are then updated using the target loss function value to obtain the target face feature generation model.

[0013] Optionally, the speaker adversarial classification module sequentially includes: a gradient inversion layer, multiple downsampling convolutional layers, a global mean pooling layer, and a Logit mapping layer; the step of inputting the output of the preset intermediate layer of the original face feature generation model into the speaker adversarial classification module to obtain the target classification loss function value includes:

[0014] The output of the preset intermediate layer of the original face feature generation model is input into the speaker adversarial classification module, and then passes through the gradient inversion layer, the multi-layer downsampling convolutional layer, the global mean pooling layer and the Logit mapping layer in sequence. After being processed by the SOFTMAX activation function, the predicted speaker label is obtained.

[0015] Calculate the loss function value between the predicted speaker label and the actual speaker label to obtain the classification loss function value;

[0016] The classification loss function value is backpropagated using gradient inversion, and the gradient value of the classification loss function value is inverted in the gradient inversion layer to obtain the target classification loss function value.

[0017] Optionally, inputting a set of the training data into the original face feature generation model to obtain the face feature loss function value includes:

[0018] Select the spectrum-scaled original speech signal feature frame sequence from a set of training data, and input the spectrum-scaled original speech signal feature frame sequence into the original face feature generation model to obtain the predicted face feature parameter frame sequence;

[0019] The original face feature parameter frame sequence is selected from the same set of training data, and the loss function value between the predicted face feature parameter frame sequence and the original face feature parameter frame sequence is calculated to obtain the face feature loss function value.

[0020] Optionally, the spectral scaling of the original speech signal feature frame sequence includes:

[0021] Get the random scaling ratio;

[0022] According to the random scaling ratio, each frame of speech signal features in the original speech signal feature frame sequence is linearly interpolated and scaled, and after the linear interpolation and scaling of any frame of speech signal features is completed, the original dimensions of each speech signal feature are returned.

[0023] Optionally, the step of performing linear interpolation scaling on each frame of the original speech signal feature frame sequence according to the random scaling ratio, and returning to the original dimension of each speech signal feature after the linear interpolation scaling of any frame of the speech signal feature is completed, includes:

[0024] When the random scaling ratio is greater than 0 and less than 1, the speech signal features of any frame are linearly interpolated and reduced to obtain the reduced speech signal features.

[0025] Obtain the last dimension of the reduced speech signal features, and add a random number to the last dimension of the speech signal features to obtain a new speech signal feature, so as to supplement the reduced speech signal features until the dimension of the reduced speech signal features reaches the original dimension.

[0026] Optionally, the step of performing linear interpolation scaling on each frame of the original speech signal feature frame sequence according to the random scaling ratio, and returning to the original dimension of each speech signal feature after the linear interpolation scaling of any frame of the speech signal feature is completed, includes:

[0027] When the random scaling ratio is greater than or equal to 1, all dimensions of the speech signal features in any frame are sequentially linearly interpolated and amplified to obtain amplified speech signal features. The portion of the amplified speech signal features that exceeds the original dimension is then removed. The amplified speech signal features include the original speech signal features and the speech signal features amplified according to the random scaling ratio.

[0028] Optionally, the step of spectrally scaling the original speech signal feature frame sequence and combining the spectrally scaled original speech signal feature frame sequence, the original face feature parameter frame sequence, and the actual speaker labels of the original video file into a set of training data includes:

[0029] Obtain the original video files of multiple speakers;

[0030] The original video files of each speaker are decomposed into the original speech signal feature frame sequence and the original face feature parameter frame sequence of each speaker.

[0031] The original speech signal feature frame sequences of each speaker are spectrally scaled, and the spectrally scaled original speech signal feature frame sequences of each speaker, their corresponding original face feature parameter frame sequences, and their corresponding actual speaker labels are used to form the training data of each speaker.

[0032] Optionally, after obtaining the target face feature parameter frame sequence, the method further includes:

[0033] Obtain the original image sequence from the preset video template;

[0034] The target face feature parameter frame sequence and the original image sequence are input into a pre-trained face rendering model to generate the target image.

[0035] A second aspect of this application provides a facial feature generation system, the system comprising:

[0036] The first acquisition module is used to acquire the speech data to be recognized;

[0037] The first input module is used to input the speech data to be recognized into a pre-trained target face feature generation model to obtain a target face feature parameter frame sequence, wherein the target face feature generation model is trained through the following steps:

[0038] The second acquisition module is used to acquire the original speech signal feature frame sequence and the original face feature parameter frame sequence, wherein the original speech signal feature frame sequence and the original face feature parameter frame sequence are aligned in chronological order.

[0039] The scaling module is used to perform spectral scaling on the original speech signal feature frame sequence, and to combine the spectrally scaled original speech signal feature frame sequence, the original face feature parameter frame sequence, and the actual speaker labels of the original video file into a set of training data.

[0040] The second input module is used to input a set of training data into the original face feature generation model to obtain the face feature loss function value, and after inputting a set of training data into the original face feature generation model, input the output result of the preset intermediate layer of the original face feature generation model into the speaker adversarial classification module to obtain the target classification loss function value.

[0041] The update module is used to add the target classification loss function value and the face feature loss function value to obtain the target loss function value, and update the model parameters of the original face feature generation model through the target loss function value to obtain the target face feature generation model.

[0042] Optionally, the speaker adversarial classification module sequentially includes: a gradient inversion layer, multiple downsampling convolutional layers, a global mean pooling layer, and a Logit mapping layer; the second input module, which inputs the output of the preset intermediate layer of the original face feature generation model into the speaker adversarial classification module to obtain the target classification loss function value, includes:

[0043] The first input submodule is used to input the output of the preset intermediate layer of the original face feature generation model into the speaker adversarial classification module, which then passes through the gradient inversion layer, the multi-layer downsampling convolutional layer, the global mean pooling layer, and the Logit mapping layer in sequence, and is calculated by the SOFTMAX activation function to obtain the predicted speaker label.

[0044] The first calculation submodule is used to calculate the loss function value between the predicted speaker label and the actual speaker label to obtain the classification loss function value;

[0045] The backpropagation submodule is used to perform gradient backpropagation on the classification loss function value, and invert the gradient value of the classification loss function value in the gradient inversion layer to obtain the target classification loss function value.

[0046] Optionally, the first input module, which inputs a set of training data into the original face feature generation model to obtain the face feature loss function value, includes:

[0047] The first selection submodule is used to select the spectrum-scaled original speech signal feature frame sequence from a set of training data, and input the spectrum-scaled original speech signal feature frame sequence into the original face feature generation model to obtain the predicted face feature parameter frame sequence.

[0048] The second selection submodule is used to select the original face feature parameter frame sequence from the same set of training data, and calculate the loss function value between the predicted face feature parameter frame sequence and the original face feature parameter frame sequence to obtain the face feature loss function value.

[0049] Optionally, the spectral scaling of the original speech signal feature frame sequence includes:

[0050] The first acquisition submodule is used to obtain the random scaling ratio;

[0051] The scaling submodule is used to perform linear interpolation scaling on each frame of speech signal features in the original speech signal feature frame sequence according to the random scaling ratio, and after the linear interpolation scaling of any frame of speech signal features is completed, return to the original dimension of each speech signal feature.

[0052] Optionally, the scaling submodule, which performs linear interpolation scaling on each frame of the original speech signal feature frame sequence according to the random scaling ratio, and returns to the original dimension of each speech signal feature after the linear interpolation scaling of any frame of the speech signal feature is completed, includes:

[0053] The reduction subunit is used to perform linear interpolation reduction on the speech signal features of any frame when the random scaling ratio is greater than 0 and less than 1, so as to obtain the reduced speech signal features.

[0054] The supplementary subunit is used to obtain the last dimension of the reduced speech signal features and add a random number to the last dimension of the speech signal features to obtain a new speech signal feature, so as to supplement the reduced speech signal features until the dimension of the reduced speech signal features reaches the original dimension.

[0055] Optionally, the scaling submodule, which performs linear interpolation scaling on each frame of the original speech signal feature frame sequence according to the random scaling ratio, and returns to the original dimension of each speech signal feature after the linear interpolation scaling of any frame of the speech signal feature is completed, includes:

[0056] The amplification subunit is used to sequentially perform linear interpolation amplification on all dimensions of the speech signal features in any frame of the speech signal features when the random scaling ratio is greater than or equal to 1, to obtain amplified speech signal features, and to delete the part of the amplified speech signal features that exceeds the original dimension. The amplified speech signal features include the original speech signal features and the speech signal features amplified according to the random scaling ratio.

[0057] Optionally, the step of spectrally scaling the original speech signal feature frame sequence and combining the spectrally scaled original speech signal feature frame sequence, the original face feature parameter frame sequence, and the actual speaker labels of the original video file into a set of training data includes:

[0058] The second acquisition submodule is used to acquire the original video files of multiple speakers;

[0059] The decomposition submodule is used to decompose the original video files of each speaker into the original speech signal feature frame sequence and the original face feature parameter frame sequence of each speaker.

[0060] The sub-module is used to perform spectral scaling on the original speech signal feature frame sequences of each speaker, and to combine the spectral-scaled original speech signal feature frame sequences of each speaker, their corresponding original face feature parameter frame sequences, and their corresponding actual speaker labels to form the training data of each speaker.

[0061] Optionally, after obtaining the target face feature parameter frame sequence, the method further includes:

[0062] The third acquisition submodule is used to acquire the original image sequence from the preset video template;

[0063] The generation submodule is used to input the target face feature parameter frame sequence and the original image sequence into a pre-trained face rendering model to generate the target image.

[0064] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the facial feature generation method as described in the first aspect of this application.

[0065] The beneficial effects of this application are:

[0066] This application provides a method for generating facial features, the method comprising: acquiring speech data to be recognized; inputting the speech data to be recognized into a pre-trained target facial feature generation model to obtain a target facial feature parameter frame sequence, wherein the target facial feature generation model is trained through the following steps: acquiring an original speech signal feature frame sequence and an original facial feature parameter frame sequence, the original speech signal feature frame sequence and the original facial feature parameter frame sequence being aligned in chronological order; performing spectral scaling on the original speech signal feature frame sequence, and merging the spectrally scaled original speech signal feature frame sequence and the original facial feature parameter frame sequence... The sequence and the actual speaker labels of the original video file constitute a set of training data. This set of training data is input into the original face feature generation model to obtain a face feature loss function value. After inputting the training data into the original face feature generation model, the output of the preset intermediate layer of the original face feature generation model is input into the speaker adversarial classification module to obtain a target classification loss function value. The target classification loss function value is added to the face feature loss function value to obtain a target loss function value. The model parameters of the original face feature generation model are updated using the target loss function value to obtain the target face feature generation model. In this application, when pre-training the target face feature generation model, a loss function is added to the model's training data. This function, through a gradient inversion underlying method, aims to intentionally make it difficult for the model to distinguish between speakers, i.e., to eliminate the individual characteristics of different speakers. Simultaneously, random augmentation is performed on the training data by randomly scaling the spectrum of the audio in the training data to change the speaker's timbre, enabling the model to output the original face feature parameters even when the input timbre changes, thereby improving the model's stability to input changes. The target recognition model pre-trained through this application has better generalization and stability. Attached Figure Description

[0067] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a flowchart illustrating the steps of a method for generating facial features according to an embodiment of this application.

[0069] Figure 2 This is a schematic diagram of a training model for a target face feature generation model provided in an embodiment of this application;

[0070] Figure 3This is a schematic diagram of the structure of a speaker adversarial classification module provided in an embodiment of this application;

[0071] Figure 4 This is a schematic diagram of a linear interpolation reduction method provided in an embodiment of this application;

[0072] Figure 5 This is a schematic diagram of a linear interpolation amplification provided in an embodiment of this application;

[0073] Figure 6 This is a schematic diagram of a facial feature generation system provided in an embodiment of this application;

[0074] Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0075] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0076] In a first aspect, this application provides a flowchart illustrating the steps of a method for generating facial features, as shown below. Figure 1 As shown, the method includes:

[0077] Step S101: Obtain the speech data to be recognized;

[0078] Step S102: Input the speech data to be recognized into a pre-trained target face feature generation model to obtain a target face feature parameter frame sequence, wherein the target face feature generation model is trained through the following steps:

[0079] Step S10: Obtain the original speech signal feature frame sequence and the original face feature parameter frame sequence, wherein the original speech signal feature frame sequence and the original face feature parameter frame sequence are aligned in chronological order;

[0080] Step S11: Spectral scaling is performed on the original speech signal feature frame sequence, and the spectrum-scaled original speech signal feature frame sequence, the original face feature parameter frame sequence, and the actual speaker labels of the original video file are combined into a set of training data.

[0081] Step S12: Input a set of training data into the original face feature generation model to obtain the face feature loss function value; and after inputting a set of training data into the original face feature generation model, input the output result of the preset intermediate layer of the original face feature generation model into the speaker adversarial classification module to obtain the target classification loss function value.

[0082] Step S13: Add the target classification loss function value to the face feature loss function value to obtain the target loss function value, and update the model parameters of the original face feature generation model through the target loss function value to obtain the target face feature generation model.

[0083] Specifically, in this embodiment, the voice data for which facial features need to be generated is acquired, and the acquired voice data is input into a pre-trained target facial feature generation model, thereby obtaining a target facial feature parameter frame sequence.

[0084] In a preferred embodiment, after obtaining the target face feature parameter frame sequence, the method further includes:

[0085] Obtain the original image sequence from the preset video template;

[0086] The target face feature parameter frame sequence and the original image sequence are input into a pre-trained face rendering model to generate the target image.

[0087] In practical applications, by inputting the target facial feature parameter frame sequence into a face rendering model and combining it with a pre-prepared original image sequence from a video template, a video image corresponding to the aforementioned speech data can be generated. Virtual anchor synthesis technology is one application scenario of this application. In this scenario, pre-defined speech is input into a target facial feature generation model optimized by this application to obtain a pre-defined target facial feature parameter frame sequence corresponding to the speech. The original image sequence is then obtained from a pre-defined video template. Both the target facial feature parameter frame sequence and the original image sequence are input into a pre-trained face rendering model. This face rendering model can be a commonly used model. The face rendering model generates an image corresponding to the pre-defined speech, and finally, a virtual anchor is obtained from the generated image.

[0088] In this embodiment, the target face feature generation model is trained through the following steps, specifically including:

[0089] In step S10, obtaining the original speech signal feature frame sequence and the original face feature parameter frame sequence is to provide necessary training data for training the target face feature generation model. Specifically, a pre-prepared video of a target person speaking can be separated into audio and video. In practical applications, the FFMPEG media encoding tool can be used for audio and video separation. For example, the video of the speech can be decomposed into multiple video segments, each video segment can be decomposed into multiple image frames and audio frames, and the decomposed multiple image frames and audio frames can be processed in chronological order.

[0090] Furthermore, the original speech signal feature frame sequence is obtained from the audio frame, and the original face feature parameter frame sequence is obtained from the image frame.

[0091] In this embodiment, facial feature parameter frames can be extracted from each image frame using 3DMM tools. These extracted frames from multiple image frames form the original facial feature parameter frame sequence. For audio frames, a Mel-frequency transform can be performed, converting the time-domain signal of the audio frame into a frequency-domain signal. The capability value of each frequency band is mapped onto a Mel scale. This process typically involves using a Mel filter bank to transform the spectrum onto a Mel scale, followed by a logarithmic transform to simulate the non-linearity of human ear perception of audio. The Mel-frequency spectral representations of each frame are then combined to form an audio frame sequence. This sequence reflects the temporal evolution of the entire audio file and contains Mel-frequency spectral information for each time segment.

[0092] In step S11, the original speech signal feature frame sequence is spectrally scaled, and then the scaled speech signal feature frame sequence, the original face feature parameter frame sequence, and the actual speaker labels are combined to form a set of training data.

[0093] Specifically, the original speech signal feature frame sequence is spectrally scaled, which can be done by scaling each frequency band of the Mel spectrum. This scaled speech signal feature frame sequence, along with the original face feature parameter frame sequence, constitutes the input part of the training data. In practical applications, the original face feature parameter frame sequence may include facial features extracted from the video, such as facial landmarks and facial expression information. The actual speaker label refers to the speaker identity label corresponding to each time segment in the video file. This is crucial for training the model to learn the relationship between speech and facial features, as well as the speaker's identity.

[0094] In step S12, you can refer to Figure 2 The diagram shown illustrates a training model for a target facial feature generation model. Figure 2As shown, a set of training data, including the scaled speech signal feature frame sequence, the original face feature parameter frame sequence, and the actual speaker label, is input into the original face feature generation model. The scaled speech signal feature frame sequence is used as the input data of the model, while the original face feature parameter frame sequence and the actual speaker label are used as the target data of the model.

[0095] Furthermore, during model training, the difference between the generated facial features and the actual facial features is compared to calculate the facial feature loss function value. In practical applications, the facial feature loss function typically uses regression loss, such as mean squared error, to measure the distance between the generated facial features and the actual facial features. Simultaneously, the output of the preset intermediate layer of the original facial feature generation model is input into the speaker adversarial classification module, which calculates the target classification loss function value.

[0096] In step S13, the target classification loss function value and the face feature loss function value are added together to form the total target loss function value. This target loss function value comprehensively considers two performance indicators: the accuracy of face feature generation and the robustness of the generated features in speaker classification. The target loss function value is used to update the model parameters of the original face feature generation model. This is usually achieved through backpropagation and optimization algorithms (such as stochastic gradient descent). The goal of the update process is to minimize the target loss function so that the generated face features not only conform to the desired speech features but also have good recognition performance for different speakers.

[0097] In this step, based on the target loss function value, the model parameters of the original face feature generation model can be gradually adjusted during training to obtain the target face feature generation model. This target face feature generation model is then used to identify the speech data to be recognized, making the generated face features more consistent with the desired speech features and more robust to speaker classification tasks. This joint training approach helps the model find a balance across multiple tasks, thereby improving overall performance.

[0098] In a preferred embodiment, the speaker adversarial classification module sequentially includes: a gradient inversion layer, a multi-layer downsampling convolutional layer, a global mean pooling layer, and a Logit mapping layer; the step of inputting the output of the preset intermediate layer of the original face feature generation model into the speaker adversarial classification module to obtain the target classification loss function value includes:

[0099] The output of the preset intermediate layer of the original face feature generation model is input into the speaker adversarial classification module, and then passes through the gradient inversion layer, the multi-layer downsampling convolutional layer, the global mean pooling layer and the Logit mapping layer in sequence. After being processed by the SOFTMAX activation function, the predicted speaker label is obtained.

[0100] Calculate the loss function value between the predicted speaker label and the actual speaker label to obtain the classification loss function value;

[0101] The classification loss function value is backpropagated using gradient inversion, and the gradient value of the classification loss function value is inverted in the gradient inversion layer to obtain the target classification loss function value.

[0102] Specifically, such as Figure 3 The diagram illustrates the structure of a speaker adversarial classification module. This module comprises multiple layers: a gradient inversion layer, multiple downsampling convolutional layers, a global mean pooling layer, and a Logit mapping layer. Each layer has a specific function. The following is a detailed explanation of each component in this embodiment:

[0103] The gradient inversion layer is a technique used to train Domain Adversarial Networks (DANNs) to force the model to generate feature representations robust to domain differences. Here, the gradient inversion layer is used to train the speaker adversarial classification module, making the generated face features more robust to speaker classification tasks through adversarial training. Multi-layer downsampling convolutional layers are typically used to extract high-level representations of input features. In the speaker adversarial classification module of this embodiment, multi-layer downsampling convolutional layers are used to further extract features from the output of preset intermediate layers. The global average pooling layer performs global average pooling on the input feature maps, averaging the values ​​of each feature map. This helps reduce the number of parameters, extract global information, and retain important features. The Logit mapping layer maps the output of the global average pooling layer to Logit values, which are unprocessed model outputs typically used for classification problems. In practical applications, the output of the Logit mapping layer also needs to be processed by the SOFTMAX activation function, which converts the Logit values ​​into a probability distribution, making them easier to interpret as probabilities for each class. This probability distribution is used to calculate the predicted speaker label.

[0104] Furthermore, after obtaining the predicted speaker labels, the loss function value between the predicted speaker labels and the actual speaker labels is calculated, typically using cross-entropy loss or other classification loss measures.

[0105] The classification loss function value is backpropagated using gradient inversion, and the gradient value of the classification loss function value is inverted in the gradient inversion layer to obtain the target classification loss function value.

[0106] In this embodiment, adversarial training helps the model learn that the generated facial features are more robust to speaker classification tasks, and the introduction of a gradient inversion layer enhances the model's generalization ability.

[0107] In a preferred embodiment, the step of inputting a set of training data into the original face feature generation model to obtain the face feature loss function value includes:

[0108] Select the spectrum-scaled original speech signal feature frame sequence from a set of training data, and input the spectrum-scaled original speech signal feature frame sequence into the original face feature generation model to obtain the predicted face feature parameter frame sequence;

[0109] The original face feature parameter frame sequence is selected from the same set of training data, and the loss function value between the predicted face feature parameter frame sequence and the original face feature parameter frame sequence is calculated to obtain the face feature loss function value.

[0110] Specifically, in this embodiment, a sequence of original speech signal feature frames after spectral scaling is selected from a set of training data. This sequence is used as input to the original face feature generation model. The original face feature generation model learns to map speech features to corresponding face features, generating a predicted sequence of face feature parameter frames. Another original sequence of face feature parameter frames is selected from the same set of training data, and the predicted sequence is compared with the original sequence. A loss function value is calculated between them. This loss function value measures the difference between the face features generated by the original face feature generation model and the actual face features. Typically, mean squared error or other appropriate loss metrics are used.

[0111] In a preferred embodiment, the spectral scaling of the original speech signal feature frame sequence includes:

[0112] Get the random scaling ratio;

[0113] According to the random scaling ratio, each frame of speech signal features in the original speech signal feature frame sequence is linearly interpolated and scaled, and after the linear interpolation and scaling of any frame of speech signal features is completed, the original dimensions of each speech signal feature are returned.

[0114] Specifically, to perform spectral scaling on the original speech signal feature frame sequence, it is necessary to obtain a random scaling ratio and then perform linear interpolation scaling on each frame of speech signal features in the original speech signal feature frame sequence using the random scaling ratio. After the linear interpolation scaling of any frame of speech signal features is completed, the original dimension of each speech signal feature is returned. This is to maintain the original feature space of the speech signal features so as to ensure consistency with subsequent processing and model input.

[0115] In some embodiments, the step of performing linear interpolation scaling on each frame of the original speech signal feature frame sequence according to the random scaling ratio, and returning to the original dimension of each speech signal feature after the linear interpolation scaling of any frame of the speech signal feature is completed, includes:

[0116] When the random scaling ratio is greater than 0 and less than 1, the speech signal features of any frame are linearly interpolated and reduced to obtain the reduced speech signal features.

[0117] Obtain the last dimension of the reduced speech signal features, and add a random number to the last dimension of the speech signal features to obtain a new speech signal feature, so as to supplement the reduced speech signal features until the dimension of the reduced speech signal features reaches the original dimension.

[0118] Specifically, refer to Figure 4 The diagram illustrates a linear interpolation reduction process. In this embodiment, for cases where the random scaling ratio is greater than 0 and less than 1, linear interpolation reduction is performed on each frame of the original speech signal feature frame sequence. This reduction process includes the following steps: For the speech signal features of each frame, linear interpolation reduction is performed using a random scaling ratio. This means that the value of each feature dimension will be reduced in a linear interpolation manner. The last dimension of the speech signal feature is extracted from the reduced speech signal features, and the last dimension of the speech signal feature is added to a randomly generated value to obtain a new speech signal feature. The generation of this random number helps to supplement the reduced speech signal features.

[0119] For example, if the original Mel spectrum obtained in step 4 is 80-dimensional [M1, M2, ..., M...] 80 ] is denoted as M i Where 1 ≤ i ≤ 80. Randomly select a scaling factor within the range of 0.5-2:

[0120] If the random scaling factor is 0.5, it means scaling to 40 dimensions [N1, N2, ..., N]. 40 Specifically, the 40 values ​​were calculated using linear interpolation.

[0121] Then N i To obtain K, add 80 dimensions. i That is, for 1≤i≤40, K i =N i For 41≤i≤80, K i =N 40 +random means filling the last 40 dimensions with a random number added to the last dimension.

[0122] In some embodiments, the step of performing linear interpolation scaling on each frame of the original speech signal feature frame sequence according to the random scaling ratio, and returning to the original dimension of each speech signal feature after the linear interpolation scaling of any frame of the speech signal feature is completed, includes:

[0123] When the random scaling ratio is greater than or equal to 1, all dimensions of the speech signal features in any frame are sequentially linearly interpolated and amplified to obtain amplified speech signal features. The portion of the amplified speech signal features that exceeds the original dimension is then removed. The amplified speech signal features include the original speech signal features and the speech signal features amplified according to the random scaling ratio.

[0124] Specifically, refer to Figure 5 The diagram illustrates a linear interpolation amplification method. This embodiment describes the steps of linearly interpolating and amplifying each frame of the original speech signal feature frame sequence when the random scaling ratio is greater than or equal to 1. This amplification process includes the following steps: For each frame of speech signal features, linear interpolation amplification is performed using a random scaling ratio. This means that the value of each feature dimension will increase in a linear interpolation manner. The portion of the amplified speech signal features that exceeds the original dimension is then removed to ensure that the dimension does not exceed the original dimension. This can be achieved by simply truncating the first few dimensions of the amplified speech signal features.

[0125] In this embodiment, the amplified speech signal features include the original speech signal features and a portion of the original speech signal features amplified according to a random scaling ratio. This ensures that the amplified features contain information from the original features and introduces some random speech signal features, thereby enabling the model to better adapt to inputs at different scales and improve its generalization ability.

[0126] For example, if the original Mel spectrum obtained in step 4 is 80-dimensional [M1, M2, ..., M...] 80 ] is denoted as M iWhere 1 ≤ i ≤ 80. Randomly select a scaling factor within the range of 0.5-2:

[0127] If the random scaling factor is 2, it means that the image will be scaled up to 160 dimensions [N1, N2, ..., N]. 160 Specifically, the 160 values ​​are calculated through linear interpolation, where N1 = M1, N2 = M1, N3 = M2, N4 = M2, ..., N 159 =M 80 N 160 =M 80 Then N i K is obtained by truncating the first 80 dimensions. i That is, for 1≤i≤80, K i =N i .

[0128] In a preferred embodiment, the step of spectrally scaling the original speech signal feature frame sequence and combining the spectrally scaled original speech signal feature frame sequence, the original face feature parameter frame sequence, and the actual speaker labels of the original video file into a set of training data includes:

[0129] Obtain the original video files of multiple speakers;

[0130] The original video files of each speaker are decomposed into the original speech signal feature frame sequence and the original face feature parameter frame sequence of each speaker.

[0131] The original speech signal feature frame sequences of each speaker are spectrally scaled, and the spectrally scaled original speech signal feature frame sequences of each speaker, their corresponding original face feature parameter frame sequences, and their corresponding actual speaker labels are used to form the training data of each speaker.

[0132] This application provides a method for generating facial features, the method comprising: acquiring speech data to be recognized; inputting the speech data to be recognized into a pre-trained target facial feature generation model to obtain a target facial feature parameter frame sequence, wherein the target facial feature generation model is trained through the following steps: acquiring an original speech signal feature frame sequence and an original facial feature parameter frame sequence, the original speech signal feature frame sequence and the original facial feature parameter frame sequence being aligned in chronological order; performing spectral scaling on the original speech signal feature frame sequence, and merging the spectrally scaled original speech signal feature frame sequence and the original facial feature parameter frame sequence... The sequence and the actual speaker labels of the original video file constitute a set of training data. This set of training data is input into the original face feature generation model to obtain a face feature loss function value. After inputting the training data into the original face feature generation model, the output of the preset intermediate layer of the original face feature generation model is input into the speaker adversarial classification module to obtain a target classification loss function value. The target classification loss function value is added to the face feature loss function value to obtain a target loss function value. The model parameters of the original face feature generation model are updated using the target loss function value to obtain the target face feature generation model. In this application, when pre-training the target face feature generation model, a loss function is added to the model's training data. This function, through a gradient inversion underlying method, aims to intentionally make it difficult for the model to distinguish between speakers, i.e., to eliminate the individual characteristics of different speakers. Simultaneously, random augmentation is performed on the training data by randomly scaling the spectrum of the audio in the training data to change the speaker's timbre, enabling the model to output the original face feature parameters even when the input timbre changes, thereby improving the model's stability to input changes. The target recognition model pre-trained through this application has better generalization and stability.

[0133] Based on the same concept, a second aspect of this application provides a facial feature generation system, the system as follows: Figure 6 As shown, it specifically includes:

[0134] The first acquisition module 201 is used to acquire the speech data to be recognized;

[0135] The first input module 202 is used to input the speech data to be recognized into a pre-trained target face feature generation model to obtain a target face feature parameter frame sequence, wherein the target face feature generation model is trained through the following steps:

[0136] The second acquisition module 203 is used to acquire the original speech signal feature frame sequence and the original face feature parameter frame sequence, wherein the original speech signal feature frame sequence and the original face feature parameter frame sequence are aligned in chronological order.

[0137] The scaling module 204 is used to perform spectral scaling on the original speech signal feature frame sequence, and to combine the spectrally scaled original speech signal feature frame sequence, the original face feature parameter frame sequence, and the actual speaker labels of the original video file into a set of training data.

[0138] The second input module 205 is used to input a set of training data into the original face feature generation model to obtain the face feature loss function value, and after inputting a set of training data into the original face feature generation model, input the output result of the preset intermediate layer of the original face feature generation model into the speaker adversarial classification module to obtain the target classification loss function value.

[0139] The update module 206 is used to add the target classification loss function value and the face feature loss function value to obtain the target loss function value, and update the model parameters of the original face feature generation model through the target loss function value to obtain the target face feature generation model.

[0140] Optionally, the speaker adversarial classification module sequentially includes: a gradient inversion layer, multiple downsampling convolutional layers, a global mean pooling layer, and a Logit mapping layer; the second input module 205, which inputs the output of the preset intermediate layer of the original face feature generation model into the speaker adversarial classification module to obtain the target classification loss function value, includes:

[0141] The first input submodule is used to input the output of the preset intermediate layer of the original face feature generation model into the speaker adversarial classification module, which then passes through the gradient inversion layer, the multi-layer downsampling convolutional layer, the global mean pooling layer, and the Logit mapping layer in sequence, and is calculated by the SOFTMAX activation function to obtain the predicted speaker label.

[0142] The first calculation submodule is used to calculate the loss function value between the predicted speaker label and the actual speaker label to obtain the classification loss function value;

[0143] The backpropagation submodule is used to perform gradient backpropagation on the classification loss function value, and invert the gradient value of the classification loss function value in the gradient inversion layer to obtain the target classification loss function value.

[0144] Optionally, the first input module 202, which inputs a set of training data into the original face feature generation model to obtain the face feature loss function value, includes:

[0145] The first selection submodule is used to select the spectrum-scaled original speech signal feature frame sequence from a set of training data, and input the spectrum-scaled original speech signal feature frame sequence into the original face feature generation model to obtain the predicted face feature parameter frame sequence.

[0146] The second selection submodule is used to select the original face feature parameter frame sequence from the same set of training data, and calculate the loss function value between the predicted face feature parameter frame sequence and the original face feature parameter frame sequence to obtain the face feature loss function value.

[0147] Optionally, the scaling module 204, which performs spectral scaling on the original speech signal feature frame sequence, includes:

[0148] The first acquisition submodule is used to obtain the random scaling ratio;

[0149] The scaling submodule is used to perform linear interpolation scaling on each frame of speech signal features in the original speech signal feature frame sequence according to the random scaling ratio, and after the linear interpolation scaling of any frame of speech signal features is completed, return to the original dimension of each speech signal feature.

[0150] Optionally, the scaling submodule, which performs linear interpolation scaling on each frame of the original speech signal feature frame sequence according to the random scaling ratio, and returns to the original dimension of each speech signal feature after the linear interpolation scaling of any frame of the speech signal feature is completed, includes:

[0151] The reduction subunit is used to perform linear interpolation reduction on the speech signal features of any frame when the random scaling ratio is greater than 0 and less than 1, so as to obtain the reduced speech signal features.

[0152] The supplementary subunit is used to obtain the last dimension of the reduced speech signal features and add a random number to the last dimension of the speech signal features to obtain a new speech signal feature, so as to supplement the reduced speech signal features until the dimension of the reduced speech signal features reaches the original dimension.

[0153] Optionally, the scaling submodule, which performs linear interpolation scaling on each frame of the original speech signal feature frame sequence according to the random scaling ratio, and returns to the original dimension of each speech signal feature after the linear interpolation scaling of any frame of the speech signal feature is completed, includes:

[0154] The amplification subunit is used to sequentially perform linear interpolation amplification on all dimensions of the speech signal features in any frame of the speech signal features when the random scaling ratio is greater than or equal to 1, to obtain amplified speech signal features, and to delete the part of the amplified speech signal features that exceeds the original dimension. The amplified speech signal features include the original speech signal features and the speech signal features amplified according to the random scaling ratio.

[0155] Optionally, the step of spectrally scaling the original speech signal feature frame sequence and combining the spectrally scaled original speech signal feature frame sequence, the original face feature parameter frame sequence, and the actual speaker labels of the original video file into a set of training data includes:

[0156] The second acquisition submodule is used to acquire the original video files of multiple speakers;

[0157] The decomposition submodule is used to decompose the original video files of each speaker into the original speech signal feature frame sequence and the original face feature parameter frame sequence of each speaker.

[0158] The sub-module is used to perform spectral scaling on the original speech signal feature frame sequences of each speaker, and to combine the spectral-scaled original speech signal feature frame sequences of each speaker, their corresponding original face feature parameter frame sequences, and their corresponding actual speaker labels to form the training data of each speaker.

[0159] Optionally, after obtaining the target face feature parameter frame sequence, the method further includes:

[0160] Obtain the original image sequence from the preset video template;

[0161] The target face feature parameter frame sequence and the original image sequence are input into a pre-trained face rendering model to generate the target image.

[0162] Based on the same inventive concept, a third aspect of the embodiments of this application provides an electronic device 100, such as... Figure 7 As shown, it includes a memory 110, a processor 120, and a computer program stored on the memory 110. The processor 120 executes the computer program to implement the method for generating facial features as described in the first aspect of this application.

[0163] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0164] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0165] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0167] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0168] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0169] The above provides a detailed description of the method, system, and electronic device for generating facial features. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for generating facial features, characterized in that, The method includes: Acquire the speech data to be recognized; The speech data to be recognized is input into a pre-trained target face feature generation model to obtain a target face feature parameter frame sequence. The target face feature generation model is trained through the following steps: Obtain the original speech signal feature frame sequence and the original face feature parameter frame sequence, which are aligned in chronological order. The original speech signal feature frame sequence is spectrally scaled, and the spectrally scaled original speech signal feature frame sequence, the original face feature parameter frame sequence, and the actual speaker labels of the original video file are combined into a set of training data. A set of training data is input into the original face feature generation model to obtain the face feature loss function value. After inputting a set of training data into the original face feature generation model, the output of the preset intermediate layer of the original face feature generation model is input into the speaker adversarial classification module to obtain the target classification loss function value. The target classification loss function value is added to the face feature loss function value to obtain the target loss function value. The model parameters of the original face feature generation model are then updated using the target loss function value to obtain the target face feature generation model. The speaker adversarial classification module sequentially includes: a gradient inversion layer, a multi-layer downsampling convolutional layer, a global mean pooling layer, and a Logit mapping layer; the step of inputting the output of the preset intermediate layer of the original face feature generation model into the speaker adversarial classification module to obtain the target classification loss function value includes: The output of the preset intermediate layer of the original face feature generation model is input into the speaker adversarial classification module, and then passes through the gradient inversion layer, the multi-layer downsampling convolutional layer, the global mean pooling layer and the Logit mapping layer in sequence. After being processed by the SOFTMAX activation function, the predicted speaker label is obtained. Calculate the loss function value between the predicted speaker label and the actual speaker label to obtain the classification loss function value; The classification loss function value is backpropagated using gradient inversion, and the gradient value of the classification loss function value is inverted in the gradient inversion layer to obtain the target classification loss function value.

2. The method for generating facial features according to claim 1, characterized in that, The step of inputting a set of training data into the original face feature generation model to obtain the face feature loss function value includes: Select the spectrum-scaled original speech signal feature frame sequence from a set of training data, and input the spectrum-scaled original speech signal feature frame sequence into the original face feature generation model to obtain the predicted face feature parameter frame sequence; The original face feature parameter frame sequence is selected from the same set of training data, and the loss function value between the predicted face feature parameter frame sequence and the original face feature parameter frame sequence is calculated to obtain the face feature loss function value.

3. The method for generating facial features according to claim 1, characterized in that, The spectral scaling of the original speech signal feature frame sequence includes: Get the random scaling ratio; According to the random scaling ratio, each frame of speech signal features in the original speech signal feature frame sequence is linearly interpolated and scaled, and after the linear interpolation and scaling of any frame of speech signal features is completed, the original dimensions of each speech signal feature are returned.

4. The method for generating facial features according to claim 3, characterized in that, The step of performing linear interpolation scaling on each frame of the original speech signal feature frame sequence according to the random scaling ratio, and returning to the original dimension of each speech signal feature after the linear interpolation scaling of any frame of the speech signal feature is completed, includes: When the random scaling ratio is greater than 0 and less than 1, the speech signal features of any frame are linearly interpolated and reduced to obtain the reduced speech signal features. Obtain the last dimension of the reduced speech signal features, and add a random number to the last dimension of the speech signal features to obtain a new speech signal feature, so as to supplement the reduced speech signal features until the dimension of the reduced speech signal features reaches the original dimension.

5. The method for generating facial features according to claim 3, characterized in that, The step of performing linear interpolation scaling on each frame of the original speech signal feature frame sequence according to the random scaling ratio, and returning to the original dimension of each speech signal feature after the linear interpolation scaling of any frame of the speech signal feature is completed, includes: When the random scaling ratio is greater than or equal to 1, all dimensions of the speech signal features in any frame are sequentially linearly interpolated and amplified to obtain amplified speech signal features. The portion of the amplified speech signal features that exceeds the original dimension is then removed. The amplified speech signal features include the original speech signal features and the speech signal features amplified according to the random scaling ratio.

6. The method for generating facial features according to claim 1, characterized in that, The process involves spectrally scaling the original speech signal feature frame sequence and combining the spectrally scaled original speech signal feature frame sequence, the original face feature parameter frame sequence, and the actual speaker labels from the original video file into a set of training data, including: Obtain the original video files of multiple speakers; The original video files of each speaker are decomposed into the original speech signal feature frame sequence and the original face feature parameter frame sequence of each speaker. The original speech signal feature frame sequences of each speaker are spectrally scaled, and the spectrally scaled original speech signal feature frame sequences of each speaker, their corresponding original face feature parameter frame sequences, and their corresponding actual speaker labels are used to form the training data of each speaker.

7. The method for generating facial features according to any one of claims 1-6, characterized in that, After obtaining the target face feature parameter frame sequence, the method further includes: Obtain the original image sequence from the preset video template; The target face feature parameter frame sequence and the original image sequence are input into a pre-trained face rendering model to generate the target image.

8. A facial feature generation system, characterized in that, The system includes: The first acquisition module is used to acquire the speech data to be recognized; The first input module is used to input the speech data to be recognized into a pre-trained target face feature generation model to obtain a target face feature parameter frame sequence, wherein the target face feature generation model is trained through the following steps: The second acquisition module is used to acquire the original speech signal feature frame sequence and the original face feature parameter frame sequence, wherein the original speech signal feature frame sequence and the original face feature parameter frame sequence are aligned in chronological order. The scaling module is used to perform spectral scaling on the original speech signal feature frame sequence, and to combine the spectrally scaled original speech signal feature frame sequence, the original face feature parameter frame sequence, and the actual speaker labels of the original video file into a set of training data. The second input module is used to input a set of training data into the original face feature generation model to obtain the face feature loss function value, and after inputting a set of training data into the original face feature generation model, input the output result of the preset intermediate layer of the original face feature generation model into the speaker adversarial classification module to obtain the target classification loss function value. The update module is used to add the target classification loss function value and the face feature loss function value to obtain the target loss function value, and update the model parameters of the original face feature generation model through the target loss function value to obtain the target face feature generation model; The speaker adversarial classification module comprises, in sequence: a gradient inversion layer, a multi-layer downsampling convolutional layer, a global mean pooling layer, and a Logit mapping layer; the second input module comprises: The first input submodule is used to input the output of the preset intermediate layer of the original face feature generation model into the speaker adversarial classification module, which then passes through the gradient inversion layer, the multi-layer downsampling convolutional layer, the global mean pooling layer, and the Logit mapping layer in sequence, and is calculated by the SOFTMAX activation function to obtain the predicted speaker label. The first calculation submodule is used to calculate the loss function value between the predicted speaker label and the actual speaker label to obtain the classification loss function value; The backpropagation submodule is used to perform gradient backpropagation on the classification loss function value, and invert the gradient value of the classification loss function value in the gradient inversion layer to obtain the target classification loss function value.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method for generating facial features as described in any one of claims 1 to 7.

Images