Method and device for generating face image based on voice

Abstract

The invention relates to the technical field of face generation, in particular to a method and device for generating a face image based on voice. The method comprises the following steps: extracting shallow audio features of target voice through an audio feature extractor; the shallow audio features comprise energy distribution of the target voice; a multi-dimensional attention mechanism for highlighting the features of voice energy is introduced into the audio feature extractor; based on the shallow audio features, a corresponding target face image is obtained through a stylized generative adversarial network; the stylized generative adversarial network comprises a generator, a discriminator and a classifier used for improving the identity accuracy of the target face image, and the generator comprises an audio feature mapping module used for extracting deep audio features; according to the method, the deep features of the voice are actively extracted through the neural network, and the problem that the generated face image is not matched with the voice is effectively avoided.

Description

Technical Field

[0001] The present application relates to the technical field of facial image generation, and in particular to a method and device for generating facial images based on speech. Background Art

[0002] When humans speak, the sound is emitted by the vibration of the vocal cords, which is amplified by the chest and mouth. At the same time, the cheekbones, chin, nose and lips also vibrate, affecting the final sound produced by the person. Therefore, a person's facial information can be derived from his voice. However, since the relationship between sound and facial information is not intuitive and there is a lack of effective analysis tools, it is difficult to use this relationship in the field of face image generation.

[0003] Through digital signal processing methods, various useful feature parameters can be extracted from speech, such as fundamental frequency, formant, energy and Mel frequency cepstral coefficients, which can reflect the speaker's physiological characteristics and emotional state. By using neural network models, face generation technology can generate realistic face graphics based on a small amount or specific data. Therefore, it is possible to establish a mapping relationship between human voice and face image through neural network models.

[0004] At present, the neural network model for predicting facial images based on speech is unable to actively extract the deep features of speech, resulting in the problem that the generated facial images do not match the speech. Summary of the invention

[0005] In order to solve the problems existing in the above-mentioned prior art, the purpose of this application is to provide a method and device for generating facial images based on speech.

[0006] In a first aspect, the present application provides a method for generating a face image based on speech, comprising:

[0007] The shallow audio features of the target speech are extracted by an audio feature extractor; the shallow audio features include the energy distribution of the target speech; the multi-dimensional attention mechanism for highlighting the features of speech energy is introduced in the audio feature extractor;

[0008] Based on shallow audio features, deep audio features are extracted through a stylized generative adversarial network to obtain the corresponding target face image; the stylized generative adversarial network includes a generator, a discriminator and a classifier for improving the identity accuracy of the target face image; the generator includes an audio feature mapping module for extracting deep audio features.

[0009] In one embodiment, the audio feature extractor includes a filter, a residual block and an enhanced gate loop block; the residual block includes a convolution layer, a multi-dimensional attention layer and a filter feature scaling layer; the step of extracting shallow audio features of the target speech by the audio feature extractor includes:

[0010] Input the target speech into the filter to obtain filtered speech data; the data size of the speech data is the same as the input data size of the residual block;

[0011] Input the speech data into the convolution layer, and obtain the initial features to input into the multi-dimensional attention layer;

[0012] In the multi-dimensional attention layer, the energy correlation between each element in the initial feature and its adjacent elements is calculated as the element weight; the element weight is applied to the initial feature to obtain the energy feature;

[0013] The feature weights in the energy feature are optimized through the filter feature scaling layer to obtain the speech feature;

[0014] The speech features are input into the enhanced gated loop block to obtain shallow audio features.

[0015] In one embodiment, the energy correlation between each element in the initial feature and the adjacent elements is calculated by the following formula:

[0016]

[0017] Among them, Φ() represents the multi-dimensional attention mechanism; Represents the initial feature M f The energy of the target element t in; μ represents the initial feature M in each channel f The mean of 2 Represents the initial feature M in each channel f The variance of ;λ is a hyperparameter; Represents the target element t and the initial feature M f Energy correlations with other elements in the

[0018] In one embodiment, the generator of the stylized generative adversarial network includes an audio feature mapping module and a plurality of pixel generation blocks arranged in sequence; the audio feature mapping module encodes shallow audio features into deep audio features through a plurality of fully connected layers; the pixel generation block obtains an output image through stylized convolution and image convolution.

[0019] In one embodiment, the method further comprises:

[0020] In the first pixel generation block, the preset constant tensor is convolved with the stylized convolution kernel to obtain the output features; the output features are convolved with the image convolution kernel to obtain the output image; the weights of the stylized convolution kernel are obtained based on the deep audio features and random noise;

[0021] In subsequent pixel generation blocks except the first pixel generation block, an enhanced output feature and an enhanced output image are obtained respectively by upsampling the output feature and the output image of the previous pixel generation block; the output feature of the current pixel generation block is obtained by convolving the enhanced output feature with the stylized convolution kernel of the current pixel generation block, and the output feature of the current pixel generation block is convolved with the imaged convolution kernel of the current pixel generation block and multiplied element-by-element with the enhanced output image to obtain the output image of the current pixel generation block; the H dimension and W dimension of the enhanced output feature are twice the output feature of the previous pixel generation block; the H dimension and W dimension of the enhanced output image are twice the output image of the previous pixel generation block;

[0022] The target face image is obtained based on the output image of the last pixel generation block.

[0023] In one embodiment, the method further comprises:

[0024] Perform an affine transformation on the deep audio code and add a random noise vector to obtain the initial weights of the stylized convolution kernel;

[0025] The initial weights of the stylized convolution kernels of each channel are normalized by the Euclidean norm to obtain the weights of the stylized convolution kernels.

[0026] In one embodiment, a training set for training an audio feature extractor and a stylized generative adversarial network includes speech data, a real face image corresponding to the speech data, and a true identity encoding of the real face image; and the method further includes:

[0027] The facial features of each real face image in the training set are extracted through a classifier, and the facial features are matched with each real identity code;

[0028] The generated identity code of the generated face image is obtained according to the facial features of the generated face image through a classifier; the generated face image is generated based on the speech data in the training set.

[0029] In one embodiment, a discriminator is used to determine whether a generated face image is a real face image; the discriminator includes a small batch standard deviation layer; and the method further includes:

[0030] The convolutional layer of the discriminator is used to extract features from the real face image and the generated face image to obtain the difference features;

[0031] The mini-batch standard deviation of the difference feature is calculated through the mini-batch standard deviation layer to obtain the standard deviation feature;

[0032] The standard deviation feature and the difference feature are combined to obtain a combined difference feature, and based on the combined difference feature, an identification result that the generated face image is a real face image is obtained.

[0033] In one embodiment, during the training process of the stylized generative adversarial network, the loss function used is a joint loss function; the joint loss function includes a loss function of the generator and a loss function of the discriminator;

[0034] The loss function L of the generator G(all) The formula is as follows:

[0035] L G(all) =L G(max) +L P2(g)

[0036] Among them, L G(max) represents the adversarial cross entropy loss of the generator; L P2(g) represents the path length penalty of the generator; the path length penalty is used to increase the information content of deep audio features;

[0037] The loss function L of the discriminator D(all) The formula is as follows:

[0038] L D(all) =L D(min) +L P1(d)

[0039] Among them, L D(min) represents the total cross entropy loss of the discriminator; L P1(d) represents the gradient penalty loss of the discriminator; the gradient penalty is used to alleviate the mode collapse of the stylized generative adversarial network and enhance the training stability of the stylized generative adversarial network.

[0040] In a second aspect, the present application provides a device for generating a face image based on speech, comprising:

[0041] An audio feature extraction module is used to extract shallow audio features of the target speech through an audio feature extractor; the shallow audio features include the energy distribution of the target speech; a multi-dimensional attention mechanism for highlighting the features of speech energy is introduced in the audio feature extractor;

[0042] The face image generation module is used to extract deep audio features based on shallow audio features through a stylized generative adversarial network to obtain the corresponding target face image; the stylized generative adversarial network includes a generator, a discriminator and a classifier for improving the identity accuracy of the target face image; the generator includes an audio feature mapping module for extracting deep audio features.

[0043] The present application provides a method for generating a face image based on speech. The method extracts shallow audio features in a target speech by introducing an audio feature extractor that incorporates a multi-dimensional attention mechanism, wherein the multi-dimensional attention mechanism can highlight the energy features in the target speech and help a stylized generative network generate a more accurate face image. The method generates a target face image by extracting deep audio features through a generator of a stylized generative adversarial network, thereby obtaining finer deep features of the target speech and generating a more realistic and accurate target face image based on the deep features. In the training process of a neural network model, the confrontation between the generator and the discriminator of the stylized generative adversarial network can improve the authenticity of the generated target face image, and the confrontation between the generator and the classifier of the stylized generative adversarial network can make the target face image closer to the real identity of the person who makes the speech. The above method for generating a face image based on speech actively extracts deep data of speech through a neural network, effectively avoiding the problem of mismatch between the generated face image and the speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flowchart of the steps of a method for generating a face image based on speech in an embodiment;

[0045] Figure 2 is a flowchart of steps for extracting shallow audio features of a target speech by an audio feature extractor in one embodiment;

[0046] Figure 3 is a flowchart of steps for generating a target face image by a generator in one embodiment;

[0047] Figure 4 is a flowchart of steps for modulating and demodulating the weights of a stylized convolution kernel in one embodiment;

[0048] Figure 5 is a flow chart of the steps of generating an output identity encoding of a face image by a classifier output in one embodiment;

[0049] Figure 6 is a flow chart of steps for a discriminator in one embodiment to obtain a discriminant result that a generated face image is a real face image;

[0050] Figure 7 The present invention is a structural block diagram of an apparatus for generating a facial image based on speech in an embodiment. DETAILED DESCRIPTION

[0051] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0053] When used herein, the singular forms "a", "an", and "said / the" may also include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "include / comprise" or "have" and the like specify the presence of stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. At the same time, the term "and / or" used in this specification includes any and all combinations of the relevant listed items.

[0054] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0055] This application provides a method for generating a face image based on speech, such as Figure 1 As shown, the following steps S12 to S14 are included:

[0056] S12, extracting shallow audio features of the target speech through an audio feature extractor; the shallow audio features include the energy distribution of the target speech; and a multi-dimensional attention mechanism for highlighting the features of speech energy is introduced in the audio feature extractor.

[0057] Specifically, the target speech is feature extracted by combining the residual neural network and the recurrent neural network and introducing an audio feature extractor with a multi-dimensional attention mechanism, and the energy distribution of different elements in the speech data is obtained as a shallow audio feature; wherein, according to the multi-dimensional attention mechanism, the energy correlation between each element in the energy distribution and other elements is calculated, and the characteristic elements are highlighted.

[0058] S14, based on the audio features, deep audio features are extracted through a stylized generative adversarial network to obtain a corresponding target face image; the stylized generative adversarial network includes a generator, a discriminator and a classifier for improving the identity accuracy of the target face image; the generator includes an audio feature mapping module for extracting deep audio features.

[0059] Specifically, the stylized generative adversarial network includes a generator, a discriminator and a classifier; among them, the audio feature mapping module of the generator can encode shallow audio features into deeper audio features with richer information through a fully connected neural network; the generator further performs upsampling and stylized convolution on the deep audio features to obtain the generated target face image.

[0060] Furthermore, during the training process of the stylized generative adversarial network, the discriminator determines whether the target face image is a real face image by comparing the features of the generated target face image with the real face images from the training set, which can help the generator generate a target face image that is closer to a real face; the classifier determines the face identity based on the facial features of the target face image, and determines whether the target face image matches the identity of the person who speaks the voice based on the identity corresponding to the voice, which can help the generator generate a target face image that is more consistent with the identity of the person who speaks the voice.

[0061] The above-mentioned method for generating facial images based on speech extracts shallow audio features in the target speech by introducing an audio feature extractor with a multi-dimensional attention mechanism, wherein the multi-dimensional attention mechanism can highlight the energy features in the target speech and can help the stylized generative network generate more accurate facial images; generates the target facial image after extracting deep audio features by the generator of the stylized generative adversarial network, can obtain more refined deep features of the target speech, and generate more realistic and accurate target facial images based on the deep features; in the training process of the neural network model, the confrontation between the generator and the discriminator of the stylized generative adversarial network can improve the authenticity of the generated target facial image, and the confrontation between the generator and the classifier of the stylized generative adversarial network can make the target facial image closer to the true identity of the person who speaks the speech; the method actively extracts deep data of the speech through the neural network, effectively avoiding the problem of mismatch between the generated facial image and the speech.

[0062] In an exemplary embodiment, the audio feature extractor includes a filter, a residual block and an enhanced gate recurrent block; the residual block includes a convolutional layer, a multi-dimensional attention layer and a filter feature scaling layer; Figure 2 As shown, extracting shallow audio features of the target speech by an audio feature extractor includes the following steps S1202 to S1210:

[0063] S1202, input the target speech into the filter to obtain filtered speech data; the data size of the speech data is the same as the input data size of the residual block.

[0064] Specifically, in the filter, the input target speech is first cropped into speech data of a data size suitable for subsequent processing; preliminary feature extraction is performed on the speech data through convolution, pooling, normalization and activation functions, and then input into the subsequent residual block.

[0065] Preferably, the data adapted for subsequent processing is 64K audio data, including 4s of audio data, and contains 16K of data per second.

[0066] S1204, input the speech data into the convolution layer, and input the obtained initial features into the multi-dimensional attention layer.

[0067] Specifically, in each residual block, the speech features of the speech data are further extracted through convolution, batch normalization and activation function to obtain the initial features, which are then input into the multi-dimensional attention layer.

[0068] S1206, in the multi-dimensional attention layer, calculate the energy correlation between each element in the initial feature and the adjacent elements as the element weight; apply the element weight to the initial feature to obtain the energy feature.

[0069] Specifically, in the multidimensional attention layer, based on the multidimensional attention mechanism formula, the energy correlation between each element in the initial feature and other elements is calculated; the energy correlation between each element in the initial feature and the adjacent elements is calculated by the following formula:

[0070]

[0071] Among them, Φ() represents the multi-dimensional attention mechanism; Represents the initial feature M f The energy of the target element t in; μ represents the initial feature M in each channel f The mean of 2 Represents the initial feature M in each channel f The variance of ;λ is a hyperparameter; Represents the target element t and the initial feature M f Energy correlations with other elements in the

[0072] Furthermore, the inverse of the energy correlation is multiplied by the corresponding element one by one to obtain the energy feature, which is as follows:

[0073]

[0074] in, Represents energy characteristics; sigmoid is a common activation function; e f is the energy correlation of each element; ⊙ represents element-by-element multiplication.

[0075] It should be explained that the lower the energy correlation is, the more obvious the difference between the target element and other elements is. Therefore, multiplying the inverse of the energy correlation by the target element can highlight the target element in the energy feature.

[0076] S1208, optimizing feature weights in energy features through a filter feature scaling layer to obtain speech features.

[0077] Specifically, the Sigmoid function is used to scale the feature weights of each element in the energy feature within the range of [0,1] to enhance important features and suppress unimportant features; the optimized energy feature is then input into the next residual block for a new round of feature extraction; until in the last residual block, the output energy feature is output as a speech feature to the enhancement gate loop block.

[0078] S1210, input the speech features into the enhanced gated loop block to obtain shallow audio features.

[0079] Specifically, by introducing an enhanced gated recurrent block with an attention mechanism, the speech features are further extracted to obtain shallow audio features.

[0080] It should be noted that the enhanced gated recurrent block includes a gated recurrent unit (GRU), a fully connected layer, and a Squeeze-and-Excitation Networks (SENet) layer; among them, GRU is a variant of a recurrent neural network that can extract hidden states in a speech sequence to capture temporal dependencies in the speech sequence that are beneficial to speech recognition; SENet is an attention mechanism that compresses the features of each channel of the input feature into a single feature value through global average pooling to form a channel feature vector, and then generates a weight tensor for each channel according to the channel feature vector through a fully connected layer and an activation function, and multiplies it with the input feature channel by channel to obtain the output feature.

[0081] In an exemplary embodiment, the generator of the stylized generative adversarial network includes an audio feature mapping module and a plurality of sequentially arranged pixel generation blocks; the audio feature mapping module encodes shallow audio features into deep audio features through a plurality of fully connected layers; the pixel generation block obtains an output image through stylized convolution and image convolution.

[0082] Specifically, in the generator, the audio features are first encoded into deep audio features containing finer features through an audio feature mapping module; then, a target face image is generated based on the deep audio features through multiple pixel generation blocks.

[0083] Furthermore, the audio feature mapping module includes multiple fully connected layers, which perform linear and nonlinear transformations layer by layer and optimize the weights of each fully connected layer in the audio feature mapping module during the backward transfer process of model training, thereby gradually encoding the audio features into deep audio features.

[0084] Preferably, the audio feature mapping module includes 6 fully connected layers; the generator includes 6 pixel generation blocks, which can expand and transform the initial tensor layer by layer through stylized convolution and image convolution to obtain a large-size output image.

[0085] In an exemplary embodiment, Figure 3 As shown, the method further includes the following steps S22 to S26:

[0086] S22, in the first pixel generation block, convolving the preset constant tensor through the stylized convolution kernel to obtain the output feature; convolving the output feature through the image convolution kernel to obtain the output image; the weight of the stylized convolution kernel is obtained based on the deep audio feature and random noise.

[0087] Preferably, the preset constant tensor (256, 4, 4) is a tensor with 256 channels, each channel having a size of 4×4.

[0088] Specifically, in the first pixel generation block, the preset constant tensor (256, 4, 4) is stylized convolutioned by the stylized convolution kernel of this layer to obtain the output feature (128, 4, 4); the output feature (128, 4, 4) is imaged convolutioned by the imaged convolution kernel of this layer to obtain the output image (3, 4, 4); the size of the stylized convolution kernel is 3×3, and the size of the imaged convolution kernel is 1×1.

[0089] S24, in subsequent pixel generation blocks except the first pixel generation block, an enhanced output feature and an enhanced output image are obtained respectively by upsampling the output feature and the output image of the previous pixel generation block; the output feature of the current pixel generation block is obtained by convolving the enhanced output feature with the stylized convolution kernel of the current pixel generation block, and the output feature of the current pixel generation block is convolved with the image convolution kernel of the current pixel generation block and multiplied element-by-element with the enhanced output image to obtain the output image of the current pixel generation block; the H dimension and W dimension of the enhanced output feature are twice the size of the output feature of the previous pixel generation block; the H dimension and W dimension of the enhanced output image are twice the size of the output image of the previous pixel generation block.

[0090] Exemplarily, in the second pixel generation block, the output feature (128, 4, 4) of the previous pixel generation block and the output image (3, 4, 4) of the previous pixel generation block are first upsampled to obtain enhanced output features (128, 8, 8) and enhanced output image (3, 8, 8), respectively; then, the enhanced output feature (128, 8, 8) is convolved with the stylized convolution kernel of this layer to obtain the output feature (64, 8, 8) of this layer, and the output feature (64, 8, 8) of this layer is convolved with the imaging convolution kernel of this layer and multiplied with the enhanced output image (3, 8, 8) to obtain the output image (3, 8, 8) of this layer.

[0091] S26, obtaining a target face image according to the output image of the last pixel generation block.

[0092] Specifically, the output image (3, 128, 128) of the sixth pixel generation block is used as the target face image.

[0093] In an exemplary embodiment, Figure 4 As shown, the method further includes the following steps S32 to S34:

[0094] S32, performs an affine transformation on the deep audio code and adds a random noise vector to obtain the initial weights of the stylized convolution kernel.

[0095] Specifically, the deep audio features are affine transformed into a style modulation vector through a fully connected layer; the fully connected layer uses the Affine function; the style modulation vector includes the scale factor y s and the offset factor y b ; Modulate the input feature x according to the style modulation vector and the random noise vector n c Get the modulated characteristic x' c , the modulation formula is as follows:

[0096] x' c =y s ·x c +y b +γ·n

[0097] Among them, γ is a trainable parameter used to control the influence of the random noise vector; by modulating the style modulation vector, the distribution of the output features is changed, which can control the global structure and texture of the generated image; adding random noise to the offset can increase the fine-grainedness of the generated image;

[0098] According to the weight size of the stylized convolution kernel required for each pixel generation block, ys is mapped to the initial weight of the stylized convolution kernel of each channel of the corresponding size through the fully connected layer, and y b +γ·n is mapped to an offset.

[0099] S34, performing Euclidean norm normalization on the initial weights of the stylized convolution kernels of each channel to obtain the weights of the stylized convolution kernels.

[0100] Specifically, the Euclidean norm of each channel is calculated, and then the initial weight of each channel is divided by the Euclidean norm of the channel to obtain the weight of the stylized convolution kernel of the channel; Euclidean norm normalization can ensure that the weight of each channel remains in a reasonable range and enhance the stability of the generator during training; the Euclidean norm calculation formula for each channel is as follows:

[0101]

[0102] in, Indicates the number of input channels is C in , the number of output channels is C out , the initial weight of the stylized convolution kernel with height h and width w; || ||2 means calculating the Euclidean norm.

[0103] In an exemplary embodiment, the training set for training the audio feature extractor and the stylized generative adversarial network includes speech data, real face images corresponding to the speech data, and real identity encodings of the real face images; Figure 5 As shown, the method further includes the following steps S42 to S44:

[0104] S42, extracting facial features of each real face image in the training set through a classifier, and matching the facial features with each real identity code.

[0105] Specifically, during the training process of the stylized generative adversarial network, the classifier is trained to recognize the real face images in the training set, and the output layer of the classifier is set to a softmax function with the number of classifications equal to the number of real identity codes, so that the classifier can classify the real face images in the training set into the corresponding real identity codes.

[0106] S44, obtaining a generated identity code of the generated face image according to the facial features of the generated face image through a classifier; the generated face image is generated based on the voice data in the training set.

[0107] Specifically, the generated face image generated by the generator based on the voice data of the training set is input into the classifier, the generated identity code of the generated face image is obtained based on the extracted facial features, and the generated identity code is compared with the real identity code of the voice data to obtain the matching result between the generated face image and the real identity; through the classifier, the ability of the generator to generate generated face images that match the identity can be improved.

[0108] For example, if there are 20 real identity codes in the training set, the output of the classifier is a tensor (20,1) with 20 channels. Each channel has only a single value, which represents the possibility of generating a face image for the identity code. The identity code corresponding to the channel with the greatest possibility is taken as the generated identity code.

[0109] In an exemplary embodiment, the discriminator is used to determine whether the generated face image is a real face image; the discriminator includes a small batch standard deviation layer; Figure 6 As shown, the method further includes the following steps S52 to S56:

[0110] S52, extracting features of the real face image and the generated face image through the convolution layer of the discriminator to obtain difference features.

[0111] Specifically, the structure of the discriminator is opposite to that of the generator and has multiple convolution blocks. During the training process of the stylized generative adversarial network, the face image and the generated face image are convolved and downsampled through multiple convolution blocks to extract features and reduce the size of the features to obtain difference features.

[0112] S54, calculating the small batch standard deviation of the difference feature through the small batch standard deviation layer to obtain the standard deviation feature.

[0113] Preferably, a mini-batch standard deviation layer is added after the convolutional block of the discriminator to improve the performance of the discriminator.

[0114] Specifically, the difference features are divided into four small batches and flattened into a two-dimensional difference feature map as input; then the average value and standard deviation of each position in the four two-dimensional difference feature maps are calculated. The formula for calculating the average value is as follows:

[0115]

[0116] in, represents the difference feature value of the jth position in the i-th mini-batch in the two-dimensional difference feature map, i∈[1,4], μ j Represents the average value of the difference feature value at the jth position in four small batches; the formula for calculating the standard deviation is as follows:

[0117]

[0118] Among them, σ j Represents the standard deviation of the difference eigenvalues ​​at the jth position in the four mini-batch layers; ε is set to 1e-8; the mean of the standard deviations at all positions is calculated to obtain a single-valued standard deviation tensor; the formula for calculating the mean of the standard deviations at all positions is as follows:

[0119]

[0120] Among them, σ represents the mean of the standard deviation at each position; B represents the size of the batch; C represents the number of channels; H represents the height of the difference feature; W represents the width of the difference feature; the standard deviation tensor is expanded so that its height and width are the same as those of the difference feature to obtain the standard deviation feature.

[0121] S56, combining the standard deviation feature and the difference feature to obtain a combined difference feature, and obtaining an identification result that the generated face image is a real face image based on the combined difference feature.

[0122] Specifically, the concat function is used to connect the difference feature and the standard deviation feature along the channel dimension to obtain the combined difference feature; the size of the combined difference feature is reduced through a downsampling operation, and finally the identification result is output through a fully connected layer; the identification result is the probability value of the generated face image being a real face image, and the closer the probability value is to 1, the more real the generated face image is.

[0123] In an exemplary embodiment, during the training process of the stylized generative adversarial network, the loss function used is a joint loss function; the joint loss function includes a loss function of the generator, a loss function of the discriminator, and a loss function of the classifier;

[0124] Specifically, the loss function L of the generator is G(all) The formula is as follows:

[0125] L G(all) =L G(max) +L P2(g)

[0126] Among them, L G(max) represents the adversarial cross entropy loss of the generator; L P2(g) represents the path length penalty of the generator; the adversarial cross entropy loss L of the generator G(max) It is obtained by adding the cross entropy loss of the discriminator to determine whether the generated face image is a real face image, and the cross entropy loss of the classifier to determine whether the identity of the generated face image matches the identity of the voice speaker; the path length penalty can not only help the audio feature mapping module of the generator to encode deeper audio features with richer information, but also enhance the realism and coherence of the generated face image.

[0127] On the other hand, the loss function L of the discriminator D(all) The formula is as follows:

[0128] L D(all) =L D(min) +L P1(d)

[0129] Among them, L D(min) represents the total cross entropy loss of the discriminator; LP1(d) represents the gradient penalty loss of the discriminator; the total cross entropy loss L of the discriminator D(min) It is obtained by adding the cross entropy loss when the discriminator judges that the generated face image is fake, and the cross entropy loss when the discriminator judges that the real face image in the training set is real; the gradient penalty is used to alleviate the mode collapse of the stylized generative adversarial network and enhance the training stability of the stylized generative adversarial network.

[0130] On the other hand, the loss function L of the classifier C(min) It is obtained by calculating the cross entropy loss of the classifier to determine the real face images in the training set that are classified into the corresponding real identity encoding.

[0131] It should be understood that although Figure 1-Figure 6 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1-Figure 6 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0132] In a second aspect, the present application provides a device 70 for generating a face image based on speech, such as Figure 7 As shown, including:

[0133] The audio feature extraction module 71 is used to extract shallow audio features of the target speech through an audio feature extractor; the shallow audio features include the energy distribution of the target speech; and a multi-dimensional attention mechanism for highlighting the features of speech energy is introduced in the audio feature extractor.

[0134] The face image generation module 72 is used to extract deep audio features based on shallow audio features through a stylized generative adversarial network to obtain a corresponding target face image; the stylized generative adversarial network includes a generator, a discriminator and a classifier for improving the identity accuracy of the target face image; the generator includes an audio feature mapping module for extracting deep audio features.

[0135] For the specific definition of a device for generating facial images based on speech, please refer to the definition of a method for generating facial images based on speech above, which will not be repeated here. Each module in the above-mentioned device for generating facial images based on speech can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0136] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0137] In the description of this specification, the description with reference to the terms "some embodiments", "other embodiments", "ideal embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example.

[0138] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for generating a face image based on speech, characterized in that: The method comprises: Extracting shallow audio features of the target speech through an audio feature extractor; the shallow audio features include energy distribution of the target speech; the audio feature extractor introduces a multi-dimensional attention mechanism for highlighting features of speech energy; Based on the shallow audio features, deep audio features are extracted through a stylized generative adversarial network to obtain a corresponding target face image; the stylized generative adversarial network includes a generator, a discriminator and a classifier for improving the identity accuracy of the target face image; the generator includes an audio feature mapping module for extracting the deep audio features.

2. The method according to claim 1, characterized in that The audio feature extractor includes a filter, a residual block and an enhanced gate loop block; the residual block includes a convolution layer, a multi-dimensional attention layer and a filter feature scaling layer; The step of extracting the shallow audio features of the target speech by an audio feature extractor comprises: Inputting the target speech into the filter to obtain filtered speech data; the data size of the speech data is the same as the input data size of the residual block; Input the speech data into the convolution layer to obtain initial features and input them into the multi-dimensional attention layer; In the multidimensional attention layer, the energy correlation between each element in the initial feature and the adjacent elements is calculated as the element weight; the element weight is applied to the initial feature to obtain the energy feature; Optimizing the feature weights in the energy feature through the filter feature scaling layer to obtain the speech feature; The speech features are input into an enhanced gated loop block to obtain the shallow audio features.

3. The method according to claim 2, characterized in that The energy correlation between each element and the adjacent elements in the initial feature is calculated by the following formula: Among them, Φ() represents the multi-dimensional attention mechanism; Represents the initial feature M f The energy of the target element t in; μ represents the initial feature M in each channel f The mean of 2 Represents the initial feature M in each channel f The variance of ; λ is a hyperparameter; Represents the target element t and the initial feature M f Energy correlation with other elements in the 4. The method according to claim 1, characterized in that The generator of the stylized generative adversarial network includes the audio feature mapping module and a plurality of pixel generation blocks arranged in sequence; the audio feature mapping module encodes the shallow audio features into the deep audio features through a plurality of fully connected layers; The pixel generation block obtains an output image through stylized convolution and image-based convolution.

5. The method according to claim 4, characterized in that The method further comprises: In the first pixel generation block, a preset constant tensor is convolved by a stylized convolution kernel to obtain an output feature; the output feature is convolved by an image convolution kernel to obtain an output image; the weight of the stylized convolution kernel is obtained based on the deep audio feature and random noise; In subsequent pixel generation blocks except the first pixel generation block, an enhanced output feature and an enhanced output image are obtained respectively by upsampling the output feature and the output image of the previous pixel generation block; the enhanced output feature is convolved by the stylized convolution kernel of the current pixel generation block to obtain the output feature of the current pixel generation block, and the output feature of the current pixel generation block is convolved by the imaged convolution kernel of the current pixel generation block and multiplied element-by-element with the enhanced output image to obtain the output image of the current pixel generation block; the H dimension and W dimension of the enhanced output feature are twice the size of the output feature of the previous pixel generation block; the H dimension and W dimension of the enhanced output image are twice the size of the output image of the previous pixel generation block; The target face image is obtained according to the output image of the last pixel generation block.

6. The method according to claim 5, characterized in that The method further comprises: Performing an affine transformation on the deep audio code and adding a random noise vector to obtain an initial weight of a stylized convolution kernel; The initial weight of the stylized convolution kernel of each channel is normalized by the Euclidean norm to obtain the weight of the stylized convolution kernel.

7. The method according to claim 6, characterized in that A training set for training the audio feature extractor and the stylized generative adversarial network includes speech data, a real face image corresponding to the speech data, and a true identity encoding of the real face image; The method further comprises: Extracting facial features of each real face image in the training set by a classifier, and matching the facial features with each real identity code; The generated identity code of the generated face image is obtained by the classifier according to the facial features of the generated face image; the generated face image is generated based on the voice data in the training set.

8. The method according to claim 7, characterized in that The discriminator is used to determine whether the generated face image is a real face image; the discriminator includes a small batch standard deviation layer; the method also includes: Extracting features from the real face image and the generated face image through the convolution layer of the discriminator to obtain difference features; Calculating the small batch standard deviation of the difference feature through the small batch standard deviation layer to obtain a standard deviation feature; The standard deviation feature and the difference feature are combined to obtain a combined difference feature, and based on the combined difference feature, an identification result that the generated face image is the real face image is obtained.

9. The method according to claim 8, characterized in that In the training process of the stylized generative adversarial network, the loss function used is a joint loss function; the joint loss function includes the loss function of the generator and the loss function of the discriminator; The loss function L of the generator G(all) The formula is as follows: L G(all) =L G(max) +L P2(g) Among them, L G(max) represents the adversarial cross entropy loss of the generator; L P2(g) represents a path length penalty of the generator; the path length penalty is used to increase the information content of the deep audio feature; The loss function L of the discriminator D(all) The formula is as follows: L D(all) =L D(min) +L P1(d) Among them, L D(min) represents the total cross entropy loss of the discriminator; L P1(d) represents the gradient penalty loss of the discriminator; the gradient penalty is used to alleviate the mode collapse of the stylized generative adversarial network and enhance the training stability of the stylized generative adversarial network.

10. A device for generating a face image based on speech, characterized in that: The device comprises: An audio feature extraction module, used for extracting shallow audio features of a target speech through an audio feature extractor; the shallow audio features include energy distribution of the target speech; the audio feature extractor introduces a multi-dimensional attention mechanism for highlighting features of speech energy; A face image generation module is used to extract deep audio features through a stylized generative adversarial network based on the shallow audio features to obtain a corresponding target face image; the stylized generative adversarial network includes a generator, a discriminator and a classifier for improving the identity accuracy of the target face image; the generator includes an audio feature mapping module for extracting deep audio features.

Images