A method and apparatus for generating facial images based on speech

Abstract

This application relates to the field of face generation technology, specifically to a method and apparatus for generating face images based on speech. The method includes: extracting shallow audio features of target speech using an audio feature extractor; the shallow audio features include the energy distribution of the target speech; the audio feature extractor incorporates a multidimensional attention mechanism to highlight features of speech energy; based on the shallow audio features, a stylized generative adversarial network (GAN) is used to obtain the corresponding target face image; the stylized GAN includes a generator, a discriminator, and a classifier to improve the accuracy of the target face image identification, the generator including an audio feature mapping module for extracting deep audio features; the method actively extracts deep features of speech through a neural network, effectively avoiding the problem of mismatch between the generated face image and the speech.

Description

Technical Field

[0001] This application relates to the field of face image generation technology, specifically to a method and apparatus for generating face images based on speech. Background Technology

[0002] When humans speak, sound is produced by the vibration of the vocal cords, amplified through the chest cavity and oral cavity, while the cheekbones, chin, nose, and lips also vibrate, affecting the final sound produced. Therefore, facial information can be derived from a person's voice. However, because the relationship between voice and facial information is not intuitive and effective analytical tools are lacking, this relationship is difficult to apply in the field of facial image generation.

[0003] Digital signal processing methods can extract various useful feature parameters from speech, such as fundamental frequency, formants, energy, and Mel-frequency cepstral coefficients. These features can reflect the speaker's physiological characteristics and emotional state. Furthermore, by employing neural network models, face generation technology can generate realistic facial images based on limited or specific data. Therefore, establishing a mapping relationship between human voice and facial images through neural network models becomes possible.

[0004] Currently, neural network models that predict face images based on speech cannot actively extract deep features of speech, resulting in a mismatch between the generated face image and the speech. Summary of the Invention

[0005] In order to solve the problems existing in the prior art, the purpose of this application is to provide a method and apparatus for generating face images based on speech.

[0006] In a first aspect, this application provides a method for generating face images based on speech, comprising:

[0007] Shallow audio features of the target speech are extracted using an audio feature extractor; these shallow audio features include the energy distribution of the target speech; a multidimensional attention mechanism is introduced into the audio feature extractor to highlight features that highlight speech energy.

[0008] Based on shallow audio features, a stylized generative adversarial network is used to extract deep audio features to obtain the corresponding target face image. The stylized generative adversarial network includes a generator, a discriminator, and a classifier to improve the accuracy of the target face image identification. The generator includes an audio feature mapping module for extracting deep audio features.

[0009] In one embodiment, the audio feature extractor includes a filter, a residual block, and an enhancement gated recurrent block; the residual block includes a convolutional layer, a multidimensional attention layer, and a filter feature scaling layer; the step of extracting shallow audio features of the target speech using the audio feature extractor includes:

[0010] The target speech is input into the filter to obtain filtered speech data; the size of the speech data is the same as the input data size of the residual block.

[0011] Input the speech data into the convolutional layer to obtain the initial features and input the multidimensional attention layer;

[0012] In the multidimensional attention layer, the energy correlation between each element in the initial feature and its neighboring elements is calculated as the element weight; the element weights are then applied to the initial feature to obtain the energy feature.

[0013] The speech features are obtained by optimizing the feature weights in the energy features through a filter feature scaling layer.

[0014] The speech features are input into the enhanced gated loop block to obtain shallow audio features.

[0015] In one embodiment, the energy correlation between each element in the initial feature and its neighboring elements is calculated by the following formula:

[0016]

[0017] Where Φ() represents the multidimensional attention mechanism; Represents the initial feature M f The energy of the target element t in the equation; μ represents the initial feature M in each channel. f The mean; σ 2 Indicates the initial feature M within each channel f The variance; λ is a hyperparameter; Represents the relationship between the target element t and the initial feature M. f The energy correlation of other elements in it.

[0018] In one embodiment, the generator of the stylized generative adversarial network includes an audio feature mapping module and multiple pixel generation blocks arranged in sequence; the audio feature mapping module encodes shallow audio features into deep audio features through multiple fully connected layers; the pixel generation blocks obtain the output image through stylized convolution and image-based convolution.

[0019] In one embodiment, the method further includes:

[0020] In the first pixel generation block, a pre-defined constant tensor is convolved with a stylized convolution kernel to obtain the output features; the output features are then convolved with an image-based convolution kernel to obtain the output image; the weights of the stylized convolution kernel are based on deep audio features and random noise.

[0021] In subsequent pixel generation blocks excluding the first pixel generation block, enhanced output features and enhanced output images are obtained by upsampling the output features and output image of the previous pixel generation block, respectively. The enhanced output features of the current pixel generation block are convolved with the stylized convolution kernel of the current pixel generation block to obtain the output features of the current pixel generation block. The output features of the current pixel generation block are then convolved with the image-based convolution kernel of the current pixel generation block and multiplied element-wise with the enhanced output image to obtain the output image of the current pixel generation block. The H and W dimensions of the enhanced output features are twice the size of the output features of the previous pixel generation block. The H and W dimensions of the enhanced output image are also twice the size of the output image of the previous pixel generation block.

[0022] The target face image is obtained by generating the output image of the block based on the last pixel.

[0023] In one embodiment, the method further includes:

[0024] An affine transformation is performed on the deep audio codec, and a random noise vector is added to obtain the initial weights of the stylized convolution kernel.

[0025] The initial weights of the stylized convolution kernels for each channel are normalized using the Euclidean norm to obtain the weights of the stylized convolution kernels.

[0026] In one embodiment, the training set for training the audio feature extractor and the stylized generative adversarial network includes speech data, real face images corresponding to the speech data, and real identity codes of the real face images; the method further includes:

[0027] Facial features are extracted from real face images in the training set using a classifier, and these facial features are then mapped to real identity codes.

[0028] The generated face image is identified by a classifier based on the facial features of the generated face image; the generated face image is generated based on the speech data in the training set.

[0029] In one embodiment, a discriminator is used to determine whether the generated face image is a real face image; the discriminator includes a mini-batch standard deviation layer; the method further includes:

[0030] The discriminator's convolutional layer extracts features from real and generated face images to obtain differential features;

[0031] The standard deviation features are obtained by calculating the small-batch standard deviation of the difference features through the small-batch standard deviation layer.

[0032] The standard deviation feature and the difference feature are combined to obtain the combined difference feature, and the identification result of the generated face image as a real face image is obtained based on the combined difference feature.

[0033] In one embodiment, during the training of the stylized generative adversarial network, a joint loss function is used; the joint loss function includes the generator's loss function and the discriminator's loss function.

[0034] The generator's loss function L G(all) The formula is as follows:

[0035] L G(all) =L G(max) +L P2(g)

[0036] Among them, L G(max) L represents the adversarial cross-entropy loss of the generator; P2(g) This represents the path length penalty for the generator; the path length penalty is used to increase the information content of deep audio features.

[0037] The loss function L of the discriminator D(all) The formula is as follows:

[0038] L D(all) =L D(min) +L P1(d)

[0039] Among them, L D(min) L represents the total cross-entropy loss of the discriminator. P1(d) This represents the gradient penalty loss of the discriminator; the gradient penalty is used to mitigate pattern collapse in stylized generative adversarial networks and enhance the training stability of stylized generative adversarial networks.

[0040] Secondly, this application provides an apparatus for generating facial images based on speech, comprising:

[0041] The audio feature extraction module is used to extract shallow audio features of the target speech through an audio feature extractor; the shallow audio features include the energy distribution of the target speech; a multi-dimensional attention mechanism is introduced into the audio feature extractor to highlight features of speech energy;

[0042] The face image generation module is used to extract deep audio features based on shallow audio features using a stylized generative adversarial network to obtain the corresponding target face image. The stylized generative adversarial network includes a generator, a discriminator, and a classifier to improve the accuracy of the target face image identification. The generator includes an audio feature mapping module for extracting deep audio features.

[0043] This application provides a method for generating face images based on speech. It extracts shallow audio features from the target speech using an audio feature extractor that incorporates a multi-dimensional attention mechanism. This multi-dimensional attention mechanism highlights energy features in the target speech, helping a stylized generative network (SGN) generate more accurate face images. After extracting deep audio features using a stylized generative adversarial network (SGAN) generator, the target face image is generated, resulting in more refined deep features of the target speech and generating more realistic and accurate face images based on these deep features. During the training of the neural network model, the adversarial interaction between the generator and discriminator in the SGN improves the realism of the generated face image, while the adversarial interaction between the generator and classifier makes the face image more closely resemble the real identity of the speaker. This method for generating face images based on speech effectively avoids the problem of mismatch between the generated face image and the speech by actively extracting deep data from the speech through a neural network. Attached Figure Description

[0044] Figure 1 This is a flowchart of the steps in a method for generating a face image based on speech in one embodiment;

[0045] Figure 2 This is a flowchart of the steps for extracting shallow audio features of target speech using an audio feature extractor in one embodiment;

[0046] Figure 3 This is a flowchart illustrating the steps involved in generating a target face image using a generator in one embodiment.

[0047] Figure 4 This is a flowchart of the steps for modulating and demodulating the weights of a stylized convolutional kernel in one embodiment;

[0048] Figure 5 This is a flowchart illustrating the steps involved in generating an output identity code for a face image from the classifier's output in one embodiment.

[0049] Figure 6 This is a flowchart illustrating the steps of a discriminator in one embodiment to obtain a discrimination result that the generated face image is a real face image;

[0050] Figure 7 This is a structural block diagram of a device for generating face images based on voice in one embodiment. Detailed Implementation

[0051] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings, which illustrate embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of this application will be thorough and complete.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0053] When used herein, the singular forms of “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising / including” or “having,” etc., specify the presence of the stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. Meanwhile, the term “and / or” as used in this specification includes any and all combinations of the associated listed items.

[0054] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0055] This application provides a method for generating face images based on speech, such as... Figure 1 As shown, the process includes the following steps S12 to S14:

[0056] S12, shallow audio features of the target speech are extracted by an audio feature extractor; the shallow audio features include the energy distribution of the target speech; a multidimensional attention mechanism is introduced in the audio feature extractor to highlight the features of speech energy.

[0057] Specifically, an audio feature extractor that combines residual neural networks and recurrent neural networks and introduces a multidimensional attention mechanism is used to extract features from the target speech, obtaining the energy distribution of different elements in the speech data as shallow audio features. Among them, according to the multidimensional attention mechanism, the energy correlation between each element in the energy distribution and other elements is calculated, and the feature elements are highlighted accordingly.

[0058] S14. Based on audio features, a stylized generative adversarial network is used to extract deep audio features to obtain the corresponding target face image. The stylized generative adversarial network includes a generator, a discriminator, and a classifier to improve the accuracy of the target face image identity. The generator includes an audio feature mapping module for extracting deep audio features.

[0059] Specifically, the stylized generative adversarial network includes a generator, a discriminator, and a classifier. The generator's audio feature mapping module, through a fully connected neural network, can encode shallow audio features into more informative deep audio features. The generator further performs upsampling and stylized convolution on the deep audio features to obtain the generated target face image.

[0060] Furthermore, during the training of the stylized generative adversarial network, the discriminator compares the features of the generated target face image with those of real face images from the training set to determine whether the target face image is a real face image, which helps the generator generate target face images that are closer to real faces; the classifier determines the identity of the face based on the facial features of the target face image, and determines whether the target face image matches the identity of the speaker based on the identity corresponding to the speech, which helps the generator generate target face images that are more consistent with the identity of the speaker.

[0061] The aforementioned method for generating face images based on speech extracts shallow audio features from the target speech by introducing an audio feature extractor with a multi-dimensional attention mechanism. This multi-dimensional attention mechanism highlights energy features in the target speech, helping the stylized generative network (SGN) generate more accurate face images. After extracting deep audio features through the generator of a stylized generative adversarial network (SGAN), the target face image is generated, resulting in more refined deep features of the target speech and generating more realistic and accurate face images based on these deep features. During the training of the neural network model, the adversarial interaction between the generator and discriminator in the SGN improves the realism of the generated face image, while the adversarial interaction between the generator and classifier makes the face image more closely resemble the real identity of the speaker. This method actively extracts deep speech data through a neural network, effectively avoiding the problem of mismatch between the generated face image and the speech.

[0062] In one exemplary embodiment, the audio feature extractor includes filters, residual blocks, and enhancement gated loop blocks; the residual blocks include convolutional layers, multidimensional attention layers, and filter feature scaling layers; such as Figure 2 As shown, the extraction of shallow audio features of target speech using an audio feature extractor includes the following steps S1202 to S1210:

[0063] S1202, the target speech is input into the filter to obtain filtered speech data; the size of the speech data is the same as the size of the input data of the residual block.

[0064] Specifically, in the filter, the input target speech is first cropped into speech data that is suitable for the data size of subsequent processing; preliminary feature extraction is performed on the speech data through convolution, pooling, normalization and activation functions, and then the data is input into the subsequent residual block.

[0065] Preferably, the data adapted for subsequent processing is 64K audio data, including 4 seconds of audio data, and contains 16K data per second.

[0066] S1204 inputs speech data into the convolutional layer and inputs the resulting initial features into the multidimensional attention layer.

[0067] Specifically, in each residual block, the speech features of the speech data are further extracted through convolution, batch normalization, and activation functions to obtain initial features, which are then input into the multidimensional attention layer.

[0068] S1206 In the multidimensional attention layer, the energy correlation between each element in the initial feature and its neighboring elements is calculated as the element weight; the element weights are applied to the initial feature to obtain the energy feature.

[0069] Specifically, in the multidimensional attention layer, based on the multidimensional attention mechanism formula, the energy correlation between each element in the initial feature and other elements is calculated; the energy correlation between each element in the initial feature and its neighboring elements is calculated by the following formula:

[0070]

[0071] Where Φ() represents the multidimensional attention mechanism; Represents the initial feature M f The energy of the target element t in the equation; μ represents the initial feature M in each channel. f The mean; σ 2 Indicates the initial feature M within each channel f The variance; λ is a hyperparameter; Represents the relationship between the target element t and the initial feature M. f The energy correlation of other elements in it.

[0072] Furthermore, by multiplying the reciprocal of the energy correlation with each corresponding element, we obtain the energy characteristic, as shown in the following formula:

[0073]

[0074] in, Represents energy characteristics; sigmoid is a common activation function; e f It represents the energy correlation of each element; ⊙ indicates element-wise multiplication.

[0075] It should be explained that the lower the energy correlation, the more obvious the difference between the target element and other elements. Therefore, multiplying the inverse of the energy correlation by the target element can highlight the target element in the energy characteristics.

[0076] S1208 optimizes the feature weights in the energy features through a filter feature scaling layer to obtain speech features.

[0077] Specifically, the Sigmoid function is used to scale the feature weights of each element in the energy feature within the range of [0,1] to enhance important features and suppress unimportant features; then the optimized energy feature is input into the next residual block for a new round of feature extraction; until the last residual block, the output energy feature is output as a speech feature to the enhancement gate loop block.

[0078] S1210 inputs speech features into the enhancement gated loop block to obtain shallow audio features.

[0079] Specifically, shallow audio features are obtained by further extracting speech features through an enhanced gating loop block that introduces an attention mechanism.

[0080] It should be noted that the enhanced gated recurrent block includes a gated recurrent unit (GRU), a fully connected layer, and a Squeeze-and-Excitation Networks (SENet) layer. GRU is a variant of a recurrent neural network that can extract hidden states in speech sequences to capture temporal dependencies that are beneficial to speech recognition. SENet is an attention mechanism that compresses the features of each channel of the input features into a single feature value through global average pooling, forming a channel feature vector. Then, a weight tensor for each channel is generated based on the channel feature vector through a fully connected layer and an activation function, and multiplied with the input features channel by channel to obtain the output features.

[0081] In an exemplary embodiment, the generator of the stylized generative adversarial network includes an audio feature mapping module and a plurality of pixel generation blocks arranged in sequence; the audio feature mapping module encodes shallow audio features into deep audio features through a plurality of fully connected layers; the pixel generation blocks obtain the output image through stylized convolution and image-based convolution.

[0082] Specifically, in the generator, the audio features are first encoded into deeper audio features with more refined features through the audio feature mapping module; then, the target face image is generated based on the deeper audio features through multiple pixel generation blocks.

[0083] Furthermore, the audio feature mapping module includes multiple fully connected layers. Through layer-by-layer linear and non-linear transformations, and by optimizing the weights of each fully connected layer in the audio feature mapping module during the backpropagation process of model training, the audio features are gradually encoded into deep audio features.

[0084] Preferably, the audio feature mapping module includes 6 fully connected layers; the generator contains 6 pixel generation blocks, which can expand and transform the initial tensor layer by layer through stylized convolution and image convolution to obtain a large-size output image.

[0085] In one exemplary embodiment, such as Figure 3 As shown, the method further includes the following steps S22 to S26:

[0086] S22, in the first pixel generation block, a pre-defined constant tensor is convolved with a stylized convolution kernel to obtain output features; the output features are then convolved with an image-based convolution kernel to obtain the output image; the weights of the stylized convolution kernel are obtained based on deep audio features and random noise.

[0087] Preferably, the preset constant tensor (256,4,4) is a tensor with 256 channels, each channel being a 4×4 size tensor.

[0088] Specifically, in the first pixel generation block, the preset constant tensor (256,4,4) is stylized by the stylized convolution kernel of this layer to obtain the output feature (128,4,4); the output feature (128,4,4) is image-convolved by the image-convolution kernel of this layer to obtain the output image (3,4,4); the stylized convolution kernel size is 3×3, and the image-convolution kernel size is 1×1.

[0089] S24, in subsequent pixel generation blocks excluding the first pixel generation block, enhanced output features and enhanced output images are obtained by upsampling the output features and output image of the previous pixel generation block, respectively; the enhanced output features of the current pixel generation block are obtained by convolving the enhanced output features with the stylized convolution kernel of the current pixel generation block, and the output features of the current pixel generation block are obtained by convolving the output features of the current pixel generation block with the image-style convolution kernel of the current pixel generation block and multiplying them element-wise with the enhanced output image; the H and W dimensions of the enhanced output features are twice the size of the output features of the previous pixel generation block; the H and W dimensions of the enhanced output image are twice the size of the output image of the previous pixel generation block.

[0090] For example, in the second pixel generation block, the output features (128,4,4) and the output image (3,4,4) of the previous pixel generation block are first upsampled to obtain the enhanced output features (128,8,8) and the enhanced output image (3,8,8) respectively; then, the enhanced output features (128,8,8) are convolved with the stylized convolution kernel of this layer to obtain the output features (64,8,8) of this layer, and the output features (64,8,8) are convolved with the image-based convolution kernel of this layer and multiplied with the enhanced output image (3,8,8) to obtain the output image (3,8,8) of this layer.

[0091] S26, the target face image is obtained from the output image of the last pixel block.

[0092] Specifically, the output image (3,128,128) of the sixth pixel generation block is used as the target face image.

[0093] In one exemplary embodiment, such as Figure 4 As shown, the method further includes the following steps S32 to S34:

[0094] S32 performs an affine transformation on the deep audio encoding and adds a random noise vector to obtain the initial weights of the stylized convolution kernel.

[0095] Specifically, deep audio features are affinely transformed into style modulation vectors through fully connected layers; the fully connected layers use the Affine function; the style modulation vectors include a scale factor y. s and offset factor y b The input feature x is modulated by the style modulation vector and the random noise vector n. c Obtain the modulated feature x' c The modulation formula is as follows:

[0096] x' c =y s ·x c +y b +γ·n

[0097] Here, γ is a trainable parameter used to control the degree of influence of the random noise vector; by modulating the style modulation vector, the distribution of output features is changed, which can control the global structure and texture of the generated image; adding random noise to the offset can add fine-grainedness to the generated image.

[0098] Based on the weight size of the stylized convolutional kernel required for each pixel block, ys is mapped through a fully connected layer to the initial weights of the stylized convolutional kernels for each channel of the corresponding size, and ys is then... b +γ·n is mapped to an offset.

[0099] S34. The initial weights of the stylized convolution kernels for each channel are normalized using the Euclidean norm to obtain the weights of the stylized convolution kernels.

[0100] Specifically, the Euclidean norm of each channel is calculated, and then the initial weights of each channel are divided by the Euclidean norm of that channel to obtain the weights of the stylized convolution kernel for that channel. Euclidean norm normalization ensures that the weights of each channel remain within a reasonable range, enhancing the stability of the generator during training. The formula for calculating the Euclidean norm of each channel is as follows:

[0101]

[0102] in, This indicates that the number of input channels is C. in The number of output channels is C out , where h is the initial weight of the stylized convolution kernel and w is the width; || ||2 represents the calculation of the Euclidean norm.

[0103] In one exemplary embodiment, the training set for training the audio feature extractor and the stylized generative adversarial network includes speech data, real face images corresponding to the speech data, and real identity codes of the real face images; such as Figure 5 As shown, the method further includes the following steps S42 to S44:

[0104] S42, extract facial features from each real face image in the training set using a classifier, and associate the facial features with each real identity code.

[0105] Specifically, during the training process of the stylized generative adversarial network, the classifier is trained to recognize each real face image in the training set. The output layer of the classifier is set to a softmax function with the number of categories equal to the number of real identity codes, so that the classifier can classify the real face images in the training set into their corresponding real identity codes.

[0106] S44, the classifier obtains the generation identity code of the generated face image based on the facial features of the generated face image; the generated face image is generated based on the speech data in the training set.

[0107] Specifically, the generated face image generated by the generator based on the speech data of the training set is input into the classifier. The generated identity code of the face image is obtained based on the extracted facial features. The generated identity code is compared with the real identity code of the speech data to obtain the matching result between the generated face image and the real identity. Through the classifier, the generator's ability to generate face images that match the identity can be improved.

[0108] For example, if there are 20 real identity codes in the training set, the output of the classifier is a tensor (20,1) with 20 channels. Each channel has only a single value, which represents the probability of generating a face image with that identity code. The identity code corresponding to the channel with the highest probability is taken as the generated identity code.

[0109] In one exemplary embodiment, the discriminator is used to determine whether the generated face image is a real face image; the discriminator includes a mini-batch standard deviation layer; such as Figure 6 As shown, the method further includes the following steps S52 to S56:

[0110] S52 extracts features from real and generated face images through the convolutional layer of the discriminator to obtain differential features.

[0111] Specifically, the discriminator has the opposite structure to the generator, with multiple convolutional blocks. During the training of the stylized generative adversarial network, the face image and the generated face image are convolved and downsampled through multiple convolutional blocks to extract features and reduce the size of the features, thus obtaining differential features.

[0112] S54 calculates the small-batch standard deviation of the difference features through the small-batch standard deviation layer to obtain the standard deviation features.

[0113] Preferably, a mini-batch standard deviation layer is added after the convolutional block of the discriminator to improve the performance of the discriminator.

[0114] Specifically, the differential features are divided into four mini-batches and flattened into two-dimensional differential feature maps as input; then, the mean and standard deviation of each position in the four two-dimensional differential feature maps are calculated. The formula for calculating the mean is as follows:

[0115]

[0116] in, Let represent the difference feature value at the j-th position in the i-th mini-batch of the two-dimensional difference feature map, where i∈[1,4]. μ j Let $\frac{j}{j}$ represent the average of the difference eigenvalues ​​at the $j$ position in the four mini-batches. The formula for calculating the standard deviation is as follows:

[0117]

[0118] Where, σ j Let represent the standard deviation of the differential feature values ​​at the j-th position in the four mini-batch layers; let ε be 1e-8; calculate the mean of the standard deviations at all positions to obtain a single-valued standard deviation tensor; the formula for calculating the mean of the standard deviations at all positions is as follows:

[0119]

[0120] Where σ represents the mean of the standard deviations at each position; B represents the size of the batch; C represents the number of channels; H represents the height of the difference feature; W represents the width of the difference feature; the standard deviation tensor is expanded so that its height and width are the same as the height and width of the difference feature to obtain the standard deviation feature.

[0121] S56, merge the standard deviation feature and the difference feature to obtain the combined difference feature, and obtain the identification result of the generated face image as a real face image based on the combined difference feature.

[0122] Specifically, the concat function is used to connect the difference features and standard deviation features along the channel dimension to obtain the combined difference features; the size of the combined difference features is reduced by downsampling, and finally the discrimination result is output through a fully connected layer; the discrimination result is the probability value of the generated face image being a real face image. The closer the probability value is to 1, the more realistic the generated face image is.

[0123] In an exemplary embodiment, during the training of the stylized generative adversarial network, the loss function used is a joint loss function; the joint loss function includes the loss function of the generator, the loss function of the discriminator, and the loss function of the classifier.

[0124] Specifically, the generator's loss function L G(all) The formula is as follows:

[0125] L G(all) =L G(max) +L P2(g)

[0126] Among them, L G(max) L represents the adversarial cross-entropy loss of the generator; P2(g) The generator's path length penalty; the generator's adversarial cross-entropy loss L. G(max) The path length penalty is obtained by adding the cross-entropy loss of the discriminator in determining whether the generated face image is a real face image to the cross-entropy loss of the classifier in determining whether the identity of the generated face image matches the identity of the person speaking the speech. The path length penalty not only helps the audio feature mapping module of the generator to encode deeper audio features with richer information, but also enhances the realism and coherence of the generated face image.

[0127] On the other hand, the loss function L of the discriminator D(all) The formula is as follows:

[0128] L D(all) =L D(min) +L P1(d)

[0129] Among them, L D(min) L represents the total cross-entropy loss of the discriminator.P1(d) The gradient penalty loss of the discriminator is represented by L; the total cross-entropy loss of the discriminator is represented by L. D(min) The gradient penalty is obtained by adding the cross-entropy loss of the discriminator judging the generated face image as fake to the cross-entropy loss of the discriminator judging the real face image in the training set as real; the gradient penalty is used to alleviate the mode collapse of the stylized generative adversarial network and enhance the training stability of the stylized generative adversarial network.

[0130] On the other hand, the loss function L of the classifier C(min) The cross-entropy loss is obtained by calculating the cross-entropy loss of the classifier in classifying real face images in the training set into their corresponding real identity codes.

[0131] It should be understood that, although Figures 1-6 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 1-6 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0132] Secondly, this application provides a device 70 for generating facial images based on voice, such as... Figure 7 As shown, it includes:

[0133] The audio feature extraction module 71 is used to extract shallow audio features of the target speech through an audio feature extractor; the shallow audio features include the energy distribution of the target speech; a multidimensional attention mechanism is introduced in the audio feature extractor to highlight the features of speech energy.

[0134] The face image generation module 72 is used to extract deep audio features based on shallow audio features through a stylized generative adversarial network to obtain the corresponding target face image. The stylized generative adversarial network includes a generator, a discriminator, and a classifier for improving the accuracy of the target face image identity. The generator includes an audio feature mapping module for extracting deep audio features.

[0135] For specific limitations regarding the device for generating facial images based on speech, please refer to the limitations of the method for generating facial images based on speech mentioned above, which will not be repeated here. Each module in the aforementioned device for generating facial images based on speech can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module. It should be noted that the module division in this embodiment is illustrative and only represents a logical functional division; other division methods may be used in actual implementation.

[0136] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0137] In the description of this specification, references to terms such as "some embodiments," "other embodiments," and "ideal embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative descriptions of the above terms do not necessarily refer to the same embodiments or examples.

[0138] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for generating facial images based on speech, characterized in that, The method includes: Shallow audio features of the target speech are extracted using an audio feature extractor; the shallow audio features include the energy distribution of the target speech; a multidimensional attention mechanism for highlighting speech energy features is introduced into the audio feature extractor; Based on the shallow audio features, a deep audio feature extraction is performed using a stylized generative adversarial network to obtain the corresponding target face image; the stylized generative adversarial network includes a generator, a discriminator, and a classifier for improving the accuracy of the target face image identification; the generator includes an audio feature mapping module for extracting the deep audio features; The stylized generative adversarial network's generator includes the audio feature mapping module and multiple pixel generation blocks arranged sequentially; the audio feature mapping module encodes the shallow audio features into deep audio features through multiple fully connected layers; the pixel generation blocks obtain the output image through stylized convolution and image-based convolution; The method further includes: In the first pixel generation block, a pre-defined constant tensor is convolved with a stylized convolution kernel to obtain output features; the output features are then convolved with an image-oriented convolution kernel to obtain the output image; the weights of the stylized convolution kernel are based on deep audio features and random noise. In subsequent pixel generation blocks excluding the first pixel generation block, enhanced output features and enhanced output images are obtained by upsampling the output features and output image of the previous pixel generation block, respectively. The enhanced output features are then convolved with the stylized convolution kernel of the current pixel generation block to obtain the output features of the current pixel generation block. Finally, the output features of the current pixel generation block are convolved with the image-based convolution kernel of the current pixel generation block and multiplied element-wise with the enhanced output image to obtain the output image of the current pixel generation block. H Dimensions and W The dimension size is twice that of the output features of the previous pixel generation block; the enhanced output image H Dimensions and W The dimension size is twice that of the output image of the previous pixel generation block; The target face image is obtained by generating the output image of the last pixel block.

2. The method according to claim 1, characterized in that, The audio feature extractor includes filters, residual blocks, and enhancement gate loop blocks; the residual blocks include convolutional layers, multidimensional attention layers, and filter feature scaling layers; The step of extracting shallow audio features of the target speech using an audio feature extractor includes: The target speech is input into the filter to obtain filtered speech data; the size of the speech data is the same as the size of the input data of the residual block. The speech data is input into the convolutional layer to obtain initial features, which are then input into the multidimensional attention layer. In the multidimensional attention layer, the energy correlation between each element in the initial feature and its neighboring elements is calculated as the element weight; the element weight is applied to the initial feature to obtain the energy feature; The feature weights in the energy features are optimized by the filter feature scaling layer to obtain the speech features; The speech features are input into the enhanced gating loop block to obtain the shallow audio features.

3. The method according to claim 2, characterized in that, The energy correlation between each element and its neighboring elements in the initial feature is calculated by the following formula: in, () indicates a multidimensional attention mechanism; Represents the initial feature target element in Energy; Indicates the initial features within each channel The mean; Indicates the initial features within each channel The variance; It's a hyperparameter; Represents the target element With the initial feature The energy correlation of other elements in it.

4. The method according to claim 1, characterized in that, The method further includes: An affine transformation is performed on the deep audio codec, and a random noise vector is added to obtain the initial weights of the stylized convolution kernel. The initial weights of the stylized convolution kernels for each channel are normalized using the Euclidean norm to obtain the weights of the stylized convolution kernels.

5. The method according to claim 4, characterized in that, The training set used to train the audio feature extractor and the stylized generative adversarial network includes speech data, real face images corresponding to the speech data, and real identity codes of the real face images. The method further includes: Facial features of each real face image in the training set are extracted using a classifier, and the facial features are mapped to each real identity code. The classifier obtains the generated identity code of the generated face image based on the facial features of the generated face image; the generated face image is generated based on the speech data in the training set.

6. The method according to claim 5, characterized in that, The discriminator is used to determine whether the generated face image is a real face image; the discriminator includes a mini-batch standard deviation layer; the method further includes: The discriminator's convolutional layer extracts features from the real face image and the generated face image to obtain differential features; The standard deviation features are obtained by calculating the small-batch standard deviation of the difference features through the small-batch standard deviation layer. The standard deviation feature and the difference feature are combined to obtain a combined difference feature, and the identification result of the generated face image as the real face image is obtained based on the combined difference feature.

7. The method according to claim 6, characterized in that, During the training of the stylized generative adversarial network, the loss function used is a joint loss function, which includes the loss function of the generator and the loss function of the discriminator. The generator's loss function The formula is as follows: in, This represents the adversarial cross-entropy loss of the generator; This represents the path length penalty of the generator; the path length penalty is used to increase the information content of the deep audio features. The loss function of the discriminator The formula is as follows: in, This represents the total cross-entropy loss of the discriminator; The gradient penalty loss represents the discriminator; the gradient penalty is used to mitigate mode collapse of the stylized generative adversarial network and enhance the training stability of the stylized generative adversarial network.

8. A device for generating facial images based on speech, characterized in that, The device includes: An audio feature extraction module is used to extract shallow audio features of a target speech using an audio feature extractor; the shallow audio features include the energy distribution of the target speech; the audio feature extractor incorporates a multidimensional attention mechanism to highlight features of speech energy; A face image generation module is used to extract deep audio features based on the shallow audio features using a stylized generative adversarial network to obtain the corresponding target face image; the stylized generative adversarial network includes a generator, a discriminator, and a classifier for improving the accuracy of the target face image identification; the generator includes an audio feature mapping module for extracting deep audio features; The stylized generative adversarial network's generator includes the audio feature mapping module and multiple pixel generation blocks arranged sequentially; the audio feature mapping module encodes the shallow audio features into deep audio features through multiple fully connected layers; the pixel generation blocks obtain the output image through stylized convolution and image-based convolution; The face image generation module is further configured to, in the first pixel generation block, convolve a preset constant tensor using a stylized convolution kernel to obtain output features; convolve the output features using an image-based convolution kernel to obtain an output image; the weights of the stylized convolution kernel are obtained based on deep audio features and random noise; in subsequent pixel generation blocks excluding the first pixel generation block, upsampling operations are performed on the output features and output image of the previous pixel generation block to obtain enhanced output features and enhanced output images respectively; the enhanced output features are convolved using the stylized convolution kernel of this pixel generation block to obtain the output features of this pixel generation block; the enhanced output features are convolved using the image-based convolution kernel of this pixel generation block and multiplied element-wise with the enhanced output image to obtain the output image of this pixel generation block; the enhanced output features... H Dimensions and W The dimension size is twice that of the output features of the previous pixel generation block; the enhanced output image H Dimensions and W The dimension size is twice that of the output image of the previous pixel generation block; the target face image is obtained based on the output image of the last pixel generation block.

Images