Speaking face video generation processing method and device and related equipment

Abstract

The invention belongs to the technical field of financial science and technology, provides a speaking face video generation processing method, a speaking face video generation processing device and related equipment, and aims to solve the problem of low authenticity of speaking face video generation in the prior art. Determining a first audio feature sequence corresponding to the voice, determining a first face image feature set corresponding to the target person source image, determining a plurality of pairs of corresponding relations between second audio features and second face image features, and generating a target face image sequence corresponding to the first audio feature sequence based on the above; and finally, generating a speaking face video corresponding to the speaker according to the voice and the target face image sequence, so that the authenticity of generation of the speaking face video can be improved, for example, in the financial field, generation of a video in which a salesman leaves and dubs can be realized, and the authenticity of the generated salesman video can be improved.

Description

Technical Field

[0001] The present application relates to the field of financial technology, and in particular to a method, device, computer equipment and computer-readable storage medium for generating and processing a speaking face video. Background Art

[0002] Talking face video generation is an important branch of the digital human generation field. It can use voice or text as input to generate facial animation videos with expressions or actions that match the above input. Since virtual digital employees are already widely used in the financial field, including but not limited to banks, it has broad application prospects in the financial field. For example, financial institutions can use talking face video generation technology to develop intelligent customer service and virtual assistant systems. These systems can provide a more realistic and humanized user interaction experience by generating talking face videos, and provide users with personalized financial consulting and services.

[0003] Although the application of virtual digital employees in the financial field is relatively common, it still faces many technical difficulties, the most important of which is how to improve the accuracy of the actions and expressions corresponding to the generated talking face video. In the process of generating talking face videos, the talking face video generation model needs to be able to accurately capture the actions or expressions of the face corresponding to the audio. For example, in the process of promoting insurance business, insurance salesmen need to make videos of salesmen explaining / speech and send the videos to users to expand business or maintain customer relations. The solution adopted in traditional technology is that the salesmen upload their own images, and then perform face-changing operations based on the pre-made unified explanation / speech video to generate the salesmen’s own video. This solution requires pre-production of videos and recordings, but after the face-changing, the sound is still the original video sound, not the salesmen’s own voice, and in terms of the authenticity of the video generation, it is impossible to accurately restore the characteristics of the real face, resulting in a poor experience of the generated video, which seriously affects the promotion of the business.

[0004] Therefore, how to improve the authenticity of speaking face video generation has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a method, apparatus, computer device and computer-readable storage medium for generating and processing a speaking face video, which can solve the technical problem of low authenticity of speaking face video generation in traditional technologies.

[0006] In a first aspect, the present application provides a method for generating and processing a speaking face video, comprising: responding to an instruction for generating a speaking face video, determining the speech and target person source image corresponding to the speaking person required to generate the speaking face video; extracting a plurality of first audio features contained in the speech based on a preset audio feature extraction model, and according to the time sequence of the first audio features in the speech, composing all the first audio features into a sequence to obtain a first audio feature sequence; extracting a plurality of first face image features contained in the target person source image based on a preset face detection model, and composing all the first face image features into a set to obtain a first face image feature set; determining the correspondence between the second audio features and the second face image features contained in a preset video training sample; generating a target face image sequence corresponding to the first audio feature sequence based on a preset face image generation model and according to the first audio feature sequence, the first face image feature set and the correspondence; generating a speaking face video corresponding to the speaking person according to the speech and the target face image sequence.

[0007] In a second aspect, the present application provides a device for generating and processing a speaking face video, comprising: a first determining unit, for responding to an instruction for generating a speaking face video, and determining the speech and target person source image corresponding to the speaking person required to generate the speaking face video; a first extracting unit, for extracting a plurality of first audio features contained in the speech based on a preset audio feature extraction model, and according to the time sequence of the first audio features in the speech, composing all the first audio features into a sequence to obtain a first audio feature sequence; a second extracting unit, for extracting a plurality of first face image features contained in the target person source image based on a preset face detection model, and composing all the first face image features into a set to obtain a first face image feature set; a second determining unit, for determining the correspondence between the second audio features and the second face image features contained in a preset video training sample; a first generating unit, for generating a target face image sequence corresponding to the first audio feature sequence based on a preset face image generation model, and according to the first audio feature sequence, the first face image feature set and the correspondence; a second generating unit, for generating a speaking face video corresponding to the speaking person according to the speech and the target face image sequence.

[0008] In a third aspect, the present application provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the speaking face video generation and processing method are implemented.

[0009] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the speaking face video generation and processing method.

[0010] The present application provides a method, apparatus, computer device and computer-readable storage medium for generating and processing a speaking face video. The method learns the correspondence between audio and face-related contents including but not limited to face features, background features and the like from video training samples based on cross-modal understanding. When generating a speaking face video, the method determines the voice corresponding to the speaking person and the source image of the target person, then extracts the face-related features contained in the source image of the target person, and extracts the audio contained in the voice. Then, based on the correspondence, the method generates a corresponding target face image according to the audio and uses it as an image frame of the speaking face video, and compares all the image frames with their correspondence. The corresponding audio is combined into a video to obtain a talking face video corresponding to the voice, thereby realizing audio-driven talking face video generation. Not only can the real voice of the person be used as the audio corresponding to the video, but also a natural and realistic virtual talking face animation is generated according to the real voice of the person and the source image of the target person, which can improve the authenticity of the generated talking face video. By way of example, in financial fields including but not limited to banking, insurance, etc., according to the scheme of audio-driven face talking video generation described in the embodiment of the present application, it is possible to realize the generation of a video in which the salesperson himself appears on the scene and does his own dubbing, without the need to pre-produce the video, thereby improving the authenticity of the generated salesperson video. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 A flowchart of a method for generating and processing a speaking face video provided in an embodiment of the present application;

[0013] Figure 2 A schematic diagram of a first sub-process of the method for generating and processing a speaking face video provided in an embodiment of the present application;

[0014] Figure 3 A schematic diagram showing an example of a comparative learning process of a method for generating and processing a speaking face video provided in an embodiment of the present application;

[0015] Figure 4 A schematic diagram of a second sub-process of the method for generating and processing a speaking face video provided in an embodiment of the present application;

[0016] Figure 5 A schematic diagram of an example process of training a preset face image generation model for a speaking face video generation processing method provided in an embodiment of the present application;

[0017] Figure 6 A schematic block diagram of a device for generating and processing a speaking face video provided in an embodiment of the present application;

[0018] Figure 7 A schematic block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0021] The embodiment of the present application provides a method for generating and processing a speaking face video, which can be applied to but not limited to computer devices such as smart phones, laptops, desktop computers, servers, self-service machines, and can be used when generating a speaking face video, including but not limited to the financial technology field.

[0022] In the face of the technical problem of low authenticity of speaking face video generation in traditional technologies, the inventors proposed the speaking face video generation processing method of the embodiment of the present application. The core idea of ​​the embodiment of the present application is: from the video training samples, learn the correspondence between audio and facial features, and when generating speaking face videos, determine the corresponding voice and target person source image of the speaker, and then generate the speaking face video based on the correspondence and the voice and target person source image, which can improve the authenticity of the speaking face video generation.

[0023] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0024] See also Figure 1 , Figure 1 The flowchart of the method for generating and processing a speaking face video provided in the embodiment of the present application is shown in FIG. Figure 1As shown, the method includes but is not limited to the following steps S11-S16:

[0025] S11, in response to an instruction for generating a speaking face video, determining a speech corresponding to a speaking person and a source image of a target person required for generating the speaking face video.

[0026] Explanatoryly, an embodiment of the present application provides an audio-driven talking face video, whereby, in the case of determining to generate a talking face video, in response to an instruction to generate a talking face video, the voice and target person source image corresponding to the talking person required to generate the talking face video are determined, wherein the talking face video represents a talking face animation, the voice represents the speaking sound of the talking person corresponding to the talking face video, the target person source image represents the source image of the talking person's face image animation in the generated talking face video, and the target person source image is generally an input picture corresponding to the target person.

[0027] For example, in financial fields including but not limited to banking, insurance, etc., when it is necessary to generate a business promotion video for a corresponding salesperson, a voice clip of the salesperson's related business promotion and a picture of the salesperson containing a facial image can be obtained as the target person source image, thereby realizing the generation of a business promotion video with the salesperson himself appearing and dubbing, and there is no need to pre-produce the video, which can improve the authenticity of the generated salesperson video.

[0028] S12. Based on a preset audio feature extraction model, extract a plurality of first audio features contained in the speech, and group all of the first audio features into a sequence according to the time sequence of the first audio features in the speech to obtain a first audio feature sequence.

[0029] Explanatoryly, an audio feature extraction model is pre-set, i.e., a preset audio feature extraction model, which includes but is not limited to Wav_net. The preset audio feature extraction model is used to extract audio features contained in speech. The Wav_net model can generate high-fidelity speech and even imitate any human speech. Therefore, when the audio features contained in speech are extracted by the Wav_net model, the accuracy of the extracted audio features can be improved.

[0030] According to the above settings, based on the preset audio feature extraction model, the audio features contained in the speech are extracted to obtain a number of first audio features, and then all the first audio features are grouped into a sequence according to the time sequence of each first audio feature in the speech to obtain a first audio feature sequence, and the first audio feature sequence is a sequence based on the time sequence corresponding to the speech. Among them, the "first" involved in the first audio feature and the first audio feature sequence is only used to distinguish different audio features and audio feature sequences, and is not used to limit audio features and audio feature sequences. Other similar descriptions in the embodiments of the present application are similar.

[0031] S13. Based on a preset face detection model, extract a plurality of first face image features contained in the target person source image, and group all of the first face image features into a set to obtain a first face image feature set.

[0032] Explanatoryly, a face detection model is pre-set, i.e., a preset face detection model, which includes but is not limited to Face_net. The preset face detection model is used to detect facial image features contained in the target person source image, where the facial image features represent relevant feature content of the face of the speaking person when speaking, and the facial image features include but are not limited to facial features and facial background features. Facial features include but are not limited to the shape of facial features, expressions, head movements, and postures of the face when speaking. Among them, the Face_net model encodes the facial image into a 128-dimensional digital vector as the extracted feature. This feature vector contains key information of the face, such as face shape, position of facial features, expression, etc. The Face_net model can be used to extract rich and accurate key facial image features of the face.

[0033] Based on a preset face detection model, a plurality of first face image features contained in a source image of a target person are extracted, and all the first face image features are grouped into a set to obtain a first face image feature set.

[0034] S14: Determine a correspondence between a second audio feature and a second facial image feature included in a preset video training sample.

[0035] Explanatoryly, video training samples are pre-set, i.e., preset video training samples, which include but are not limited to sample video images and sample video voices, wherein the sample video image represents an image screen corresponding to a sample video in the preset video training sample, the sample video image includes a number of image frames, the sample video voice represents the sound corresponding to the sample video in the preset video training sample, and the image frames corresponding to the sample video image and the audio frames corresponding to the sample video voice correspond to each other based on the time sequence of the sample video, i.e., the image frames correspond to the audio frames.

[0036] Based on the above description, the second facial image features corresponding to the facial image features contained in the image frame are extracted, and the second audio features corresponding to the audio features contained in the audio frame are extracted. The correlation between the second audio features and the second facial image features is learned by cross-modal contrastive learning, and the corresponding relationship between the second audio features and the second facial image features is obtained. The corresponding relationship is generally expressed as the correlation between several pairs of second audio features and second facial image features, that is, the mapping relationship between the speech sound corresponding to each second audio feature and the face corresponding to the second facial image feature when speaking. Among them, cross-modal contrastive learning includes but is not limited to contrastive learning between different types of data such as images and audio. Contrastive Learning aims to learn useful feature representations by comparing different types of sample data in the same data representation space. Its core idea is to learn meaningful representations of data by maximizing the similarity between samples of the same type (or reducing the distance between them) and minimizing the similarity between samples of different types (or increasing the distance between them). For example, the second audio features and the second facial image features are represented by corresponding vectors respectively, and useful feature representations are learned by vector comparison.

[0037] It should be noted that the "second" involved in the second audio feature and the second facial image feature is only used to distinguish different audio features and facial image features, and is not used to limit the audio features and facial image features. Other similar descriptions in the embodiments of the present application are similar.

[0038] S15. Based on a preset facial image generation model and according to the first audio feature sequence, the first facial image feature set and the corresponding relationship, generate a target facial image sequence corresponding to the first audio feature sequence.

[0039] Explanatory, a face image generation model is pre-set, i.e., a preset face image generation model, which includes but is not limited to StyleGAN, StyleGAN2, and StyleGAN3, and a preset face image generation model is used to generate a corresponding target face image. Among them, the StyleGAN series model can freely control the style and detail changes of the image, etc., to generate the target face image desired by the user, and realize fine control over the target face image.

[0040] According to the above settings, based on the preset facial image generation model and according to the corresponding relationship, the target first facial image features corresponding to each first audio feature in the first audio feature sequence are first determined from the first facial image feature set, and then based on the target first facial image features, the corresponding target facial image is generated, thereby generating target facial images corresponding to all the first audio features in the first audio feature sequence, and all the target facial images are sorted according to the order of all the first audio features in the first audio feature sequence to form a sequence, and a target facial image sequence is obtained, that is, a target facial image sequence corresponding to the first audio feature sequence is generated.

[0041] Furthermore, since the target person source image is the most realistic and natural picture of the speaking person, the target person source image can be used as the first target face image of the target face image sequence, which can further improve the authenticity of the speaking face video generation.

[0042] S16. Generate a speaking face video corresponding to the speaking person according to the speech and the target face image sequence.

[0043] Explanatoryally, since the order of all first audio features in the first audio feature sequence corresponds to the time order of speech, and the target face image sequence corresponds to the first audio feature sequence, the order of all target face images in the target face image sequence corresponds to the time order of speech, that is, the order of all target face images in the target face image sequence is consistent with the time order of speech. Therefore, the target face images contained in the target face image sequence are taken as image frames of the video, the speech is taken as the audio corresponding to the video, and the speech and all target face images are combined into a video to obtain a speaking face video corresponding to the speaking person.

[0044] Illustratively, in financial fields including but not limited to banking, insurance, etc., according to the solution for audio-driven facial speaking video generation described in the embodiments of the present application, by obtaining a voice of a salesperson related to business promotion and a picture of the salesperson containing a facial image as the target person source image, it is possible to generate a business promotion video with the salesperson himself appearing and dubbing, without the need for pre-production of the video, thereby improving the authenticity of the generated salesperson video.

[0045] In the embodiment of the present application, by responding to an instruction to generate a speaking face video, the speech corresponding to the speaking person and the target person source image required to generate the speaking face video are determined, and a first audio feature sequence corresponding to the speech is determined, and a first face image feature set corresponding to the target person source image is determined, and a correspondence between several pairs of second audio features and second face image features is determined, and then a target face image sequence corresponding to the first audio feature sequence is generated according to the first audio feature sequence, the first face image feature set and the correspondence, and finally a speaking face video corresponding to the speaking person is generated according to the speech and the target face image sequence, so that on the basis of cross-modal understanding, audio-face-related information including but not limited to face features, background information, etc. is learned from video training samples. The corresponding relationship between facial image features such as scene features, when generating a talking face video, determine the voice corresponding to the talking person and the source image of the target person, then extract the facial image features related to the face contained in the source image of the target person, and extract the audio contained in the voice, and then based on the corresponding relationship, generate the corresponding target face image according to the audio and use it as the image frame of the talking face video, and combine all image frames and their corresponding audio into a video to obtain the talking face video corresponding to the voice, and realize the speaking face video generation based on audio drive, which can not only use the real voice of the person as the audio corresponding to the video, but also generate natural and realistic virtual talking face animation according to the real voice of the person and the source image of the target person, which can improve the authenticity of the speaking face video generation.

[0046] In one embodiment, determining the correspondence between the second audio feature and the second facial image feature included in the preset video training sample includes:

[0047] Based on a preset contrast learning model and according to preset video training samples, a correspondence relationship between a plurality of pairs of second audio features and second facial image features is determined.

[0048] Explanatory, a contrastive learning model is pre-set, i.e., a preset contrastive learning model, which includes but is not limited to SimCLR. The preset contrastive learning model is used to learn the correlation between cross-modal audio and facial image features corresponding to the facial image, thereby achieving cross-modal understanding.

[0049] Based on the setting of the above preset contrast learning model, a preset video training sample is obtained, the preset video training sample includes a sample video image and a sample video voice, and the sample video image corresponds to the sample video voice, that is, the sample video image and the sample video voice are the image and audio of the same sample video, and then, taking the video frame of the sample video as a unit, extracting face images including but not limited to 2D face images, 3D face images and other face images of the sample video image, and encoding the face images based on but not limited to the picture encoder Face_net to obtain sample face image features, and extracting the audio frames contained in the sample video voice including but not limited to MFCC features (Mel Frequency Convolution Channel in English) CepstralCoefficients, Mel frequency cepstral coefficients), then based on including but not limited to the audio encoder Wav_net, the extracted MFCC features and other features are encoded to obtain sample audio features, the sample audio features and the sample facial image features are representations of the same data representation space, for example, the sample audio features and the sample facial image features are both represented by vectors, since the sample audio features and the sample facial image features are representations of the same data representation space, the sample audio features and the sample facial image features are comparable, and then, based on the preset contrast learning model, the correlation between the sample audio features and the sample facial image features is learned, thereby achieving cross-modal understanding and learning across audio and images, and determining the correspondence between several pairs of second audio features and second facial image features.

[0050] The embodiment of the present application determines the correspondence between several pairs of second audio features and second facial image features based on a preset contrast learning model and according to preset video training samples, thereby achieving cross-modal understanding and learning of audio and image, and further achieving audio-driven speaking face video generation, which can improve the authenticity of speaking face video generation.

[0051] In one embodiment, see Figure 2 and Figure 3 , Figure 2 This is a schematic diagram of a first sub-process of the method for generating and processing a speaking face video provided in an embodiment of the present application. Figure 3 This is a schematic diagram of an example of the comparative learning process of the speaking face video generation and processing method provided in the embodiment of the present application. Figure 2 As shown, in this embodiment, based on a preset contrast learning model and according to preset video training samples, a correspondence relationship between a plurality of pairs of second audio features and second facial image features is determined, including:

[0052] S21. Extracting, according to a preset video training sample, a sample 3D face image and a corresponding sample Mel-frequency cepstral coefficient contained in the preset video training sample;

[0053] S22, encoding the sample Mel-frequency cepstral coefficients based on a preset waveform network model to obtain a number of sample audio features;

[0054] S23, based on a preset FaceNet model, extracting features of the sample 3D face image to obtain features of a number of sample face images;

[0055] S24. Based on a preset contrast learning model, learn the correlation between the sample audio features and the corresponding sample facial image features to obtain a plurality of corresponding relationships between second audio features and second facial image features.

[0056] Explanatoryly, a waveform network model is pre-set, the waveform network model is a Wav_net model, that is, a preset waveform network model, the preset waveform network model is an audio encoder, and a FaceNet model is pre-set, that is, a preset FaceNet model, the preset FaceNet model is an image encoder.

[0057] Video training samples are pre-set, i.e., preset video training samples. The preset video training samples include sample video images and sample video voices, and the sample video images of the same sample video correspond to the sample video voices. The preset video training samples may include sample videos of different speaking persons. Thus, when determining the correspondence between the second audio feature and the second facial image feature, the commonality of the correlation between the sample audio features of different speaking persons and their corresponding sample facial image features can be fully learned, and the correspondence between the second audio feature and the second facial image feature with as universal a meaning as possible can be obtained. When implementing the audio-driven speaking facial video generation provided in the embodiment of the present application, the authenticity, naturalness, fluency, and realism of the speaking facial video generation can be further improved.

[0058] As described above, the preset video training sample includes a sample video image and a sample video voice. According to the preset video training sample, a sample 3D face image corresponding to the sample video image included in the preset video training sample is extracted, and the sample mel frequency cepstral coefficient (i.e., MFCC feature) corresponding to the sample video voice included in the preset video training sample is extracted. Then, based on the preset waveform network model, the sample mel frequency cepstral coefficient is encoded to obtain a number of sample audio features, and, based on the preset FaceNet model, the sample 3D face image is encoded to extract features to obtain a number of sample face image features, such as Figure 3 As shown, the sample face image feature F 3d , the sample audio feature is recorded as F wav .

[0059] Then, based on the preset contrast learning model, the correlation between the sample audio features and their corresponding sample facial image features is learned, that is, the audio information is used to learn the potential features of the speaker contained in the sample video image, including but not limited to facial expressions, movements, head postures, etc., to obtain the correspondence between several pairs of second audio features and second facial image features.

[0060] Furthermore, the sample facial image features include sample facial features and sample facial background features; based on a preset contrast learning model, the correlation between the sample audio features and the corresponding sample facial image features is learned to obtain a plurality of corresponding relationships between the second audio features and the second facial image features, including:

[0061] Based on a preset contrast learning model, learning a sample first feature corresponding to a first correlation between the sample audio feature and the corresponding sample facial feature, and learning a sample second feature corresponding to a second correlation between the sample audio feature and the corresponding sample facial background feature;

[0062] The sample first feature and the sample second feature are used together as sample features corresponding to the correlation between the sample audio feature and the corresponding sample facial image feature, to obtain the corresponding relationship between the second audio feature and the second facial image feature.

[0063] Explanatoryly, since a human face is generally placed in a corresponding background, the background is also an important component of the face image. Therefore, the sample face image features include sample face features and sample face background features. When determining the correspondence between the second audio feature and the second face image feature, based on the preset contrast learning model, not only the sample first feature corresponding to the first correlation between the sample audio feature and the corresponding sample face feature is learned, but also the sample second feature corresponding to the second correlation between the sample audio feature and the corresponding sample face background feature is learned. Then, the sample first feature and the sample second feature are used together as the sample features corresponding to the correlation between the sample audio feature and the corresponding sample face image feature to obtain the correspondence between the second audio feature and the second face image feature.

[0064] The embodiment of the present application realizes cross-modal understanding and learning of audio and image by utilizing contrastive learning, and can not only learn the sample first feature corresponding to the first correlation between the sample audio feature and the corresponding sample facial feature, but also learn the sample second feature corresponding to the second correlation between the sample audio feature and the corresponding sample facial background feature, thereby learning the potential features of the sample facial image from multiple angles, and can retain the micro-expressions of the person in the target person source image as the driving force, including but not limited to details such as blinking and eyebrow movements, thereby realizing audio-driven speaking face video generation, making the final speaking face video generation effect more natural and realistic, and consistent with the human eye observation effect, thereby improving the authenticity of the speaking face video generation.

[0065] In one embodiment, based on a preset facial image generation model and according to the first audio feature sequence, the first facial image feature set and the corresponding relationship, generating a target facial image sequence corresponding to the first audio feature sequence includes:

[0066] Based on the first facial image feature set and the corresponding relationship, and according to the first audio feature sequence, determining a target first facial image feature corresponding to the first audio feature;

[0067] Based on a preset StyleGAN model and according to the first audio feature and the corresponding target first facial image feature, generating a target facial image corresponding to the target first facial image feature;

[0068] According to the order of all the first audio features in the first audio feature sequence, all the target face images are grouped into a sequence to obtain a target face image sequence.

[0069] Explanatory, a StyleGAN model is pre-set, that is, a preset StyleGAN model, the preset StyleGAN model includes but is not limited to StyleGAN, StyleGAN2, StyleGAN3, and the preset StyleGAN model is used to generate a corresponding target face image.

[0070] For each first audio feature included in the first audio feature sequence, according to the corresponding relationship, the corresponding first facial image feature is determined from the first facial image feature set as the target first facial image feature, so as to determine the target first facial image feature corresponding to the first audio feature. The target first facial image feature may be a plurality of first facial image features included in the first facial image feature set. The target first facial image feature corresponding to each first audio feature included in the first audio feature sequence is obtained. Then, a feature pair consisting of the first audio feature and its corresponding target first facial image feature is input into a preset StyleGAN model to generate a target facial image corresponding to the target first facial image feature based on the preset StyleGAN model. There is a corresponding relationship between the target facial image and the first audio feature. Then, according to the order of all the first audio features in the first audio feature sequence, all the target facial images are grouped into a sequence to obtain a target facial image sequence. The target facial image sequence corresponds to the first audio feature sequence in order, that is, the target facial image contained in the target facial image sequence and the first audio features at the corresponding sequential positions contained in the first audio feature sequence are in a one-to-one correspondence.

[0071] In the embodiment of the present application, by presetting the StyleGAN model and generating a target facial image corresponding to the target first facial image feature based on the first audio feature and its corresponding target first facial image feature, fine control of the generation of the target facial image driven by audio can be achieved, so that the final speaking face video generation effect is more natural and realistic, and conforms to the observation effect of the human eye, thereby improving the authenticity of the speaking face video generation.

[0072] In one embodiment, see Figure 4 and Figure 5 , Figure 4 This is a schematic diagram of a second sub-flow of the method for generating and processing a speaking face video provided in an embodiment of the present application. Figure 5 This is a schematic diagram of an example process of training a preset face image generation model for the speaking face video generation processing method provided in the embodiment of the present application. Figure 4 As shown, in this embodiment, based on the preset facial image generation model and according to the first audio feature sequence, the first facial image feature set and the corresponding relationship, before generating the target facial image sequence corresponding to the first audio feature sequence, the method further includes:

[0073] S41, generating a sample target face image corresponding to the sample target person source image according to the sample video voice and the corresponding sample target person source image included in the preset video training sample;

[0074] S42, determining a sample real face image corresponding to the sample target generated face image according to the sample audio frame included in the sample video voice;

[0075] S43, calculating the perception loss value corresponding to the image perception loss between the sample target generated face image and the sample real face image, the identity loss value corresponding to the identity loss, the generation adversarial loss value corresponding to the generation adversarial loss, and the reconstruction loss value corresponding to the reconstruction loss;

[0076] S44. Take the sum of the perception loss value, the identity loss value, the generative adversarial loss value and the reconstruction loss value as the total loss value corresponding to the total loss function of the preset face image generation model, and train the preset face image generation model.

[0077] Explanatory, please continue to see Figure 5 , and as mentioned above, based on the preset contrast learning model and according to the preset video training samples, first determine the correspondence between several pairs of second audio features and second facial image features, then extract features of the sample video speech contained in the preset video training samples to obtain several sample audio features, the sample audio features correspond to the sample audio frames contained in the sample video speech, and extract features of the sample target person source image to obtain several sample target person source image features, and then as mentioned above, based on the preset facial image generation model, and according to the obtained several sample audio features, several sample target person source image features and the above correspondence, generate a sample target generated facial image corresponding to the sample target person source image, and there is a correspondence between the sample audio features and the sample target generated facial image.

[0078] The preset video training samples also include sample real face images corresponding to the image frames of the sample video, and based on the correspondence between the image frames of the sample video and the sample audio frames, that is, there is a correspondence between the sample real face image-the image frame of the sample video-the sample audio frame-the sample target generated face image, then the sample real face image corresponding to the sample target generated face image can be determined based on the sample audio frame contained in the sample video voice.

[0079] According to the sample target generated face image and its corresponding sample real face image, the perceptual loss value corresponding to the perceptual loss between the sample target generated face image and the sample real face image, the identity loss value corresponding to the identity loss, the generative adversarial loss value corresponding to the generative adversarial loss, and the reconstruction loss value corresponding to the reconstruction loss are calculated. Among them, the image perceptual loss represents the perceptual difference between the sample target generated face image and the sample real face image, which can be recorded as L LPIPS, image perception loss is also called "perceptual loss" or learned perceptual image patch similarity (LPIPS), which is used to measure the difference between two images; identity loss represents the difference in person identity information between the sample target generated face image and the sample real face image, which can be recorded as L ID ; Generative adversarial loss represents the difference between the sample target generated face image and the sample real face image generated by the generator in the generative adversarial network, as well as the ability of the discriminator to distinguish between the two. PatchGAN can be used as a discriminator. Its receptive field targets a small area in the input, so that the preset face image generation model can pay more attention to the image details. It can be recorded as L GAN ; The reconstruction loss represents the error in the process of compressing the sample target generated face image into a low-dimensional representation as much as possible during the learning process of the preset face image generation model, and then restoring it to the original data, which can be recorded as L recon .

[0080] Finally, the sum of the perception loss value, identity loss value, generation adversarial loss value and reconstruction loss value is used as the total loss value corresponding to the total loss function of the preset face image generation model to train the preset face image generation model, that is, the total loss value L = L LPIPS +L ID +L GAN +L recon .

[0081] In an embodiment of the present application, the preset face image generation model is trained by combining image perception loss, identity loss, generative adversarial loss and reconstruction loss as the total loss function of the preset face image generation model. Since the error between the sample target generated face image and the sample real face image is measured from multiple angles, the realism of the target face image generated by the preset face image generation model can be improved, thereby improving the authenticity of the speaking face video generation.

[0082] In one embodiment, based on a preset audio feature extraction model, a plurality of first audio features contained in the speech are extracted, including:

[0083] Determine the Mel-frequency cepstral coefficients corresponding to the speech;

[0084] Based on a preset waveform network model, the Mel-frequency cepstral coefficients are encoded to obtain a plurality of first audio features.

[0085] Explanatoryly, the speech signal is first converted into a set of coefficients reflecting sound characteristics according to the Mel Frequency Cepstral Coefficients (i.e. MFCC features) to extract the sound signal characteristics of the speech, thereby determining the Mel Frequency Cepstral Coefficients corresponding to the speech, i.e. MFCC features, and then the preset waveform network model is used as the audio encoder, and the MFCC features are encoded based on the preset waveform network model, i.e. Wav_net, to obtain a number of first audio features.

[0086] In the embodiment of the present application, a plurality of first audio features are obtained by encoding Mel-frequency cepstral coefficients based on a preset waveform network model. Since all audio samples in the preset waveform network model are affected by all previous time steps, the plurality of first audio features obtained can reflect the speech context correlation based on the temporal order of speech. In the embodiment of the present application, when the audio and facial image features are combined to realize the audio-driven speaking face video generation, the speech context correlation based on the temporal order of speech reflected between the plurality of first audio features can be converted into the context correlation between the corresponding plurality of target facial images. In the case of combining the audio and the target facial image into a speaking face video, the correlation of the audio corresponds to the correlation consistency of the plurality of target facial images and is reflected in the speaking face video, so that the final speaking face video generation effect is smoother, more natural and more realistic, and conforms to the human eye observation effect, thereby improving the authenticity of the speaking face video generation.

[0087] In one embodiment, based on a preset face detection model, a plurality of first face image features contained in the target person source image are extracted, including:

[0088] Determine a 3D face image corresponding to the target person source image;

[0089] Based on a preset FaceNet model, feature extraction is performed on the 3D face image to obtain a plurality of first face image features.

[0090] Explanatory, an input target person source image is obtained, the target person source image represents an image that is the source of the target person, and then a 3D face image model is extracted from the target person source image to determine the 3D face image corresponding to the target person source image, and then based on the preset FaceNet model, the 3D face image is encoded and feature extracted to obtain a number of first face image features, wherein the core idea of ​​FaceNet is to use a convolutional neural network (CNN) to extract features from the target person source image, and map the face image contained in the target person source image to a high-dimensional vector space, so that the distance between face vectors of the same identity is as small as possible, and the distance between face vectors of different identities is as large as possible. This high-dimensional vector space is called an embedding space, and the vector of each face image in the embedding space is called an embedding vector. Therefore, the FaceNet model can detect key information of the face, including but not limited to details such as face shape, facial features, and expressions, to achieve high-precision detection of face image details and improve the accuracy of face detection.

[0091] In the embodiment of the present application, a 3D face image corresponding to a source image of a target person is determined, and based on a preset FaceNet model, features of the 3D face image are extracted to obtain a plurality of first face image features. Since the 3D face image is complete and three-dimensional, it can express the rich features of the source image of the target person but not limited to the target person at all angles, and the FaceNet model can achieve high-precision detection of face image details, so that the plurality of first face image features obtained can reflect the rich and high-precision information corresponding to the face image including but not limited to key and detailed features, and especially when a plurality of first audio features are obtained based on a preset waveform network model, the audio is combined with the face image features to achieve When generating a talking face video driven by audio, not only can the speech context correlation based on the time sequence of speech reflected between a number of first audio features be converted into the context correlation between a number of corresponding target face images, but also the number of target face images and the context correlation between them can be reflected richly and accurately. When the audio and the target face images are combined into a talking face video, the correlation of the audio corresponds to the consistency of the correlation of the number of target face images, and the number of target face images show rich and high-precision target person image information, so that the final talking face video generation effect is richer, smoother, more natural and more realistic, which conforms to the observation effect of the human eye and can further improve the authenticity of the talking face video generation.

[0092] It should be noted that the speaking face video generation and processing methods described in the above embodiments can recombine the technical features included in different embodiments as needed to obtain a combined implementation plan, but all are within the protection scope required by this application.

[0093] See also Figure 6 , Figure 6 The schematic block diagram of the apparatus for generating and processing a video of a speaking person's face provided in the embodiment of the present application. Corresponding to the above-mentioned method for generating and processing a video of a speaking person's face, the embodiment of the present application also provides an apparatus for generating and processing a video of a speaking person's face. Figure 6 As shown, the apparatus for generating and processing a speaking face video includes a unit for executing the aforementioned speaking face video generation and processing method, and the apparatus for generating and processing a speaking face video can be configured in a computer device. Figure 6 The speaking face video generation and processing device 60 includes a first determination unit 61, a first extraction unit 62, a second extraction unit 63, a second determination unit 64, a first generation unit 65 and a second generation unit 66.

[0094] The first determination unit 61 is used to respond to the instruction of generating a speaking face video and determine the speech corresponding to the speaking person and the target person source image required to generate the speaking face video;

[0095] A first extraction unit 62 is used to extract a plurality of first audio features contained in the speech based on a preset audio feature extraction model, and to assemble all of the first audio features into a sequence according to the time sequence of the first audio features in the speech to obtain a first audio feature sequence;

[0096] A second extraction unit 63 is used to extract a plurality of first facial image features contained in the target person source image based on a preset face detection model, and to form a set of all the first facial image features to obtain a first facial image feature set;

[0097] A second determining unit 64, configured to determine a correspondence between a second audio feature and a second facial image feature included in a preset video training sample;

[0098] A first generating unit 65 is used to generate a target face image sequence corresponding to the first audio feature sequence based on a preset face image generation model and according to the first audio feature sequence, the first face image feature set and the corresponding relationship;

[0099] The second generating unit 66 is used to generate a speaking face video corresponding to the speaking person according to the speech and the target face image sequence.

[0100] In one embodiment, the second determining unit 64 is specifically configured to determine the correspondence between a plurality of pairs of second audio features and second facial image features based on a preset contrast learning model and according to preset video training samples.

[0101] In one embodiment, the second determining unit 64 includes:

[0102] A first extraction subunit is used to extract a sample 3D face image and a corresponding sample Mel-frequency cepstral coefficient contained in the preset video training sample according to the preset video training sample;

[0103] A first encoding subunit, configured to encode the sample Mel-frequency cepstral coefficients based on a preset waveform network model to obtain a plurality of sample audio features;

[0104] A second extraction subunit is used to extract features of the sample 3D face image based on a preset FaceNet model to obtain a number of sample face image features;

[0105] The first learning subunit is used to learn the correlation between the sample audio features and the corresponding sample facial image features based on a preset contrast learning model, and obtain the corresponding relationship between several pairs of second audio features and second facial image features.

[0106] In one embodiment, the sample face image features include sample face features and sample face background features; the first learning subunit includes:

[0107] A second learning subunit is used to learn, based on a preset contrast learning model, a sample first feature corresponding to a first correlation between the sample audio feature and the corresponding sample facial feature, and to learn a sample second feature corresponding to a second correlation between the sample audio feature and the corresponding sample facial background feature;

[0108] The first determination subunit is used to use the sample first feature and the sample second feature together as sample features corresponding to the correlation between the sample audio feature and the corresponding sample facial image feature, to obtain the corresponding relationship between the second audio feature and the second facial image feature.

[0109] In one embodiment, the first generating unit 65 includes:

[0110] A second determining subunit, configured to determine, based on the first facial image feature set and the corresponding relationship and according to the first audio feature sequence, a target first facial image feature corresponding to the first audio feature;

[0111] A first generating subunit, configured to generate a target facial image corresponding to the target first facial image feature based on a preset StyleGAN model and according to the first audio feature and the target first facial image feature corresponding thereto;

[0112] The composition subunit is used to compose all the target face images into a sequence according to the order of all the first audio features in the first audio feature sequence to obtain a target face image sequence.

[0113] In one embodiment, the speaking face video generation and processing device 60 further includes:

[0114] A third generating unit is used to generate a sample target generated face image corresponding to the sample target person source image according to the sample video voice and the corresponding sample target person source image included in the preset video training sample;

[0115] A third determining unit, configured to determine, based on the sample audio frames included in the sample video speech, a sample real face image corresponding to the sample target generated face image;

[0116] A calculation unit, used to calculate a perceptual loss value corresponding to an image perceptual loss between the sample target generated face image and the sample real face image, an identity loss value corresponding to an identity loss, a generation adversarial loss value corresponding to a generation adversarial loss, and a reconstruction loss value corresponding to a reconstruction loss;

[0117] A training unit is used to train the preset face image generation model by taking the sum of the perceptual loss value, the identity loss value, the generative adversarial loss value and the reconstruction loss value as the total loss value corresponding to the total loss function of the preset face image generation model.

[0118] In one embodiment, the second extraction unit 63 includes:

[0119] A third determining subunit is used to determine a 3D face image corresponding to the target person source image;

[0120] The third extraction subunit is used to extract features from the 3D face image based on a preset FaceNet model to obtain a plurality of first face image features.

[0121] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned speaking face video generation and processing device and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and brevity of description, it will not be repeated here.

[0122] At the same time, the division and connection method of each unit in the above-mentioned speaking face video generation and processing device is only used for illustration. In other embodiments, the speaking face video generation and processing device can be divided into different units as needed, and the units in the speaking face video generation and processing device can also adopt different connection orders and methods to complete all or part of the functions of the above-mentioned speaking face video generation and processing device.

[0123] The above-mentioned speaking face video generation and processing device can be implemented in the form of a computer program. The computer program can be used in Figure 7 Runs on the computer device shown.

[0124] See also Figure 7 , Figure 7 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a computer device such as a desktop computer or a server, or may be a component or part in other devices.

[0125] See also Figure 7 The computer device 500 includes a processor 502, a memory and a network interface 505 connected via a system bus 501, wherein the memory may include a non-volatile storage medium 503 and an internal memory 504, and the memory may also be a volatile storage medium.

[0126] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 can execute the above-mentioned method for generating and processing a speaking face video.

[0127] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0128] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute the above-mentioned speaking face video generation processing method.

[0129] The network interface 505 is used to communicate with other devices over the network. Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the present application scheme, and does not constitute a limitation on the computer device 500 to which the present application scheme is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. For example, in some embodiments, the computer device may only include a memory and a processor. In such an embodiment, the structure and function of the memory and the processor are the same as those of the present application scheme. Figure 7The embodiments shown are consistent and will not be described again here.

[0130] The processor 502 is used to run a computer program 5032 stored in a memory to implement the following steps: responding to an instruction for generating a speaking face video, determining the speech and target person source image corresponding to the speaking person required to generate the speaking face video; extracting a plurality of first audio features contained in the speech based on a preset audio feature extraction model, and according to the time sequence of the first audio features in the speech, composing all the first audio features into a sequence to obtain a first audio feature sequence; extracting a plurality of first face image features contained in the target person source image based on a preset face detection model, and composing all the first face image features into a set to obtain a first face image feature set; determining the correspondence between the second audio features and the second face image features contained in a preset video training sample; generating a target face image sequence corresponding to the first audio feature sequence based on a preset face image generation model and according to the first audio feature sequence, the first face image feature set and the correspondence; generating a speaking face video corresponding to the speaking person according to the speech and the target face image sequence.

[0131] In one embodiment, the processor 502 implements the following steps when determining the correspondence between the second audio feature and the second facial image feature included in the preset video training sample:

[0132] Based on a preset contrast learning model and according to preset video training samples, a correspondence relationship between a plurality of pairs of second audio features and second facial image features is determined.

[0133] In one embodiment, when the processor 502 determines the correspondence between a plurality of pairs of second audio features and second facial image features based on a preset contrast learning model and according to preset video training samples, the processor 502 specifically implements the following steps:

[0134] According to the preset video training sample, extracting the sample 3D face image and the corresponding sample Mel-frequency cepstral coefficients contained in the preset video training sample;

[0135] Based on a preset waveform network model, encoding the sample Mel-frequency cepstral coefficients to obtain a number of sample audio features;

[0136] Based on a preset FaceNet model, extract features from the sample 3D face image to obtain several sample face image features;

[0137] Based on a preset contrast learning model, the correlation between the sample audio features and the corresponding sample facial image features is learned to obtain a plurality of corresponding relationships between second audio features and second facial image features.

[0138] In one embodiment, the sample facial image features include sample facial features and sample facial background features; when the processor 502 learns the correlation between the sample audio features and the sample facial image features corresponding thereto based on a preset contrast learning model and obtains the correspondence between several pairs of second audio features and second facial image features, the processor 502 specifically implements the following steps:

[0139] Based on a preset contrast learning model, learning a sample first feature corresponding to a first correlation between the sample audio feature and the corresponding sample facial feature, and learning a sample second feature corresponding to a second correlation between the sample audio feature and the corresponding sample facial background feature;

[0140] The sample first feature and the sample second feature are used together as sample features corresponding to the correlation between the sample audio feature and the corresponding sample facial image feature, to obtain the corresponding relationship between the second audio feature and the second facial image feature.

[0141] In one embodiment, when the processor 502 generates a target facial image sequence corresponding to the first audio feature sequence based on a preset facial image generation model and according to the first audio feature sequence, the first facial image feature set and the corresponding relationship, the processor 502 specifically implements the following steps:

[0142] Based on the first facial image feature set and the corresponding relationship, and according to the first audio feature sequence, determining a target first facial image feature corresponding to the first audio feature;

[0143] Based on a preset StyleGAN model and according to the first audio feature and the corresponding target first facial image feature, generating a target facial image corresponding to the target first facial image feature;

[0144] According to the order of all the first audio features in the first audio feature sequence, all the target face images are grouped into a sequence to obtain a target face image sequence.

[0145] In one embodiment, before implementing the preset facial image generation model and generating the target facial image sequence corresponding to the first audio feature sequence according to the first audio feature sequence, the first facial image feature set and the corresponding relationship, the processor 502 further implements the following steps:

[0146] According to the sample video voice and the corresponding sample target person source image contained in the preset video training sample, a sample target generated face image corresponding to the sample target person source image is generated;

[0147] Determining, based on the sample audio frames included in the sample video speech, a sample real face image corresponding to the sample target generated face image;

[0148] Calculate the perceptual loss value corresponding to the image perceptual loss, the identity loss value corresponding to the identity loss, the generative adversarial loss value corresponding to the generative adversarial loss, and the reconstruction loss value corresponding to the reconstruction loss between the sample target generated face image and the sample real face image;

[0149] The sum of the perceptual loss value, the identity loss value, the generative adversarial loss value and the reconstruction loss value is used as the total loss value corresponding to the total loss function of the preset face image generation model to train the preset face image generation model.

[0150] In one embodiment, when the processor 502 extracts a plurality of first face image features contained in the target person source image based on a preset face detection model, the processor 502 specifically implements the following steps:

[0151] Determine a 3D face image corresponding to the target person source image;

[0152] Based on a preset FaceNet model, feature extraction is performed on the 3D face image to obtain a plurality of first face image features.

[0153] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0154] Those skilled in the art will appreciate that all or part of the processes in the method for implementing the above embodiment can be completed by a computer program, which can be stored in a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0155] Therefore, the present application also provides a computer-readable storage medium. The computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, and the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the following steps:

[0156] A computer program product, when running on a computer, enables the computer to execute the steps of the method for generating and processing a speaking face video described in the above embodiments.

[0157] The computer-readable storage medium may be an internal storage unit of the aforementioned device, such as a hard disk or memory of the device. The computer-readable storage medium may also be an external storage device of the device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the device and an external storage device.

[0158] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0159] The storage medium is a physical, non-transient storage medium, for example, it can be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, etc., which can store computer programs.

[0160] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0161] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0162] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0163] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for an electronic device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0164] The software tools or components not provided by our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0165] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for generating and processing a speaking face video, characterized in that: include: In response to an instruction to generate a speaking face video, determining a speech corresponding to a speaking person and a source image of a target person required to generate the speaking face video; Based on a preset audio feature extraction model, extract a plurality of first audio features contained in the speech, and according to the time sequence of the first audio features in the speech, assemble all the first audio features into a sequence to obtain a first audio feature sequence; Based on a preset face detection model, extract a plurality of first face image features contained in the target person source image, and form a set of all the first face image features to obtain a first face image feature set; Determine a correspondence between a second audio feature and a second facial image feature included in a preset video training sample; Based on a preset facial image generation model and according to the first audio feature sequence, the first facial image feature set and the corresponding relationship, generating a target facial image sequence corresponding to the first audio feature sequence; A speaking face video corresponding to the speaking person is generated according to the speech and the target face image sequence.

2. The method for generating and processing a speaking face video according to claim 1, characterized in that: Determining a correspondence between a second audio feature and a second facial image feature included in a preset video training sample includes: Based on a preset contrast learning model and according to preset video training samples, a correspondence relationship between a plurality of pairs of second audio features and second facial image features is determined.

3. The method for generating and processing a speaking face video according to claim 2, characterized in that: Based on a preset contrast learning model and according to preset video training samples, a corresponding relationship between a plurality of pairs of second audio features and second facial image features is determined, including: According to the preset video training sample, extracting the sample 3D face image and the corresponding sample Mel-frequency cepstral coefficients contained in the preset video training sample; Based on a preset waveform network model, encoding the sample Mel-frequency cepstral coefficients to obtain a number of sample audio features; Based on a preset FaceNet model, extract features from the sample 3D face image to obtain several sample face image features; Based on a preset contrast learning model, the correlation between the sample audio features and the corresponding sample facial image features is learned to obtain a plurality of corresponding relationships between second audio features and second facial image features.

4. The method for generating and processing a speaking face video according to claim 3, characterized in that: The sample face image features include sample face features and sample face background features; Based on a preset contrast learning model, the correlation between the sample audio feature and the corresponding sample facial image feature is learned to obtain a plurality of corresponding relationships between the second audio feature and the second facial image feature, including: Based on a preset contrast learning model, learning a sample first feature corresponding to a first correlation between the sample audio feature and the corresponding sample facial feature, and learning a sample second feature corresponding to a second correlation between the sample audio feature and the corresponding sample facial background feature; The sample first feature and the sample second feature are used together as sample features corresponding to the correlation between the sample audio feature and the corresponding sample facial image feature, to obtain the corresponding relationship between the second audio feature and the second facial image feature.

5. The method for generating and processing a speaking face video according to any one of claims 1 to 4, characterized in that: Based on a preset facial image generation model and according to the first audio feature sequence, the first facial image feature set and the corresponding relationship, generating a target facial image sequence corresponding to the first audio feature sequence, including: Based on the first facial image feature set and the corresponding relationship, and according to the first audio feature sequence, determining a target first facial image feature corresponding to the first audio feature; Based on a preset StyleGAN model and according to the first audio feature and the corresponding target first facial image feature, generating a target facial image corresponding to the target first facial image feature; According to the order of all the first audio features in the first audio feature sequence, all the target face images are grouped into a sequence to obtain a target face image sequence.

6. The method for generating and processing a speaking face video according to claim 5, characterized in that: Before generating a target facial image sequence corresponding to the first audio feature sequence based on a preset facial image generation model and according to the first audio feature sequence, the first facial image feature set and the corresponding relationship, the method further includes: According to the sample video voice and the corresponding sample target person source image contained in the preset video training sample, a sample target generated face image corresponding to the sample target person source image is generated; Determining, based on the sample audio frames included in the sample video speech, a sample real face image corresponding to the sample target generated face image; Calculate the perceptual loss value corresponding to the image perceptual loss, the identity loss value corresponding to the identity loss, the generative adversarial loss value corresponding to the generative adversarial loss, and the reconstruction loss value corresponding to the reconstruction loss between the sample target generated face image and the sample real face image; The sum of the perceptual loss value, the identity loss value, the generative adversarial loss value and the reconstruction loss value is used as the total loss value corresponding to the total loss function of the preset face image generation model to train the preset face image generation model.

7. The method for generating and processing a speaking face video according to any one of claims 1 to 4, characterized in that: Based on a preset face detection model, a plurality of first face image features contained in the target person source image are extracted, including: Determine a 3D face image corresponding to the target person source image; Based on a preset FaceNet model, feature extraction is performed on the 3D face image to obtain a plurality of first face image features.

8. A device for generating and processing a video of a speaking face, characterized in that: include: A first determination unit is used to respond to an instruction to generate a speaking face video and determine a speech corresponding to a speaking person and a target person source image required to generate the speaking face video; A first extraction unit, configured to extract a plurality of first audio features contained in the speech based on a preset audio feature extraction model, and assemble all of the first audio features into a sequence according to a time sequence of the first audio features in the speech to obtain a first audio feature sequence; A second extraction unit is used to extract a plurality of first facial image features contained in the target person source image based on a preset face detection model, and to form a set of all the first facial image features to obtain a first facial image feature set; A second determining unit, used to determine a correspondence between a second audio feature and a second facial image feature included in a preset video training sample; A first generating unit, configured to generate a target face image sequence corresponding to the first audio feature sequence based on a preset face image generation model and according to the first audio feature sequence, the first face image feature set and the corresponding relationship; The second generating unit is used to generate a speaking face video corresponding to the speaking person according to the speech and the target face image sequence.

9. A computer device, characterized in that: The computer device comprises a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program to execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 can be implemented.

Images