Speaking face generation method and device, computer equipment and storage medium

Abstract

The invention discloses a speaking face generation method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a speaker video, and selecting a reference image sequence set from the speaker video; performing multi-modal feature extraction based on the reference image sequence set to obtain face diversified features; mapping and splicing the diversified features of each face to obtain multi-modal fusion features; constructing and generating affine change parameters based on the multi-modal fusion features, and performing affine change on the face diversified features by using the affine change parameters to obtain deep deformation features; performing cross-modal attention calculation on the multi-modal fusion features and the deep deformation features, adaptively generating attention weights corresponding to the deep deformation features, and obtaining enhanced deformation fusion features; and decoding the enhanced deformation fusion feature, the face consistency feature and the expression feature to obtain a speaking face image with an expression at a specific moment. According to the invention, the generation quality and generalization ability of the speaking face picture are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method, device, computer equipment and medium for generating a speaking face. Background Art

[0002] With the rapid development of computer vision and artificial intelligence technology, generating realistic multi-expression talking faces has become a hot topic of research. In the fields of multimedia communications, virtual reality, game animation, etc., the technology that can generate matching talking face animations based on speech has broad application prospects. Existing talking face generation methods usually rely on a large amount of video data and spatially deform image features through adaptive affine transformation to simulate the movement of facial muscles when speaking.

[0003] In the process of implementing the present invention, the inventors found that the existing methods have at least the following problems:

[0004] Although these methods can generate talking face images that match the given audio to a certain extent, they have obvious limitations in expression generation. Specifically, it is difficult for existing technologies to add or change specific expressions in the generated talking face images, mainly because they lack effective modeling and control mechanisms for expression features. In addition, expression generation is not just a simple mapping of facial muscle movements, it also involves the expression of complex emotions and psychological states.

[0005] Therefore, how to effectively integrate facial expression information into the speaking face generation process to make the generated speaking face more realistic is a technical problem that needs to be solved urgently. Summary of the invention

[0006] The embodiments of the present invention provide a method, an apparatus, a computer device and a storage medium for generating a speaking face, so as to improve the authenticity and expression diversity of the generated speaking face.

[0007] In order to solve the above technical problems, the present application embodiment provides a speaking face generation method, including:

[0008] Obtain a speaker video, and randomly select N basic images and audio corresponding to each basic image from the speaker video as a reference image sequence set, where N is a positive integer;

[0009] Performing multimodal feature extraction based on the reference image sequence set to obtain diverse facial features, wherein the diverse facial features include facial consistency features, audio features, and expression features;

[0010] Mapping and splicing each of the diverse facial features to obtain multimodal fusion features;

[0011] Generate affine change parameters based on the multimodal fusion features, and use the affine change parameters to perform affine change on the diverse features of the face to obtain deep deformation features;

[0012] Performing cross-modal attention calculation on the multimodal fusion feature and the deep deformation feature, adaptively generating an attention weight corresponding to the deep deformation feature, and obtaining an enhanced deformation fusion feature;

[0013] The enhanced deformation fusion feature, the face consistency feature and the expression feature are decoded to obtain a speaking face image with expression at a specific moment.

[0014] Optionally, the performing multimodal feature extraction based on the reference image sequence set to obtain diversified facial features includes:

[0015] Using a convolutional neural network to perform expression classification on the reference image sequence set to obtain expression label text information;

[0016] Using a deep pre-training model to extract features from the audio corresponding to each of the basic images to obtain voiceprint features;

[0017] Performing image feature extraction on the basic image to obtain the facial features;

[0018] The expression text information, the voiceprint features and the facial features are aligned to obtain the expression features, the audio features and the facial consistency features.

[0019] Optionally, mapping and splicing each of the diverse facial features to obtain a multimodal fusion feature includes:

[0020] Mapping feature vectors corresponding to the expression feature, the audio feature, and the face consistency feature to the same latent space;

[0021] The three feature vectors are concatenated in the latent space to obtain the multimodal fusion feature.

[0022] Optionally, the affine transformation parameter includes a scaling factor s i , rotation coefficient θ i and the translation coefficient t i , the generating affine change parameters based on the multimodal fusion features comprises:

[0023] The affine change parameters are output using the following formula:

[0024]

[0025] Among them, n p represents multiple groups of fully connected linear neural networks, is the multimodal fusion feature of the i-th basic image in the reference image sequence set, affine(·) represents affine transformation, and for the two-dimensional coordinates (x, y), the calculation formula of the transformed new coordinates (x', y') is:

[0026] Optionally, performing cross-modal attention calculation on the multimodal fusion feature and the deep deformation feature, adaptively generating an attention weight corresponding to the deep deformation feature, and obtaining the enhanced deformation fusion feature includes:

[0027] The attention weight is calculated using the following formula:

[0028] in Indicates that the multimodal fusion features and deep deformation features are mapped to the same latent space And nn a Then, according to the product of the two latent space vectors, an attention score is finally output;

[0029] The enhanced deformation fusion feature is expressed as:

[0030]

[0031] in Represents the attention weight of the j-th base image.

[0032] In order to solve the above technical problems, the embodiment of the present application further provides a speaking face generation device, comprising:

[0033] An acquisition module, used to acquire a speaker video, and randomly select N basic images and audio corresponding to each basic image from the speaker video as a reference image sequence set, where N is a positive integer;

[0034] An extraction module, configured to perform multimodal feature extraction based on the reference image sequence set to obtain diversified facial features, wherein the diversified facial features include facial consistency features, audio features, and expression features;

[0035] A splicing module, used for mapping and splicing each of the diversified facial features to obtain multimodal fusion features;

[0036] An affine module, used to generate affine change parameters based on the multimodal fusion features, and use the affine change parameters to perform affine change on the diverse features of the face to obtain deep deformation features;

[0037] An enhancement module, used to perform cross-modal attention calculation on the multimodal fusion feature and the deep deformation feature, adaptively generate an attention weight corresponding to the deep deformation feature, and obtain an enhanced deformation fusion feature;

[0038] The generation module is used to decode the enhanced deformation fusion feature, the face consistency feature and the expression feature to obtain a speaking face image with expression at a specific moment.

[0039] Optionally, the extraction module includes:

[0040] An expression feature classification unit, used for performing expression classification on the reference image sequence set using a convolutional neural network to obtain expression label text information;

[0041] A voiceprint feature extraction unit, configured to extract features from the audio corresponding to each of the basic images using a deep pre-training model to obtain voiceprint features;

[0042] An image feature extraction unit, used to extract image features from the basic image to obtain the facial features;

[0043] The feature alignment unit is used to perform feature alignment processing on the expression text information, the voiceprint feature and the face feature to obtain the expression feature, the audio feature and the face consistency feature.

[0044] Optionally, the splicing module includes:

[0045] A mapping unit, used for mapping feature vectors corresponding to the expression feature, the audio feature and the face consistency feature to the same latent space;

[0046] The concatenation unit is used to concatenate the three feature vectors in the latent space to obtain the multimodal fusion feature.

[0047] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above speaking face generation method when executing the computer program.

[0048] In order to solve the above technical problems, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above speaking face generation method are implemented.

[0049] The speaking face generation method, device, computer equipment and storage medium provided by the embodiments of the present invention obtain a speaking video, and randomly select N basic images and audio corresponding to each basic image from the speaking video as a reference image sequence set; perform multimodal feature extraction based on the reference image sequence set to obtain face diversity features, which include face consistency features, audio features and expression features; map and splice each face diversity feature to obtain a multimodal fusion feature; construct and generate affine change parameters based on the multimodal fusion features, and use the affine change parameters to perform affine change on the face diversity features to obtain deep deformation features; perform cross-modal attention calculation on the multimodal fusion features and the deep deformation features, adaptively generate attention weights corresponding to the deep deformation features, and obtain enhanced deformation fusion features; decode the enhanced deformation fusion features, face consistency features and expression features to obtain a speaking face image with expression at a specific moment. It achieves the adaptive generation of affine transformation parameters through multimodal feature fusion, and enhances the deformed spatial features with the help of attention mechanism to ensure that they can be effectively utilized by subsequent image generation modules to generate talking face images with various expressions, thereby improving the generation quality and generalization ability of talking face images. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0051] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0052] Figure 2 is a flow chart of an embodiment of a method for generating a speaking face of the present application;

[0053] Figure 3 It is a schematic diagram of the overall process of the speaking face generation method of the present application;

[0054] Figure 4 is a structural schematic diagram of an embodiment of a speaking face generation device according to the present application;

[0055] Figure 5 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0057] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] See also Figure 1 ,like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0060] Users can use terminal devices 101 , 102 , 103 to interact with server 105 via network 104 to receive or send messages, etc.

[0061] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0062] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0063] It should be noted that the speaking face generation method provided in the embodiment of the present application is executed by a server, and accordingly, the speaking face generation device is arranged in the server.

[0064] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. The terminal devices 101, 102, 103 in the embodiment of the present application may specifically correspond to application systems in actual production.

[0065] See also Figure 2 , Figure 2 A method for generating a speaking face provided by an embodiment of the present invention is shown. Figure 1 The server in is used as an example to illustrate the details as follows:

[0066] S201: Obtain a speaker video, and randomly select N basic images and audio corresponding to each basic image from the speaker video as a reference image sequence set, where N is a positive integer.

[0067] Specifically, we first need to obtain the speaker video V, the corresponding audio T and the character expression E in the video. Accordingly, we randomly select N images from the video V as the reference image sequence set I ref , and obtain the image I corresponding to time i according to the time sequence i and T i .

[0068] Furthermore, the video frame rate is unified (the same as extracting image features) and resampled to 24FPS, and the video images are classified for expression using a pre-trained classifier (based on the VGG-16 convolutional neural network model) to obtain the expression label text information.

[0069] Furthermore, the Deepspeech pre-trained model is used to extract voiceprint features, which are then aligned with the source video that is uniformly processed to 24FPS.

[0070] S202: Perform multimodal feature extraction based on the reference image sequence set to obtain diverse facial features, where the diverse facial features include facial consistency features, audio features, and expression features.

[0071] It should be noted that when extracting different types of features, three encoders are used to extract them separately, which is a parallel relationship. After extraction, the features are fused and aligned according to the time series to improve the efficiency of feature extraction and fusion.

[0072] In a specific embodiment, Figure 3 As shown, Figure 3 This is a schematic diagram of the overall process of the speaking face generation method of the present application. Multimodal feature extraction is performed based on a reference image sequence set to obtain diverse facial features including:

[0073] A convolutional neural network is used to classify the expressions of the reference image sequence set to obtain the text information of the expression label;

[0074] A deep pre-training model is used to extract features from the audio corresponding to each basic image to obtain voiceprint features;

[0075] Perform image feature extraction on the basic image to obtain facial features;

[0076] Perform feature alignment processing on expression text information, voiceprint features and facial features to obtain expression features, audio features and facial consistency features.

[0077] In this embodiment, the multimodal expression, frame image, and sound information of the source video are separated and fused and aligned. The input source video is introduced in the form of expression, sound, and frame image, and feature extraction is performed through a pre-trained encoder. Through the feature fusion module, the expression features, image features, and audio features are deeply fused, and multiple sets of affine transformation parameters are adaptively generated. The expression information is implicitly applied to the deep spatial features, and multiple deep deformation features are obtained to provide feature information diversity for subsequent image generation.

[0078] S203: Map and splice each diverse facial feature to obtain a multimodal fusion feature.

[0079] In a specific optional implementation manner, mapping and splicing each diverse facial feature to obtain a multimodal fusion feature includes:

[0080] Map the feature vectors corresponding to the expression features, audio features and face consistency features to the same latent space;

[0081] The three feature vectors are concatenated in the latent space to obtain multimodal fusion features.

[0082] Specifically, a feature fusion module is constructed to first obtain the feature vectors F of the three different modes. c , F t and F e Map it to a latent space, perform feature concatenation, and then obtain multimodal fusion features of a specific dimension through linear mapping in Represents feature concatenation, and nn represents a linear neural network that maps feature vectors of different dimensions to the same latent space. Right now

[0083] In this embodiment, a transformation method with multiple sets of parameters is used to perform adaptive affine transformation on multimodal fusion features. After aligning the features of expression, sound and feature frame images, three sets of coefficients of scaling, rotation and translation are calculated through multiple sets of fully connected linear neural network layers. A transformation operator is applied to the diverse features of the input video face to deform the diverse feature vector into a new deep deformation feature vector. This is conducive to the rapid transformation of multimodal features.

[0084] S204: Generate affine change parameters based on the multimodal fusion feature construction, and use the affine change parameters to perform affine change on the diverse features of the face to obtain deep deformation features.

[0085] In a specific optional implementation, the affine transformation parameter includes a scaling factor s i , rotation coefficient θ i and the translation coefficient t i , the affine change parameters generated based on multimodal fusion features include:

[0086] The affine change parameters are output using the following formula:

[0087]

[0088] Among them, n p represents multiple groups of fully connected linear neural networks, is the multimodal fusion feature of the i-th base image in the reference image sequence set, affine(·) represents the affine transformation, and for the two-dimensional coordinates (x, y), the calculation formula for the transformed new coordinates (x', y') is:

[0089] Among them, the scaling factor represents the multiplication operator of the image feature matrix, which can proportionally amplify the feature matrix value; the rotation coefficient is (-1,1). By performing trigonometric function transformation on the rotation coefficient, a deflection angle can be obtained. By performing matrix multiplication operation on the feature matrix, a deflected matrix value can be obtained; the translation coefficient is a set of matrix constants, representing the offset in a specified direction. By performing matrix addition operation on the feature matrix, the offset in the corresponding direction can be obtained.

[0090] S205: Perform cross-modal attention calculation on the multimodal fusion features and the deep deformation features, adaptively generate attention weights corresponding to the deep deformation features, and obtain enhanced deformation fusion features.

[0091] In a specific optional implementation, cross-modal attention calculation is performed on the multimodal fusion feature and the deep deformation feature, and the attention weight corresponding to the deep deformation feature is adaptively generated to obtain the enhanced deformation fusion feature, including:

[0092] The attention weight is calculated using the following formula:

[0093] in Indicates that the multimodal fusion features and deep deformation features are mapped to the same latent space

[0094] And nn a Then, according to the product of the two latent space vectors, an attention score is finally output;

[0095] The enhanced deformation fusion feature is expressed as:

[0096]

[0097] in Represents the attention weight of the j-th base image.

[0098] Specifically, in order to efficiently utilize the deep deformation features The diversity of , reducing its redundancy, and constructing a cross-modal attention enhancement module. Using multimodal fusion features Depth Deformation Features Perform cross-modal attention calculations to adaptively give these deep deformation features different attention weights and further obtain enhanced deformation fusion features in represents the attention weight of the jth reference image. The detailed calculation of attention is as follows: i =softmax(nn a in Indicates that the multimodal fusion features and deep deformation features are mapped to the same latent space And nn a Then, according to the product of the two latent space vectors, an attention score is finally output. The calculation formula of the softmax function is:

[0099]

[0100] It should be noted that most existing methods only focus on expanding the diversity of deep deformation features to improve the richness of the final image generation, that is, the ability to generate facial images corresponding to different audios, which greatly increases the redundancy of deep deformation features. While reducing the quality of image generation, the model often requires a large amount of data to ignore some redundant data, which seriously affects the generalization of the model. This embodiment uses a cross-modal attention mechanism to enhance the deep deformation features, while reducing feature redundancy, so that subsequent image generators can more effectively use deep deformation features related to facial expressions and audio information, thereby improving the image generation quality and generalization ability of the model.

[0101] S206: Decoding the enhanced deformation fusion features, the face consistency features and the expression features to obtain a speaking face image with expression at a specific moment.

[0102] Specifically, the deformation fusion features are decoded by the convolutional neural network layer through the feature decoder Decoder (·), and the source image features and deformation features (face consistency features and expression features) are merged according to the channel and sent to the feature decoder to generate the speaking face image with expression corresponding to the specific time i. Repeat the above steps to obtain a speaker video of any length based on the audio length.

[0103] In this embodiment, a video of a speaker is obtained, and N basic images and the audio corresponding to each basic image are randomly selected from the video of the speaker as a reference image sequence set; multimodal feature extraction is performed based on the reference image sequence set to obtain face diversity features, which include face consistency features, audio features and expression features; each face diversity feature is mapped and spliced ​​to obtain a multimodal fusion feature; an affine change parameter is constructed based on the multimodal fusion feature, and the face diversity feature is affinely changed using the affine change parameter to obtain a deep deformation feature; cross-modal attention calculation is performed on the multimodal fusion feature and the deep deformation feature, and the attention weight corresponding to the deep deformation feature is adaptively generated to obtain an enhanced deformation fusion feature; the enhanced deformation fusion feature, the face consistency feature and the expression feature are decoded to obtain a speaker face image with expression at a specific moment. The cross-modal attention mechanism is implemented to enhance the deep deformation feature, so that while reducing feature redundancy, the subsequent image generator can more effectively utilize the deep deformation features related to expression and audio information, thereby improving the quality and generalization ability of the speaker face image generation.

[0104] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0105] Figure 4 FIG. 1 is a block diagram showing a principle of a speaking face generation device corresponding to the speaking face generation method in the above embodiment. Figure 4 As shown, the speaking face generation device includes an acquisition module 31, an extraction module 32, a splicing module 33, an affine module 34, an enhancement module 35 and a generation module 36. The detailed description of each functional module is as follows:

[0106] An acquisition module 31 is used to acquire a speaker video and randomly select N basic images and audio corresponding to each basic image from the speaker video as a reference image sequence set, where N is a positive integer;

[0107] An extraction module 32, configured to perform multimodal feature extraction based on a reference image sequence set to obtain diversified facial features, wherein the diversified facial features include facial consistency features, audio features, and expression features;

[0108] A splicing module 33 is used to map and splice each diverse facial feature to obtain a multi-modal fusion feature;

[0109] An affine module 34 is used to generate affine change parameters based on multimodal fusion features, and use the affine change parameters to perform affine changes on the diverse features of the face to obtain deep deformation features;

[0110] An enhancement module 35 is used to perform cross-modal attention calculation on the multimodal fusion features and the deep deformation features, adaptively generate attention weights corresponding to the deep deformation features, and obtain enhanced deformation fusion features;

[0111] The generation module 36 is used to decode the enhanced deformation fusion features, the face consistency features and the expression features to obtain a speaking face image with expression at a specific moment.

[0112] Optionally, the extraction module 32 includes:

[0113] An expression feature classification unit is used to classify the expression of the reference image sequence set using a convolutional neural network to obtain expression label text information;

[0114] A voiceprint feature extraction unit, used to extract features from the audio corresponding to each basic image using a deep pre-training model to obtain voiceprint features;

[0115] An image feature extraction unit, used to extract image features from a basic image to obtain facial features;

[0116] The feature alignment unit is used to perform feature alignment processing on the expression text information, voiceprint features and face features to obtain expression features, audio features and face consistency features.

[0117] Optionally, the splicing module 33 includes:

[0118] A mapping unit, used to map feature vectors corresponding to expression features, audio features, and face consistency features into the same latent space;

[0119] The concatenation unit is used to concatenate three feature vectors in the latent space to obtain multimodal fusion features.

[0120] The specific definition of the speaking face generation device can be found in the definition of the speaking face generation method above, which will not be repeated here. Each module in the above speaking face generation device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0121] To solve the above technical problems, the present application also provides a computer device. Figure 5 , Figure 5 This is a basic structural block diagram of the computer device in this embodiment.

[0122] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components connected to the memory 41, the processor 42, and the network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0123] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0124] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as the program code of the speaking face generation method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0125] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the program code stored in the memory 41 or process data, such as running the program code of the speaking face generation method.

[0126] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0127] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor to enable the at least one processor to perform the steps of the speaking face generation method as described above.

[0128] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0129] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to replace some of the technical features therein with equivalents. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for generating a speaking face, characterized in that: include: Obtain a speaker video, and randomly select N basic images and audio corresponding to each basic image from the speaker video as a reference image sequence set, where N is a positive integer; Performing multimodal feature extraction based on the reference image sequence set to obtain diverse facial features, wherein the diverse facial features include facial consistency features, audio features, and expression features; Mapping and splicing each of the diverse facial features to obtain multimodal fusion features; Generate affine change parameters based on the multimodal fusion features, and use the affine change parameters to perform affine change on the diverse features of the face to obtain deep deformation features; Performing cross-modal attention calculation on the multimodal fusion feature and the deep deformation feature, adaptively generating an attention weight corresponding to the deep deformation feature, and obtaining an enhanced deformation fusion feature; The enhanced deformation fusion feature, the face consistency feature and the expression feature are decoded to obtain a speaking face image with expression at a specific moment.

2. The method for generating a speaking face according to claim 1, wherein: The multimodal feature extraction based on the reference image sequence set to obtain diversified facial features includes: Using a convolutional neural network to perform expression classification on the reference image sequence set to obtain expression label text information; Using a deep pre-training model to extract features from the audio corresponding to each of the basic images to obtain voiceprint features; Performing image feature extraction on the basic image to obtain the facial features; The expression text information, the voiceprint features and the facial features are aligned to obtain the expression features, the audio features and the facial consistency features.

3. The method for generating a speaking face according to claim 1, wherein: The mapping and splicing of each of the diverse facial features to obtain multimodal fusion features includes: Mapping feature vectors corresponding to the expression feature, the audio feature, and the face consistency feature to the same latent space; The three feature vectors are concatenated in the latent space to obtain the multimodal fusion feature.

4. The method for generating a speaking face according to claim 1, wherein: The affine transformation parameters include the scaling factor s i , rotation coefficient θ i and the translation coefficient t i , the generating affine change parameters based on the multimodal fusion features comprises: The affine change parameters are output using the following formula: Among them, n p represents multiple groups of fully connected linear neural networks, is the multimodal fusion feature of the i-th basic image in the reference image sequence set, affine(·) represents affine transformation, and for the two-dimensional coordinates (x, y), the calculation formula of the transformed new coordinates (x', y') is:

5. The method for generating a speaking face according to claim 1, wherein: The performing cross-modal attention calculation on the multimodal fusion feature and the deep deformation feature, adaptively generating the attention weight corresponding to the deep deformation feature, and obtaining the enhanced deformation fusion feature includes: The attention weight is calculated using the following formula: where nn am : nn aa : Indicates that the multimodal fusion features and deep deformation features are mapped to the same latent space And nn a Then, according to the product of the two latent space vectors, an attention score is finally output; The enhanced deformation fusion feature is expressed as: in Represents the attention weight of the j-th base image.

6. A speaking face generation device, characterized in that: include: An acquisition module, used to acquire a speaker video, and randomly select N basic images and audio corresponding to each basic image from the speaker video as a reference image sequence set, where N is a positive integer; An extraction module, configured to perform multimodal feature extraction based on the reference image sequence set to obtain diversified facial features, wherein the diversified facial features include facial consistency features, audio features, and expression features; A splicing module, used for mapping and splicing each of the diversified facial features to obtain multimodal fusion features; An affine module, used to generate affine change parameters based on the multimodal fusion features, and use the affine change parameters to perform affine change on the diverse features of the face to obtain deep deformation features; An enhancement module, used to perform cross-modal attention calculation on the multimodal fusion feature and the deep deformation feature, adaptively generate an attention weight corresponding to the deep deformation feature, and obtain an enhanced deformation fusion feature; The generation module is used to decode the enhanced deformation fusion feature, the face consistency feature and the expression feature to obtain a speaking face image with expression at a specific moment.

7. The speaking face generation device according to claim 6, characterized in that: The extraction module comprises: An expression feature classification unit, used for performing expression classification on the reference image sequence set using a convolutional neural network to obtain expression label text information; A voiceprint feature extraction unit, configured to extract features from the audio corresponding to each of the basic images using a deep pre-training model to obtain voiceprint features; An image feature extraction unit, used to extract image features from the basic image to obtain the facial features; The feature alignment unit is used to perform feature alignment processing on the expression text information, the voiceprint feature and the face feature to obtain the expression feature, the audio feature and the face consistency feature.

8. The speaking face generation device according to claim 6, characterized in that: The splicing module comprises: A mapping unit, used for mapping feature vectors corresponding to the expression feature, the audio feature and the face consistency feature to the same latent space; The concatenation unit is used to concatenate the three feature vectors in the latent space to obtain the multimodal fusion feature.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for generating a speaking face according to any one of claims 1 to 5 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for generating a speaking face according to any one of claims 1 to 5 is implemented.

Images