Methods, devices, computer equipment, and storage media for generating speaking faces

Abstract

This invention discloses a method, apparatus, computer device, and storage medium for generating speaking faces, comprising: acquiring a video of a speaker and selecting a set of reference image sequences from the video; extracting multimodal features based on the set of reference image sequences to obtain diverse facial features; mapping and stitching together each diverse facial feature to obtain a multimodal fusion feature; constructing affine transformation parameters based on the multimodal fusion feature and applying the affine transformation parameters to perform affine transformations on the diverse facial features to obtain deep deformation features; performing cross-modal attention calculation on the multimodal fusion feature and the deep deformation feature, adaptively generating attention weights corresponding to the deep deformation feature to obtain enhanced deformation fusion features; and decoding the enhanced deformation fusion feature, facial consistency features, and expression features to obtain a speaking face image with expression at a specific moment. This invention improves the quality and generalization ability of generated speaking face images.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more particularly to a method, apparatus, computer device, and medium for generating speaking faces. Background Technology

[0002] With the rapid development of computer vision and artificial intelligence technologies, generating realistic, multi-expression speaking faces has become a research hotspot. In fields such as multimedia communication, virtual reality, and game animation, the technology of generating matching speaking face animations based on speech has broad application prospects. Existing speaking face generation methods typically rely on large amounts of video data, using adaptive affine transformations to spatially deform image features to simulate facial muscle movements during speech.

[0003] In the process of realizing this invention, the inventors discovered that the existing methods have at least the following problems:

[0004] While these methods can generate speaking face images that match given audio to some extent, they have significant limitations in expression generation. Specifically, existing techniques struggle to add or alter specific expressions in the generated speaking face images, primarily because they lack effective modeling and control mechanisms for facial expression features. Furthermore, expression generation is not merely a simple mapping of facial muscle movements; it involves the expression of complex emotions and psychological states.

[0005] Therefore, how to effectively integrate facial expression information into the process of generating a speaker's face to make the generated speaker's face more realistic is a technical problem that urgently needs to be solved. Summary of the Invention

[0006] This invention provides a method, apparatus, computer device, and storage medium for generating speaking faces, in order to improve the realism and expression diversity of the generated speaking faces.

[0007] To address the aforementioned technical problems, embodiments of this application provide a method for generating a speaking face, including:

[0008] Acquire a video of the speaker, and randomly select N base images and the audio corresponding to each base image from the video of the speaker as a reference image sequence set, where N is a positive integer;

[0009] Multimodal feature extraction is performed based on the reference image sequence set to obtain facial diversity features, which include facial consistency features, audio features, and expression features.

[0010] Each of the aforementioned diverse facial features is mapped and stitched together to obtain multimodal fusion features;

[0011] Based on the multimodal fusion features, affine transformation parameters are generated, and the diverse facial features are subjected to affine transformation using the affine transformation parameters to obtain deep deformation features.

[0012] Cross-modal attention calculation is performed on the multimodal fusion feature and the deep deformation feature, and attention weights corresponding to the deep deformation feature are adaptively generated to obtain the enhanced deformation fusion feature;

[0013] The enhanced deformation fusion feature, the face consistency feature, and the expression feature are decoded to obtain a speaking face image with expression at a specific moment.

[0014] Optionally, the step of extracting multimodal features based on the reference image sequence set to obtain diverse facial features includes:

[0015] A convolutional neural network is used to classify facial expressions in the reference image sequence set to obtain facial expression label text information;

[0016] A deep pre-trained model is used to extract features from the audio corresponding to each of the base images to obtain voiceprint features;

[0017] Image feature extraction is performed on the base image to obtain the facial features;

[0018] The facial expression text information, the voiceprint features, and the facial features are subjected to feature alignment processing to obtain the facial expression features, the audio features, and the facial consistency features.

[0019] Optionally, the mapping and concatenation of each of the diverse facial features to obtain multimodal fusion features includes:

[0020] The feature vectors corresponding to the facial expression features, the audio features, and the face consistency features are mapped to the same latent space;

[0021] The three feature vectors are concatenated in the latent space to obtain the multimodal fusion feature.

[0022] Optionally, the affine transformation parameters include the scaling factor s. i Rotation coefficient θ i With translation coefficient t i The step of constructing and generating affine transformation parameters based on the multimodal fusion features includes:

[0023] The affine transformation parameters are output using the following formula:

[0024]

[0025] Where, nn p This represents multiple fully connected linear neural networks. The multimodal fusion feature of the i-th base image in the reference image sequence set is given by affine(·), which represents an affine transformation. For two-dimensional coordinates (x, y), the formula for calculating the transformed new coordinates (x', y') is as follows:

[0026] Optionally, the step of performing cross-modal attention calculation on the multimodal fusion feature and the deep deformation feature, and adaptively generating the attention weights corresponding to the deep deformation feature to obtain the enhanced deformation fusion feature includes:

[0027] The attention weights are calculated using the following formula:

[0028] in This means mapping multimodal fusion features and deep deformation features to the same latent space. And nn a Then, based on the product of the two latent space vectors, an attention score is finally output;

[0029] Enhanced deformation fusion features are represented as follows:

[0030]

[0031] in This represents the attention weight for the j-th base image.

[0032] To address the aforementioned technical problems, this application also provides a speaking face generation device, comprising:

[0033] The acquisition module is used to acquire the speaker's video and randomly select N base images and the audio corresponding to each base image from the speaker's video as a reference image sequence set, where N is a positive integer;

[0034] The extraction module is used to perform multimodal feature extraction based on the reference image sequence set to obtain facial diversity features, which include facial consistency features, audio features, and expression features.

[0035] The stitching module is used to map and stitch together each of the aforementioned diverse facial features to obtain multimodal fusion features;

[0036] The affine module is used to construct and generate affine transformation parameters based on the multimodal fusion features, and to use the affine transformation parameters to perform affine transformations on the diverse facial features to obtain deep deformation features;

[0037] The enhancement module is used to perform cross-modal attention calculation on the multimodal fusion feature and the deep deformation feature, adaptively generate the attention weights corresponding to the deep deformation feature, and obtain the enhanced deformation fusion feature;

[0038] The generation module is used to decode the enhanced deformation fusion feature, the face consistency feature and the expression feature to obtain a speaking face image with expression at a specific moment.

[0039] Optionally, the extraction module includes:

[0040] The facial expression feature classification unit is used to perform facial expression classification on the reference image sequence set using a convolutional neural network to obtain facial expression label text information;

[0041] The voiceprint feature extraction unit is used to extract features from the audio corresponding to each of the base images using a deep pre-trained model to obtain voiceprint features.

[0042] An image feature extraction unit is used to extract image features from the base image to obtain the facial features;

[0043] The feature alignment unit is used to perform feature alignment processing on the facial expression text information, the voiceprint features and the facial features to obtain the facial expression features, the audio features and the facial consistency features.

[0044] Optionally, the splicing module includes:

[0045] The mapping unit is used to map the feature vectors corresponding to the facial expression features, the audio features, and the face consistency features to the same latent space;

[0046] The concatenation unit is used to concatenate the three feature vectors in the latent space to obtain the multimodal fusion feature.

[0047] To address the aforementioned technical problems, this application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the aforementioned speaking face generation method.

[0048] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method for generating a speaking face.

[0049] The present invention provides a method, apparatus, computer device, and storage medium for generating a speaking face. This involves acquiring a speaker's video and randomly selecting N base images and corresponding audio from each base image as a reference image sequence set. Multimodal feature extraction is performed based on the reference image sequence set to obtain diverse facial features, including facial consistency features, audio features, and facial expression features. Each diverse facial feature is mapped and concatenated to obtain a multimodal fusion feature. Affine transformation parameters are constructed based on the multimodal fusion feature, and these parameters are used to perform affine transformations on the diverse facial features to obtain deep deformation features. Cross-modal attention calculation is performed on the multimodal fusion feature and the deep deformation feature to adaptively generate attention weights corresponding to the deep deformation feature, resulting in an enhanced deformation fusion feature. The enhanced deformation fusion feature, facial consistency feature, and facial expression feature are decoded to obtain a speaking face image with facial expressions at a specific moment. This method achieves adaptive generation of affine transformation parameters through multimodal feature fusion and enhances the deformed spatial features with the help of an attention mechanism to ensure that they can be effectively utilized by the subsequent image generation module, thereby generating speaking face images with various expressions and improving the quality and generalization ability of speaking face image generation. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0052] Figure 2 This is a flowchart of an embodiment of the speaking face generation method of this application;

[0053] Figure 3 This is a schematic diagram of the overall process of the speaking face generation method of this application;

[0054] Figure 4 This is a schematic diagram of a structure of an embodiment of the speaking face generation device according to this application;

[0055] Figure 5 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0057] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Please see Figure 1 ,like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0060] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc.

[0061] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0062] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0063] It should be noted that the speaking face generation method provided in this application embodiment is executed by the server, and correspondingly, the speaking face generation device is set in the server.

[0064] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included. The terminal devices 101, 102, and 103 in this embodiment can specifically correspond to application systems in actual production.

[0065] Please see Figure 2 , Figure 2 This invention illustrates a method for generating a speaking face according to an embodiment of the present invention, which is applied to... Figure 1 Taking the server-side as an example, the details are as follows:

[0066] S201: Obtain the speaker's video, and randomly select N base images and the audio corresponding to each base image from the speaker's video as a reference image sequence set, where N is a positive integer.

[0067] Specifically, we first need to acquire the speaker's video V, corresponding to the audio T and the speaker's facial expression E in the video. Accordingly, we randomly select N images from the video V as a reference image sequence set I. ref And obtain the image I corresponding to time i according to the time sequence. i and T i .

[0068] Furthermore, the video is resampled to 24 FPS (similar to image feature extraction) with a uniform frame rate, and a pre-trained classifier (based on the VGG-16 convolutional neural network model) is used to classify the video images for facial expressions and obtain the text information of the facial expression labels.

[0069] Furthermore, a Deepspeech pre-trained model was used to extract voiceprint features, which were then aligned with the source video, which was uniformly processed to 24 FPS.

[0070] S202: Multimodal feature extraction is performed based on the reference image sequence set to obtain facial diversity features, which include facial consistency features, audio features, and expression features.

[0071] It should be noted that when extracting different types of features, three encoders are used to extract them separately in a parallel manner. After extraction, the features are then fused and aligned according to the time series to improve the efficiency of feature extraction and fusion.

[0072] In one specific implementation, such as Figure 3 As shown, Figure 3 This is a schematic diagram of the overall process of the speaking face generation method of this application. Based on a set of reference image sequences, multimodal feature extraction is performed to obtain diverse facial features, including:

[0073] A convolutional neural network is used to classify facial expressions in a set of reference image sequences to obtain facial expression label text information.

[0074] A deep pre-trained model is used to extract features from the audio corresponding to each base image to obtain voiceprint features;

[0075] Image features are extracted from the base image to obtain facial features;

[0076] Feature alignment processing is performed on facial expression text information, voiceprint features, and facial features to obtain facial expression features, audio features, and facial consistency features.

[0077] In this embodiment, multimodal facial expressions, frame images, and audio information from the source video are separated, fused, and aligned. The input source video is introduced in three forms: facial expressions, audio, and frame images. Feature extraction is performed using a pre-trained encoder. Through a feature fusion module, facial expression features, image features, and audio features are deeply fused to adaptively generate multiple sets of affine transformation parameters. This implicitly applies facial expression information to the depth space features, resulting in multiple depth deformation features that provide diverse feature information for subsequent image generation.

[0078] S203: Map and stitch together the diverse features of each face to obtain multimodal fusion features.

[0079] In one specific optional implementation, mapping and stitching together each type of facial diversity feature to obtain multimodal fusion features includes:

[0080] Map the feature vectors corresponding to facial expression features, audio features, and face consistency features to the same latent space;

[0081] In the latent space, the three feature vectors are concatenated to obtain multimodal fusion features.

[0082] Specifically, a feature fusion module is constructed, which first combines the obtained feature vectors F from three different modalities. c F t and F e The feature is mapped to a latent space and concatenated. Then, a linear mapping is used to obtain multimodal fusion features of a specific dimension. in The term "nn" represents feature concatenation, where "nn" indicates that a linear neural network maps feature vectors of different dimensions to the same latent space. Right now

[0083] In this embodiment, a multi-parameter transformation method is used to perform adaptive affine transformation on multimodal fusion features. After aligning the features of facial expressions, voice, and feature frame images, three sets of coefficients for scaling, rotation, and translation are calculated through multiple fully connected linear neural network layers. Transformation operators are applied to the diverse facial features of the input video, transforming the diverse feature vectors into new depth-deformed feature vectors. This facilitates rapid transformation of multimodal features.

[0084] S204: Based on multimodal fusion features, affine transformation parameters are generated, and the affine transformation parameters are used to perform affine transformations on diverse facial features to obtain deep deformation features.

[0085] In one specific optional implementation, the affine transformation parameter includes the scaling factor s. i Rotation coefficient θ i With translation coefficient t i The affine transformation parameters generated based on multimodal fusion features include:

[0086] The affine transformation parameters are output using the following formula:

[0087]

[0088] Where, nn p This represents multiple fully connected linear neural networks. For the multimodal fusion features of the i-th base image in the reference image sequence set, affine(·) represents the affine transformation. For two-dimensional coordinates (x, y), the formula for calculating the transformed new coordinates (x', y') is:

[0089] Among them, the scaling factor represents the multiplication operator of the image feature matrix, which can enlarge the feature matrix value proportionally; the rotation factor is (-1,1). Performing a trigonometric function transformation on the rotation factor yields a deflection angle. Performing matrix multiplication on the feature matrix yields a deflected matrix value; the translation factor is a set of matrix constants representing the offset in a specified direction. Performing matrix addition on the feature matrix yields the offset in the corresponding direction.

[0090] S205: Perform cross-modal attention calculation on the multimodal fusion features and deep deformation features, adaptively generate the attention weights corresponding to the deep deformation features, and obtain the enhanced deformation fusion features.

[0091] In one specific optional implementation, cross-modal attention calculation is performed on the multimodal fusion features and deep deformation features to adaptively generate attention weights corresponding to the deep deformation features, resulting in enhanced deformation fusion features including:

[0092] The attention weights are calculated using the following formula:

[0093] in This means mapping multimodal fusion features and deep deformation features to the same latent space.

[0094] And nn a Then, based on the product of the two latent space vectors, an attention score is finally output;

[0095] Enhanced deformation fusion features are represented as follows:

[0096]

[0097] in This represents the attention weight for the j-th base image.

[0098] Specifically, in order to efficiently utilize depth deformation features To reduce redundancy and enhance diversity, a cross-modal attention enhancement module is constructed. Multimodal fusion features are used. With depth deformation features Cross-modal attention computation is performed, adaptively assigning different attention weights to these deep deformation features, and further obtaining enhanced deformation fusion features. in Let a represent the attention weight for the j-th reference image. The detailed calculation of attention is as follows: a i =softmax(nn) a in This means mapping multimodal fusion features and deep deformation features to the same latent space. And nn a The final output is an attention score based on the product of the two latent space vectors. The formula for calculating the softmax function is as follows:

[0099]

[0100] It's important to note that most existing methods focus solely on expanding the diversity of depth morphing features to enhance the richness of the final image generation, specifically the ability to generate facial images corresponding to different audio frequencies. However, this significantly increases the redundancy of depth morphing features. While reducing image generation quality, this also means the model often requires a large amount of data to ignore redundant information, severely impacting its generalization ability. This embodiment utilizes a cross-modal attention mechanism to enhance depth morphing features. This reduces feature redundancy and allows the subsequent image generator to more effectively utilize depth morphing features related to facial expressions and audio information, thereby improving the model's image generation quality and generalization ability.

[0101] S206: Decode the enhanced deformation fusion features, face consistency features, and expression features to obtain a speaking face image with expression at a specific moment.

[0102] Specifically, the obtained deformed fusion features are decoded by the feature decoder Decoder(·) through a convolutional neural network layer. The source image features and deformed features (face consistency features and expression features) are merged according to channels and then fed into the feature decoder to generate a speaking face image with expression corresponding to a specific time i. By repeating the above steps, you can obtain speaker videos of any length based on the audio length.

[0103] In this embodiment, a speaker's video is acquired, and N base images and their corresponding audio are randomly selected from the video as a reference image sequence set. Multimodal feature extraction is performed based on this reference image sequence set to obtain diverse facial features, including facial consistency features, audio features, and facial expression features. Each type of diverse facial feature is mapped and concatenated to obtain a multimodal fusion feature. Affine transformation parameters are generated based on these multimodal fusion features, and these parameters are used to perform affine transformations on the diverse facial features to obtain deep deformation features. Cross-modal attention calculation is performed on the multimodal fusion features and deep deformation features to adaptively generate attention weights corresponding to the deep deformation features, resulting in enhanced deformation fusion features. The enhanced deformation fusion features, facial consistency features, and facial expression features are decoded to obtain a speaking face image with facial expressions at a specific moment. This cross-modal attention mechanism enhances the deep deformation features, reducing feature redundancy and enabling subsequent image generators to more effectively utilize deep deformation features related to facial expressions and audio information, thus improving the quality and generalization ability of the generated speaking face image.

[0104] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0105] Figure 4 A schematic block diagram of a speaking face generation device corresponding to the speaking face generation method in the above embodiments is shown. Figure 4 As shown, the speaking face generation device includes an acquisition module 31, an extraction module 32, a stitching module 33, an affine module 34, an enhancement module 35, and a generation module 36. Detailed descriptions of each functional module are as follows:

[0106] The acquisition module 31 is used to acquire the speaker's video and randomly select N basic images and the audio corresponding to each basic image from the speaker's video as a reference image sequence set, where N is a positive integer;

[0107] Extraction module 32 is used to extract multimodal features based on a set of reference image sequences to obtain diverse facial features, including facial consistency features, audio features and expression features;

[0108] The stitching module 33 is used to map and stitch together the diverse features of each face to obtain multimodal fusion features;

[0109] Affine module 34 is used to construct and generate affine transformation parameters based on multimodal fusion features, and to perform affine transformations on diverse facial features using affine transformation parameters to obtain deep deformation features;

[0110] Enhancement module 35 is used to perform cross-modal attention calculation on multimodal fusion features and deep deformation features, adaptively generate attention weights corresponding to deep deformation features, and obtain enhanced deformation fusion features;

[0111] The generation module 36 is used to decode the enhanced deformation fusion features, face consistency features and expression features to obtain a speaking face image with expression at a specific moment.

[0112] Optionally, the extraction module 32 includes:

[0113] The facial expression feature classification unit is used to classify facial expressions in a set of reference image sequences using a convolutional neural network to obtain facial expression label text information.

[0114] The voiceprint feature extraction unit is used to extract features from the audio corresponding to each base image using a deep pre-trained model to obtain voiceprint features.

[0115] The image feature extraction unit is used to extract image features from the base image to obtain facial features;

[0116] The feature alignment unit is used to perform feature alignment processing on facial expression text information, voiceprint features and facial features to obtain facial expression features, audio features and facial consistency features.

[0117] Optionally, the splicing module 33 includes:

[0118] The mapping unit is used to map the feature vectors corresponding to facial expression features, audio features, and face consistency features to the same latent space.

[0119] The concatenation unit is used to concatenate the three feature vectors in the latent space to obtain multimodal fusion features.

[0120] Specific limitations regarding the speaking face generation device can be found in the limitations of the speaking face generation method described above, and will not be repeated here. Each module in the aforementioned speaking face generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0121] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 5 , Figure 5 This is a basic structural block diagram of the computer device in this embodiment.

[0122] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with components connected to the memory 41, processor 42, and network interface 43 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0123] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0124] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or D-interface display memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as the program code for a speaking face generation method. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0125] In some embodiments, the processor 42 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run program code stored in the memory 41 or process data, for example, to run program code for a speaking face generation method.

[0126] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0127] This application also provides another embodiment, namely, a computer-readable storage medium storing an interface display program that can be executed by at least one processor to cause the at least one processor to perform the steps of the speaking face generation method described above.

[0128] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0129] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A method for generating a speaking face, characterized in that, include: Acquire a video of the speaker, and randomly select N base images and the audio corresponding to each base image from the video of the speaker as a reference image sequence set, where N is a positive integer; Multimodal feature extraction is performed based on the reference image sequence set to obtain facial diversity features, which include facial consistency features, audio features, and expression features. Each of the aforementioned diverse facial features is mapped and stitched together to obtain multimodal fusion features; Based on the multimodal fusion features, affine transformation parameters are generated, and the diverse facial features are subjected to affine transformation using the affine transformation parameters to obtain deep deformation features. Cross-modal attention calculation is performed on the multimodal fusion feature and the deep deformation feature, and attention weights corresponding to the deep deformation feature are adaptively generated to obtain the enhanced deformation fusion feature; The enhanced deformation fusion feature, the face consistency feature obtained by feature alignment processing, and the expression feature are decoded to obtain a speaking face image with expression at a specific moment; The step of extracting multimodal features based on the reference image sequence set to obtain diverse facial features includes: A convolutional neural network is used to classify facial expressions in the reference image sequence set to obtain facial expression label text information; A deep pre-trained model is used to extract features from the audio corresponding to each of the base images to obtain voiceprint features; Image features are extracted from the base image to obtain facial features; The text information of the emoticon tag, the voiceprint feature, and the facial feature are subjected to feature alignment processing to obtain mutually independent emoticon features, audio features, and facial consistency features.

2. The method for generating a speaking face as described in claim 1, characterized in that, The mapping and concatenation of each of the aforementioned diverse facial features to obtain multimodal fusion features includes: The feature vectors corresponding to the facial expression features, the audio features, and the face consistency features are mapped to the same latent space; The three feature vectors are concatenated in the latent space to obtain the multimodal fusion feature.

3. The method for generating a speaking face as described in claim 1, characterized in that, The affine transformation parameters include scaling factors. Rotation coefficient With translation coefficient The step of constructing and generating affine transformation parameters based on the multimodal fusion features includes: The affine transformation parameters are output using the following formula: in, This represents multiple fully connected linear neural networks. The multimodal fusion feature of the i-th base image in the reference image sequence set. This represents an affine transformation for two-dimensional coordinates. New coordinates after transformation The calculation formula is .

4. The method for generating a speaking face as described in claim 1, characterized in that, The step of performing cross-modal attention calculation on the multimodal fusion feature and the deep deformation feature, and adaptively generating the attention weights corresponding to the deep deformation feature to obtain the enhanced deformation fusion feature includes: The attention weights are calculated using the following formula: ,in , This means that multimodal fusion features and deep deformation features are mapped to the same latent space. ,and Then, based on the product of the two latent space vectors, an attention score is finally output; Enhanced deformation fusion features are represented as follows: in Indicates the first Attention weights for the basic image.

5. A device for generating a speaking face, characterized in that, include: The acquisition module is used to acquire the speaker's video and randomly select N base images and the audio corresponding to each base image from the speaker's video as a reference image sequence set, where N is a positive integer; The extraction module is used to perform multimodal feature extraction based on the reference image sequence set to obtain facial diversity features, which include facial consistency features, audio features, and expression features. The stitching module is used to map and stitch together each of the aforementioned diverse facial features to obtain multimodal fusion features; The affine module is used to construct and generate affine transformation parameters based on the multimodal fusion features, and to use the affine transformation parameters to perform affine transformations on the diverse facial features to obtain deep deformation features; The enhancement module is used to perform cross-modal attention calculation on the multimodal fusion feature and the deep deformation feature, adaptively generate the attention weights corresponding to the deep deformation feature, and obtain the enhanced deformation fusion feature; The generation module is used to decode the enhanced deformation fusion features, the face consistency features obtained from feature alignment processing, and the expression features to obtain a speaking face image with expression at a specific moment; wherein, the extraction module includes: The facial expression feature classification unit is used to perform facial expression classification on the reference image sequence set using a convolutional neural network to obtain facial expression label text information; The voiceprint feature extraction unit is used to extract features from the audio corresponding to each of the base images using a deep pre-trained model to obtain voiceprint features. The image feature extraction unit is used to extract image features from the base image to obtain facial features; The feature alignment unit is used to perform feature alignment processing on the expression tag text information, the voiceprint feature and the face feature to obtain mutually independent expression features, audio features and face consistency features.

6. The speaking face generation device as described in claim 5, characterized in that, The splicing module includes: The mapping unit is used to map the feature vectors corresponding to the facial expression features, the audio features, and the face consistency features to the same latent space; The concatenation unit is used to concatenate the three feature vectors in the latent space to obtain the multimodal fusion feature.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the speaking face generation method as described in any one of claims 1 to 4.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the speaking face generation method as described in any one of claims 1 to 4.

Images