Action driving method, device, equipment and storage medium for target object

Abstract

This application provides a method, apparatus, device, and storage medium for driving the action of a target object, relating to the field of artificial intelligence technology. The method includes: acquiring source speech and a target video, wherein the target video includes a target object; performing facial parameter conversion processing on the speech parameters of the source speech at each moment to obtain source parameters of the source speech at the corresponding moment; extracting parameters from the target video to obtain target parameters; reconstructing the target object in the target video based on a combination parameter obtained by combining the source parameters and the target parameters to obtain a reconstructed image; and generating a synthesized video from the reconstructed image, wherein the synthesized video contains the target object, and the action of the target object corresponds to the source speech. This application enables a smoother and more realistic synthesized video with speech-driven target object action, improving the visual effect of video synthesis.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and includes, but is not limited to, a method, apparatus, device, and storage medium for driving the action of a target object. Background Technology

[0002] In recent years, the mainstream methods in the field of lip-sync video generation have been based on deep learning. One approach involves using recurrent neural networks to learn key mouth points from speech features, then generating mouth textures based on these key point information, and finally combining this information with the target video frame to obtain the lip-sync video frame. Another approach is to first learn a universal, shared "speech-expression" space from multiple audio clips from different sources, and then obtain the final lip-sync video frame based on the obtained expression parameters.

[0003] However, the lip-synced speaking video frames generated by the methods in the related technologies all suffer from problems such as unsmoothness and unrealistic appearance. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for driving the action of a target object, relating to the field of artificial intelligence technology. By determining the source parameters corresponding to the source speech and the target parameters corresponding to the target video, and then reconstructing the target object's image based on the combined parameters, a synthesized video is obtained. Thus, because the synthesized video, which ultimately drives the target object's action based on the combined parameters, is obtained, the final synthesized video is smoother and more realistic.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides an action-driven method for a target object, the method comprising:

[0007] Acquire source audio and target video, wherein the target video includes a target object;

[0008] The speech parameters of the source speech at each time moment are processed by face parameter conversion to obtain the source parameters of the source speech at the corresponding time moment;

[0009] The target video is subjected to parameter extraction to obtain target parameters;

[0010] Based on the combination parameters obtained by combining the source parameters and the target parameters, the target object in the target video is reconstructed to obtain a reconstructed image;

[0011] A synthetic video is generated from the reconstructed image, wherein the synthetic video contains the target object, and the action of the target object corresponds to the source speech.

[0012] This application embodiment provides a motion driving device for a target object, the device comprising:

[0013] The acquisition module is used to acquire source audio and target video, wherein the target video includes a target object;

[0014] The face parameter conversion module is used to perform face parameter conversion processing on the speech parameters of the source speech at each time moment to obtain the source parameters of the source speech at the corresponding time moment.

[0015] The parameter extraction module is used to extract parameters from the target video to obtain target parameters;

[0016] The image reconstruction module is used to reconstruct the image of the target object in the target video based on the combination parameters obtained by combining the source parameters and the target parameters, so as to obtain a reconstructed image;

[0017] A generation module is used to generate a synthesized video from the reconstructed image, wherein the synthesized video contains the target object, and the action of the target object corresponds to the source speech.

[0018] This application provides an action-driven system for a target object, which includes at least a terminal and a server;

[0019] The terminal is used to send an action-driven request for the target object to the server. The action-driven request includes source audio and target video, and the target video includes the target object.

[0020] The server is used to respond to the action-driven request and implement the action-driven method for the target object described above.

[0021] This application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to implement the aforementioned action-driven method for the target object.

[0022] This application provides an action-driving device for a target object, comprising: a memory for storing executable instructions; and a processor for implementing the aforementioned action-driving method for the target object when executing the executable instructions stored in the memory.

[0023] This application provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-described action-driven method for the target object.

[0024] The embodiments of this application have the following beneficial effects: Face parameter conversion processing is performed on the speech parameters of the source speech at each moment to obtain the source parameters of the source speech at the corresponding moment; target parameters are extracted from the target video; and the target object is reconstructed based on the combined parameters of the source and target parameters to obtain a reconstructed image. Finally, a synthesized video is generated from the reconstructed image. Thus, because the synthesized video of the target object's actions is obtained based on the combined parameters of the source and target parameters, the final synthesized video is smoother and more realistic, improving the visual effect of the video synthesis. Attached Figure Description

[0025] Figure 1 This is a system framework diagram of an action-driven method for a target object in related technologies;

[0026] Figure 2 This is an optional architecture diagram of the action-driven system for the target object provided in the embodiments of this application;

[0027] Figure 3 This is a schematic diagram of the server structure provided in an embodiment of this application;

[0028] Figure 4 This is an optional flowchart illustrating the action-driven method for the target object provided in the embodiments of this application;

[0029] Figure 5 This is an optional flowchart illustrating the action-driven method for the target object provided in the embodiments of this application;

[0030] Figure 6 This is an optional flowchart illustrating the action-driven method for the target object provided in the embodiments of this application;

[0031] Figure 7 This is a schematic diagram illustrating the implementation flow of the training method for the image rendering model provided in this application embodiment;

[0032] Figure 8 This is a system framework diagram of the action-driven method for the target object provided in the embodiments of this application;

[0033] Figure 9 This is a framework diagram of the text-to-speech module provided in the embodiments of this application;

[0034] Figure 10 This is a framework diagram of the speech-to-face parameter network provided in the embodiments of this application;

[0035] Figure 11 This is a diagram showing the effect of the Dlib algorithm provided in the embodiments of this application;

[0036] Figure 12 This is a framework diagram of the image rendering model provided in the embodiments of this application;

[0037] Figure 13 This is a diagram of the condition-based GAN framework provided in the embodiments of this application;

[0038] Figure 14 It is a virtual human speaking video synthesized using methods from related technologies;

[0039] Figure 15 It is a synthesized video generated by the action-driven method of the target object in the embodiments of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application pertain. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit the application.

[0042] Before explaining the embodiments of this application, the action-driven method for the target object in the related art will be described first:

[0043] Currently, the field of lip-syncing video generation is mainly divided into two categories: text-driven and speech-driven. As the name suggests, text-driven generation takes a text and a video of a target person as input, converts the text into speech using Text-to-Speech (TTS) technology, learns facial features from the speech features, and finally outputs a video of the target person reading the input text. Speech-driven generation, on the other hand, skips the TTS step and directly inputs a speech and a video of the target person. It can be said that text-driven methods are an extension of speech-driven methods. Most mainstream methods are based on deep learning. For example, the Audio2Obama method first uses a recurrent neural network to learn 20 lip key points from speech features, then generates lip textures based on the key point information, and finally combines these with the target video frames to obtain lip-syncing video frames. The text-driven ObamaNet method mainly consists of three modules: a text-to-speech module, a speech-keypoint module, and a keypoint-video frame module. The text-to-speech module uses Char2Wav from the TTS algorithm, the speech-keypoint module uses a recurrent neural network to learn keypoint information from speech features, and the keypoint-video frame module uses the U-Net network with skip connections to achieve information transmission. This method is also the first deep learning-based text-driven lip-sync video generation model in recent years.

[0044] While the aforementioned methods achieve considerable results, they all rely on experiments with the same individual, resulting in poor model scalability. Therefore, other methods are focusing on designing networks that can adapt to different people's voices. For example, one approach first learns a universal, shared "speech-expression" space based on multiple audio clips from different sources. Then, it reconstructs a 3D face based on the obtained expression parameters, resulting in a corresponding UV map, which is a graph mapping 3D face coordinates to a 2D plane. This method also uses a U-Net network to render video frames. Another approach proposes a speech identity removal network to transform the speech features of different speakers into a global domain. Then, a recurrent neural network is used to learn expression parameters from the speech features. These expression parameters are combined with the 3D face parameters of the target person to reconstruct a 3D mesh, which is then input into a U-Net network to obtain the final video frame. Another approach focuses on improving the rendering module by proposing a memory-enhanced generative adversarial network (GAN) to preserve the identity and spatial feature pairs of different speakers, thereby enabling the synthesis of videos featuring different speakers.

[0045] In related technologies, a motion-driven method for target objects based on a speech-driven model is also proposed. This method first learns a general and shared "speech-expression" space based on sound segments from multiple different sources. This space is composed of multiple mixed shapes, and the expression parameters of different people can be constructed by linear combinations of different mixed shapes in the space. Then, 3D face reconstruction is performed based on the obtained expression parameters to obtain the corresponding UV map. Finally, U-Net based on dilated convolution is used to render video frames. Figure 1 This is a system framework diagram of an action-driven method for a target object in related technologies, such as... Figure 1 As shown, the system framework of the target object action-driven method consists of a generalized network 11 and a specialized network 12. The specific processing flow of this technical solution system framework is as follows: First, sound segments 111 from different sources are input into a speech recognition system (DeepSpeech RNN) 112 for speech feature extraction. The resulting speech features are then processed by a convolutional neural network (CNN) 113 to map the speech features of different people to a general, shared latent audio expression space 114. For the speech features of different people, they can be constructed by linear combinations of different blendshapes in this space. The output of the generalized network 11 enters the content-aware filtering 121 of the specialized network 12 to obtain smooth audio-expression parameters 122, which in turn yields the reconstructed 3D face model 123 and UV map 124. Finally, the UV Map 124 and the background image 125 are input into the Neural Rendering Network 126 to obtain the final output image 127.

[0046] The aforementioned methods in related technologies suffer from at least the following problems: They are speech-driven methods, unable to output a synchronized lip-sync video given text; the facial parameters used are limited to UV maps obtained from 3D face models, but these UV maps only provide prior information about lip shapes, offering no auxiliary information for tooth details; and during network training, they only penalize frames corresponding to predicted and actual values, neglecting the differences between consecutive input frames, resulting in jitter in the final video. Furthermore, all of these methods produce unsmooth and unrealistic synchronized lip-sync video frames.

[0047] The main challenges in generating 3D virtual human lip-synced speech videos currently include two aspects: face reconstruction and video frame rendering. To address the first challenge, this application proposes a speech-to-face parameter network that can simultaneously learn 2D mouth keypoints and 3D facial expression parameters from speech features. This obtains both the precise positional information provided by 2D keypoints and retains the depth information advantage of 3D facial parameters. Combining 2D and 3D parameters to reconstruct the face ensures accuracy. After obtaining the reconstructed face, it needs to be blended with the background. To address the second challenge, this application proposes a two-stage rendering network. The first rendering network renders the mouth texture region from the reconstructed face, while the second rendering network combines the mouth texture region with the background to render the final video frame. The advantages of using a two-stage rendering network are: 1) Training the two rendering networks separately reduces training difficulty while ensuring the accuracy of the mouth texture generated by the first rendering network; 2) When training the second rendering network, the mouth region is penalized again, achieving correction of the mouth shape and optimization of details such as teeth and wrinkles. In addition, a video frame similarity loss was used when training the rendering network to ensure that the difference between consecutive output frames is not large, thus avoiding video jitter and unsmooth and unrealistic video issues.

[0048] The action-driven method for a target object provided in this application first acquires source speech and a target video, the target video including the target object; then, it performs face parameter conversion processing on the speech parameters of the source speech at each moment to obtain the source parameters of the source speech at the corresponding moment; it extracts parameters from the target video to obtain target parameters; based on the combined parameters obtained by combining the source parameters and target parameters, it reconstructs the target object in the target video to obtain a reconstructed image; finally, it generates a synthesized video by using the reconstructed image, the synthesized video containing the target object, and the action of the target object corresponding to the source speech. Thus, because the synthesized video of the target object's action is obtained based on the combined parameters of the source parameters and target parameters, the final synthesized video is smoother and more realistic, improving the visual effect of the video synthesis.

[0049] The following describes exemplary applications of the motion-driving device for the target object in the embodiments of this application. In one implementation, the motion-driving device for the target object provided in the embodiments of this application can be implemented as any terminal with video playback function, such as a laptop computer, tablet computer, desktop computer, mobile device (e.g., mobile phone, portable music player, personal digital assistant, dedicated messaging device, portable gaming device), or intelligent robot. In another implementation, the motion-driving device for the target object provided in the embodiments of this application can also be implemented as a server. The following will describe exemplary applications when the motion-driving device for the target object is implemented as a server.

[0050] See Figure 2 , Figure 2This is an optional architecture diagram of the target object action-driven system 20 provided in this application embodiment. To achieve the synthesis of a synthetic video that simultaneously contains a target object and source speech, i.e., to generate a synthetic video in which source speech drives the action of the target object, the target object action-driven system 20 provided in this application embodiment includes a terminal 100, a network 200, and a server 300. The terminal 100 acquires the target video and source speech, generates an action-driven request for the target object based on the target video and source speech, and sends the action-driven request to the server 300 through the network 200. In response to the action-driven request, the server 300 performs face parameter conversion processing on the speech parameters of the source speech at each moment to obtain the source parameters of the source speech at the corresponding moment; and extracts parameters from the target video to obtain target parameters; then, based on the combined parameters obtained by combining the source parameters and target parameters, it performs image reconstruction on the target object in the target video to obtain a reconstructed image; and generates a synthetic video from the reconstructed image, wherein the synthetic video contains the target object, and the action of the target object corresponds to the source speech. After obtaining the synthetic video, the synthetic video is sent to the terminal 100 through the network 200. After acquiring the synthesized video, terminal 100 plays the synthesized video on its current interface 100-1.

[0051] The action-driven method for the target object provided in this application also relates to the field of artificial intelligence technology, using artificial intelligence technology to synthesize synthetic videos. In this application embodiment, it can be achieved at least through computer vision technology, speech technology, and natural language processing technology within artificial intelligence. Computer vision (CV) is a science that studies how to enable machines to "see," specifically, it refers to machine vision that uses cameras and computers to replace human eyes for target recognition and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition. Key technologies in speech technology include Automatic Speech Recognition (ASR), Text-to-Speech (TTS), and voiceprint recognition. Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech emerging as one of the most promising methods. Natural Language Processing (NLP) is an important area within computer science and artificial intelligence. It studies various theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science integrating linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus has a close connection with linguistic research. NLP technologies typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0052] The action-driven method for the target object provided in this application embodiment can also be implemented based on a cloud platform and through cloud technology. For example, the server 300 mentioned above can be a cloud server.

[0053] Figure 3 This is a schematic diagram of the structure of the server 300 provided in the embodiments of this application. Figure 3The server 300 shown includes at least one processor 310, memory 350, at least one network interface 320, and a user interface 330. The various components in the server 300 are coupled together via a bus system 340. It is understood that the bus system 340 is used to implement communication between these components. In addition to a data bus, the bus system 340 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 3 The general labeled all buses as Bus System 340.

[0054] The processor 310 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0055] User interface 330 includes one or more output devices 331 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 330 also includes one or more input devices 332, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0056] Memory 350 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Memory 350 may optionally include one or more storage devices physically located remote from processor 310. Memory 350 may include volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 350 described in this application embodiment is intended to include any suitable type of memory. In some embodiments, memory 350 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as illustrated below.

[0057] Operating system 351 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0058] The network communication module 352 is used to reach other computing devices via one or more (wired or wireless) network interfaces 320, exemplary network interfaces 320 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0059] The input processing module 353 is used to detect and translate one or more user inputs or interactions from one or more input devices 332.

[0060] In some embodiments, the apparatus provided in this application may be implemented in software. Figure 3 A motion-driving device 354 for a target object, stored in memory 350, is shown. This motion-driving device 354 can be a motion-driving device for a target object in server 300, and can be software in the form of programs and plugins, including the following software modules: acquisition module 3541, face parameter conversion module 3542, parameter extraction module 3543, image reconstruction module 3544, and generation module 3545. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0061] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the action-driven method of the target object provided in this application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0062] The following will describe the action-driven method for the target object provided in this application embodiment, using the exemplary application and implementation of the server 300 provided in the embodiments of this application. This method can be a video synthesis method. See also Figure 4 , Figure 4 This is an optional flowchart illustrating the action-driven method for a target object provided in an embodiment of this application. The following will be combined with... Figure 4 The steps shown are explained.

[0063] Step S401: Obtain source audio and target video, the target video including the target object.

[0064] Here, the server can receive a target object action-driven request sent by the user through the terminal. This action-driven request is used to request the synthesis of the source audio and the target video to generate a synthesized video that simultaneously contains the target object and the source audio, and the source audio drives the action of the target object. That is, the synthesized video to be generated contains the target object in the target video, and the audio corresponding to the target object is the source audio.

[0065] The source speech can be pre-recorded by the user, downloaded from the internet, or obtained by converting specific text. In some embodiments, the sound features of the source speech can be the sound features of a specific object or the sound features of a target object in the target video.

[0066] Step S402: Perform face parameter conversion processing on the speech parameters of the source speech at each time step to obtain the source parameters of the source speech at the corresponding time step.

[0067] Here, the source parameters at each moment include, but are not limited to, facial expression parameters and mouth keypoint parameters. Facial expression parameters are the facial expressions corresponding to the speech parameters at that moment. For example, when the speech parameters correspond to a cheerful speech, the facial expression parameter could be a smiling facial expression parameter; when the speech parameters correspond to a sad speech, the facial expression parameter could be a frowning facial expression parameter. Mouth keypoint parameters are the mouth shape parameters used to express the speech parameters at that moment.

[0068] In this embodiment, the facial expression parameters are 3D facial expression parameters, and the mouth key point parameters are 2D key point parameters.

[0069] Step S403: Extract parameters from the target video to obtain the target parameters.

[0070] Here, a preset algorithm can be used to extract parameters from the target video, that is, to extract parameters from the target object in the target video. The target parameters include, but are not limited to, the target mouth key point parameters and the target face parameters. Of course, the target parameters can also include the target object's pose parameters, position parameters, shape parameters and motion parameters, etc.

[0071] Step S404: Based on the combined parameters obtained by combining the source parameters and the target parameters, the target object in the target video is reconstructed to obtain a reconstructed image.

[0072] Here, the source parameters and target parameters are first combined to obtain the combined parameters. The combined parameters are used to characterize the pose, position, shape, action, and lip movements of the target object in the final synthesized video.

[0073] In this embodiment of the application, the target object is reconstructed according to the combination parameters to obtain a reconstructed image, which is used to generate the final synthesized video.

[0074] Step S405: Generate a synthesized video from the reconstructed image.

[0075] Here, the synthesized video contains a target object, and the actions of the target object correspond to the source speech.

[0076] In this embodiment, a corresponding reconstructed image is generated for each moment of the speech parameters, and each reconstructed image is rendered to generate a synthetic image. Since the speech parameters have a certain duration, there can be at least one reconstructed image, and the duration of the synthetic video is equal to the duration of the source speech, or the duration of the synthetic video is greater than the duration of the source speech. When there is only one reconstructed image, the final generated synthetic video is a single synthetic image; when there are multiple reconstructed images, the duration of the final generated synthetic video is the same as the duration of the source speech, and the synthetic video is formed by connecting multiple synthetic images in chronological order.

[0077] In some embodiments, the target video may have at least one video frame containing a target object. When the target video includes one video frame, the video frame contains the target object. The video compositing request is used to request the generation of a composite video containing the target object, and the composite video is a dynamic video obtained based on one video frame. When the target video includes multiple video frames, at least one video frame contains the target object. The video compositing request is used to request the generation of a composite video containing the target object, and the composite video is a dynamic video obtained based on multiple video frames.

[0078] In some embodiments, when the target video includes multiple video frames, the duration of the target video can be the same as or different from the duration of the source audio. When the duration of the target video is the same as the duration of the source audio, a synthetic image can be formed based on the audio parameters of the source audio at each moment corresponding to each video frame, ultimately forming a synthetic video with the same duration as the target video.

[0079] The embodiments of this application can be applied to the following scenarios: In the education industry, if you want to generate a teaching video about a certain knowledge point, you can input the source audio (i.e., classroom teacher audio) corresponding to the knowledge point and the target video with the teacher's lecture into the server. The server can use the method of the embodiments of this application to directly generate a teaching video (i.e., a synthesized video) in which the teacher explains the knowledge point and output it.

[0080] The action-driven method for a target object provided in this application involves performing facial parameter conversion processing on the speech parameters of the source speech at each moment to obtain the source parameters of the source speech at the corresponding moment, extracting parameters from the target video to obtain target parameters, and reconstructing the target object image based on the combined parameters of the source and target parameters to obtain a reconstructed image. Finally, a synthesized video is generated from the reconstructed image, wherein the synthesized video contains the target object, and the actions of the target object correspond to the source speech. Thus, because the synthesized video of the target object's actions is obtained based on the combined parameters of the source and target parameters, the final synthesized video is smoother and more realistic, improving the visual effect of the video synthesis.

[0081] In some embodiments, the motion-driven system for the target object includes at least a terminal and a server. Through the interaction between the terminal and the server, a response to the terminal's motion-driven request is achieved, generating the synthetic video desired by the user.

[0082] In some embodiments, the action-driven request includes source audio and target video. In other embodiments, the action-driven request may also include source text, through which the source audio can be obtained. Figure 5 This is an optional flowchart illustrating an action-driven method for a target object provided in an embodiment of this application, such as... Figure 5 As shown, the method includes the following steps:

[0083] Step S501: The terminal acquires the source audio and the target video.

[0084] Here, the source audio can be audio collected by the user through a voice acquisition device on the terminal, or audio downloaded by the user through the terminal. The target video can be a video of any length, containing the target object.

[0085] Step S502: The terminal acquires the source text and the target video.

[0086] Here, the source text is the text used to generate the source speech. In this embodiment, not only can the input source speech be processed to generate a synthetic video with the source speech, but the input source text can also be parsed and converted to generate source speech, thereby forming a synthetic video with the source speech.

[0087] In step S503, the terminal performs text parsing on the source text to obtain the linguistic features of the source text.

[0088] Here, linguistic features include, but are not limited to, linguistic features such as pinyin, pauses, punctuation marks, and tones. In some embodiments, the source text can also be parsed using artificial intelligence technology to obtain its linguistic features.

[0089] In step S504, the terminal extracts acoustic parameters from the linguistic features to obtain the acoustic parameters of the source text in the time domain. Here, the acoustic parameters are the parameter representations of the source text in the time domain. The acoustic parameters of the source text in the time domain are obtained by extracting acoustic parameters from the linguistic features.

[0090] In step S505, the terminal performs conversion processing on the acoustic parameters to obtain the speech waveform of the source text in the frequency domain.

[0091] Here, the speech waveform is the acoustic representation corresponding to the acoustic parameters, and the speech waveform is the parametric representation of the source text in the frequency domain.

[0092] In step S506, the terminal determines the voice corresponding to the voice waveform as the source voice.

[0093] In step S507, the terminal encapsulates the source audio and target video to form an action-driven request.

[0094] In some embodiments, the terminal may also encapsulate the source text in an action-driven request and send the action-driven request to the server, so that the server can implement the steps of converting the source text into source speech in steps S503 to S506.

[0095] In step S508, the terminal sends the action-driven request to the server.

[0096] In step S509, the server parses the action-driven request to obtain the source audio and the target video.

[0097] In step S510, the server performs face parameter conversion processing on the voice parameters of the source voice at each moment to obtain the source parameters of the source voice at the corresponding moment.

[0098] Step S511: The server extracts parameters from the target video to obtain the target parameters.

[0099] In step S512, the server reconstructs the image of the target object in the target video based on the combination parameters obtained by combining the source parameters and the target parameters, and obtains the reconstructed image.

[0100] In step S513, the server generates a synthesized video by reconstructing the image, wherein the synthesized video contains a target object and the action of the target object corresponds to the source speech.

[0101] It should be noted that steps S510 to S513 are the same as steps S402 to S405 described above, and will not be repeated in this embodiment.

[0102] In step S514, the server sends the synthesized video to the terminal.

[0103] Step S515: The terminal plays the synthesized video on the current interface.

[0104] In some embodiments, the source parameters include: facial expression parameters and mouth keypoint parameters. Based on Figure 4 , Figure 6 This is an optional flowchart illustrating an action-driven method for a target object provided in an embodiment of this application, such as... Figure 6 As shown, step S402 can be achieved through the following steps:

[0105] Step S601: Extract features from the source speech to obtain the speech feature vector of the source speech.

[0106] Step S602: Perform convolution and fully connected processing on the speech feature vector in sequence to obtain the facial expression parameters and mouth key point parameters of the source speech at the corresponding time.

[0107] In some embodiments, step S602 can be implemented by the following steps:

[0108] Step S6021: The speech feature vector is sequentially convolved by at least two first convolutional layers with specific convolutional kernels to obtain the convolutional processed vector.

[0109] Step S6022: The convolution processing vector is processed sequentially through at least two fully connected layers to obtain a fully connected processing vector.

[0110] Here, the fully connected processing vector includes the vector corresponding to the facial expression parameters and the vector corresponding to the mouth keypoint parameters. The sum of the dimensions of the vector corresponding to the facial expression parameters and the vector corresponding to the mouth keypoint parameters is equal to the dimension of the fully connected processing vector.

[0111] Please continue to refer to Figure 6 In some embodiments, step S403 can be implemented by the following steps:

[0112] Step S603: Extract mouth parameters and face parameters from the target object in the current video frame of the target video in sequence to obtain the target mouth key point parameters and target face parameters.

[0113] Here, the target mouth key point parameters and target face parameters are parameters of the target object. When the target video has multiple video frames, the target mouth key point parameters and target face parameters of the target object can be extracted from each video frame.

[0114] Step S604: Determine the target mouth key point parameters and target face parameters as target parameters.

[0115] Please continue to refer to Figure 6 In some embodiments, step S404 can be implemented by the following steps:

[0116] Step S605: Combine the source parameters and the target parameters to obtain the combined parameters.

[0117] Here, combining the source parameters and the target parameters can be achieved by extracting the parameters used to generate the final composite image and deleting the parameters not used to generate the final composite image, thus obtaining the combined parameters.

[0118] Step S606: Based on the combination parameters, the target object in the target video is reconstructed to obtain the mouth contour map and UV map.

[0119] In this embodiment of the application, the reconstructed image includes a mouth contour map and a UV map. The mouth contour map is used to reflect the mouth contour of the target object in the final generated synthetic image, and the UV map is used to combine with the mouth contour map to generate the mouth region texture of the target object in the synthetic image.

[0120] Step S607: Determine the mouth contour map and UV map as the reconstructed image.

[0121] In this embodiment, the source parameters include: expression parameters and mouth key point parameters; the target parameters include target mouth key point parameters and target face parameters; the target face parameters include at least: target pose parameters, target shape parameters, and target expression parameters; correspondingly, step S605 can be implemented through the following steps:

[0122] Step S6051: Replace the target facial expression parameter in the target face parameters with the facial expression parameter to obtain the replaced face parameters. Step S6052: Replace the target mouth keypoint parameter with the mouth keypoint parameter to obtain the replaced mouth keypoint parameter. Step S6053: Determine the replaced face parameters and the replaced mouth keypoint parameter as the combined parameters.

[0123] Correspondingly, the process of generating a synthetic video by reconstructing the image in step S405 can be achieved through the following steps:

[0124] Step S6054: Input the replaced face parameters, replaced mouth keypoint parameters, and background image corresponding to the target video at each time step into the image rendering model. Step S6055: Render the mouth region using the replaced face parameters and replaced mouth keypoint parameters through the first rendering network in the image rendering model to obtain a mouth region texture image.

[0125] In some embodiments, the first rendering network includes at least one second convolutional layer, at least one first downsampling layer, and at least one first upsampling layer; the mouth region rendering process in step S6055 can be implemented by the following steps: sequentially passing the second convolutional layer and the first downsampling layer to perform convolution processing and downsampling processing on the replaced face parameters and the replaced mouth keypoint parameters to extract the depth features of the reconstructed image; and passing the first upsampling layer to perform upsampling processing on the replaced face parameters and the replaced mouth keypoint parameters to restore the resolution of the reconstructed image and obtain the mouth region texture image.

[0126] Step S6056: The texture image of the mouth area and the background image are stitched together using the second rendering network in the image rendering model to obtain the composite image at the current moment.

[0127] In some embodiments, the second rendering network includes at least one third convolutional layer, at least one second downsampling layer, and at least one second upsampling layer; the stitching process in step S6056 can be implemented by the following steps: sequentially passing the third convolutional layer and the second downsampling layer to perform convolutional processing and downsampling processing on the mouth region texture image and the background image to extract the depth features of the mouth region texture image and the background image; and passing the second upsampling layer to perform upsampling processing on the mouth region texture image and the background image to restore the resolution of the mouth region texture image and the background image, and obtain the synthesized image at the current time.

[0128] Step S6057: Based on the synthesized image at each moment, determine the synthesized video that simultaneously contains the target object and the source speech.

[0129] In some embodiments, the image rendering model described above is used to render the reconstructed image at each time step to generate a synthetic image at the corresponding time step, wherein the synthetic image contains not only the target object but also the source speech at the corresponding time step. The image rendering model includes at least a first rendering network and a second rendering network. The first rendering network is used to perform feature extraction and mouth shape region rendering on the reconstructed image and the target image, respectively, and the second rendering network is used to stitch the mouth shape region texture image and the target image together. The training method of the image rendering model provided in the embodiments of this application is described below.

[0130] Figure 7This is a schematic diagram illustrating the implementation flow of the training method for the image rendering model provided in this application embodiment, as shown below. Figure 7 As shown, the method includes the following steps:

[0131] Step S701: Input the sample reconstructed image and the sample target image into the image rendering model.

[0132] In some embodiments, the sample reconstructed image can be obtained through the following steps: performing face parameter conversion processing on the speech parameters of the sample speech at the current time to obtain sample speech parameters; extracting parameters from the sample target image to obtain sample target parameters; combining the sample speech parameters and sample target parameters to obtain sample combination parameters; and reconstructing the target object in the sample target image based on the sample combination parameters to obtain the sample reconstructed image.

[0133] In other embodiments, the sample reconstructed image can also be obtained through the following steps: parsing the sample text to obtain its linguistic features; extracting acoustic parameters from the linguistic features to obtain the acoustic parameters of the sample text in the time domain; converting these acoustic parameters to obtain the speech waveform of the sample text in the frequency domain, and identifying the speech waveform as the sample speech. Then, performing face parameter conversion on the speech parameters of the sample speech at the current time to obtain the sample speech parameters; extracting parameters from the sample target image to obtain the sample target parameters; combining the sample speech parameters and the sample target parameters to obtain sample combination parameters; and reconstructing the target object in the sample target image based on the sample combination parameters to obtain the sample reconstructed image.

[0134] The sample target image includes the sample target object, and the final generated sample composite image also includes the sample target object.

[0135] Step S702: Through the first rendering network of the image rendering model, feature extraction and mouth region rendering are performed on the sample reconstructed image and the sample target image respectively to obtain the sample mouth texture image.

[0136] Here, the first rendering network includes at least one second convolutional layer, at least one first downsampling layer, and at least one first upsampling layer.

[0137] During feature extraction, the parameters corresponding to the input sample reconstructed image and sample target image can be convolved using a second convolutional layer. A first downsampling layer then downsamples the convolutional parameters to extract the depth features of the sample reconstructed image and sample target image, thus obtaining the first sample image features. During mouth region rendering, the extracted first sample image features can be upsampled using a first upsampling layer to restore the resolution of the sample reconstructed image and sample target image, resulting in a sample mouth texture image.

[0138] In this embodiment, a second convolutional layer is connected before each first downsampling layer and after each first upsampling layer. That is, a convolutional process is performed before each downsampling process and after each upsampling process. In some embodiments, a skip connection is introduced between the first downsampling layer and the first upsampling layer to preserve feature information at different resolutions.

[0139] Step S703: The sample mouth texture image and the sample target image are stitched together using the second rendering network in the image rendering model to obtain the sample composite image.

[0140] Here, the second rendering network includes at least one third convolutional layer, at least one second downsampling layer, and at least one second upsampling layer.

[0141] During the stitching process, the parameters corresponding to the input sample mouth texture image and sample target image are first convolved using a third convolutional layer. Then, a second downsampling layer downsamples the convolutional parameters to extract the depth features of the sample mouth texture image and sample target image, thus obtaining the second sample image features. Next, a second upsampling layer upsamples the extracted second sample image features to restore the resolution of the sample mouth texture image and sample target image, resulting in the composite sample image.

[0142] In this embodiment, a third convolutional layer is connected before each second downsampling layer and after each second upsampling layer. That is, a convolutional process is performed before each downsampling process and after each upsampling process. In some embodiments, a skip connection may be introduced between the second downsampling layer and the second upsampling layer to preserve feature information at different resolutions.

[0143] Step S704: Input the synthesized image of the sample into the preset loss model to obtain the loss result.

[0144] In some embodiments, step S704 can be implemented through the following steps:

[0145] Step S7041: Obtain the real synthetic image corresponding to the sample reconstructed image and the sample target image.

[0146] Step S7042: The sample synthesized image and the real synthesized image are stitched together and input into the preset loss model. The preset loss model is used to calculate the similarity loss between the sample synthesized image and the real synthesized image before and after the frame to obtain the loss result.

[0147] In this embodiment of the application, when calculating the similarity loss between consecutive frames, the following loss functions can be calculated: the loss between the two losses of the image rendering model regarding the real synthesized image and the sample synthesized image, the generative adversarial loss, the L1 loss, the difference between the feature maps output by the real synthesized image and the sample synthesized image in N activation layers calculated using the L1 loss, and the final loss obtained by linearly weighting the difference, and the similarity loss between consecutive frames. The loss result is calculated based on at least one of the above loss functions. That is, the loss result can be obtained by weighted summing the loss between the two losses of the image rendering model regarding the real synthesized image and the sample synthesized image, the generative adversarial loss, the L1 loss, the difference between the feature maps output by the real synthesized image and the sample synthesized image in N activation layers calculated using the L1 loss, and the final loss obtained by linearly weighting the difference, and the similarity loss between consecutive frames.

[0148] Step S705: Correct the parameters in the first and second rendering networks based on the loss results to obtain the trained image rendering model.

[0149] In this embodiment, when training the image rendering model, a generative adversarial strategy can be adopted, and the model training can be based on the similarity between consecutive frames to calculate the loss result of the image rendering model in each prediction. This allows for accurate training of the image rendering model, and the trained model considers the continuous changes between consecutive frames, making the changes between two consecutive video frames in the generated synthetic video smoother. This results in a smoother and more realistic synthetic video, improving the visual effect of the synthetic video generated by the image rendering model.

[0150] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0151] This application provides an action-driven method for a target object. This action-driven method is a text-driven or voice-driven 3D virtual human mouth-syncing speaking video generation technology. It predicts mouth shapes by combining 2D and 3D facial parameters, and then uses a rendering network trained with video frame difference loss to synthesize the final output image. This application solves the problems that the voice-driven model is limited to voice input and that the synthesized video is unrealistic and jittery.

[0152] In this embodiment, 2D / 3D face parameters can be learned using text or speech, and then a realistic video of a specific target person speaking in sync with their lip movements can be synthesized. In the implementation process, firstly, TTS technology is used to convert the input text into corresponding speech. Then, a convolutional neural network is used to learn 2D / 3D face parameters from the speech features. Simultaneously, 2D / 3D face parameters are extracted from a video of a target person. By replacing the target person's parameters with the learned parameters, a new 2D / 3D face model is reconstructed. This reconstructed face model (i.e., the reconstructed image) is input into a rendering network to generate video frames, thereby achieving the generation of a video of the target person speaking in sync with their lip movements.

[0153] The method described in this application can be applied to at least several projects and product applications, including smart speaker screens, smart TVs, AI education, virtual anchors, and live streaming. It can synthesize synchronized speaking videos corresponding to a specific target person based on input text or speech, significantly improving the human-computer interaction and user experience of smart products. In typical application scenarios, such as AI education, the method described in this application can automatically generate a personalized 3D virtual teacher who speaks synchronously based on text or speech input from the teacher's end, enabling instruction to students. This improves the user experience for students while reducing the workload for teachers. Alternatively, in the field of virtual anchors, by synthesizing a personalized virtual anchor, the anchor can attract attention by broadcasting game content, enhance interaction through talk shows, or gain high click rates through song and dance covers.

[0154] Figure 8 This is a system framework diagram of the action-driven method for the target object provided in the embodiments of this application, such as... Figure 8As shown, the system can be input with either source text 801 or source speech 802. If the input is source text 801, it will be processed by the text-to-speech module 803 to generate the corresponding source speech. Then, the source speech will be processed by the speech-to-face parameter network 804 to obtain the corresponding face parameters. The face parameters here include 2D mouth key points and 3D expression parameters. The obtained face parameters will be combined with the target parameters obtained by the face parameter extraction module 805 to reconstruct a new face model 806. The UV Map 8061 and the reconstructed mouth key points 8062 can be obtained from the face model 806. Then, the UV Map 8061 and the reconstructed mouth key points 8062 will be input into the two-stage image rendering model 807 trained by the similarity loss between consecutive frames to generate the final output image 808 (i.e., the synthesized image).

[0155] The following section provides a detailed explanation of each part of the system framework for the action-driven method for the target object.

[0156] Text-to-speech module 803: This module aims to convert a given input source text into the corresponding source speech, which will then be used as input to the speech-to-face parameter network.

[0157] Figure 9 This is a framework diagram of the text-to-speech module provided in the embodiments of this application, such as... Figure 9 As shown, the text-to-speech module is mainly divided into three sub-modules: text analysis module 901, acoustic model module 902, and vocoder module 903. Text analysis module 901 is primarily responsible for parsing the input text (i.e., the source text), determining the pronunciation, tone, and intonation of each character, and mapping the text to linguistic features. These linguistic features include, but are not limited to, pinyin, pauses, punctuation marks, and tones. Acoustic model module 902 is primarily responsible for mapping the linguistic features to acoustic parameters, where the acoustic parameters are the time-domain representation of the source text. Vocoder module 903 is responsible for converting the acoustic parameters into a speech waveform, where the speech waveform is the frequency-domain representation of the source text.

[0158] Voice-to-Face Parameter Network 804: Figure 10 This is a framework diagram of the speech-to-face parameter network provided in the embodiments of this application, such as... Figure 10 As shown, A I (Input Audio) represents the input audio segment (i.e., the source audio), obtained from the user's speech or the text-to-speech module mentioned above. A (Audio Features) represents speech features, c1-c4 (Convolution layers) represent four convolutional layers, f1-f3 (Fully connected layers) represent three fully connected layers, and T S(Three-dimensional facial expression parameters of source) represents the source 3D facial expression parameters, K S (2D mouth Keypointsof source) represents the 2D mouth keypoints of the source.

[0159] The purpose of speech-to-face parameter network is to predict the corresponding source 3D facial expression parameters and 2D mouth key points from the input speech segment. The 3D facial expression parameters have 10-dimensional coefficients, while the 2D mouth key points are based on 20 key points used in the Dlib algorithm. Since the 2D key points are composed of two coordinate systems (x, y), the 20 key points correspond to a 40-dimensional vector.

[0160] For the input source speech A I First, the speech features F are extracted using a recurrent neural network (RNN) proposed in the DeepSpeech method. A Then it enters a convolutional neural network (CNN) consisting of four convolutional layers c1-c4 and three fully connected layers f1-f3. Finally, the CNN obtains two sets of face parameters, namely the 3D face expression parameters T. S and 2D mouth key points K S Among them, the extracted speech features F A It can be a 16×29 tensor, with convolutional layers c1-c4 all using 3×1 convolutional kernels, and F... A The dimensions are reduced to 8×32, 4×32, 2×64 and 1×64 respectively. The feature map output by the convolutional layer c4 will pass through three fully connected layers f1-f3 to obtain vectors of 128, 64 and 50 dimensions respectively.

[0161] Face parameter extraction module 805: This module aims to extract the 2D mouth keypoint locations and 3D face parameters of the target person from video frames. The 2D mouth keypoints are obtained using the Dlib algorithm; given an image, this algorithm predicts 68 keypoints on the face, such as... Figure 11The image shown is a diagram illustrating the effect of the Dlib algorithm provided in this embodiment. The left image 1101 is the original image, and the points on the face in the right image 1102 are key points predicted by the Dlib algorithm. In this embodiment, only the predicted 20 mouth key points can be used as 2D face parameters. The 3D face parameters are calculated by predicting 62 dimensions for each face image, including 12-dimensional pose parameters, 40-dimensional shape parameters, and 10-dimensional expression parameters. The 2D mouth key points and 3D face expression parameters obtained by the face parameter extraction module are replaced by the results obtained from the speech-to-face parameter network, while the pose and shape parameters of the target person are retained, resulting in a recombined 3D face parameter. Then, the recombined 3D face parameters are used to reconstruct the target person's face and obtain the corresponding UV Map. The new 2D mouth key point information will be directly used as one of the inputs for subsequent rendering.

[0162] Image rendering model 807: Figure 12 This is a framework diagram of the image rendering model provided in the embodiments of this application, such as... Figure 12 As shown, given 2D mouth keypoints, a UV map, and a background image, the rendering network aims to synthesize the final lip-synced speech video frames. In the implementation process, the 20 reconstructed mouth keypoints are first connected to obtain a polygon as the mouth contour, i.e., K. R (Reconstructed model keypoints), and then based on a specific algorithm, a UV Map is mapped from the 3D face parameters, i.e., U R K R and U R Both have a resolution of 256×256, and the two are stitched together as input to the image rendering model. The image rendering model consists of two stages. The first stage (i.e., the first rendering network) synthesizes the mouth region texture r1 and the target video background frame b. g The background image is stitched together and used as the input to the second rendering network; in the second stage (i.e., the second rendering network), the background image is combined to synthesize the final output r2. Both rendering networks use the U-Net network structure. The U-Net network continuously applies downsampling and convolution operations to extract depth features from the input, and then restores its resolution through progressive upsampling layers. Skip connections are introduced between downsampling and upsampling to preserve feature information at different resolutions.

[0163] In some embodiments, conditional generative adversarial networks (GANs) may be used when training the rendering network, such as... Figure 13The diagram shown is a conditional GAN ​​framework provided in this application embodiment. For the predicted value F (i.e., the synthesized image F) and the real value R (i.e., the real image R) of the rendering network, they are concatenated with the input I (i.e., the input image I) of the rendering network and then sent to the discriminator 1301 to obtain two losses L for the real value and the predicted value. D_fake and L D_real The final loss function L of discriminator 1301 D This can be expressed by the following formula (1-1):

[0164] L D =(L D_fake +L D_real )*0.5 (1-1);

[0165] The rendering network can be viewed as a generator, and its loss function includes the generative adversarial loss L. G_GAN L G_GAN L in the discriminator D_fake The difference is that the generator aims to maximize this value to fool the discriminator, while the discriminator aims to minimize it to distinguish the synthesized image. Furthermore, to make the synthesized image F closer to the real image R, the generator also employs L1 loss, as shown in the following formula (1-2):

[0166] L G_L1 =L1(F,R) (1-2);

[0167] Among them, L G_L1 This represents the loss value corresponding to L1 loss.

[0168] Furthermore, constraints were imposed on the synthetic and real images at the feature level. For example, the synthetic and real images were input into the VGG19 network respectively, and then the difference between the feature maps output by the two images at the five activation layers was calculated using L1 loss and linearly weighted to obtain the final loss L. G_VGG As shown in the following formula (1-3):

[0169]

[0170] Among them, Relu fi and Relu ri These represent the feature maps of the synthetic image and the real image at the i-th activation layer, respectively.

[0171] The aforementioned losses are calculated individually for each frame, without any constraints between frames. This can lead to unevenness or jitter in the final synthesized video. Therefore, this application also introduces a frame-to-frame similarity loss L. G_Smi This reduces the difference between consecutive frames in the synthesized video and the real video. Please continue to refer to... Figure 8 For the synthesized t-th frame, first calculate the difference between the synthesized t-th frame and the (t-1)-th frame, denoted as d. fake Similarly, the difference between frame t and frame (t-1) in the real video is calculated and denoted as d. real L G_Smi The purpose is to reduce d fake and d real The difference, i.e., min[L1(d) fake ,d real )).

[0172] Therefore, the final loss function L of the generator (i.e., the image rendering model) G The formula is as follows (1-4):

[0173] L G =L G_GAN +α*L G_L1 +β*L G_VGG +γ*L G_Smi (1-4);

[0174] In this context, α, β, and γ are all hyperparameters.

[0175] The method provided in this application, compared with other related virtual human synchronized speaking video generation algorithms, can synthesize a more temporally smooth and realistic result. Among other things, Figure 14 It is a virtual human speaking video synthesized using methods in related technologies, such as Figure 14 As shown, synthesized video frames often appear unsmooth and unrealistic.

[0176] The embodiments of this application overcome the aforementioned difficulties by combining 2D and 3D face parameters and introducing similarity loss between consecutive frames, resulting in a final synthesized video with the following effect: Figure 15 The image shows ten consecutive video frames, arranged from left to right and top to bottom. Figure 15 As can be seen, the synthesized video generated by the embodiments of this application is smoother and more realistic, with better visual effects.

[0177] It should be noted that the method in this application embodiment is a text-driven method. By combining mature TTS technology, it can generate a video of a target person speaking, given a piece of text and any video of the target person. Typical application scenarios for this application embodiment include the AI ​​education industry that has emerged in recent years. Unlike current voice-driven virtual teacher generation solutions, this application embodiment extends the input requirements to text or voice, which can further enhance the user experience.

[0178] In some embodiments, the speech-to-face parameter network described above employs a convolutional neural network to predict face parameters based on the speech features extracted using DeepSpeech. However, for this module, the embodiments of this application do not limit the model type of the deep convolutional network. For example, recurrent neural networks or generative adversarial networks can also be used instead of convolutional neural networks, and the choice can be made according to the accuracy and efficiency requirements of the actual application or product. Similarly, the two rendering networks in the image rendering model can not only adopt the U-Net structure, but other encoder-decoder structures can also be used, such as the hourglass network.

[0179] The following continues to describe an exemplary structure of the target object motion driving device 354 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 3 As shown, the software module stored in the target object's motion driving device 354 in the memory 350 can be the target object's motion driving device in the server 300, and the device includes:

[0180] Acquisition module 3541 is used to acquire source audio and target video, wherein the target video includes a target object;

[0181] The face parameter conversion module 3542 is used to perform face parameter conversion processing on the speech parameters of the source speech at each moment to obtain the source parameters of the source speech at the corresponding moment.

[0182] Parameter extraction module 3543 is used to extract parameters from the target video to obtain target parameters;

[0183] The image reconstruction module 3544 is used to reconstruct the image of the target object in the target video based on the combination parameters obtained by combining the source parameters and the target parameters, so as to obtain a reconstructed image;

[0184] The generation module 3545 is used to generate a synthesized video from the reconstructed image, wherein the synthesized video contains the target object, and the action of the target object corresponds to the source speech.

[0185] In some embodiments, the acquisition module is further configured to: acquire source text, perform text parsing on the source text to obtain linguistic features of the source text; extract acoustic parameters from the linguistic features to obtain acoustic parameters of the source text in the time domain; perform conversion processing on the acoustic parameters to obtain speech waveforms of the source text in the frequency domain; and determine the speech corresponding to the speech waveform as the source speech.

[0186] In some embodiments, the source parameters include: facial expression parameters and mouth key point parameters; the face parameter conversion module is further configured to: extract features from the source speech to obtain a speech feature vector of the source speech; and sequentially perform convolution and fully connected processing on the speech feature vector to obtain the facial expression parameters and mouth key point parameters of the source speech at the corresponding time.

[0187] In some embodiments, the face parameter conversion module is further configured to: sequentially perform the convolution processing on the speech feature vector through at least two first convolutional layers with specific convolutional kernels to obtain a convolution processing vector; sequentially perform the fully connected processing on the convolution processing vector through at least two fully connected layers to obtain a fully connected processing vector; wherein the fully connected processing vector includes a vector corresponding to the expression parameters and a vector corresponding to the mouth keypoint parameters, and the sum of the dimensions of the vector corresponding to the expression parameters and the vector corresponding to the mouth keypoint parameters is equal to the dimension of the fully connected processing vector.

[0188] In some embodiments, the parameter extraction module is further configured to: extract mouth parameters and extract face parameters sequentially from the target object in the current video frame of the target video to obtain target mouth key point parameters and target face parameters; and determine the target mouth key point parameters and the target face parameters as the target parameters.

[0189] In some embodiments, the image reconstruction module is further configured to: combine the source parameters and the target parameters to obtain the combined parameters; reconstruct the target object in the target video according to the combined parameters to obtain a mouth contour map and a UV map; and determine the mouth contour map and the UV map as the reconstructed image.

[0190] In some embodiments, the source parameters include: expression parameters and mouth keypoint parameters; the target parameters include target mouth keypoint parameters and target face parameters; the target face parameters include at least: target pose parameters, target shape parameters, and target expression parameters; the image reconstruction module is further configured to: replace the target expression parameters in the target face parameters with the expression parameters to obtain replaced face parameters; replace the target mouth keypoint parameters with the mouth keypoint parameters to obtain replaced mouth keypoint parameters; and determine the replaced face parameters and the replaced mouth keypoint parameters as the combined parameters.

[0191] In some embodiments, the generation module is further configured to: input the replaced face parameters, the replaced mouth keypoint parameters, and the background image corresponding to the target video at each moment into an image rendering model; render the mouth region of the replaced face parameters and the replaced mouth keypoint parameters through a first rendering network in the image rendering model to obtain a mouth region texture image; stitch the mouth region texture image and the background image through a second rendering network in the image rendering model to obtain a synthesized image at the current moment; and determine the synthesized video that simultaneously contains the target object and the source speech based on the synthesized image at each moment.

[0192] In some embodiments, the first rendering network includes at least one second convolutional layer, at least one first downsampling layer, and at least one first upsampling layer; the rendering module is further configured to: sequentially perform convolution processing and downsampling processing on the replaced face parameters and the replaced mouth keypoint parameters through the second convolutional layer and the first downsampling layer to extract the depth features of the reconstructed image; and, through the first upsampling layer, perform upsampling processing on the replaced face parameters and the replaced mouth keypoint parameters to restore the resolution of the reconstructed image and obtain the mouth region texture image.

[0193] In some embodiments, the second rendering network includes at least one third convolutional layer, at least one second downsampling layer, and at least one second upsampling layer; the rendering module is further configured to: sequentially perform convolution processing and downsampling processing on the mouth region texture image and the background image through the third convolutional layer and the second downsampling layer to extract the depth features of the mouth region texture image and the background image; and, through the second upsampling layer, perform upsampling processing on the mouth region texture image and the background image to restore the resolution of the mouth region texture image and the background image, and obtain the synthesized image at the current time.

[0194] In some embodiments, the image rendering model is trained through the following steps: inputting a sample reconstructed image and a sample target image into the image rendering model; performing feature extraction and mouth region rendering on the sample reconstructed image and the sample target image respectively through the first rendering network of the image rendering model to obtain a sample mouth texture image; performing a stitching process on the sample mouth texture image and the sample target image through the second rendering network of the image rendering model to obtain a sample composite image; inputting the sample composite image into a preset loss model to obtain a loss result; and correcting the parameters in the first rendering network and the second rendering network according to the loss result to obtain the trained image rendering model.

[0195] In some embodiments, the image rendering model is trained through the following steps: obtaining a real synthetic image corresponding to the sample reconstructed image and the sample target image; stitching the sample synthetic image and the real synthetic image together and inputting them into the preset loss model; calculating the frame similarity loss between the sample synthetic image and the real synthetic image through the preset loss model to obtain the loss result.

[0196] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0197] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described in this application.

[0198] This application provides a storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to perform the method provided in this application, for example... Figure 4 The method shown.

[0199] In some embodiments, the storage medium may be a computer-readable storage medium, such as a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or a compact disk-read-only memory (CD-ROM); or it may be a device that includes one or any combination of the above-mentioned memories.

[0200] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0201] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0202] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An action driving method of a target object, characterized by, The method comprises: acquiring source speech and target video, the target video comprising a target object; performing face parameter conversion processing on speech parameters of the source speech at each time, to obtain source parameters of the source speech at the corresponding time; the source parameters comprising expression parameters and mouth key point parameters; performing parameter extraction on the target video to obtain target parameters; the target parameters comprising target mouth key point parameters and target face parameters; the target face parameters comprising at least target posture parameters, target shape parameters and target expression parameters; replacing the target expression parameters in the target face parameters with the expression parameters to obtain replaced face parameters; and replacing the target mouth key point parameters with the mouth key point parameters to obtain replaced mouth key point parameters; inputting the replaced face parameters, the replaced mouth key point parameters and a background image corresponding to the target video at each time into an image rendering model; wherein the image rendering model is trained by the following steps: inputting a sample reconstructed image and a sample target image into the image rendering model; performing feature extraction and mouth shape region rendering on the sample reconstructed image and the sample target image respectively by a first rendering network of the image rendering model, to obtain a sample mouth shape texture image; performing splicing processing on the sample mouth shape texture image and the sample target image by a second rendering network in the image rendering model, to obtain a sample synthesized image; acquiring a real synthesized image corresponding to the sample reconstructed image and the sample target image; inputting the sample synthesized image and the real synthesized image after splicing into a preset loss model, to determine a loss between the real synthesized image and the sample synthesized image; determining a loss value of a generative adversarial loss of the image rendering model, a loss value of an L1 loss and a loss value of a front-back frame similarity loss; determining a difference between feature maps output by N activation layers of the real synthesized image and the sample synthesized image calculated by the L1 loss; performing linear weighting on the difference, to obtain a loss value of a final loss; performing weighted summation on the loss value between the two losses, the loss value of the generative adversarial loss, the loss value of the L1 loss, the loss value of the final loss and the loss value of the front-back frame similarity loss, to obtain a loss result; correcting parameters in the first rendering network and the second rendering network according to the loss result, to obtain a trained image rendering model; performing mouth shape region rendering on the replaced face parameters and the replaced mouth key point parameters by the first rendering network in the image rendering model, to obtain a mouth shape region texture image; performing splicing processing on the mouth shape region texture image and the background image by the second rendering network in the image rendering model, to obtain a synthesized image at the current time; According to the synthetic image at each moment, a synthetic video with the target object and the source voice is determined, wherein the target object is in the synthetic video, and the action of the target object corresponds to the source voice.

2. The method of claim 1, wherein, The source voice is obtained by: obtaining a source text, and performing text analysis on the source text to obtain linguistic features of the source text; extracting acoustic parameters of the linguistic features to obtain acoustic parameters of the source text in the time domain; performing conversion processing on the acoustic parameters to obtain a voice waveform of the source text in the frequency domain; determining the voice corresponding to the voice waveform as the source voice.

3. The method of claim 1, wherein, The face parameter conversion processing on the voice parameter of the source voice at each moment to obtain the source parameter of the source voice at the corresponding moment includes: extracting features of the source voice to obtain a voice feature vector of the source voice; sequentially performing convolution processing and full connection processing on the voice feature vector to obtain the expression parameter and the mouth key point parameter of the source voice at the corresponding moment.

4. The method of claim 3, wherein, The convolution processing and full connection processing on the voice feature vector to obtain the expression parameter and the mouth key point parameter of the source voice at the corresponding moment include: sequentially performing the convolution processing on the voice feature vector through at least two first convolution layers with specific convolution kernels to obtain a convolution processing vector; sequentially performing the full connection processing on the convolution processing vector through at least two full connection layers to obtain a full connection processing vector; wherein the full connection processing vector includes a vector corresponding to the expression parameter and a vector corresponding to the mouth key point parameter, and the sum of the dimensions of the vector corresponding to the expression parameter and the vector corresponding to the mouth key point parameter is equal to the dimension of the full connection processing vector.

5. The method of claim 1, wherein, The parameter extraction on the target video to obtain a target parameter includes: sequentially performing mouth parameter extraction and face parameter extraction on the target object in the current video frame of the target video to obtain target mouth key point parameters and target face parameters; determining the target mouth key point parameters and the target face parameters as the target parameters.

6. The method of claim 1, wherein, The first rendering network includes at least one second convolution layer, at least one first down-sampling layer, and at least one first up-sampling layer; The mouth shape region rendering on the replaced face parameters and the replaced mouth key point parameters through the first rendering network in the image rendering model to obtain a mouth shape region texture image includes: sequentially performing convolution processing and down-sampling processing on the replaced face parameters and the replaced mouth key point parameters through the second convolution layer and the first down-sampling layer to extract depth features of a reconstructed image; and performing up-sampling processing on the replaced face parameters and the replaced mouth key point parameters through the first up-sampling layer to restore the resolution of the reconstructed image and obtain the mouth shape region texture image.

7. The method of claim 1, wherein, The second rendering network includes at least one third convolution layer, at least one second down-sampling layer, and at least one second up-sampling layer; The second rendering network in the image rendering model is used to splice the mouth region texture image and the background image to obtain a synthesized image at a current time, including: The third convolution layer and the second down-sampling layer are sequentially used to perform convolution processing and down-sampling processing on the mouth region texture image and the background image to extract deep features of the mouth region texture image and the background image; and The second up-sampling layer is used to perform up-sampling processing on the mouth region texture image and the background image to restore resolutions of the mouth region texture image and the background image, and obtain the synthesized image at the current time.

8. An action driving apparatus of a target object, characterized by comprising: The device comprises: An acquisition module configured to acquire source speech and target video, the target video comprising a target object; A face parameter conversion module configured to perform face parameter conversion processing on speech parameters of the source speech at each time to obtain source parameters of the source speech at corresponding times, the source parameters comprising expression parameters and mouth key point parameters; A parameter extraction module configured to perform parameter extraction on the target video to obtain target parameters, the target parameters comprising target mouth key point parameters and target face parameters, the target face parameters comprising at least target posture parameters, target shape parameters and target expression parameters; An image reconstruction module configured to replace the target expression parameters in the target face parameters with the expression parameters to obtain replaced face parameters, and replace the target mouth key point parameters with the mouth key point parameters to obtain replaced mouth key point parameters. The generating module is configured to input the replaced face parameter, the replaced mouth key point parameter and a background image corresponding to the target video at each moment into an image rendering model; wherein the image rendering model is trained by the following steps: inputting a sample reconstructed image and a sample target image into the image rendering model; performing feature extraction and mouth shape region rendering on the sample reconstructed image and the sample target image respectively by a first rendering network of the image rendering model to obtain a sample mouth shape texture image; performing splicing processing on the sample mouth shape texture image and the sample target image by a second rendering network in the image rendering model to obtain a sample synthesized image; obtaining a real synthesized image corresponding to the sample reconstructed image and the sample target image; inputting the sample synthesized image and the real synthesized image after splicing into a preset loss model to determine a loss between the real synthesized image and the sample synthesized image; determining a loss value of a generative adversarial loss of the image rendering model, a loss value of an L1 loss and a loss value of a front and back frame similarity loss; determining a difference between feature maps output by N activation layers of the real synthesized image and the sample synthesized image calculated by the L1 loss; performing linear weighting on the difference to obtain a loss value of a final loss; performing weighted summation on the loss value between the two losses, the loss value of the generative adversarial loss, the loss value of the L1 loss, the loss value of the final loss and the loss value of the front and back frame similarity loss to obtain a loss result; correcting parameters in the first rendering network and the second rendering network according to the loss result to obtain a trained image rendering model; performing mouth shape region rendering on the replaced face parameter and the replaced mouth key point parameter by the first rendering network in the image rendering model to obtain a mouth shape region texture image; performing splicing processing on the mouth shape region texture image and the background image by the second rendering network in the image rendering model to obtain a synthesized image at the current moment; determining a synthesized video having both the target object and the source speech according to the synthesized image at each moment, wherein the synthesized video has the target object and the action of the target object corresponds to the source speech.

9. The apparatus of claim 8, wherein the image reconstruction module is further configured to combine the source parameter and the target parameter to obtain a combined parameter, and reconstruct an image of the target object in the target video according to the combined parameter to obtain a mouth contour map and a UV map; the mouth contour map and the UV map are determined as the reconstructed image.

10. An action driving system of a target object, characterized by, at least comprising: a terminal and a server; the terminal is configured to send a motion driving request of the target object to the server, the motion driving request comprising a source speech and a target video, and the target video comprising a target object; The server is configured to implement the action driving method of the target object according to any one of claims 1-7 in response to the action driving request.

11. A computer readable storage medium, characterized in that, The storage stores executable instructions configured to cause the processor to implement the action driving method of the target object according to any one of claims 1-7 when the executable instructions are executed by the processor.

12. A computer program product comprising executable instructions or a computer program, characterized in that, The executable instructions or the computer program are configured to implement the action driving method of the target object according to any one of claims 1-7 when executed by the processor.

Images