Method, apparatus, device and storage medium for video generation

Abstract

Embodiments of the present disclosure provide a video generation method, device, equipment, storage medium and program product. The method comprises: obtaining a reference image and a dialogue voice, the reference image comprising a target object, and the dialogue voice comprising target voice corresponding to the target object and interaction voice for interacting with the target object; generating reference motion feature information and reference visual feature information corresponding to a face of the target object based on a reference video; extracting interaction motion feature information of the dialogue voice; determining a motion feature information sequence corresponding to the dialogue voice based on at least the interaction motion feature information; and generating a target video based on the reference motion feature information, the reference visual feature information and the motion feature information sequence. In this way, natural conversion of the target object between different states (for example, listening and speaking) can be achieved without manual role designation or displayed role switching.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computer technology, and particularly to methods, apparatus, devices, and computer-readable storage media for video generation. Background Technology

[0002] In recent years, researchers have paid considerable attention to audio-driven face generation in order to construct dialogue agents. However, most studies focus only on one-sided communication, such as speaking or listening, ignoring the duality of human-to-human interaction. Speaking face generation techniques aim to synthesize facial animations of the speaker based on a reference image and audio. While these works can produce vivid videos with precise lip-sync, they only emphasize the speaker's role and neglect the listener's feedback. Listening face generation techniques aim to react to the speaker's actions; however, these works limit listener responses to nonverbal facial movements, which differs significantly from real-life interaction scenarios. Improving interactivity in face generation has been a long-standing concern. Summary of the Invention

[0003] In a first aspect of this disclosure, a video generation method is provided. The method includes: acquiring a reference image and dialogue speech, the reference image including a target object, and the dialogue speech including target speech corresponding to the target object and interactive speech for interacting with the target object; generating reference motion feature information and reference visual feature information corresponding to the face of the target object based on the reference video; extracting interactive motion feature information of the dialogue speech; determining a sequence of motion feature information corresponding to the dialogue speech based at least on the interactive motion feature information; and generating a target video based on the reference motion feature information, the reference visual feature information, and the sequence of motion feature information.

[0004] In a second aspect of this disclosure, an apparatus for video generation is provided. The apparatus includes: an input acquisition module configured to acquire a reference image and dialogue speech, the reference image including a target object, and the dialogue speech including target speech corresponding to the target object and interactive speech for interacting with the target object; a feature information generation module configured to generate reference motion feature information and reference visual feature information corresponding to the face of the target object based on the reference video; an interactive motion feature information extraction module configured to extract interactive motion feature information of the dialogue speech; a motion feature information sequence determination module configured to determine a motion feature information sequence corresponding to the dialogue speech based at least on the interactive motion feature information; and a target video generation module configured to generate a target video based on the reference motion feature information, the reference visual feature information, and the motion feature information sequence.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the electronic device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The medium stores a computer program that, when executed by a processor, implements the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes a computer program that, when executed by a processor, implements the method of the first aspect.

[0008] It should be understood that the content described in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0011] Figure 2 The reasoning process of a video generation model according to some embodiments of the present disclosure is illustrated;

[0012] Figure 3 A schematic diagram of the architecture of a motion extraction model according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic diagram illustrating the extraction of style feature information according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A flowchart of a video generation method according to some embodiments of the present disclosure is shown;

[0015] Figure 6 An exemplary structural block diagram of an apparatus for video generation according to some embodiments of the present disclosure is shown; and

[0016] Figure 7 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0018] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0019] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0020] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0021] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.

[0022] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0023] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0024] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0025] A neural network is a machine learning network based on deep learning. A neural network processes input and provides a corresponding output, typically consisting of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the input to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each node processing the input from the layer above.

[0026] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as an input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. The testing phase can sometimes be integrated into the training phase. In the application or inference phase, the trained model can be used to process actual model inputs based on the trained parameter values ​​to determine the corresponding model output.

[0027] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In environment 100, electronic device 110 applies video generation model 105 to perform video generation. Video generation model 105 is configured to generate video 116 based on reference image 112 and audio 114.

[0028] In some embodiments, reference image 112 may include a reference object (e.g., a person), and speech 114 includes speech of what the reference object is expected to say. Video 116 may represent the reference object speaking according to speech 114.

[0029] In environment 100, electronic device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, electronic device 110 can also support any type of user-facing interface (such as "wearable" circuitry). Video generation model 105 can be implemented, for example, in various types of computing systems / servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, and so on.

[0030] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0031] As mentioned earlier, face generation lacks interactivity. Recent studies have begun to explore binary interaction face generation, where the generated face needs to fulfill the roles of listener and speaker, capable of speaking or listening. However, these studies require manually assigning roles between listener and speaker, failing to achieve smooth and natural role transitions.

[0032] Many practical applications are increasingly focusing on audio-driven face generation in binary interactions. Some related technologies have designed role converters to perform role switching between listener and speaker. However, explicit role switching can lead to unnaturalness and inconsistency between different states. Furthermore, this paradigm cannot cover all states in binary dialogues, such as when a dialogue agent and dialogue partner speak simultaneously. Some related technologies employ pre-training methods to jointly simulate the actions of the speaker and listener to capture binary context. In applications, pre-trained models require additional fine-tuning for downstream tasks, such as generating faces separately and listening to faces. Therefore, in binary dialogues, manual role assignment is necessary, which can lead to inappropriate switching. In addition, there are other studies on binary interactions, but they are all individual-specific and lack generalization ability.

[0033] To address the aforementioned issues, an embodiment of this disclosure proposes a video generation scheme. Specifically, it involves acquiring a reference image and dialogue speech, the reference image including a target object, and the dialogue speech including target speech corresponding to the target object and interactive speech for interacting with the target object; generating reference motion feature information and reference visual feature information corresponding to the target object's face based on the reference video; extracting interactive motion feature information from the dialogue speech; determining a sequence of motion feature information corresponding to the dialogue speech based at least on the interactive motion feature information; and generating a target video based on the reference motion feature information, reference visual feature information, and the sequence of motion feature information, the target video including the target object speaking according to the target speech, and also including at least one of the target object's sound and movement during the interactive speech.

[0034] According to the scheme disclosed herein, the interactive motion feature information of the dialogue speech can simultaneously include the motion feature information of the target speech and the motion feature information of the interactive speech, and the target object can exhibit corresponding motion feature information at specific moments. In this way, the target object can be naturally switched between different states (e.g., listening and speaking) without the need for manual role specification or explicit role switching.

[0035] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0036] Figure 2 The inference process 200 of a video generation model 105 according to some embodiments of the present disclosure is illustrated. For example... Figure 2 As shown, in order to generate the target video 202, a reference image 205 (made by I) can first be obtained for generating the target video 202. self (Representation) and dialogue voice 210. In some embodiments, reference image 205 may include target object (e.g., person, cartoon character, animal, etc.). Dialogue voice 210 may include target voice 212 corresponding to the target object (by A) self (representation) and interactive voice 214 for interacting with the target object (by A) other (Indicated). Target speech 212 can be the speech corresponding to the content to be said by the target object, and interactive speech 214 can be the speech corresponding to the content to be said by the target object's dialogue partner.

[0037] In some embodiments, the target speech 212 and the interactive speech 214 can be acquired in real time or predetermined. In some examples, in real-time dialogue scenarios, the target speech 212 and the interactive speech 214 can be acquired in real time. In speech-based video generation scenarios, the target speech 212 and the interactive speech 214 can be pre-recorded for video generation. In some examples, in text-based speech generation scenarios, the target speech 212 and the interactive speech 214 can be generated based on text dialogue, and then the target speech 212 and the interactive speech 214 can be used to generate video.

[0038] After obtaining the reference image 205, reference motion feature information 215 (derived from m) corresponding to the face of the target object can be generated based on the reference image 205. self The system includes reference motion feature information 215 and reference visual feature information 220. In some examples, reference motion feature information 215 may characterize features related to the facial movements of the target object (e.g., eye and / or lip movements). Reference visual feature information 220 may characterize features unrelated to the facial movements of the target object (e.g., appearance).

[0039] In some embodiments, the reference motion feature information 215 may be located in the motion feature latent space 240. In some examples, the motion of a person in the reference image 205 (e.g., lip movements, facial expressions, and head poses) can be mapped to this space and transformed into a low-dimensional feature vector (e.g., the reference motion feature information 215).

[0040] In some embodiments, the reference motion feature information 215 may be a one-dimensional vector. According to embodiments of this disclosure, setting the reference motion feature information 215 as a one-dimensional vector allows it to contain as little person information as possible about the target object. In this way, person information and motion feature information can be decoupled, improving the generalization ability of the motion feature information.

[0041] In some embodiments, a visual encoder (not shown) can be used to extract reference visual feature information 220 corresponding to the face of the target object from the reference image 205. For example, the visual encoder can extract three-dimensional appearance information from the reference image 205 as reference visual feature information 220.

[0042] In some embodiments, a mask image 225 can be obtained by occluding regions in the reference image 205 that are unrelated to the facial movement of the target object. In some examples, most facial pixels in the reference image 205 can be occluded, leaving only the eye and lip regions, and then the mask image 225 can be obtained. Using a motion encoder (not shown), reference motion feature information 215 corresponding to the target object's face is extracted from the mask image 225. In this way, by preserving the most expressive parts of the facial expression (e.g., eyes and lips) in the mask image 225, interference from background, hairstyle, clothing, facial features, and other information unrelated to facial movement from different images can be eliminated, thereby improving the accuracy of the reference motion feature information 215.

[0043] In some embodiments, a trained 3D facial landmark model (not shown) can be used to project points related to the facial contours of the target object onto a mask image. In some examples, to provide facial orientation and contour information, the trained 3D facial landmark model can be used to project facial contour information (e.g., points related to the facial contours of the target object) onto a mask image 225. Then, a motion encoder is used to extract reference motion feature information 215 from the projected mask image 225. In this way, the risk of leakage of the target object's identity information can be reduced, while also providing more facial expression details than pure facial landmarks.

[0044] After generating reference motion feature information 215 and reference visual feature information 220, interactive motion feature information 230 of the dialogue speech 210 can be extracted (by f m (Representation). In some examples, the interactive motion feature information 230 may include both the motion feature information of the target speech and the motion feature information of the interactive speech. The interactive motion feature information 230 may be extracted using a motion extraction model 232.

[0045] The following will be referenced Figure 3 Description of extracting interactive motion feature information 230. Figure 3 A schematic diagram of the architecture of a motion extraction model 232 according to some embodiments of the present disclosure is shown. Figure 3 As shown, it can be obtained from the first motion feature library 305 (by M) v The method retrieves the first motion features of the target speech. These first motion features are associated with the speaker's movements; for example, they may include lip movements, mouth movements, facial muscle movements, etc. The first motion feature library 305 stores the correspondence between multiple speech sounds and multiple first motion features. In some examples, the first motion feature library 305 includes multiple learnable embedding representations (e.g., first motion features) to record the movements of a specific speaker (e.g., the movement corresponding to the target speech), and these embedding representations are obtained by e.1:K It means that, among them Let represent the k-th embedding representation, and d represent the dimension. Based on the embedding representations stored in the first motion feature library 305, the first motion feature can be determined.

[0046] After obtaining the first motion feature, the motion feature information of the target speech 212 can be determined based on the target speech 212 and the first motion feature. In some examples, the target speech 212 can be used as a query, and the first motion feature obtained from the first motion feature library 305 can be used as the key and value. Through the cross-attention layer 310, the motion feature information of the target speech can be determined.

[0047] Then, from the second motion feature library 315 (by M) nv The system acquires second motion features corresponding to the interactive speech, which are associated with the movements of non-speakers. For example, the second motion features may include the non-speaker's ear movements, head turning movements, feedback movements, etc. The second motion feature library stores the correspondence between multiple speech sounds and multiple second motion features. In some examples, the first motion feature library 305 includes multiple learnable embedding representations (e.g., second motion features) to record the movements of specific non-speakers (e.g., movements corresponding to the interactive speech), and these embedding representations are obtained by e. 1:K It means that, among them Let represent the k-th embedding representation, and d represent the dimension. Based on the embedding representations stored in the second motion feature library 315, the second motion features can be determined.

[0048] After obtaining the second motion features, the motion feature information of the interactive speech 212 can be determined based on the interactive speech 214 and the second motion features. In some examples, the interactive speech 214 can be used as a query, and the second motion features obtained from the second motion feature library 315 can be used as the key and value. Through the cross-attention layer 320, the motion feature information of the target speech can be determined.

[0049] After determining the motion feature information of the target speech and the interactive speech, interactive motion feature information 230 can be obtained by fusing the motion feature information of the target speech and the interactive speech. In some examples, when the target object speaks, the target speech 212 contains rich information, while the interactive speech 214 contains very little information. The motion feature information of the target speech and the interactive speech is fused using the fusion unit 325. In the fused motion feature information (also called interactive motion feature information 230), the motion feature information of the target speech dominates and drives the target object to appear to be speaking. The fusion unit 325 may involve element-wise summation and multiple multilayer perceptron (MLP) layers. In some examples, when the target object's dialogue partner speaks, the interactive speech 214 contains rich information, while the target speech 212 contains very little information. The motion feature information of the target speech and the interactive speech is fused using the fusion unit 325. In the fused motion feature information, the motion feature information of the interactive speech dominates and drives the target object to appear to be listening.

[0050] In this way, interactive motion feature information 230 can be dynamically constructed based on the content of the dialogue speech, so that the target object can present the corresponding state (e.g., speaking state or listening state). It should be noted that before using the target speech 212 and interactive speech 214 to determine the corresponding motion feature information, the target speech 212 and interactive speech 214 can be encoded using a speech encoder to obtain the corresponding feature representation.

[0051] In some embodiments, the interactive motion feature information 230 can be extracted by the motion extraction model 232. The motion features in the first motion feature library 305 and the second motion feature library 315 are determined during the training of the motion extraction model. In some examples, during the training of the motion extraction model 232, the correspondence between speech and motion features stored in the first motion feature library 305 and the second motion feature library 315 can be updated, thereby obtaining more accurate motion features corresponding to the target speech or interactive speech from the first motion feature library 305 and the second motion feature library 315.

[0052] In some embodiments, the style feature information 234 indicating the speaking style can be used as a basis (by s m (This is used to adjust the first motion feature.) In some examples, such as... Figure 3 As shown, style feature information 234 can be introduced through style modulation layer 330 to explicitly edit the first motion feature, giving it a specific style. Then, based on the target speech and the adjusted first motion feature, motion feature information of the target speech can be determined.

[0053] In some embodiments, the second motion feature can be adjusted based on style feature information 234. In some examples, style feature information 234 can be introduced through style modulation layer 335 to explicitly edit the second motion feature, giving it a specific style. Then, motion feature information of the interactive speech is determined based on the interactive speech and the adjusted second motion feature. Since style feature information 234 contains global information such as emotion and attitude, the realism and vividness of the motion feature information in both the target speech and the interactive speech can be enhanced.

[0054] In some embodiments, style feature information 234 can be extracted from a reference video, which may include the speech of a reference subject. In some examples, the speech of the reference subject has a specific style, such as calm, excited, nervous, confident, etc. Figure 4 A schematic diagram 400 illustrates the extraction of style feature information 234 according to some embodiments of the present disclosure. For example... Figure 4 As shown, reference video 405 includes multiple images (by I1, I2, ..., I...). n Using a motion encoder (not shown in the figure), multiple images in the reference video 405 can be encoded into a reference motion feature sequence 410 (composed of m1, m2, ..., m...). n (To be represented). Next, using the motion style encoder 415, a style feature sequence 420 can be extracted from the reference motion feature sequence 410. By compressing the style feature sequence 420 along the time dimension, style feature information 234 can be obtained. It should be noted that during the training phase, the style feature information 234 can come from any video segment of the driven individual. During the inference phase, the style feature information 234 can be extracted from any video or set to empty.

[0055] Continue to refer to Figure 2 After extracting the interactive motion feature information 230, the motion feature information sequence 235 corresponding to the dialogue speech can be determined based on the interactive motion feature information 230.

[0056] In some embodiments, the motion feature information sequence 235 can be determined iteratively. For a predetermined round in multiple iterations, a reference motion feature information sequence 250, including multiple copies of the reference motion feature information 215, is generated by copying the reference motion feature information 215. Noise is added to the reference motion feature information sequence 250 to obtain a noisy reference motion feature information sequence 255. Next, a diffusion model 260 can be used to perform a denoising operation on the noisy reference motion feature information sequence 255 based on the interactive motion feature information 230 and a portion of the motion feature information sequence determined in the previous round of the predetermined round, to determine the motion feature information sequence 235. In some examples, the diffusion model 260 can be used to map the interactive motion feature information 230 into the motion feature latent space 240. Given a data distribution q(m) 1:M ,f m ), where f m Represents interactive motion feature information 230, m 1:N Representing a sequence of motion feature information corresponding to N frames, the diffusion model 260 can estimate the conditional distribution q(m). 1:N |f m The diffusion model 260 can have a small number of blocks (e.g., 3 blocks, 4 blocks, 5 blocks, etc.), making the video generation model 105 proposed in this disclosure lightweight enough to enable real-time interaction.

[0057] In some embodiments, each block in the diffusion model 260 may include a self-attention layer 262, a motion attention layer 264, and a temporal attention layer 266. In each denoising step, the diffusion model 260 predicts the noise to be added to the reference motion feature information sequence 250. The diffusion time step is converted to a sinusoidal embedding and then concatenated with the noisy motion latent code in the temporal dimension. In the motion attention layer 264, the output of the self-attention layer 262 can be used as a query, with the interactive motion feature information 230 as the key and value. Furthermore, the temporal attention layer 266 can use a portion of the motion features 265 of the motion feature information sequence determined in the previous round as a condition for determining the motion feature information sequence 235, thereby ensuring a smooth transition between motion feature information sequences generated in adjacent rounds.

[0058] After determining the motion feature information sequence 235, a target video 202 can be generated based on the reference motion feature information, the reference visual feature information, and the motion feature information sequence 235. The target video 202 may include the target object speaking according to the target speech 212, and also includes at least one of the target object's voice and movement during the interactive speech 214. For example, the target object may respond to the interactive speech 214 with gestures (e.g., nodding, smiling, etc.) during the interactive speech 214. The target object may respond to the interactive speech 214 with language during the interactive speech 214. In some examples, a motion flow prediction model can be used to predict the motion flow based on the reference motion feature information 215 and the motion feature information sequence 235. The reference visual feature information 220 performs a distortion operation on the motion flow, and the target video 202 can be generated by a decoder. The above process can be represented as follows:

[0059] Flow s→d =F(E m (I self ),E m (V drt )) (1)

[0060]

[0061] Where E m (I self ) represents reference motion characteristic information 215, E m (V drt ) represents the motion feature information sequence 235, Flow s→d Represents motion flow, Warp(·) represents the warp operation, E face (I self ) represents reference visual feature information 220, This indicates the target video is 202.

[0062] In some embodiments, the motion feature information sequence 235 may be located in the motion feature latent space 240. The motion feature latent space 240 may be determined based on the training of the motion encoder, visual encoder, and decoder used for video generation.

[0063] In some embodiments, a motion encoder, a visual encoder, and a decoder can be trained using a first sample video 270. First, the first sample video 270 can be acquired, and it may include multiple sample images, each including a sample object. For each sample image in the first sample video, the motion encoder being trained can be used to encode the sample image to obtain sample motion feature information (e.g., facial motion-related feature information) of the sample object in the motion feature latent space 240. The visual encoder being trained can be used to encode the sample image to obtain sample visual feature information (e.g., appearance-related feature information) of the sample object. The decoder being trained can be used to generate a reconstructed image corresponding to the sample image based on the sample motion feature information and the sample visual feature information. Then, the motion encoder, visual encoder, and decoder can be trained based on a first training objective configured to reduce or minimize the difference between the sample image and the reconstructed image. When training the motion encoder, visual encoder, and decoder, the motion encoder needs to continuously encode the sample images into sample motion feature information in the motion feature latent space, and the decoder needs to continuously decode the sample motion feature information into the reconstructed image. Therefore, the quality of the motion feature latent space can be continuously improved.

[0064] In some embodiments, interactive motion feature information 230 can be extracted by a trained motion extraction model 232, and motion feature information sequence 235 can be generated by a trained diffusion model 260. The motion extraction model and diffusion model can be trained using a second sample video. First, a second sample video can be acquired, which includes sample dialogue speech and multiple sample images. Based on the multiple sample images, a sample motion feature information sequence can be generated. In one example, using a motion encoder, the multiple sample images can be encoded into a motion feature information sequence in a motion latent space 240. Using the training motion extraction model, sample interactive motion feature information can be extracted from the sample dialogue speech. Using the training diffusion model, at least based on the sample interactive motion feature information, a reconstructed motion feature information sequence for the sample dialogue speech can be determined. In one example, the reconstructed motion feature information sequence is also located in the motion latent space 240. Then, the motion extraction model and diffusion model can be trained based on a second training objective, which is configured to reduce or minimize the difference between the sample motion feature information sequence and the reconstructed motion feature information sequence.

[0065] Figure 5 A flowchart of a video generation method 500 according to some embodiments of the present disclosure is shown. Method 500 can be implemented in... Figure 1 110 electronic devices. (Refer to...) Figure 1 The environment 100 is used to describe method 500.

[0066] In box 510, electronic device 110 acquires a reference image and dialogue voice. The reference image includes a target object, and the dialogue voice includes target voice corresponding to the target object and interactive voice for interacting with the target object.

[0067] In box 520, electronic device 110 generates reference motion feature information and reference visual feature information corresponding to the face of the target object based on the reference image.

[0068] In frame 530, electronic device 110 extracts interactive motion feature information of the dialogue speech.

[0069] In box 540, electronic device 110 determines the sequence of motion feature information corresponding to the dialogue speech based at least on interactive motion feature information.

[0070] In frame 550, electronic device 110 generates target video based on reference motion feature information, reference visual feature information, and a sequence of motion feature information.

[0071] In some embodiments, extracting interactive motion feature information includes: obtaining a first motion feature of the target speech from a first motion feature library, wherein the first motion feature library stores the correspondence between multiple speech voices and multiple first motion features; determining motion feature information of the target speech based on the target speech and the first motion feature; obtaining a second motion feature corresponding to the interactive speech from a second motion feature library, wherein the second motion feature library stores the correspondence between multiple speech voices and multiple second motion features; determining motion feature information of the interactive speech based on the interactive speech and the second motion feature; and obtaining interactive motion feature information by fusing the motion feature information of the target speech and the motion feature information of the interactive speech.

[0072] In some embodiments, determining motion feature information of target speech includes: adjusting a first motion feature based on style feature information indicating speaking style; and determining motion feature information of target speech based on target speech and the adjusted first motion feature; and wherein determining motion feature information of interactive speech includes: adjusting a second motion feature based on style feature information; and determining motion feature information of interactive speech based on interactive speech and the adjusted second motion feature.

[0073] In some embodiments, method 500 further includes extracting style feature information from a reference video.

[0074] In some embodiments, generating reference motion feature information and reference visual feature information corresponding to the face of the target object based on the reference video includes: extracting reference visual feature information corresponding to the face of the target object from the reference image using a visual encoder; obtaining a mask image by occluding regions in the reference image that are unrelated to the facial motion of the target object; and extracting reference motion feature information corresponding to the face of the target object from the mask image using a motion encoder.

[0075] In some embodiments, extracting reference motion feature information corresponding to the face of the target object from the mask image includes: projecting points related to the facial contour of the target object onto the mask image using a trained 3D facial keypoint model; and extracting reference motion feature information from the projected mask image using a motion encoder.

[0076] In some embodiments, the motion feature information sequence is determined iteratively, and wherein, for a predetermined round in a plurality of iterations, determining the motion feature information sequence includes: generating a reference motion feature information sequence comprising multiple copies of the reference motion feature information by copying the reference motion feature information; adding noise to the reference motion feature information sequence to obtain a noisy reference motion feature information sequence; and performing a denoising operation on the noisy reference motion feature information sequence based on interactive motion feature information and partial motion features of the motion feature information sequence determined in the previous round of the predetermined round, using a diffusion model, to determine the motion feature information sequence.

[0077] In some embodiments, reference motion feature information and a sequence of motion feature information are located in a motion feature latent space, which is determined based on the training of a motion encoder, a visual encoder, and a decoder used for video generation.

[0078] In some embodiments, the motion encoder, visual encoder, and decoder are trained by: acquiring a first sample video, the first sample video including multiple sample images, the multiple sample images including sample objects; for each sample image in the first sample video, encoding the sample image using the motion encoder being trained to obtain sample motion feature information of the sample object in the motion feature latent space; encoding the sample image using the visual encoder being trained to obtain sample visual feature information of the sample object; generating a reconstructed image corresponding to the sample image based on the sample motion feature information and the sample visual feature information using the decoder being trained; and training the motion encoder, visual encoder, and decoder based on a first training objective, the first training objective being configured to reduce or minimize the difference between the sample image and the reconstructed image.

[0079] In some embodiments, interactive motion feature information is extracted by a trained motion extraction model, and the motion feature information sequence is generated by a trained diffusion model, wherein the motion extraction model and the diffusion model are trained by: acquiring a second sample video, the second sample video including sample dialogue speech and multiple sample images; generating a sample motion feature information sequence based on the multiple sample images; extracting sample interactive motion feature information from the sample dialogue speech using the trained motion extraction model; determining a reconstructed motion feature information sequence for the sample dialogue speech using the trained diffusion model, at least based on the sample interactive motion feature information; and training the motion extraction model and the diffusion model based on a second training objective configured to reduce or minimize the difference between the sample motion feature information sequence and the reconstructed motion feature information sequence.

[0080] In some embodiments, the target speech and interactive speech are captured in real time or predetermined.

[0081] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 6 An exemplary structural block diagram of an apparatus 600 for video generation according to some embodiments of the present disclosure is shown. The apparatus 600 may be implemented as or included in an electronic device 110. Various modules / components in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0082] like Figure 6 As shown, the device 600 includes an input acquisition module 610 configured to acquire a reference image and dialogue speech, wherein the reference image includes a target object and the dialogue speech includes target speech corresponding to the target object and interactive speech for interacting with the target object; a feature information generation module 620 configured to generate reference motion feature information and reference visual feature information corresponding to the face of the target object based on the reference video; an interactive motion feature information extraction module 630 configured to extract interactive motion feature information of the dialogue speech; a motion feature information sequence determination module 640 configured to determine a motion feature information sequence corresponding to the dialogue speech based at least on the interactive motion feature information; and a target video generation module 650 configured to generate a target video based on the reference motion feature information, reference visual feature information, and motion feature information sequence.

[0083] In some embodiments, the interactive motion feature information extraction module 630 is further configured to: obtain a first motion feature of the target speech from a first motion feature library, the first motion feature library storing the correspondence between multiple speech and multiple first motion features; determine motion feature information of the target speech based on the target speech and the first motion feature; obtain a second motion feature corresponding to the interactive speech from a second motion feature library, the second motion feature library storing the correspondence between multiple speech and multiple second motion features; determine motion feature information of the interactive speech based on the interactive speech and the second motion feature; and obtain interactive motion feature information by fusing the motion feature information of the target speech and the motion feature information of the interactive speech.

[0084] In some embodiments, the interactive motion feature information extraction module 630 is further configured to adjust a first motion feature based on style feature information indicating the speaking style; and to determine motion feature information of the target speech based on the target speech and the adjusted first motion feature. The interactive motion feature information extraction module 630 is further configured to adjust a second motion feature based on style feature information; and to determine motion feature information of the interactive speech based on the interactive speech and the adjusted second motion feature.

[0085] In some embodiments, the apparatus 600 further includes a style feature information extraction module configured to extract style feature information from a reference video.

[0086] In some embodiments, the feature information generation module 620 is further configured to use a visual encoder to extract reference visual feature information corresponding to the face of the target object from a reference image; obtain a mask image by occluding regions in the reference image that are unrelated to the facial movement of the target object; and use a motion encoder to extract reference motion feature information corresponding to the face of the target object from the mask image.

[0087] In some embodiments, the feature information generation module 620 is further configured to project points related to the facial contour of the target object onto a mask image using a trained 3D facial key point model; and to extract reference motion feature information from the projected mask image using a motion encoder.

[0088] In some embodiments, the motion feature information sequence is determined iteratively, and wherein for a predetermined round in a plurality of iterations, the motion feature information sequence determination module 640 is further configured to generate a reference motion feature information sequence including multiple copies of the reference motion feature information by copying the reference motion feature information; add noise to the reference motion feature information sequence to obtain a noisy reference motion feature information sequence; and perform a denoising operation on the noisy reference motion feature information sequence based on the interactive motion feature information and a portion of the motion feature information sequence determined in the previous round of the predetermined round using a diffusion model to determine the motion feature information sequence.

[0089] In some embodiments, reference motion feature information and a sequence of motion feature information are located in a motion feature latent space, which is determined based on the training of a motion encoder, a visual encoder, and a decoder used for video generation.

[0090] In some embodiments, the motion encoder, visual encoder, and decoder are trained by: acquiring a first sample video, the first sample video including multiple sample images, the multiple sample images including sample objects; for each sample image in the first sample video, encoding the sample image using the motion encoder being trained to obtain sample motion feature information of the sample object in the motion feature latent space; encoding the sample image using the visual encoder being trained to obtain sample visual feature information of the sample object; generating a reconstructed image corresponding to the sample image based on the sample motion feature information and the sample visual feature information using the decoder being trained; and training the motion encoder, visual encoder, and decoder based on a first training objective, the first training objective being configured to reduce or minimize the difference between the sample image and the reconstructed image.

[0091] In some embodiments, interactive motion feature information is extracted by a trained motion extraction model, and the motion feature information sequence is generated by a trained diffusion model, wherein the motion extraction model and the diffusion model are trained by: acquiring a second sample video, the second sample video including sample dialogue speech and multiple sample images; generating a sample motion feature information sequence based on the multiple sample images; extracting sample interactive motion feature information from the sample dialogue speech using the trained motion extraction model; determining a reconstructed motion feature information sequence for the sample dialogue speech using the trained diffusion model, at least based on the sample interactive motion feature information; and training the motion extraction model and the diffusion model based on a second training objective configured to reduce or minimize the difference between the sample motion feature information sequence and the reconstructed motion feature information sequence.

[0092] In some embodiments, the target speech and interactive speech are captured in real time or predetermined.

[0093] The units and / or modules included in device 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units and / or modules in device 600 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0094] It should be understood that one or more steps in the above methods can be performed by suitable electronic devices or combinations of electronic devices. Such electronic devices or combinations of electronic devices may include, for example, […]. Figure 1 Electronic device 110.

[0095] Figure 7 A block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 7 The electronic device 700 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 7 The electronic device 700 shown can be used to achieve Figure 1 Electronic devices 110 or Figure 6 Device 600.

[0096] like Figure 7 As shown, electronic device 700 is in the form of a general-purpose electronic device. Components of electronic device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processing unit 710 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 700.

[0097] Electronic device 700 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 700.

[0098] Electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 7 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 720 may include computer program product 725 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0099] The communication unit 740 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 700 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0100] Input device 750 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 760 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 700 can also communicate with one or more external devices (not shown) via communication unit 740 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 700, or with any device that enables electronic device 700 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0101] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0102] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0103] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0104] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0105] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some, as newer, implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0106] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A video generation method, comprising: Acquire a reference image and dialogue voice, wherein the reference image includes a target object, and the dialogue voice includes target voice corresponding to the target object and interactive voice for interacting with the target object; Based on the reference image, generate reference motion feature information and reference visual feature information corresponding to the face of the target object; Extract the interactive motion feature information of the dialogue speech; At least based on the interactive motion feature information, determine the motion feature information sequence corresponding to the dialogue speech; as well as The target video is generated based on the reference motion feature information, the reference visual feature information, and the motion feature information sequence.

2. The method according to claim 1, wherein extracting the interactive motion feature information includes: The first motion feature of the target speech is obtained from the first motion feature library, wherein the first motion feature library stores the correspondence between multiple speech and multiple first motion features; Based on the target speech and the first motion feature, determine the motion feature information of the target speech; The second motion feature corresponding to the interactive voice is obtained from the second motion feature library, which stores the correspondence between the multiple voices and the multiple second motion features; Based on the interactive voice and the second motion feature, determine the motion feature information of the interactive voice; as well as The interactive motion feature information is obtained by fusing the motion feature information of the target speech and the motion feature information of the interactive speech.

3. The method according to claim 2, wherein determining the motion feature information of the target speech includes: The first motion feature is adjusted based on the style feature information of the speaking style. as well as Based on the target speech and the adjusted first motion feature, determine the motion feature information of the target speech; and The motion feature information for determining the interactive voice includes: The second motion feature is adjusted based on the style feature information; as well as Based on the interactive speech and the adjusted second motion features, the motion feature information of the interactive speech is determined.

4. The method according to claim 3, further comprising: The style feature information is extracted from the reference video.

5. The method according to claim 1, wherein generating reference motion feature information and reference visual feature information corresponding to the face of the target object based on the reference video includes: Using a visual encoder, facial reference visual feature information of the target object is extracted from the reference image; A mask image is obtained by occluding regions in the reference image that are unrelated to the facial movements of the target object; as well as Using a motion encoder, the facial reference motion feature information of the target object is extracted from the mask image.

6. The method according to claim 5, wherein extracting reference motion feature information corresponding to the face of the target object from the mask image includes: The trained 3D facial key point model is used to project points related to the facial contour of the target object onto the mask image; as well as The motion encoder is used to extract the reference motion feature information from the projected mask image.

7. The method of claim 1, wherein the motion feature information sequence is determined iteratively, and wherein determining the motion feature information sequence for a predetermined round among a plurality of iteration rounds comprises: By copying the reference motion feature information, a reference motion feature information sequence including multiple copies of the reference motion feature information is generated; Noise is added to the reference motion feature information sequence to obtain a noisy reference motion feature information sequence; Using a diffusion model, based on the interactive motion feature information and some motion features of the motion feature information sequence determined in the previous round of the predetermined round, a denoising operation is performed on the noisy reference motion feature information sequence to determine the motion feature information sequence.

8. The method according to claim 1, wherein the reference motion feature information and the sequence of motion feature information are located in a motion feature latent space, the motion feature latent space being determined based on the training of a motion encoder, a visual encoder, and a decoder for video generation.

9. The method of claim 8, wherein the motion encoder, the visual encoder, and the decoder are trained in the following manner: Acquire a first sample video, which includes multiple sample images, and the multiple sample images include sample objects; For each sample image in the first sample video, The sample image is encoded using a motion encoder that is being trained to obtain the sample motion feature information of the sample object in the motion feature latent space; The sample image is encoded using a visual encoder that is being trained to obtain the sample visual feature information of the sample object; Using the decoder that is being trained, a reconstructed image corresponding to the sample image is generated based on the sample motion feature information and the sample visual feature information; The motion encoder, the visual encoder, and the decoder are trained based on a first training objective, which is configured to reduce or minimize the difference between the sample image and the reconstructed image.

10. The method of claim 1, wherein the interactive motion feature information is extracted by a trained motion extraction model, the motion feature information sequence is generated by a trained diffusion model, and wherein the motion extraction model and the diffusion model are trained by: Acquire a second sample video, which includes sample dialogue audio and multiple sample images; Based on the multiple sample images, a sequence of sample motion feature information is generated; Using a motion extraction model that is being trained, sample interaction motion feature information is extracted from the sample dialogue speech; Using the diffusion model that is being trained, at least based on the sample interaction motion feature information, determine the sequence of reconstructed motion feature information for the sample dialogue speech; as well as The motion extraction model and the diffusion model are trained based on a second training objective, which is configured to reduce or minimize the difference between the sample motion feature information sequence and the reconstructed motion feature information sequence.

11. The method according to claim 1, wherein the target speech and the interactive speech are acquired in real time or predetermined.

12. An apparatus for video generation, comprising: The input acquisition module is configured to acquire a reference image and dialogue speech, wherein the reference image includes a target object, and the dialogue speech includes target speech corresponding to the target object and interactive speech for interacting with the target object; The feature information generation module is configured to generate reference motion feature information and reference visual feature information corresponding to the face of the target object based on the reference video; The interactive motion feature information extraction module is configured to extract the interactive motion feature information of the dialogue speech; The motion feature information sequence determination module is configured to determine the motion feature information sequence corresponding to the dialogue speech based at least on the interactive motion feature information; as well as The target video generation module is configured to generate a target video based on the reference motion feature information, the reference visual feature information, and the motion feature information sequence.

13. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the device to perform the method according to any one of claims 1 to 11 when executed by the at least one processing unit.

14. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images