Video generation method, apparatus, device, medium, and product

Abstract

Disclosed in embodiments of the present disclosure are a video generation method, an apparatus, a device, a medium, and a product. The method comprises: first acquiring a reference image, audio driving information and a historical image corresponding to the audio driving information; performing image feature prediction processing according to image features of the reference image and image features of the historical image to obtain predicted image features corresponding to the audio driving information, so that the predicted image features can represent image features matching the audio driving information; performing image generation processing according to the reference image and the predicted image features to obtain a generated image corresponding to the audio driving information, so that the generated image can better represent an image matching the audio driving information; and finally, generating a video according to the generated image and the historical image, so that the corresponding time of the generated image in the video is later than the corresponding time of the historical image in the video.

Description

Video generation method, device, equipment, medium, product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed on March 26, 2024, with application number 202410353871.5 and invention name “A video generation method, device, equipment, medium, product”. The entire contents of the application are incorporated by reference into this application. Technical Field

[0003] The embodiments of the present disclosure relate to the field of data processing technology, and in particular to a video generation method, apparatus, device, medium, and product. Background Art

[0004] For some application scenarios, such as video generation scenarios, the scenario has the following requirements: based on a reference image and a piece of audio, generate an image sequence corresponding to the audio, such as a video, so that some appearance features of the objects in the image sequence are consistent with the appearance features described by the reference image. Summary of the Invention

[0005] The embodiments of the present disclosure provide a video generation method, apparatus, device, medium, and product.

[0006] In order to achieve the above objectives, the technical solutions provided by the embodiments of the present disclosure are as follows:

[0007] The present disclosure provides a video generation method, the method comprising:

[0008] Acquire a reference image, audio driving information, and a historical image corresponding to the audio driving information;

[0009] Performing image feature prediction processing based on the image features of the reference image and the image features of the historical image to obtain predicted image features corresponding to the audio driving information;

[0010] Performing image generation processing based on the reference image and the predicted image features to obtain a generated image corresponding to the audio driving information; the audio features of the audio driving information are used to participate in the image generation processing or the image feature prediction processing;

[0011] A video is generated based on the generated image and the historical image, wherein the time corresponding to the generated image in the video is later than the time corresponding to the historical image in the video.

[0012] In one possible implementation, when the audio feature is used to participate in the image generation process, the process of determining the generated image includes:

[0013] performing a first encoding process on the reference image to obtain a first image feature, and performing a second encoding process on the reference image to obtain a second image feature; the first image feature is used to represent local information of pixels in the reference image; and the second image feature is used to represent global information of the reference image;

[0014] Obtaining a reference feature based on the first image feature and the second image feature;

[0015] An image generation process is performed based on the reference features, the predicted image features, and the audio features of the audio driving information to obtain a generated image corresponding to the audio driving information.

[0016] In one possible implementation, the predicted image features are determined using a time series prediction module in a data processing model;

[0017] The generated image is determined by using the image generation module in the data processing model.

[0018] In one possible implementation, when the audio features of the audio driving information are used to participate in the image feature prediction process, the training process of the data processing model includes the following two training stages performed sequentially:

[0019] Training an image generation module in the data processing model;

[0020] A first part of the data processing model is trained, where the first part includes the time series prediction module and a part of the image generation module that is related to output data of the time series prediction module.

[0021] In one possible implementation, when the audio features of the audio driving information are used to participate in the image generation process, the training process of the data processing model includes the following three training stages performed in sequence:

[0022] Training an image generation module in the data processing model;

[0023] Training a first portion of the data processing model, the first portion including the time series prediction module and a portion of the image generation module related to output data of the time series prediction module;

[0024] The second part of the data processing model is trained; the second part refers to the part of the data processing model related to the audio feature.

[0025] In a possible implementation manner, the image features of the reference image and the image features of the historical image are both obtained through a first encoding process.

[0026] In a possible implementation, the reference image is determined based on a pre-built virtual image;

[0027] The audio driving information is determined based on the i-th audio frame in the audio sequence; the audio sequence is converted from text data; i is a positive integer, 2≤i≤the total number of audio frames in the audio sequence;

[0028] The generated image corresponding to the audio driving information refers to the i-th frame image in the video;

[0029] The video is used to describe the changes in facial states of the virtual image under the audio sequence.

[0030] In a possible implementation manner, the first frame image in the video is the reference image; or, the first frame image in the video is generated based on the reference image and the first frame audio in an audio sequence.

[0031] The present disclosure provides a video generation device, including:

[0032] an acquiring unit, configured to acquire a reference image, audio driving information, and a historical image corresponding to the audio driving information;

[0033] a prediction unit, configured to perform image feature prediction processing based on the image features of the reference image and the image features of the historical image to obtain predicted image features corresponding to the audio driving information;

[0034] A generation unit is used to perform image generation processing based on the reference image and the predicted image features to obtain a generated image corresponding to the audio driving information; the audio features of the audio driving information are used to participate in the image generation processing or the image feature prediction processing.

[0035] An embodiment of the present disclosure provides an electronic device, the device comprising: a processor and a memory;

[0036] The memory is used to store instructions or computer programs;

[0037] The processor is configured to execute the instructions or computer program in the memory, so that the electronic device executes the video generation method provided by the embodiment of the present disclosure.

[0038] An embodiment of the present disclosure provides a computer-readable medium, wherein the computer-readable medium stores instructions or a computer program. When the instructions or the computer program are executed on a device, the device executes the video generation method provided by the embodiment of the present disclosure.

[0039] An embodiment of the present disclosure provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the video generation method provided by the embodiment of the present disclosure.

[0040] Embodiments of the present disclosure Embodiments of the present disclosure BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] FIG1 is a flow chart of a video generation method provided by an embodiment of the present disclosure;

[0043] FIG2 is a schematic diagram of a data processing flow provided by an embodiment of the present disclosure;

[0044] FIG3 is a schematic diagram of another data processing flow provided by an embodiment of the present disclosure;

[0045] FIG4 is a schematic structural diagram of a video generating device provided by an embodiment of the present disclosure;

[0046] FIG5 is a schematic structural diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] In order to enable those skilled in the art to better understand the embodiments of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of them. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the embodiments of the present disclosure.

[0048] To better understand the technical solutions provided by the embodiments of the present disclosure, the video generation method provided by the embodiments of the present disclosure is described below with reference to some accompanying drawings. As shown in FIG1 , the video generation method provided by the embodiments of the present disclosure includes steps S1-S4 below. FIG1 is a flow chart of a video generation method provided by the embodiments of the present disclosure.

[0049] S1: Acquire a reference image, audio driving information, and a historical image corresponding to the audio driving information.

[0050] The reference image is used to provide some characteristics of the object during the image generation process, such as characteristics such as appearance that do not change frequently over time. It should be noted that the embodiments of this disclosure do not limit the implementation of the object. For example, it can be implemented as a face, animal, object, or virtual image.

[0051] In addition, the embodiments of the present disclosure do not limit the implementation of the reference image. For ease of understanding, the following description is combined with three cases.

[0052] Case 1: For some scenarios, such as model training scenarios, the reference image mentioned above may refer to a frame of video image randomly extracted from a sample video. The sample video refers to the video required for use in the model training process; and the embodiment of the present disclosure does not limit the method for obtaining the sample video. For example, it can be implemented using any existing or future sample video acquisition method. It can be seen that in one possible implementation, when the video generation method provided by the embodiment of the present disclosure is applied to the model training process, the reference image may be an image extracted from a sample video.

[0053] Case 2: For some scenarios, such as performing a video generation task, the reference image mentioned above may refer to an image input by the user in some way, such as an image captured by a camera, an image manually uploaded, an image selected through a selection operation, etc. Therefore, in one possible implementation, when the video generation method provided in the embodiments of the present disclosure is applied to a video generation task, the reference image may refer to an image provided by the user.

[0054] Case 3, for some scenarios, such as performing a video generation task based on a virtual image, the above-mentioned reference image can be determined based on a pre-constructed virtual image, so that the reference image can express some characteristics of the virtual image, such as hairstyle, clothes and other appearance characteristics. It should be noted that the embodiment of the present disclosure does not limit the determination process of the reference image. For example, it can be implemented by any existing or future method that can capture images for virtual images, such as taking photos. In addition, the embodiment of the present disclosure does not limit the timing of capturing the reference image. For example, the reference image can be captured when the virtual image is in an expressionless state. In addition, the embodiment of the present disclosure does not limit the implementation method of the virtual image. For example, it can be implemented by a virtual image in a three-dimensional space or a virtual image in a two-dimensional space.

[0055] The audio driving information is used to provide some constraints in the image generation process, such as constraints on facial expressions such as mouth shape; and the embodiments of the present disclosure do not limit the implementation method of the audio driving information. For example, it may include the i-th frame of audio in an audio sequence, so that the audio driving information may refer to the audio required for image generation processing for the i-th frame of audio in the audio sequence, so that the generated image obtained based on the audio driving information is used to represent the image generation result for the i-th frame of audio, so that the generated image can represent an image adapted to the i-th frame of audio. The i-th frame of audio refers to the audio in the i-th arrangement position in the audio sequence. i is a positive integer, and 2≤i≤the total number of audio frames in the audio sequence.

[0056] In addition, in order to better improve the image generation effect, the embodiment of the present disclosure also provides a possible implementation method of the above-mentioned audio driving information. Under this implementation method, the audio driving information not only includes the i-th frame audio in an audio sequence, but also includes the audio that exists in the audio sequence and is relatively close to the i-th frame audio, so that these "audio that is relatively close to the i-th frame audio" can represent the context information of the i-th frame audio, thereby enabling the audio driving information to more accurately represent the semantic information described by the i-th frame audio, thereby enabling the generated image obtained based on the audio driving information to be more adapted to the i-th frame audio, which is conducive to improving the image generation effect. It should be noted that the embodiment of the present disclosure does not limit the implementation method of the "audio that is relatively close to the i-th frame audio". For example, the absolute value of the difference between the position of the "audio that is relatively close to the i-th frame audio" in the audio sequence and the position of the i-th frame audio in the audio sequence is less than a preset difference, such as 5. It can be seen that in one possible implementation, the audio driving information may include multiple frames of audio, and the absolute value of the difference between the position of each audio frame in the audio driving information and the position of the i-th audio frame in the audio sequence is less than a preset difference. Based on this, it can be seen that in one possible implementation, the audio driving information can be implemented using an audio segment, such as audio segment i shown in Figure 2 or Figure 3.

[0057] In addition, the embodiments of the present disclosure do not limit the implementation methods of the audio sequences involved in the above two sections. For ease of understanding, the following description is combined with three cases.

[0058] Case 1: For some scenarios, such as model training scenarios, when the reference image is extracted from a sample video, the above audio sequence can be extracted from the sample video, so that the audio sequence can represent part or all of the audio information carried by the sample video, so that subsequent model training processing can be performed based on the audio sequence.

[0059] Case 2: For some scenarios, such as performing a video generation task, the audio sequence mentioned above may refer to audio input by the user in some way, such as audio captured by a microphone, manually uploaded audio, or audio selected through a selection operation. Therefore, in one possible implementation, when the video generation method provided in the embodiments of the present disclosure is applied to a video generation task, the audio sequence may refer to audio provided by the user.

[0060] Case 3, for some scenarios, such as performing a video generation task based on a virtual image, the above audio sequence may refer to automatically generated audio, such as audio generated based on text data. The text data is used to describe the constraint information required for reference when performing image generation processing, such as expression constraints, etc.; and the embodiment of the present disclosure does not limit the method of obtaining the text data. For example, the text data may refer to the text content input by the user. For example, the text data may refer to the text obtained and output by a model by processing the user input content. The user input content refers to the content provided by the user to the model, such as text, audio, video, image, etc. The model is used to convert the user input content into text. In addition, the embodiment of the present disclosure does not limit the method of generating audio. For example, the audio can be obtained by any existing or future method that can convert text into audio, such as Text To Speech (TTS).

[0061] The historical image corresponding to the audio driving information is used to describe the historical information required to perform image generation processing based on the audio driving information, so that the historical image can represent the changing trends of some states, thereby enabling the historical image to provide some timing constraints to the image generation processing, such as constraints on facial expressions, etc., so that the state presented by the next frame of the image, such as the expression state, can be predicted based on the historical image.

[0062] In addition, the embodiments of the present disclosure do not limit the implementation of the historical images corresponding to the above audio driving information. For ease of understanding, some application scenarios are described below.

[0063] Case 1: For some scenarios, such as model training, when the audio driving information mentioned above refers to the audio required for image generation processing for the i-th audio frame in an audio sequence, and the audio sequence is extracted from a sample video, the historical images corresponding to the audio driving information can refer to video images extracted from the sample video, and the times corresponding to these extracted video images in the sample video are all earlier than the times corresponding to the i-th audio frame in the sample video. Therefore, if the i-th audio frame corresponds to the i-th video frame in the sample video, the historical images corresponding to the audio driving information can include the i-1-th video frame, the i-2-th video frame, ..., and the iT-th video frame in the sample video. Where T is a positive integer and can be set based on the actual application scenario. It should be noted that for any image, the time corresponding to the image in the sample video can be used to represent the temporal state of the image in the sample video. Furthermore, the present disclosure does not limit the implementation of the time corresponding to the image in the sample video; for example, it can be implemented using an arrangement position.

[0064] Case 2: For some scenarios, such as the execution of a video generation task, when the audio driving information mentioned above refers to the audio required for image generation processing for the i-th frame of audio in an audio sequence, the historical image corresponding to the audio driving information may include the image generation result corresponding to at least one frame of audio in the audio sequence, and the time corresponding to each audio frame in the at least one frame of audio in the audio sequence is earlier than the time corresponding to the i-th frame of audio in the audio sequence. Thus, in one possible implementation, when the audio driving information mentioned above includes the i-th frame of audio in the audio sequence, and the generated image corresponding to the audio driving information refers to the image generation result corresponding to the i-th frame of audio, the historical image corresponding to the audio driving information may include the image generation result corresponding to the i-1-th frame of audio in the audio sequence, the image generation result corresponding to the i-2-th frame of audio, ..., and the image generation result corresponding to the iT-th frame of audio in the audio sequence. For any frame of audio, the image generation result corresponding to the audio refers to the image generated for that audio and adapted to the audio. It should be noted that, for any image, the time corresponding to the image in the audio sequence can be used to characterize the temporal state of the image in the audio sequence; and the embodiments of the present disclosure do not limit the implementation method of the time corresponding to the image in the audio sequence, for example, it can be implemented using the arrangement position.

[0065] It should be noted that, for any video generation task, the embodiments of the present disclosure do not limit the method for obtaining the image generation result corresponding to the first frame of audio in the audio sequence involved in the task. For example, the image generation result corresponding to the first frame of audio can be determined with the help of any existing or future video generation method. For another example, the image generation result corresponding to the first frame of audio can also be determined with the help of the image generation process described by any one of the implementation methods S1-S3 in the embodiments of the present disclosure, and the historical images involved in the image generation process corresponding to the first frame of audio can be implemented using reference images or some images uploaded by users.

[0066] Based on the relevant content of S1 above, it can be seen that for any frame of audio in an audio segment, if you want to generate an image that matches this frame of audio, you not only need to obtain the audio driving information including this frame of audio, so that the audio driving information can express the semantics described by this frame of audio as accurately as possible, but also need to obtain a reference image and one or more historical images corresponding to this frame of audio, so that the reference image can provide some object states required for the image generation process, such as appearance, etc., and enable these historical images to provide some historical states required for reference in the image generation process, such as facial expressions and other states.

[0067] S2: Perform image feature prediction processing based on the image features of the reference image and the image features of the historical image corresponding to the audio driving information to obtain the predicted image features corresponding to the audio driving information.

[0068] The image feature of the reference image is used to characterize the image information carried by the reference image. For example, the image feature of the reference image can be implemented using feature 3 shown in FIG. 2 or FIG. 3 .

[0069] In addition, the embodiment of the present disclosure does not limit the method for obtaining the image features of the reference image. For example, it can be implemented using any existing or future image coding method.

[0070] For another example, in order to better improve the image generation effect, the image features of the above reference image can be obtained through a first encoding process, so that the image features can better represent the image information carried by the reference image, such as pixel-level information. It can be seen that, under one possible implementation, the image features of the reference image can be used to characterize the local information of the pixels in the reference image. It should be noted that the embodiment of the present disclosure does not limit the implementation method of the first encoding process. For example, the first encoding process can be implemented using a variational autoencoder (VAE), such as encoding 1 shown in Figure 2 or Figure 3.

[0071] For the historical images corresponding to the audio-driven information above, if the number of historical images is T, the image features of the t-th historical image are used to characterize the image information carried by the t-th historical image, t is a positive integer, t≤T, T represents the number of images in the historical images corresponding to the audio-driven information, so that the image features of the historical images corresponding to the audio-driven information, such as feature 2 shown in Figure 2 or Figure 3, can represent the information carried by these historical images, so that the "image features of the historical images corresponding to the audio-driven information" can more accurately represent the changing trends of some states, which is conducive to improving the prediction effect of image features with time series, thereby improving the image generation effect.

[0072] In addition, the embodiment of the present disclosure does not limit the method for obtaining the image features of the t-th historical image. For example, it can be implemented using any existing or future image coding method. For another example, in order to better improve the image generation effect, the image features of the t-th historical image above can be obtained through a first encoding process, so that the image features can better represent the image information carried by the t-th historical image, such as pixel-level information. It can be seen that under one possible implementation method, the image features of the t-th historical image can be determined using VAE, so that the image features of the t-th historical image can be used to characterize the local information of the pixels in the t-th historical image.

[0073] The predicted image features corresponding to the audio driving information are used to characterize certain characteristics of the image adapted to the audio driving information, such as the facial expression state. Thus, when the audio driving information refers to the audio required for image generation processing based on the i-th audio frame in an audio sequence, the predicted image features corresponding to the audio driving information can be used to characterize certain characteristics of the image adapted to the i-th audio frame, such as the facial expression state.

[0074] In addition, for the predicted image features corresponding to the audio-driven information, such as feature 4 shown in Figure 2 or Figure 3, the predicted image features can be determined based on the image features of the reference image and the image features of the historical image corresponding to the audio-driven information, so that the predicted image features meet some constraints provided by the reference image, such as object appearance constraints, etc., and the predicted image features also meet some constraints provided by the historical image, such as some state change trend constraints, etc., so that the predicted image features can better represent some image features adapted to the audio-driven information, such as facial state, appearance and other features.

[0075] In addition, in order to better improve the image generation effect, the embodiment of the present disclosure also provides a possible implementation method of the image feature prediction processing in S2 above, under which the audio features of the audio driving information above are used to participate in the image feature prediction processing. It can be seen that in one possible implementation method, S2 can specifically be: performing image feature prediction processing based on the image features of the reference image, the image features of the historical image corresponding to the audio driving information, and the audio features of the audio driving information to obtain the predicted image features corresponding to the audio driving information, so that the predicted image features not only meet some constraints provided by the reference image and the historical image, but also meet some constraints provided by the audio driving information, such as constraints on expression state, so that the predicted image features are more adapted to the audio driving information, thereby helping to improve the image generation effect.

[0076] For the audio features of the audio driving information in the previous paragraph, such as feature 1 shown in Figure 2 or Figure 3, the audio features are used to describe the audio information carried by the audio driving information, such as the semantic information and contextual information of the i-th frame audio; and the embodiments of the present disclosure do not limit the method of obtaining the audio features. For example, it can adopt any existing or future method that can perform feature extraction and processing on audio, such as with the help of the WavLM model (WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing). It can be seen that in one possible implementation, the audio features of the audio driving information may refer to the audio information obtained by processing the audio driving information using the WavLM model.

[0077] In addition, the disclosed embodiments do not limit the method for obtaining the predicted image features corresponding to the above audio-driven information. For example, the predicted image features can be determined by using the timing prediction module in the data processing model. It can be seen that, under one possible implementation, S2 above can specifically be: inputting the image features of the reference image and the image features of the historical image corresponding to the audio-driven information into the timing prediction module, so that the timing prediction module can perform timing prediction processing based on these image features, and obtain and output the predicted image features corresponding to the audio-driven information. In addition, under another possible implementation, S2 can specifically be: inputting the image features of the reference image, the image features of the historical image, and the audio features of the audio-driven information into the timing prediction module, so that the timing prediction module can perform timing prediction processing based on these features, and obtain and output the predicted image features corresponding to the audio-driven information.

[0078] For the data processing model involved in the previous paragraph, the data processing model is used to generate a new image based on the reference image, audio driving information and the historical image corresponding to the audio driving information; and the embodiment of the present disclosure does not limit the working principle of the data processing model. For example, the data processing model can adopt the working principle shown in Figure 2 or Figure 3.

[0079] In addition, the embodiments of the present disclosure do not limit the implementation of the above data processing model. For example, the data processing model may include at least a time series prediction module and an image generation module, and the input data of the image generation module includes the output data of the time series prediction module. The time series prediction module is used to perform time series prediction processing on the input data of the time series prediction module, such as predicting the current frame information based on the change trend described by the historical frame information; and the embodiments of the present disclosure do not limit the implementation of the time series prediction module. For example, the time series prediction module can be implemented using the attention mechanism. The image generation module is used to perform image generation processing based on the input data of the image generation module; and the embodiments of the present disclosure do not limit the implementation of the image generation module. For example, it can be implemented using the image generation mechanism used by the diffusion model.

[0080] Based on the relevant content of S2 above, it can be known that after obtaining the reference image, audio driving information, and the historical image corresponding to the audio driving information, the reference image, the audio driving information, and the historical image can be input into the data processing model, so that the timing prediction module in the data processing model can perform image feature prediction processing based on the image features of the reference image and the image features of the historical image to obtain the predicted image features corresponding to the audio driving information, such as feature 4 shown in Figure 2; or, the timing prediction module can perform image feature prediction processing based on the image features of the reference image, the image features of the historical image, and the audio features of the audio driving information to obtain the predicted image features corresponding to the audio driving information, such as feature 4 shown in Figure 3, so that other modules in the data processing model can subsequently perform image generation processing based on the predicted image features.

[0081] S3: Perform image generation processing based on the reference image and the predicted image features corresponding to the audio driving information to obtain a generated image corresponding to the audio driving information; the audio features of the audio driving information are used to participate in image generation processing or image feature prediction processing.

[0082] The generated image corresponding to the audio driving information refers to the result of image generation processing based on the audio driving information. It can be seen that when the audio driving information refers to the audio required as a basis for image generation processing for the i-th frame of audio in an audio sequence, the generated image corresponding to the audio driving information can refer to the image generation result corresponding to the i-th frame of audio, so that the generated image can represent an image adapted to the i-th frame of audio, so that a video can be subsequently constructed based on the generated image and the i-th frame of audio, so that the video can include the generated image, the i-th frame of audio, and the corresponding relationship between the generated image and the i-th frame of audio. i is a positive integer, and 2≤i≤the total number of audio frames in the audio sequence.

[0083] In addition, the embodiments of the present disclosure do not limit the method for determining the generated image corresponding to the above-mentioned audio driving information. For example, for some application scenarios, such as the application scenario shown in Figure 3, when the audio features of the audio driving information have been used to participate in the image feature prediction processing, the determination process of the generated image corresponding to the audio driving information may specifically include the following steps 11 to 14.

[0084] Step 11: Perform a first encoding process on the reference image to obtain a first image feature; the first image feature is used to represent local information of pixels in the reference image.

[0085] Among them, the first encoding process is used to extract some non-high-level image information from an image, such as local information of each pixel, etc.; and the embodiment of the present disclosure does not limit the implementation method of the first encoding process, for example, it can be implemented using VAE.

[0086] The first image feature refers to the result obtained by performing a first encoding process on the reference image, so that the first image feature is used to represent some non-high-level image information in the reference image, such as local information of each pixel, etc., so that the first image feature can better represent the local information of the pixels in the reference image.

[0087] Step 12: Perform a second encoding process on the reference image to obtain a second image feature; the second image feature is used to represent the global information of the reference image.

[0088] Among them, the second encoding process is used to extract some high-level image information from an image, such as the global information of the image; and the embodiment of the present disclosure does not limit the implementation method of the second encoding process. For example, it can be implemented using a contrastive language-image pre-training (CLIP) encoder, also known as a CLIP Encoder.

[0089] The second image feature refers to the result obtained by performing a second encoding process on the reference image, so that the second image feature is used to represent some high-level image information in the reference image, such as image global information, etc., so that the second image feature can better represent the global information of the reference image, and then the combination of the second image feature and the first image feature above can more comprehensively represent the image information carried by the reference image.

[0090] It should be noted that the embodiment of the present disclosure does not limit the correlation between the execution time of step 12 and the execution time of step 11. For example, the former may be earlier than the latter. Another example is that the latter is earlier than the former. Another example is that the two are the same.

[0091] Step 13: Obtain a reference feature based on the first image feature and the second image feature.

[0092] Among them, the reference features are used to comprehensively describe the information carried by the reference image, such as global information at the image level and local information at the pixel level.

[0093] In addition, the embodiment of the present disclosure does not limit the implementation of the above step 13. For example, it can specifically be: directly fusing the first image feature and the second image feature to obtain a reference feature, so that the reference feature can represent the fusion result between the first image feature and the second image feature. It should be noted that the embodiment of the present disclosure does not limit the implementation of the fusion process. For example, it can be implemented using a concatenation method, such as the concatenation method represented by the "+" in Figure 3.

[0094] In fact, for some application scenarios, such as implementing image generation processing with the help of a diffusion generator, in order to ensure the image generation effect, the embodiment of the present disclosure also provides some possible implementation methods of the above step 13, which are explained below with reference to two examples.

[0095] Example 1. In one possible implementation, step 13 above may specifically include: first fusing the first image feature and the second image feature to obtain a fused feature; then using a multilayer perceptron (MLP) to process the fused feature to obtain a reference feature, so that the reference feature satisfies the input constraints of the generator, such as size constraints, to avoid defects caused by the reference feature not satisfying the constraints, thereby improving the image generation effect.

[0096] Example 2. In one possible implementation, step 13 above may specifically include: first using MLP to process the first image feature to obtain a first processed feature, so that the first processed feature satisfies the input constraint of the generator, and using MLP to process the second image feature to obtain a second processed feature, so that the second processed feature also satisfies the input constraint of the generator; then, fusing the first processed feature and the second processed feature to obtain a reference feature, so that the reference feature also satisfies the input constraint of the generator. This can avoid defects caused by the reference feature not satisfying the constraint, thereby helping to improve the image generation effect.

[0097] Step 14: Perform image generation processing based on the above reference features and the predicted image features corresponding to the audio driving information to obtain a generated image corresponding to the audio driving information.

[0098] It should be noted that the embodiments of the present disclosure do not limit the implementation of the above step 14. For ease of understanding, the following description is provided with reference to examples.

[0099] As an example, in one possible implementation, step 14 above can be implemented with the aid of an image generation module in a data processing model. For the relevant content of the image generation module, please refer to the above. In addition, in one possible implementation, the image generation module may include a generator and a decoder, and the input data of the decoder includes the output data of the generator. It should be noted that the embodiment of the present disclosure does not limit the implementation of the generator. For example, the generator can be implemented using a diffusion generator, such as the generator shown in Figure 3. In addition, the embodiment of the present disclosure does not limit the implementation of the decoder. For example, the decoder can be implemented using a VAE decoder.

[0100] Based on the content of the previous paragraph, it can be seen that in one possible implementation method, the above step 14 can be specifically as follows: first use MLP to process the predicted image features corresponding to the above audio-driven information to obtain the processed predicted features, so that the processed predicted features meet the input constraints of the generator; then the generator in the image generation module uses the reference features and the processed predicted features as guiding conditions to denoise the noise data, obtain and output denoised features (denoised Token), such as feature 5 shown in Figure 3, so that the denoised features can better represent what information is carried in the image adapted to the audio-driven information; then, use the decoder in the image generation module to decode the denoised features to obtain the generated image corresponding to the audio-driven information, such as the generated image shown in Figure 3, so that the generated image can be used as an image adapted to the audio-driven information.

[0101] It should be noted that the embodiments of the present disclosure do not limit the working principle of the generator in the above paragraph. For example, it can be specifically: the above noise data is used as the input data of the generator; and the above reference features and the processed prediction features are used as guiding conditions of the generator, so that the reference features and the processed prediction features can participate in the denoising process of each Unet in the generator through the cross attention mechanism.

[0102] It should also be noted that for the noise data in the above two paragraphs, the noise data refers to the noise required for image generation processing based on audio-driven information; and the embodiments of the present disclosure do not limit the noise data. For example, for some scenarios, such as model training scenarios, if the audio-driven information refers to the audio used for image generation processing based on the i-th frame of audio in the sample video, then the process of obtaining the above noise data can be specifically as follows: first extract the video image corresponding to the i-th frame of audio from the sample video, such as the i-th frame of video image; then perform multiple rounds of noise addition processing on the i-th frame of video image to obtain the noise data. For another example, for some scenarios, such as the execution scenario of a certain video generation task, the above noise data can refer to randomly generated Gaussian noise.

[0103] Based on the relevant content of steps 11 to 14 above, it can be seen that in one possible implementation, after the predicted image features corresponding to the audio-driven information are determined based on the audio features of the audio-driven information, image generation processing can be performed based on the reference image and the predicted image features corresponding to the audio-driven information to obtain a generated image corresponding to the audio-driven information. In particular, because the predicted image features satisfy the constraints described by the audio-driven information, the generated image obtained based on the predicted image features also satisfies the constraints described by the audio-driven information, thereby improving the image generation effect.

[0104] In fact, in some application scenarios, such as the one shown in Figure 2, to better improve the image generation effect, the audio features of the audio-driven information can be used to participate in the image generation process. Based on this, the embodiment of the present disclosure also provides a possible implementation of the above S3. In this implementation, S3 can specifically be: performing image generation processing based on the reference image, the predicted image features corresponding to the audio-driven information, and the audio features of the audio-driven information, to obtain a generated image corresponding to the audio-driven information, such as the generated image shown in Figure 2.

[0105] In addition, the embodiment of the present disclosure does not limit the determination process of the generated image in the above paragraph. For example, in one possible implementation, the determination process of the generated image corresponding to the above audio driving information may specifically include the following steps 21 to 24.

[0106] Step 21: Perform a first encoding process on the reference image to obtain a first image feature; the first image feature is used to represent local information of pixels in the reference image.

[0107] It should be noted that for the relevant content of step 21, please refer to the relevant content of step 11 above.

[0108] Step 22: Perform a second encoding process on the reference image to obtain a second image feature; the second image feature is used to represent the global information of the reference image.

[0109] It should be noted that for the relevant content of step 22, please refer to the relevant content of step 12 above.

[0110] Step 23: Obtain a reference feature based on the first image feature and the second image feature.

[0111] It should be noted that for the relevant content of step 23, please refer to the relevant content of step 13 above.

[0112] Step 24: Perform image generation processing based on the above reference features, the predicted image features corresponding to the audio driving information, and the audio features of the audio driving information to obtain a generated image corresponding to the audio driving information.

[0113] It should be noted that the embodiment of the present disclosure does not limit the implementation of the above step 24. For example, the step 24 can be implemented with the help of the image generation module in the data processing model. The relevant content of the image generation module can be found above.

[0114] It can be seen that in one possible implementation, when the above image generation module can include a generator and a decoder, the above step 24 can specifically be: first, using an MLP to process the predicted image features corresponding to the above audio driving information to obtain processed predicted features, so that the processed predicted features meet the input constraints of the generator, and using an MLP to process the audio features of the audio driving information to obtain processed audio features, so that the processed audio features also meet the input constraints of the generator; then, the generator in the image generation module uses the reference features, the processed predicted features, and the processed audio features as guidance conditions to denoise the noise data, obtain and output a denoised feature (denoised token), such as feature 5 shown in Figure 2, so that the denoised feature can better represent the information carried in the image adapted to the audio driving information; then, using the decoder in the image generation module to decode the denoised feature to obtain a generated image corresponding to the audio driving information, such as the generated image shown in Figure 2, so that the generated image can serve as an image adapted to the audio driving information. For the relevant content of the noise data, please refer to the above.

[0115] It should be noted that the embodiments of the present disclosure do not limit the working principle of the generator in the above paragraph. For example, it can be specifically: the above noise data is used as the input data of the generator; and the above reference features, processed prediction features, and processed audio features are all used as guiding conditions of the generator, so that the reference features, processed prediction features, and processed audio features can all participate in the denoising process of each Unet in the generator through the cross attention mechanism.

[0116] Based on the relevant contents of steps 21 to 24 above, it can be known that in one possible implementation, as shown in Figure 2, when the audio features of the audio-driven information do not participate in the image feature prediction processing, image generation processing can be performed based on the reference image, the predicted image features corresponding to the audio-driven information, and the audio features of the audio-driven information to obtain a generated image corresponding to the audio-driven information, so that the generated image meets the constraints described by the audio-driven information, which is conducive to improving the image generation effect.

[0117] S4: Generate a video based on the generated image corresponding to the audio driving information and the historical image corresponding to the audio driving information, wherein the time corresponding to the generated image in the video is later than the time corresponding to the historical image in the video.

[0118] It should be noted that the embodiments of the present disclosure are not limited to the implementation method of S4 above. For example, it may specifically include: generating a video based on the generated image corresponding to the audio driving information and the historical image corresponding to the audio driving information, so that the video includes the generated image and the historical image, and the time corresponding to the generated image in the video is later than the time corresponding to the historical image in the video.

[0119] Based on the relevant contents of S1 to S4 above, it can be known that for the audio driving information including the i-th frame audio in the audio sequence, a reference image, the audio driving information, and the historical image corresponding to the audio driving information are first obtained; then, image feature prediction processing is performed based on the image features of the reference image and the image features of the historical image to obtain the predicted image features corresponding to the audio driving information, so that the predicted image features can represent the image features adapted to the i-th frame audio, so that the predicted image features can represent some states in the image adapted to the i-th frame audio, such as states that can change frequently with time, such as facial expressions; then, image generation processing is performed based on the reference image and the predicted image features to obtain the generated image corresponding to the audio driving information, so that the generated image can better represent the image adapted to the i-th frame audio; finally, a video is generated based on the generated image and the historical image, so that the time corresponding to the generated image in the video is later than the time corresponding to the historical image in the video, so that automatic video generation can be achieved.

[0120] Among them, if the audio features of the audio driving information are used to participate in the image generation process, the audio features can affect the image generation process, so that the generated image obtained based on the image generation process can meet the constraints described by the audio features, and then make the generated image more adapted to the i-th frame audio, which is conducive to improving the video generation effect.

[0121] In addition, if the audio features of the audio driving information are used to participate in the image feature prediction processing, the audio features can affect the image feature prediction processing, so that the predicted image features obtained based on the image feature prediction processing can meet the constraints described by the audio features, and then the predicted image features can more accurately represent the image features of the image adapted to the i-th frame audio, so that the generated image obtained based on the predicted image features is more adapted to the i-th frame audio, which is conducive to improving the video generation effect.

[0122] In addition, the embodiments of the present disclosure do not limit the execution subject of the video generation method provided in the embodiments of the present disclosure. For example, the video generation method provided in the embodiments of the present disclosure can be applied to a terminal device. For another example, the video generation method provided in the embodiments of the present disclosure can also be implemented with the help of a data interaction process between a terminal device and a server. The terminal device can be a smart phone, a computer, a personal digital assistant (PDA), a tablet computer, etc. The server can be a stand-alone server, a cluster server, or a cloud server.

[0123] In addition, the embodiments of the present disclosure do not limit the application scenarios of the above video generation method. For ease of understanding, three examples are used below for illustration.

[0124] Example 1: In some scenarios, such as the scenario shown in Figure 3, the video generation method provided by the embodiments of the present disclosure can be used to implement a model training process. Based on this, in order to better improve the image generation effect, the embodiments of the present disclosure also provide a possible implementation of the model training process. In this implementation, when the audio features of the above audio driving information participate in the image feature prediction process, the model training process may include the following steps 31 and 32.

[0125] Step 31: Train the image generation module in the data processing model.

[0126] It should be noted that the embodiments of the present disclosure do not limit the implementation of step 31. For example, it can be implemented using any existing or future method that can train an image generation module, such as a diffusion model. For ease of understanding, the following examples are provided for illustration.

[0127] As an example, when the above image generation module includes a diffusion generator, the above step 31 may specifically include the following steps 311 to 314.

[0128] Step 311: extract two frames of video images from the sample video, one frame as the conditional image and the other frame as the label image.

[0129] Among them, the conditional image is used as a guiding condition in the current round of training.

[0130] The label image is used as guidance information for the image generation results during the current round of training.

[0131] Step 312: Perform noise processing on the label image to obtain noise data.

[0132] It should be noted that the embodiment of the present disclosure does not limit the implementation method of the noise addition process in the above step 312. For example, it can be implemented by adopting any existing or future noise addition process method, such as the noise addition module in the diffusion model.

[0133] Step 313: The image generation module in the data processing model performs image generation processing based on the noise data and the conditional image to obtain an image generation result corresponding to the noise data.

[0134] It should be noted that the embodiments of the present disclosure do not limit the implementation of the above step 313. For example, when the image generation module in the data processing model is constructed based on the diffusion model, the step 313 can specifically be: the image generation module uses the image features of the conditional image as a guiding condition to denoise the noise data, and obtain and output the image generation result corresponding to the noise data.

[0135] Step 314: Update the image generation module in the data processing model based on the generated image corresponding to the label image and the noise data, and return to continue executing the above step 311 and subsequent steps until the first stop condition is reached.

[0136] The first stop condition can be set based on the actual application scenario. For example, the first stop condition can include that the module loss of the image generation module is lower than a first loss threshold. In another example, the first stop condition can include that the rate of change of the module loss of the image generation module is lower than a first rate of change threshold. In another example, the first stop condition can include that the number of updates of the image generation module is lower than a first number threshold.

[0137] The module loss of the image generation module is used to characterize the performance of the image generation module. The module loss of the image generation module is calculated based on the difference between the labeled image and the generated image corresponding to the noise data. It should be noted that the embodiments of this disclosure do not limit the loss calculation method; for example, episilon loss can be used for implementation.

[0138] Based on the relevant contents of steps 311 to 314 above, it can be seen that for the image generation module in the data processing model, when the image generation module is constructed based on the diffusion model, the image generation module can be implemented using any existing or future diffusion model training method.

[0139] Based on the relevant content of step 31 above, it can be known that in some application scenarios, such as the scenario shown in Figure 3, for the above data processing model, the image generation module in the data processing model can be trained separately first to make the trained image generation module have better performance, so that the training process for other modules in the data processing model can be implemented subsequently by fixing some or all parameters in the trained image generation module, such as the training process shown in steps 32 to 34 below.

[0140] Step 32: Training the first part of the data processing model, where the first part includes the time series prediction module and the part of the image generation module that is related to the output data of the time series prediction module.

[0141] Among them, the first part refers to the part existing in the above data processing model and related to the predicted image features, such as the part used to generate the predicted image features and the part used to process the predicted image features. For example, under one possible implementation, the first part may include the part of the timing prediction module and the image generation module that is related to the output data of the timing prediction module. Among them, the timing prediction module is used to generate the predicted image features; the "part of the image generation module that is related to the output data of the timing prediction module" is used to implement some processing for the predicted image features; and the embodiment of the present disclosure does not limit the implementation of the "part of the image generation module that is related to the output data of the timing prediction module", for example, the "part of the image generation module that is related to the output data of the timing prediction module" may include the cross attention module part existing in the image generation module that is related to the output data of the timing prediction module.

[0142] For another example, in one possible implementation, the first part described above may include a time series prediction module, an MLP for processing the output data of the time series prediction module, and a portion of the image generation module related to the output data of the time series prediction module. The "MLP for processing the output data of the time series prediction module" may refer to the MLP in the data processing model described above that is related to the output data of the time series prediction module, such as the MLP shown in FIG3 .

[0143] In addition, the embodiments of the present disclosure do not limit the implementation of step 32 above. For example, the image generation process involved in step 32 is similar to the image generation process shown in S1-S3 above. Based on this, the embodiments of the present disclosure also provide a possible implementation of step 32 above. In this implementation, step 32 may specifically include the following steps 321-323.

[0144] Step 321: Obtain a reference image, audio driving information, a historical image corresponding to the audio driving information, and a label image corresponding to the audio driving information. The label image's corresponding time in the sample video is later than the historical image's corresponding time in the sample video; the audio driving information's corresponding time in the sample video includes the label image's corresponding time in the sample video.

[0145] The label image corresponding to the audio driving information refers to guidance information required for reference when performing image generation processing based on the audio driving information; and the label image corresponding to the audio driving information is compatible with the audio driving information. For example, when the audio driving information refers to the audio required for image generation processing based on the i-th audio frame in the sample video, the label image corresponding to the audio driving information can refer to the i-th video frame in the sample video, so that the time corresponding to the label image in the sample video is consistent with the time corresponding to the i-th audio frame in the sample video.

[0146] In addition, for the relevant content of “reference image, audio driving information, and historical image corresponding to the audio driving information” in step 321 , please refer to S1 above.

[0147] In addition, the embodiment of the present disclosure does not limit the implementation method of the above step 321. For example, the step 321 can be specifically as follows: the most representative video image in a sample video, such as a video image used to represent a state of no expression, is used as a reference image; the i-th audio segment in the sample video, such as {i-3th frame audio, i-2th frame audio, i-1th frame audio, i-th frame audio, i+1th frame audio, i+2th frame audio, i+3th frame audio}, is used as audio driving information, so that the audio driving information includes the i-th frame audio and the i-th frame audio. The audio context information is used so that the audio driving information can represent the audio required for image generation processing of the i-th audio frame in the sample video; the i-th video frame in the sample video is used as the label image corresponding to the audio driving information; and the historical image corresponding to the audio driving information is determined based on the i-1-th video frame, the i-2-th video frame, ..., and the iT-th video frame in the sample video, so that the historical image includes the i-1-th video frame, the i-2-th video frame, ..., and the iT-th video frame. Where i is a positive integer.

[0148] Step 322: The data processing model determines a generated image corresponding to the audio-driven information based on the reference image, the audio-driven information, and the historical images corresponding to the audio-driven information. The audio features of the audio-driven information are used to participate in image feature prediction processing implemented by the time series prediction module in the data processing model.

[0149] It should be noted that the implementation of the above step 322 can refer to the relevant contents shown in S2-S3 above. For ease of understanding, the following description is made with reference to examples.

[0150] As an example, as shown in FIG3 , the above step 322 may specifically be: first, the timing prediction module in the data processing model performs image feature prediction processing based on the image features of the reference image, the image features of the historical image corresponding to the audio driving information, and the audio features of the audio driving information, and obtains and outputs the predicted image features corresponding to the audio driving information, so that the predicted image features satisfy the constraints described by the reference image, the historical image and the audio driving information; then, the image generation module in the data processing model performs image generation processing based on the reference image and the predicted image features corresponding to the audio driving information, and obtains the generated image corresponding to the audio driving information, such as the generated image shown in FIG3 .

[0151] Step 323: Update the first portion of the data processing model based on the generated image corresponding to the audio driving information and the label image corresponding to the audio driving information, and return to step 321 and subsequent steps until the second stop condition is met. The first portion includes the portion of the time series prediction module and the image generation module related to the output data of the time series prediction module.

[0152] The second stopping condition can be set based on the actual application scenario. For example, the second stopping condition can include that the model loss of the data processing model is lower than a second loss threshold. In another example, the second stopping condition can include that the rate of change of the model loss of the data processing model is lower than a second rate of change threshold. In another example, the second stopping condition can include that the number of updates to the first part of the data processing model is lower than a second number threshold.

[0153] The model loss of the data processing model is used to characterize the performance of the data processing model, so that the model loss of the data processing model can, to a certain extent, characterize the performance of the first part of the data processing model. Furthermore, the model performance of the data processing model is calculated based on the loss between the generated image corresponding to the audio driving information and the labeled image corresponding to the audio driving information. Furthermore, the disclosed embodiments do not limit the method for calculating the model loss of the data processing model.

[0154] Based on the relevant contents of steps 321 to 323 above, it can be known that in some application scenarios, such as the scenario shown in Figure 3, for the above data processing model, if the audio features are used to participate in the image feature prediction processing implemented by the time series prediction module in the data processing model, then the second training stage of the data processing model can be specifically as follows: first determine the reference image, audio driving information, the historical image corresponding to the audio driving information, and the label image corresponding to the audio driving information from the sample video; then, the data processing model determines the generated image corresponding to the audio driving information based on the reference image, the audio driving information, and the historical image corresponding to the audio driving information; then, based on the generated image and the label image corresponding to the audio driving information, update the first part of the data processing model, and fix the parameters of the other parts of the data processing model except the first part, so that the next round of training can be performed based on the updated data processing model, and iterate until the second training stage is completed when the second stop condition is reached.

[0155] Based on the relevant contents of steps 31 to 32 above, it can be seen that in some application scenarios, such as the scenario shown in Figure 3, for the above data processing model, if the audio features are used to participate in the image feature prediction processing implemented by the time series prediction module in the data processing model, then the training process of the data processing model can be implemented in a two-stage training method, and the two-stage training method can be specifically as follows: first, the image generation module in the data processing model is trained separately so that the trained image generation module has better image generation performance to achieve the first stage of training; then, under the premise of fixing most of the weight parameters of the UNet trained in the first stage, the parts of the data processing model related to the predicted image features, such as the time series prediction module, the MLP for processing the output data of the time series prediction module, and the parts of the image generation module related to the output data of the time series prediction module, etc., are trained to achieve the second stage of training, so that the finally trained data processing model has better performance, so that when the image generation task is performed based on the data processing model, a better image generation effect can be presented.

[0156] Example 2: In some scenarios, such as the scenario shown in Figure 2, the video generation method provided by the embodiments of the present disclosure can be used to implement a model training process. Based on this, to further improve the image generation effect, the embodiments of the present disclosure also provide a possible implementation of the model training process. In this implementation, when audio features are involved in the image generation process, the model training process can include steps 41-43, which are performed in sequence below.

[0157] Step 41: Train the image generation module in the data processing model.

[0158] It should be noted that for the relevant content of step 41, please refer to step 31 above.

[0159] Step 42: Training the first part of the data processing model, where the first part includes the time series prediction module and the part of the image generation module that is related to the output data of the time series prediction module.

[0160] It should be noted that for the relevant content of step 42, please refer to step 32 above.

[0161] Step 43: Train the second part of the data processing model; the second part refers to the part of the data processing model that is related to the audio features of the audio driving information.

[0162] Among them, the second part refers to the part related to the audio features in the above data processing model, such as the part for processing audio features. For example, under one possible implementation, the second part may include an MLP for processing audio features, and a part related to audio features in the image generation module. Among them, the "MLP for processing audio features" may refer to the MLP related to the audio features in the data processing model, such as MLP2 shown in Figure 2. The "part related to the audio features in the image generation module" is used to implement some processing for the audio features; and the embodiment of the present disclosure does not limit the implementation of the "part related to the audio features in the image generation module", for example, the "part related to the audio features in the image generation module" may include the cross attention module part related to the audio features in the image generation module.

[0163] In addition, the embodiment of the present disclosure does not limit the implementation of the above step 43. For example, the implementation of step 43 is similar to the training process in the first part above. For ease of understanding, the following is an example.

[0164] As an example, the above step 43 may include the following steps 431 to 433 .

[0165] Step 431: Obtain a reference image, audio driving information, a historical image corresponding to the audio driving information, and a label image corresponding to the audio driving information. The label image corresponds to a later time in the sample video than the historical image; and the audio driving information corresponds to a time in the sample video including the label image.

[0166] It should be noted that for the relevant content of step 431, please refer to the above step 321.

[0167] Step 432: The data processing model determines a generated image corresponding to the audio-driven information based on the reference image, the audio-driven information, and the historical images corresponding to the audio-driven information. The audio features of the audio-driven information are used to participate in image feature prediction processing implemented by the time series prediction module in the data processing model.

[0168] It should be noted that for the relevant content of step 432, please refer to the above step 322.

[0169] Step 433: Update the second part of the data processing model based on the generated image corresponding to the audio drive information and the label image corresponding to the audio drive information, and return to continue executing the above step 431 and subsequent steps until the third stop condition is reached.

[0170] The third stopping condition can be set based on the actual application scenario. For example, the third stopping condition can include the model loss of the data processing model being lower than a third loss threshold. In another example, the third stopping condition can include the rate of change of the model loss of the data processing model being lower than a third rate of change threshold. In another example, the third stopping condition can include the number of updates of the second part of the data processing model being lower than a third number threshold.

[0171] Based on the relevant contents of steps 431 to 433 above, it can be known that in some application scenarios, such as the scenario shown in Figure 2, for the above data processing model, after completing the two-stage training of the image processing module and the first part in the data processing model, the weights of the two-stage training can be fixed, and the second part of the data processing model can be further updated to improve the processing performance of the data processing model for speech, which is conducive to improving the image generation effect.

[0172] Based on the relevant contents of steps 41 to 43 above, it can be known that in some application scenarios, such as the scenario shown in FIG2 , for the above data processing model, if the audio features are used to participate in the image generation processing implemented by the image generation module in the data processing model, then the training process of the data processing model can be implemented in a three-stage training method, and the three-stage training method can be specifically as follows: first, the image generation module in the data processing model is trained separately so that the trained image generation module has better image generation performance to achieve the first stage of training; then, under the premise of fixing most of the weight parameters of the UNet trained in the first stage, the image features in the data processing model are trained. The relevant parts, such as the time series prediction module, the MLP for processing the output data of the time series prediction module, and the parts of the image generation module related to the output data of the time series prediction module, are trained to realize the second stage of training; finally, under the premise of fixing the weight parameters of the first stage training and the weight parameters of the second stage training, the parts of the data processing model used to process audio features, such as the MLP for processing audio features, and the parts of the image generation module related to audio features, are trained to realize the third stage of training, so that the finally trained data processing model has better performance, so that better image generation effect can be presented when performing image generation tasks based on the data processing model.

[0173] Example 3: In some scenarios, such as a virtual image driving scenario, the video generation method provided by the embodiment of the present disclosure can be used to implement the virtual image driving process. Based on this, the embodiment of the present disclosure also provides a virtual image driving process, which can specifically include the following steps 51 to 54.

[0174] Step 51: Obtain a reference image, audio driving information, and historical images corresponding to the audio driving information. The reference image is determined based on a pre-constructed virtual image. The audio driving information is determined based on the i-th frame of audio in an audio sequence; the audio sequence is converted from text data; the historical images corresponding to the audio driving information include image generation results corresponding to at least one frame of audio in the audio sequence, where the time corresponding to each audio frame in the at least one frame of audio in the audio sequence is earlier than the time corresponding to the i-th frame of audio in the audio sequence; i is a positive integer, and 2≤i≤the total number of audio frames in the audio sequence.

[0175] The term "avatar" refers to a pre-built virtual image for a user. The embodiments of this disclosure do not limit the implementation of this virtual image; for example, it can be a three-dimensional digital human. Furthermore, the embodiments of this disclosure do not limit the method for obtaining this virtual image; for example, it can be determined based on user-provided information, such as an image. Alternatively, the virtual image can be selected by the user from a selection of candidate virtual images.

[0176] In addition, the embodiment of the present disclosure does not limit the method for obtaining the reference image in the above step 51. For example, it can be specifically: taking a photo of the virtual image to obtain the reference image.

[0177] In addition, for the relevant content of step 51, please refer to S1 above.

[0178] Step 52: Perform image feature prediction processing based on the image features of the reference image and the image features of the historical image corresponding to the audio driving information to obtain the predicted image features corresponding to the audio driving information.

[0179] It should be noted that for the relevant content of step 52, please refer to S2 above.

[0180] Step 53: Perform image generation processing based on the reference image and the predicted image features corresponding to the audio-driven information to obtain a generated image corresponding to the audio-driven information; the audio features of the audio-driven information are used in the image generation process or image feature prediction process. The generated image corresponding to the audio-driven information refers to the image generation result corresponding to the i-th audio frame, so that the generated image corresponding to the audio-driven information is considered the i-th frame image in the final generated video. i is a positive integer, and 2≤i≤the total number of audio frames in the audio sequence.

[0181] It should be noted that for the relevant content of step 53, please refer to S3 above.

[0182] Step 54: Generate a video based on the generated image corresponding to the audio driving information and the historical image corresponding to the audio driving information, so that the video is used to describe the changes in the facial state of the virtual image under the audio sequence.

[0183] It should be noted that the embodiments of the present disclosure do not limit the implementation method of the above step 54. For example, it can specifically be: generating a video based on the audio sequence and the image generation results corresponding to each audio in the audio sequence, so that the video includes the audio sequence and the image generation results corresponding to each audio in the audio sequence, so that the video can describe the state of the virtual image under different frames of audio in the audio sequence, such as facial expression state, etc.

[0184] In addition, for the final generated video, the first frame image in the video (for example, the image generation result corresponding to the first frame audio in the audio sequence above) can be determined based on the reference image; and the embodiment of the present disclosure does not limit the determination process of the first frame image. For example, it can be specifically: directly use the reference image as the first frame image. For another example, it can be specifically: perform image generation processing based on the reference image and the first frame audio in the audio sequence to obtain the first frame image. It should be noted that the embodiment of the present disclosure does not limit the implementation method of the image generation processing. For example, it can be implemented using any existing or future single-frame image generation method.

[0185] Based on the relevant content of steps 51 to 54 above, it can be seen that in some application scenarios, the video generation method provided by the embodiments of the present disclosure can be used to determine a driving video for a virtual character. In particular, because the video generation method has good performance, the resulting driving video can better meet user needs, which is conducive to improving the user experience.

[0186] Based on the video generation method provided in the embodiments of the present disclosure, the embodiments of the present disclosure also provide a video generation device, which will be explained and illustrated below in conjunction with Figure 4. Figure 4 is a schematic diagram of the structure of a video generation device provided in the embodiments of the present disclosure. It should be noted that for the technical details of the video generation device provided in the embodiments of the present disclosure, please refer to the relevant content of the video generation method above.

[0187] As shown in FIG4 , the video generation device 400 provided in an embodiment of the present disclosure includes:

[0188] The data acquisition unit 401 is configured to acquire a reference image, audio driving information, and a historical image corresponding to the audio driving information;

[0189] A feature prediction unit 402 is configured to perform image feature prediction processing based on the image features of the reference image and the image features of the historical image to obtain predicted image features corresponding to the audio driving information;

[0190] An image generation unit 403 is configured to perform image generation processing based on the reference image and the predicted image features to obtain a generated image corresponding to the audio driving information; the audio features of the audio driving information are used to participate in the image generation processing or the image feature prediction processing;

[0191] The video generating unit 404 is configured to generate a video based on the generated image and the historical image, wherein the time corresponding to the generated image in the video is later than the time corresponding to the historical image in the video.

[0192] In one possible implementation, when the audio feature is used to participate in the image generation process, the image generation unit 403 is specifically used to: perform a first encoding process on the reference image to obtain a first image feature, and perform a second encoding process on the reference image to obtain a second image feature; the first image feature is used to characterize the local information of the pixels in the reference image; the second image feature is used to characterize the global information of the reference image; a reference feature is obtained based on the first image feature and the second image feature; and image generation processing is performed based on the reference feature, the predicted image feature, and the audio feature of the audio driving information to obtain a generated image corresponding to the audio driving information.

[0193] In a possible implementation, the predicted image features are determined using a time series prediction module in a data processing model; and the generated image is determined using an image generation module in the data processing model.

[0194] In one possible implementation, when the audio features of the audio driving information are used to participate in the image feature prediction process, the training process of the data processing model includes the following two training stages performed sequentially:

[0195] Training an image generation module in the data processing model;

[0196] A first part of the data processing model is trained, where the first part includes the time series prediction module and a part of the image generation module that is related to output data of the time series prediction module.

[0197] In one possible implementation, when the audio features of the audio driving information are used to participate in the image generation process, the training process of the data processing model includes the following three training stages performed in sequence:

[0198] Training an image generation module in the data processing model;

[0199] Training a first portion of the data processing model, the first portion including the time series prediction module and a portion of the image generation module related to output data of the time series prediction module;

[0200] The second part of the data processing model is trained; the second part refers to the part of the data processing model related to the audio feature.

[0201] In a possible implementation manner, the image features of the reference image and the image features of the historical image are both obtained through a first encoding process.

[0202] In one possible implementation, the audio driving information is determined based on the i-th frame of audio in an audio sequence; the audio sequence is converted from text data; i is a positive integer, 2≤i≤the total number of audio frames in the audio sequence; the generated image corresponding to the audio driving information refers to the i-th frame image in the video; the video is used to describe the changes in facial state of the virtual image under the audio sequence.

[0203] In a possible implementation manner, the first frame image in the video is the reference image; or, the first frame image in the video is generated based on the reference image and the first frame audio in an audio sequence.

[0204] Based on the relevant content of the above-mentioned video generating device 400, it can be known that the working principle of the video generating device 400 is: first obtain a reference image, the audio driving information, and a historical image corresponding to the audio driving information; then perform image feature prediction processing based on the image features of the reference image and the image features of the historical image to obtain the predicted image features corresponding to the audio driving information, so that the predicted image features can represent the image features adapted to the audio driving information, thereby enabling the predicted image features to represent some states in the image adapted to the audio driving information, such as states that can change frequently over time, such as facial expressions; then, perform image generation processing based on the reference image and the predicted image features to obtain the generated image corresponding to the audio driving information, so that the generated image can better represent the image adapted to the audio driving information; finally, generate a video based on the generated image and the historical image, so that the time corresponding to the generated image in the video is later than the time corresponding to the historical image in the video, thereby achieving automatic video generation. Among them, if the audio features of the audio-driven information are used to participate in the image generation process, the audio features are enabled to influence the image generation process, so that the generated image obtained based on the image generation process can meet the constraints described by the audio features, and thus the generated image is more compatible with the audio-driven information, which is beneficial to improving the video generation effect. In addition, if the audio features of the audio-driven information are used to participate in the image feature prediction process, the audio features are enabled to influence the image feature prediction process, so that the predicted image features obtained based on the image feature prediction process can meet the constraints described by the audio features, and thus the predicted image features can more accurately represent the image features of the image adapted to the audio-driven information, so that the generated image obtained based on the predicted image features is more compatible with the audio-driven information, which is beneficial to improving the video generation effect.

[0205] In addition, an embodiment of the present disclosure also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes any implementation of the video generation method provided by the embodiment of the present disclosure.

[0206] Referring to FIG5 , a schematic diagram of the structure of an electronic device 500 suitable for implementing embodiments of the present disclosure is shown. Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG5 is merely an example and should not limit the functionality or scope of use of embodiments of the present disclosure.

[0207] As shown in Figure 5, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0208] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although FIG5 shows the electronic device 500 with various devices, it should be understood that not all of the devices shown are required to be implemented or present. More or fewer devices may alternatively be implemented or present.

[0209] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0210] The electronic device provided by the embodiment of the present disclosure and the method provided by the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0211] The embodiments of the present disclosure further provide a computer-readable medium having instructions or computer programs stored therein. When the instructions or computer programs are executed on a device, the device executes any implementation of the video generation method provided by the embodiments of the present disclosure.

[0212] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0213] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0214] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0215] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device can perform the method.

[0216] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0217] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0218] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit / module does not, in some cases, limit the unit itself.

[0219] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0220] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0221] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems or devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0222] It should be understood that in the embodiments of the present disclosure, "at least one (item)" refers to one or more, and "more" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0223] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0224] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0225] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the disclosed embodiments. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the disclosed embodiments. Therefore, the disclosed embodiments are not limited to the embodiments shown herein, but are intended to be embodied in the widest possible manner consistent with the principles and novel features disclosed herein.

Claims

1. A video generation method, comprising: Acquire a reference image, audio driving information, and a historical image corresponding to the audio driving information; Performing image feature prediction processing based on the image features of the reference image and the image features of the historical image to obtain predicted image features corresponding to the audio driving information; Performing image generation processing based on the reference image and the predicted image features to obtain a generated image corresponding to the audio driving information; the audio features of the audio driving information are used to participate in the image generation processing or the image feature prediction processing; A video is generated based on the generated image and the historical image, wherein the time corresponding to the generated image in the video is later than the time corresponding to the historical image in the video.

2. The method according to claim 1, wherein When the audio feature is used to participate in the image generation process, the process of determining the generated image includes: performing a first encoding process on the reference image to obtain a first image feature, and performing a second encoding process on the reference image to obtain a second image feature; the first image feature is used to represent local information of pixels in the reference image; and the second image feature is used to represent global information of the reference image; Obtaining a reference feature based on the first image feature and the second image feature; An image generation process is performed based on the reference features, the predicted image features, and the audio features of the audio driving information to obtain a generated image corresponding to the audio driving information.

3. The method according to claim 1, wherein The predicted image features are determined using a time series prediction module in a data processing model; The generated image is determined by using the image generation module in the data processing model.

4. The method according to claim 3, wherein: When the audio features of the audio driving information are used to participate in the image feature prediction process, the training process of the data processing model includes the following two training stages that are performed in sequence: Training an image generation module in the data processing model; A first part of the data processing model is trained, where the first part includes the time series prediction module and a part of the image generation module that is related to output data of the time series prediction module.

5. The method according to claim 3, wherein When the audio features of the audio driving information are used to participate in the image generation process, the training process of the data processing model includes the following three training stages that are performed in sequence: Training an image generation module in the data processing model; Training a first portion of the data processing model, the first portion including the time series prediction module and a portion of the image generation module related to output data of the time series prediction module; The second part of the data processing model is trained; the second part refers to the part of the data processing model related to the audio feature.

6. The method according to claim 1, wherein The image features of the reference image and the image features of the historical image are both obtained through a first encoding process.

7. The method according to claim 1, wherein The reference image is determined based on a pre-constructed virtual image; The audio driving information is determined based on the i-th audio frame in the audio sequence; the audio sequence is converted from text data; i is a positive integer, 2≤i≤the total number of audio frames in the audio sequence; The generated image corresponding to the audio driving information refers to the i-th frame image in the video; The video is used to describe the changes in facial states of the virtual image under the audio sequence.

8. The method according to claim 1, wherein The first frame image in the video is the reference image; or, The first frame image in the video is generated based on the reference image and the first frame audio in the audio sequence.

9. A video generation device, comprising: a data acquisition unit, configured to acquire a reference image, audio driving information, and a historical image corresponding to the audio driving information; a feature prediction unit, configured to perform image feature prediction processing based on the image features of the reference image and the image features of the historical image to obtain predicted image features corresponding to the audio driving information; An image generation unit, configured to perform image generation processing based on the reference image and the predicted image features to obtain a generated image corresponding to the audio driving information; the audio features of the audio driving information are used to participate in the image generation processing or the image feature prediction processing; The video generating unit is configured to generate a video based on the generated image and the historical image, wherein the time corresponding to the generated image in the video is later than the time corresponding to the historical image in the video.

10. An electronic device comprising: processor and memory; The memory is used to store instructions or computer programs; The processor is configured to execute the instructions or computer program in the memory, so that the electronic device executes the method according to any one of claims 1 to 8.

11. A computer-readable medium storing instructions or computer programs, which, when executed on a device, cause the device to execute the method according to any one of claims 1 to 8.

12. A computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the method according to any one of claims 1 to 8.

Images