Method, device, apparatus and medium for generating video

Abstract

Methods, devices, equipment and media for generating videos are provided. In one method, a first reference image and a second reference image are determined from a plurality of reference images in a reference video. A reference text for describing the reference video is received. Based on the first reference image, the second reference image and the reference text, a generation model is obtained, and the generation model is used to generate a target video based on the first image, the second image and the text. Using the exemplary implementation of the present disclosure, the second reference image can be used as guiding data to determine the direction of development of the story in the video. In this way, the generation model can clearly grasp the changes in the content of each image in the video, thereby helping to generate a richer and more realistic video.

Description

Technical Field

[0001] Exemplary implementations of the present disclosure generally relate to computer vision, and in particular, to methods, devices, apparatuses, and computer-readable storage media for generating videos using machine learning models. Background Art

[0002] Machine learning technology has been widely used in many technical fields. In the field of computer vision, a variety of technical solutions have been proposed for automatically generating videos using machine learning models. For example, a corresponding video can be generated based on a pre-specified image and a text used to describe the video content. However, the images in the currently generated videos are usually less dynamic, for example, the objects in the video lack obvious movements and dynamic effects, etc., making it difficult to achieve a realistic motion visual effect. At this point, it is expected that a video including the desired content can be generated in a more convenient and effective way. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for generating a video is provided. In the method, a first reference image and a second reference image are determined from a plurality of reference images in a reference video. A reference text for describing the reference video is received. Based on the first reference image, the second reference image and the reference text, a generation model is obtained, and the generation model is used to generate a target video based on the first image, the second image and the text.

[0004] In a second aspect of the present disclosure, a device for generating a video is provided. The device includes: an image determination module configured to determine a first reference image and a second reference image from a plurality of reference images in a reference video; a text determination module configured to receive a reference text for describing the reference video; and an acquisition module configured to acquire a generation model based on the first reference image, the second reference image, and the reference text, the generation model being used to generate a target video based on the first image, the second image, and the text.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, and when the instructions are executed by the at least one processing unit, the electronic device executes the method according to the first aspect of the present disclosure.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor implements the method according to the first aspect of the present disclosure.

[0007] It should be understood that the content described in this content section is not intended to limit the key features or important features of the implementation of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In the following, in conjunction with the accompanying drawings and with reference to the following detailed description, the above and other features, advantages and aspects of various implementations of the present disclosure will become more apparent. In the accompanying drawings, the same or similar figure labels represent the same or similar elements, wherein:

[0009] Figure 1 A block diagram of a technical solution for generating a video is shown;

[0010] Figure 2 A block diagram illustrating a process for generating a video according to some implementations of the present disclosure is shown;

[0011] Figure 3 A block diagram showing a process for determining a second reference picture according to some implementations of the present disclosure;

[0012] Figure 4 A block diagram illustrating a process for determining a generative model according to some implementations of the present disclosure;

[0013] Figure 5 A block diagram showing a process of determining reference features of a reference video according to some implementations of the present disclosure;

[0014] Figure 6 A block diagram showing a process for generating a target video according to some implementations of the present disclosure;

[0015] Figure 7 A block diagram showing an operation process of a diffusion model in a process of generating a target video according to some implementations of the present disclosure;

[0016] Figure 8 A block diagram showing generation of a target video based on input data according to some implementations of the present disclosure is shown;

[0017] Fig. 9 A block diagram showing generation of a target video based on input data according to some implementations of the present disclosure is shown;

[0018] Fig.10 A block diagram showing generation of a target video based on input data according to some implementations of the present disclosure is shown;

[0019] Fig.11 A flowchart of a method for generating a video according to some implementations of the present disclosure is shown;

[0020] Fig.12 A block diagram showing an apparatus for generating a video according to some implementations of the present disclosure; and

[0021] Fig.13 A block diagram of a device capable of implementing multiple implementations of the present disclosure is shown. DETAILED DESCRIPTION

[0022] The implementation of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain implementations of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be interpreted as being limited to the implementations described herein. On the contrary, these implementations are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and implementations of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0023] In the description of the implementation methods of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "an implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". The following may also include other explicit and implicit definitions. As used herein, the term "model" can represent the association relationship between various data. For example, the above-mentioned association relationship can be obtained based on a variety of technical solutions currently known and / or to be developed in the future.

[0024] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0025] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0026] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0027] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0028] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0029] The term "in response to" as used herein refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of executing a subsequent action executed in response to the event or condition is not necessarily strongly related to the time when the event occurs or the condition is satisfied. For example, in some cases, the subsequent action may be executed immediately when the event occurs or the condition is satisfied; while in other cases, the subsequent action may be executed some time after the event occurs or the condition is satisfied.

[0030] Example Environment

[0031] Machine learning technology has been widely used in many technical fields. In the field of computer vision, it has been proposed to use machine learning models to automatically generate videos. Existing video generation methods mainly focus on text-to-video generation or video generation based on a single image. Although the generated video can involve object motion, the action duration is short and only has short-term dynamics, poor dynamics, and cannot present the desired information.

[0032] See also Figure 1 Describe a way to generate Figure 1 A block diagram 100 is shown according to a technical solution for generating a video. Figure 1 As shown, a machine learning model 120 can be obtained, where the machine learning model 120 can be generated based on reference data in a pre-constructed training data set. The text 110 can be used to specify the content of the video to be generated, and the image 112 can be used to specify the environment of the video (for example, as the first frame image, or an image at another location). In this example, the text 110 can, for example, indicate "A cat is walking in the street". At this point, the machine learning model 120 can generate a video 130 based on the texts 110 and 112, and the video content is a cat walking along the street.

[0033] However, the images in the currently generated videos are usually difficult to achieve realistic visual effects. Creating highly dynamic action videos, complex camera-era actions, visual effects, close-ups of expressions, or shot transitions is a difficult challenge. Current video generation methods, which focus mainly on text-to-video generation, tend to produce videos with minimal motion amplitudes, can only generate video clips of shorter lengths, and the morphology of the objects (e.g., cats) in the videos may be similar to those in image 112, and the cat's movements may be stiff and have very small amplitudes. At this point, it is desirable to generate a video including desired content in a more convenient and effective manner.

[0034] Generate a summary of the video

[0035] Since videos require an additional time dimension and consist of many keyframes, it is difficult to accurately describe each keyframe using simple language. In addition, the types of motion in videos are very diverse. Existing technical solutions are not only resource-intensive, but also pose considerable challenges to the generation model. In order to understand complex text descriptions and generate matching videos, the size of the model will increase dramatically, and the amount of annotation data required will also increase significantly.

[0036] In order to at least partially address the deficiencies in the prior art, according to an exemplary implementation of the present disclosure, a technical solution for video generation based on a diffusion model (e.g., called PixelDance) is proposed. In summary, a machine learning architecture based on a diffusion model is proposed, which can combine the image instructions of the first and last frames in a video with the text instructions for video generation. Comprehensive experimental results show that the video generated using the technical solution of the present disclosure exhibits excellent visual effects in synthesizing videos with complex scenes and complex motions.

[0037] According to an example implementation of the present disclosure, the first frame image instruction can set the scene for video generation (and establish the characters). The first frame also enables the model to generate continuous videos, in which case the model uses the last frame of the previous video as the first frame instruction of the subsequent video to generate the subsequent video. Further, the last frame instruction describing the end state of the video can serve as an additional control mechanism. In this way, the alignment of user expectations with text can be enhanced (that is, the generated video is made more in line with user expectations), and the model can be enabled to construct complex shots, ultimately generating rich video content, thereby ensuring diversity and coherence.

[0038] Generally speaking, the <text (text), first frame (first frame), last frame (last frame)> instructions can be used as conditions. Given the above three instructions, the model can focus on learning the dynamics of people, animals, objects, and other entities in the world during the training phase; and during inference, the model can "generalize" the learned laws of motion in the physical world to areas not covered during training, such as realizing the movement of anime characters, special effects shots, and so on.

[0039] Specifically, the above information can be integrated into the diffusion model. For example, text information is encoded through a pre-trained text encoder and then embedded into the diffusion model using a cross-attention mechanism. Image instructions are encoded using a pre-trained variational autoencoder (VAE) encoder and concatenated with the perturbed video content or Gaussian noise as the input to the diffusion model. During the training process, the first-frame image of the video can be directly used as the first-frame instruction to force the model to strictly follow the instruction, thereby maintaining the continuity between consecutive videos. During the inference process, this instruction can be conveniently obtained from the text-to-image model or directly provided by the user.

[0040] Correspondingly, the model should be prevented from copying the last-frame instruction when generating a video. This is because it is challenging to obtain a perfect last frame during the inference process. Therefore, the model is designed to adapt to the rough sketches provided by the user as guidance to generate the corresponding video. This sketch can be created using basic image editing tools. The user can also provide a hand-drawn contour map as the last-frame guidance. It should be understood that the sketch here can be a sketch drawn using image editing tools.

[0041] See Figure 2 Describe the overview of an exemplary implementation according to the present disclosure, Figure 2 FIG. 200 is a block diagram showing a process for generating a video according to some implementations of the present disclosure. As Figure 2 shown, a generation model 220 can be constructed as Figure 2 shown, and this generation model can describe the association relationship between the input data and the output data. At this time, the input data can include, for example: text for describing the video content, a start image (for example, which can be called the first image) for representing the first-frame image in the video, and a guiding image (for example, which can be called the second image) for representing the last-frame image in the video. The output data can include, for example, a video.

[0042] According to an example implementation of the present disclosure, the generation model 220 can be established based on various architectures that are currently known and / or will be developed in the future. During the training phase, sample data can be extracted from the reference video 230, and then the generation model 220 can be trained. As Figure 2As shown, a reference video 230 including a plurality of reference images may be obtained, and a first reference image 212 and a second reference image 214 may be obtained from the plurality of reference images. Here, the first reference image 212 (e.g., the first frame image) may be obtained from the head position of the reference video 230, and the second reference image 214 (e.g., the last frame image, or an image within a predetermined range before the last frame image) may be obtained from the tail position of the reference video 230. Further, a reference text 210 for describing the reference video may be received.

[0043] like Figure 2 As shown, the reference video 230 may include a cat, and the cats in the first reference image 212 and the second reference image 214 may be located at different positions in the picture. At this time, the text 110 may indicate "A cat is walking in the street". Further, the generation model 220 may be trained based on the first reference image 212, the second reference image 214 and the reference text 210. At this time, the trained generation model 220 may generate a target video based on the first image, the second image and the text.

[0044] In the process of processing the last frame instruction, a variety of strategies can be used to regulate the influence of the last frame. During training, the last frame instruction can be randomly selected from the last three true value frames of the reference video. Noise can be introduced into the instruction to improve the robustness of the model to instruction dependence. During the training phase, the last frame instruction can be randomly discarded with a certain probability. Accordingly, a simple and effective inference strategy is proposed. For example, in the first τ denoising steps of the reverse process of the diffusion model, the last frame instruction can be used to guide the video generation towards the desired end state. Subsequently, in the remaining steps, the last frame instruction can be discarded, allowing the model to generate more temporally coherent video content. The strength of the last frame guidance can be adjusted by τ.

[0045] It should be understood that Figure 2 The process of obtaining the generation model 220 is only schematically shown. Alternatively and / or additionally, multiple reference videos can be obtained and corresponding data can be extracted from the multiple reference videos, thereby continuously updating the generation model 220 in an iterative manner. For example, reference videos of different objects performing different actions can be obtained, so that the generation model 220 can master more abundant video generation knowledge. According to an example implementation of the present disclosure, the video generation process can be implemented in a multilingual environment. Although Figure 2 The reference texts in are expressed in English. Alternatively and / or additionally, the reference texts may be expressed based on other languages, for example, Chinese, French, Japanese, etc.

[0046] Using the exemplary implementation of the present disclosure, in the process of determining the generation model 220, the first reference image 212 can be used as the starting image of the video, and the second reference image 214 can be used as guiding data to determine the development direction of the story in the video. For example, the generated video can use the second reference image 214 as the last frame. In this way, the generation model 220 can clearly grasp the changes in the content of each image in the video, thereby helping to generate a richer and more realistic video.

[0047] Detailed process of generating videos

[0048] The outline of generating a video has been described, and more details about generating a video will be described below. According to an example implementation of the present disclosure, in order to determine the first reference image, a reference image located at the head of the reference video can be determined as the first reference image. In other words, the first frame image in the reference video can be used as the first reference image. In this way, the process of constructing training data can be simplified, and the training data can be obtained in a fast and efficient manner.

[0049] According to an example implementation of the present disclosure, the second reference image can be determined based on a variety of methods. For example, the last frame image in the reference video can be used as the second reference image. Alternatively and / or additionally, in order to introduce a disturbance factor into the training data, the second reference image can be determined from a set of reference images within a predetermined range at the end of the reference video.

[0050] See also Figure 3 Describe in more detail, Figure 3 A block diagram 300 of a process for determining a second reference video according to some implementations of the present disclosure is shown. Figure 3 As shown, it is assumed that the reference video 230 includes multiple reference images: reference image 310 located at the first frame position, reference images 312, ..., 314, 316, and reference image 318 located at the tail frame position. For example, the predetermined range 320 can represent the last 3 frames (or other numbers) at the end of the reference video. Assuming that the reference video includes N image frames, any one of the Nth frame, the N-1th frame, and the N-2th frame in the reference video can be used as the second reference image. At this time, the second reference image can include any one of the reference images 314, 316 and 318. In this way, the influence of the last frame on the training process can be weakened to a certain extent, thereby obtaining a more coherent video.

[0051] See also Figure 4 More information describing the generative model, Figure 4 A block diagram 400 of a process for determining a generative model according to some implementations of the present disclosure is shown. Figure 4As shown, the generation model 220 may include: an encoder model 410, a connection model 420, a diffusion model 430, and a decoder model 440. Here, the encoder model 410 and the decoder model 440 may have a predetermined structure and parameters. During the subsequent update of the parameters of the generation model 220, the parameters of the encoder model 410 and the decoder model 440 will not be updated. According to an example implementation of the present disclosure, a VAE and a corresponding decoder may be used.

[0052] exist Figure 4 In the example, the generation model 220 can be obtained based on the first reference image 212, the second reference image 214, and the reference text 210. Specifically, the encoder model 410 can be used to determine the first reference feature 412 (for example, represented as Z) of the reference video 230, and the first reference feature 412 can include multiple reference image features of multiple reference images. Further, the encoder model 410 can be used to determine the second reference feature 414 (for example, represented as c image ), the second reference feature includes a first reference image feature of the first reference image and a second reference image feature of the second reference image.

[0053] According to an example implementation of the present disclosure, the first reference feature 412 may include multiple dimensions, and each dimension may correspond to a reference image, that is, a feature from a reference image (having a predetermined dimension, for example, Where C represents the number of channels, H represents the image height, and W represents the image width). According to an example implementation of the present disclosure, the first reference feature and the second reference feature may have the same dimension. In this case, the number of dimensions is equal to the number of images in the reference video. For example, the reference video may be specified to include 16 (or other number of) images.

[0054] See also Figure 5 Describes more details about the identified features. Figure 5 A block diagram 500 is shown of a process for determining reference features of a reference video according to some implementations of the present disclosure. Figure 5As shown, each reference image in the reference video 230 may be processed one by one. For example, the encoder model may be used to extract features of the first reference image 212 to generate a first reference image feature. The first reference image feature may be placed at the first position (i.e., position 510) of the first reference feature 412. Subsequently, the encoder model may be used to extract reference image features of another reference image after the first reference image 212, and the reference image feature may be placed at the second position of the first reference feature 412, and so on. Each reference image in the reference video 230 may be continuously processed until the second reference image feature of the last reference image (i.e., the second reference image 214) in the reference video 230 is placed at the last position (i.e., position 512) of the first reference feature 412.

[0055] Similarly, encoder model 310 may be used to determine a second reference feature of the reference video. Encoder model 410 may be used to extract features of first reference image 212 to generate a first reference image feature. The first reference image feature may be placed at a first position (i.e., position 520) of second reference feature 442. Encoder model 410 may be used to place a second reference image feature of the last reference image in reference video 230 (i.e., second reference image 214) at a last position 522 of second reference feature 414.

[0056] like Figure 5 As shown, the position of the first reference image feature in the second reference feature 414 (i.e., the first position) corresponds to the position of the first reference image in the reference video (i.e., the first position), and the position of the second reference image feature in the second reference feature 414 (i.e., the last position) corresponds to the position of the second reference image in the reference video (i.e., the last position). In this way, the first reference feature and the second reference feature can be easily aligned in the subsequent connection process, thereby facilitating the improvement of the accuracy of the diffusion model.

[0057] According to an exemplary implementation of the present disclosure, features at other positions other than the first position and the second position in the second reference feature may be set to empty. Specifically, Figure 5 Position 530 in the middle may be set to be empty, for example, it may be filled with data "0". In this way, the second reference feature may strengthen the influence of the first reference image and the second reference image on the video generation process, so that the generation model may learn more about the conversion relationship between the first reference image and the second reference image.

[0058] Specifically, VAE can be used to encode the reference video and obtain the first reference feature 412, that is, each frame in the reference video is encoded using VAE. VAE can be used to encode the first reference image and the second reference image, and then the position of the intermediate frame is padded with 0 to obtain the second reference feature 414 (the dimension is the same as the first reference feature 412).

[0059] return Figure 4 , a diffusion model 430 may be determined based on the first reference feature 412, the second reference feature 414, and the reference text 210. Specifically, noise may be added to the first reference feature 412 and the second reference feature 414, respectively, based on the principle of the diffusion model (for example, different degrees of noise may be added to the two features in different ways), thereby generating a first noise reference feature 412' and a second noise reference feature 414'. Further, a connection model 420 may be used to connect the first noise reference feature 412' and the second noise reference feature 414', and generate a feature 422. Further, corresponding training data may be constructed based on the principle of the diffusion model, thereby updating the parameters of the diffusion model 430.

[0060] According to an example implementation of the present disclosure, the diffusion model 430 can be implemented based on a variety of structures currently known and / or to be developed in the future. The diffusion process of the diffusion model 430 involves a forward noise process and a reverse denoising process. In the forward noise process, noise data can be added to the features in multiple steps. For example, the initial feature can be represented as Z_0. In each step, noise data can be continuously added, the noise feature of step t can be represented as Z_t, and the noise feature of step t+1 can be represented as Z_(t+1). For example, noise data can be added to Z_t at step t. Then, random Gaussian data can be obtained at step T.

[0061] In the reverse denoising process, the inverse process of the above-mentioned denoising process can be performed in multiple steps to gradually obtain the initial features. Specifically, the random Gaussian data and the reference text and the corresponding second reference features can be input into the diffusion model so that the denoising process can be performed step by step in multiple steps. For example, in the t-th step, a portion of the noise in the noise feature Z_t can be removed to form a noise feature Z_(t-1) that is relatively clean relative to the noise feature Z_t. Each denoising process can be performed in an iterative manner to reversely derive the initial features (i.e., the video features without the addition of noise).

[0062] According to an example implementation of the present disclosure, in the process of determining the diffusion model, the processing model 420 may be used to connect the first reference feature and the second reference feature to generate a reference feature of the reference video, and a noise processing may be performed on the reference feature to generate a noise reference feature 422 of the reference video, and then the diffusion model 430 is used to determine a reconstruction feature 432 of the reference video based on the noise reference feature 422 and the reference text 210 (for example, represented as ). Further, a loss function is constructed based on the difference between the reconstructed features and the reference features, and then the diffusion model 430 is updated using the loss function.

[0063] According to an example implementation of the present disclosure, a 2D UNet (U-type network) can be used to implement a diffusion model. The model can be constructed by spatial downsampling and then spatial upsampling by inserting jump connections. Specifically, the model may include two basic blocks, namely a 2D convolution block and a 2D attention. The 2D UNet is extended to 3D by inserting a temporal layer, where a 1D convolution layer follows the 2D convolution layer along the temporal dimension, and a 1D attention layer follows the 2D attention layer along the temporal dimension. The model can be trained with images and videos to maintain high-fidelity spatial generation capabilities and disable 1D temporal operations on image inputs. Bidirectional self-attention can be used in all attention layers. The text instructions are encoded using a pre-trained text encoder, and the embedded c text By injecting the cross-attention layer in UNet, the hidden state is used as the query, c text as key and value.

[0064] Regarding image command injection, the image commands of the first and last frames can be combined with text guidance. Given the image commands on the first and last frames, for example, represented as {I first ,I last}, the above representation can be encoded into the same latent space as the input space of the diffusion model, resulting in {f first ,f last},in To insert instructions without losing temporal position information, the final image condition can then be constructed as in Condition c image Then, with the noised latent features z along the channel dimension t connections, which are used as input to the diffusion model.

[0065] According to an exemplary implementation of the present disclosure, a first reference image, a second reference image, and a reference text can be determined from a large number of reference videos, and then the first reference feature Z, the second reference feature c of each reference video can be determined using the method described above. imageAnd the corresponding reconstruction features Here, the reconstruction feature is determined by the diffusion model 430 and includes the influence of the forward noise process. At this time, the reconstruction feature Z can be based on the first reference feature Z and the corresponding reconstruction feature The loss function is constructed by taking the difference between .

[0066] A large number of reference videos can be continuously used to iteratively update the diffusion model 430 until the expected stopping condition is met. It should be understood that in this process, no manual annotation data is required, but the first frame, the last frame and the corresponding text description of the reference video can be directly used as training data. In this way, the workload of obtaining manual annotation data can be eliminated, thereby improving the efficiency of obtaining the generation model 220.

[0067] It should be understood that although the second reference feature c is shown above image An example of simultaneously including relevant feature information of the first reference image and the second reference image, alternatively and / or additionally, in some cases the second reference feature c image Only the feature information of the first reference image may be included. Specifically, the second reference image feature in the second reference feature may be set to null according to a predetermined condition. In other words, Figure 5 The feature at position 522 can be removed, for example, the value at this position can be set to 0.

[0068] By using the example implementation of the present disclosure, the second reference image features can be removed according to a predetermined ratio (e.g., 25% or other values). In this way, the application scenario of the second reference image may not be considered in the process of updating the generated model, thereby making the generated model suitable for more application scenarios, thereby improving the accuracy of the generated model in various situations.

[0069] According to an example implementation of the present disclosure, the generative model 220 may be trained based on a variety of methods currently known and / or to be developed in the future, so that the generative model 220 may grasp the association relationship between the first image, the second image, the text describing the video content, and the corresponding video.

[0070] According to an example implementation of the present disclosure, the generation model 230 further includes a decoder model 440. After the generation model 220 has been obtained, the requirements for generating a video can be input to the generation model 220 to generate a corresponding target video. According to an example implementation of the present disclosure, a first image and text can be input to the generation model, and then the generation model is used to generate a corresponding target video. According to an example implementation of the present disclosure, a first image, a second image and text can be input to the generation model, and then the generation model is used to generate a corresponding target video. Alternatively and / or additionally, the text can even be empty, and the target video can still be generated at this time.

[0071] In order to obtain a target video that is more in line with the expectation, the first image and the second image used to generate the target video, and the text describing the content of the target video can be received. Then, the reconstruction features of the target video can be generated based on the first image, the second image, and the text according to the trained diffusion model. Further, the target video can be generated based on the reconstruction features according to the decoder model. In this way, the diffusion model 430 can be fully utilized to perform the reverse denoising process, so as to obtain the reconstruction features of the target video in a more accurate manner.

[0072] See also Figure 6 Describes more details about the generated video. Figure 6 A block diagram 600 of a process for generating a target video according to some implementations of the present disclosure is shown. Figure 6 As shown, a first image 612, a second image 614, and a text 610 for describing the image content can be received. Here, the first image 612 may include, for example, an ice surface, so as to be used as the first frame of the target video 630 to be generated. The second image 614 may include, for example, an ice surface and a polar bear on the ice surface. The second image 614 may be used as guidance information to generate the last frame of the target video 630. At this time, the target video 630 generated by the generation model 220 under the condition of the text 610, and the content of the video is "A polar bear is walking on the ice surface".

[0073] The first image 612, the second image 614, and the text 610 may be input to the generation model 220. At this point, the generation model 220 will output a target video 630, and the first frame of the target video 630 corresponds to the first image 612, and the last frame corresponds to the second image 614. It should be understood that since the target video 630 has not yet been generated, a noise feature is used as the first feature 620 of the target video 630. According to an example implementation of the present disclosure, the first feature 620 may be determined using Gaussian noise.

[0074] Further, the second feature 622 of the target video 630 may be generated based on the first image 612 and the second image 614 according to the encoder model 410. For example, the encoder model 410 may be used to respectively determine the first image feature of the first image 612 and the second image feature of the second image 614. Then, the first image feature may be placed at the first position of the second feature 622, and the second image feature may be placed at the last position of the second feature 622. It should be understood that the process of generating the second feature 622 here is similar to that in the training stage, and thus will not be described in detail.

[0075] In the case where the first feature 620 and the second feature 622 have been determined, the reconstructed feature 626 of the target video can be obtained according to the diffusion model using the first feature 620, the second feature 622 and the text 610. Similar to the process of determining the reconstructed feature in the training phase, the first noise feature 620' and the second noise feature 622' can be obtained, and the first noise feature 620' and the second noise feature 622' can be connected using the connection model 420 to form the noise feature 624. Further, the inverse denoising phase of the diffusion model 430 can be used to generate the reconstructed feature 626.

[0076] It should be understood that, since the diffusion model 430 has been trained based on the above loss function, the reconstructed features 626 generated by the diffusion model 430 can accurately describe various aspects of the target video 630. Further, the reconstructed features 626 can be input to the decoder model 440, and the decoder model 440 will output the corresponding target video 630, and the target video 630 will accurately match the first image 612, the second image 614 and the text 610 in the input data.

[0077] It should be understood that in the process of generating the video, the reverse denoising process of the diffusion model 430 can be performed in an iterative manner, see Figure 7 Describe in more detail. Figure 7 A block diagram 700 is shown of the operation process of a diffusion model in the process of generating a target video according to some implementations of the present disclosure. Figure 7 As shown, the reverse denoising process can be iteratively performed in multiple steps. Assuming that the total number of multiple steps is T, in the first step (t=1), the first feature 620, the second feature 622 and the text 610 can be used to obtain the reconstructed feature 626 of the target video.

[0078] Then, in a second step (t=2) after the first step in the multiple steps, the first feature may be set as the reconstructed feature of the target video, To replace Z, and then perform subsequent processing. In the second step, the diffusion model can use the first feature, the second feature and the text to obtain new reconstructed features. In the subsequent third step (t=3), similar processing can be performed until the predetermined Tth step is reached. At this point, the reverse denoising process ends, and the output reconstructed features are the features that describe the target video 630 to be generated. Using the example implementation of the present disclosure, noise can be gradually removed in multiple reverse denoising steps to generate more accurate reconstructed features.

[0079] According to an exemplary implementation of the present disclosure, the guiding role of the second image 614 can be further weakened. Specifically, the portion of the second feature 622 associated with the second image 614 can be considered only in a portion of the multiple steps. For example, in a group of steps in the multiple steps, the portion can be removed from the second feature, that is, the portion of the second feature corresponding to the second image is set to empty. Figure 6 In the example, the feature at the last position in the second feature 622 can be set to 0.

[0080] Specifically, in the first τ steps of all T denoising steps, the last frame condition can be applied to guide the video generation towards the desired end state, and discarded in subsequent steps to generate a more credible and temporally consistent video:

[0081]

[0082] In the above formula, τ determines the model's reliance on the last frame instruction, and adjusting τ can achieve various applications. For example, the model can generate high-quality videos without the last frame condition (i.e., τ = 0). In addition, classifier-free guidance can be applied, which mixes the model score estimates under text prompts and without text prompts.

[0083] According to an example implementation of the present disclosure, the number of steps in which the second image 614 is not considered can be determined according to a predetermined ratio. For example, in the step of τ=T*40% (or other values) among all T steps, the influence of the second image 614 can be ignored. Assuming T=50, in the first 30 (50*(1-40%)) steps, the complete second feature 622 is used; further in the last 20 (50*40%) steps, the value at the last position in the second feature 622 can be set to 0. In this way, the role of the first image 612 and the text 610 in generating the target video can be strengthened, thereby making the generated target video 630 more in line with expectations.

[0084] According to an exemplary implementation of the present disclosure, the second image is an image that specifies the end image of the target video. Figure 8Describe in more detail, Figure 8 A block diagram 800 is shown for generating a target video based on input data according to some implementations of the present disclosure. Figure 8 As shown, input data 810 may include a first image 812, a second image 814, and text 816. At this time, the second image 814 may guide the generation of the last frame in the target video 820. For example, the last frame may be the same as the second image 814, and at this time, the first frame of the target video 820 corresponds to the first image 812, the last frame corresponds to the second image 814, and the content of the video is "A polar bear is walking on the ice surface".

[0085] Fig. 9 A block diagram 900 of generating a target video based on input data according to some implementations of the present disclosure is shown. Fig. 9 As shown, input data 910 may include a first image 912, a second image 914, and text 916. At this time, the second image 914 may guide the generation of the last frame in the target video 920. For example, the last frame may be the same as the second image 914, in which case the first frame of the target video 920 corresponds to the first image 912, the last frame corresponds to the second image 914, and the content of the video is "apolar bear is walking on the ice surface, fireworks are bursting open, an astronaut is following the bear". Using the example implementation of the present disclosure, the second image may accurately describe the ending screen of the target video, and thus the video generation may be controlled in a more precise manner.

[0086] According to an example implementation of the present disclosure, the second image may be a sketch of the content of the ending image of the specified target video. It should be understood that in some cases, it is difficult to obtain the ending screen of the target video to be generated, in which case the content of the ending image may be specified based on a sketch (e.g., hand-drawn). Fig.10 A block diagram 1000 is shown for generating a target video based on input data according to some implementations of the present disclosure. Fig.10 As shown, the input data 1010 may include a first image 1012, a second image 1014, and text 1016. At this time, the second image 1014 may guide the generation of the last frame in the target video 1020.

[0087] At this time, the polar bear in the last frame can wave in the posture specified by the second image 1014. At this time, the first frame of the target video 1020 corresponds to the first image 1012, the last frame corresponds to the second image 1014, and the content of the video is "a polar bear walking on the ice surface gradually stands up and waves". By using the example implementation of the present disclosure, the posture and position of the object in the video can be specified in a more convenient and effective way, thereby facilitating the generation of a video including richer visual content.

[0088] According to an example implementation of the present disclosure, multiple videos can be generated in a continuous manner. Specifically, when a target video has been generated in the manner described above, the end image of the target video can be set to the first image of another target video. Further, a second image for generating another target video and another text describing the content of the other target video can be received, and according to the generation model, another target video is generated based on the first image and the second image of the other target video and the other text.

[0089] Specifically, in the case where the target video 1020 has been obtained, the end image 1030 in the target video 1020 can be used as the first image for generating the next video. Further, a second image (e.g., an image of a polar bear sleeping on the ice) can be input, and the text "a polar bear gradually lies down and falls asleep" can be input. At this time, the new video generated subsequently will start with the end image 1030 in the target video 1020, and the last frame of the new video corresponds to the image of the polar bear sleeping.

[0090] According to an example implementation of the present disclosure, a target video and another target video can be connected. Using the example implementation of the present disclosure, the end image of the current video can be used as the start image of the next video to be generated in a step-by-step cumulative manner. In this way, a long video with a richer plot can be gradually generated. Using the example implementation of the present disclosure, the complexity of video creation can be greatly simplified. For example, shorter videos can be constructed in the manner of movie storyboards, and then these videos can be edited to generate longer videos.

[0091] In summary, according to an example implementation of the present disclosure, a new video generation architecture PixelDance based on a diffusion model is proposed, which combines image instructions of the first frame and the last frame with text guidance. The corresponding training process and reasoning process can be implemented based on the PixelDance architecture. Using the example implementation of the present disclosure, flexible control of the video generation process and last frame guidance of different strengths can be provided.

[0092] The proposed technical solution can be implemented on a variety of datasets, and experiments show that the videos generated by PixelDance have strong advantages in synthesizing complex scenes and / or actions. Specifically, the generative model can be trained using various available datasets, where each video in the dataset is associated with a paired text that usually provides a rough description and shows a weak correlation with the video content. Since image instructions help learn complex video distributions, PixelDance can make full use of various datasets without the need for labeled data, demonstrating its excellent ability in generating videos with complex scenes and motion.

[0093] Example Process

[0094] Fig.11 A flow chart of a method 1100 for generating a video according to some implementations of the present disclosure is shown. At block 1110, a first reference image and a second reference image are determined from a plurality of reference images in a reference video. At block 1120, a reference text for describing the reference video is received. At block 1130, a generation model is obtained based on the first reference image, the second reference image, and the reference text, and the generation model is used to generate a target video based on the first image, the second image, and the text.

[0095] According to an example implementation of the present disclosure, determining the first reference image includes: determining a reference image located at a head of a reference video as the first reference image.

[0096] According to an example implementation of the present disclosure, determining the second reference image includes: determining the second reference image from a group of reference images located within a predetermined range at the end of the reference video.

[0097] According to an example implementation of the present disclosure, a generation model includes an encoder network and a diffusion model, and obtaining the generation model based on a first reference image, a second reference image, and a reference text includes: using the encoder model to determine a first reference feature of a reference video, the first reference feature including multiple reference image features of multiple reference images; using the encoder model to determine a second reference feature of the reference video, the second reference feature including a first reference image feature of the first reference image and a second reference image feature of the second reference image; and determining the diffusion model based on the first reference feature, the second reference feature, and the reference text.

[0098] According to an example implementation of the present disclosure, a first position of the first reference image feature in the second reference feature corresponds to a position of the first reference image in the reference video, and a second position of the second reference image feature in the second reference feature corresponds to a position of the second reference image in the reference video.

[0099] According to an example implementation of the present disclosure, the dimension of the second reference feature is equal to the dimension of the first reference feature, and features at positions other than the first position and the second position in the second reference feature are set to empty.

[0100] According to an exemplary implementation of the present disclosure, the method further includes: setting the second reference image feature to empty according to a predetermined condition.

[0101] According to an example implementation of the present disclosure, determining a diffusion model based on a first reference feature, a second reference feature, and a reference text includes: performing noise processing on the first reference feature and the second reference feature to generate a first noise reference feature and a second noise reference feature of a reference video, respectively; connecting the first noise reference feature and the second noise reference feature to generate a noise reference feature of the reference video; using the diffusion model, determining a reconstructed feature of the reference video based on the noise reference feature and the reference text; and updating the diffusion model based on the difference between the reconstructed feature and the reference feature.

[0102] According to an example implementation of the present disclosure, the generation model further includes a decoder model, and the method further includes: receiving a first image for generating a target video, and text describing the content of the target video; generating, according to a diffusion model, a reconstructed feature of the target video based on the first image and the text; and generating, according to the decoder model, the target video based on the reconstructed feature.

[0103] According to an example implementation of the present disclosure, generating the reconstructed features of the target video further includes: receiving a second image used to generate the target video; and generating the reconstructed features of the target video based on the second image.

[0104] According to an example implementation of the present disclosure, generating a reconstructed feature of a target video includes: using a noise feature as a first feature of the target video; generating a second feature of the target video based on a first image and a second image according to an encoder model; and obtaining a reconstructed feature of the target video using the first feature, the second feature, and text according to a diffusion model.

[0105] According to an example implementation of the present disclosure, obtaining the reconstruction features of the target video includes: in a first step among multiple steps, using the first feature, the second feature and the text to obtain the reconstruction features of the target video; in a second step after the first step among the multiple steps,

[0106] The first feature is set as a reconstruction feature of the target video; and according to the diffusion model, based on the first feature, the second feature and the text, the reconstruction feature of the target video is obtained.

[0107] According to an exemplary implementation of the present disclosure, the method further includes: in a set of steps among the plurality of steps, setting a portion of the second feature corresponding to the second image to empty.

[0108] According to an exemplary implementation of the present disclosure, the second image includes at least any one of the following: an image for specifying an end image of the target video, and a sketch for specifying content of the end image of the target video.

[0109] According to an example implementation of the present disclosure, the method further includes: setting the end image of the target video to the first image of another target video; receiving a second image for generating another target video and another text describing the content of the other target video; and generating another target video based on the first image and the second image of the other target video and the other text according to a generation model.

[0110] According to an exemplary implementation of the present disclosure, the method further includes: connecting the target video and another target video.

[0111] Example devices and equipment

[0112] Fig.12 A block diagram of an apparatus 1200 for generating a video according to some implementations of the present disclosure is shown. The apparatus includes: an image determination module 1210, configured to determine a first reference image and a second reference image from a plurality of reference images in a reference video; a text receiving module 1220, configured to receive a reference text determined to describe the reference video; and an acquisition module 1230, configured to acquire a generation model based on the first reference image, the second reference image and the reference text, the generation model being used to generate a target video based on the first image, the second image and the text.

[0113] According to an example implementation of the present disclosure, the image determination module includes: a first image determination module configured to determine the first reference image including: determining a reference image located at a head of a reference video as the first reference image.

[0114] According to an example implementation of the present disclosure, the image determination module includes: a second image determination module configured to determine a second reference image from a group of reference images located within a predetermined range at the end of a reference video.

[0115] According to an example implementation of the present disclosure, a generation model includes an encoder model and a diffusion model, and an acquisition module, which is configured to include: a first encoding module, which is configured to use the encoder model to determine a first reference feature of a reference video, the first reference feature including multiple reference image features of multiple reference images; a second encoding module, which is configured to use the encoder model to determine a second reference feature of the reference video, the second reference feature including a first reference image feature of the first reference image and a second reference image feature of the second reference image; and a determination module, which is configured to determine the diffusion model based on the first reference feature, the second reference feature, and a reference text.

[0116] According to an example implementation of the present disclosure, a first position of the first reference image feature in the second reference feature corresponds to a position of the first reference image in the reference video, and a second position of the second reference image feature in the second reference feature corresponds to a position of the second reference image in the reference video.

[0117] According to an example implementation of the present disclosure, the dimension of the second reference feature is equal to the dimension of the first reference feature, and features at positions other than the first position and the second position in the second reference feature are set to empty.

[0118] According to an exemplary implementation of the present disclosure, the apparatus further includes: a setting module configured to set the second reference image feature to empty according to a predetermined condition.

[0119] According to an example implementation of the present disclosure, a determination module includes: a noise module configured to perform noise processing on a first reference feature and a second reference feature, respectively, to generate a first noise reference feature and a second noise reference feature of a reference video; a connection module configured to connect the first noise reference feature and the second noise reference feature to generate a noise reference feature of the reference video; a reconstruction module configured to determine a reconstruction feature of the reference video based on the noise reference feature and a reference text using a diffusion model; and an update module configured to update the diffusion model based on the difference between the reconstructed feature and the reference feature.

[0120] According to an example implementation of the present disclosure, the generation model further includes a decoder model, and the device further includes: a receiving module configured to receive a first image for generating a target video and text describing the content of the target video; a generation module configured to generate a reconstruction feature of the target video based on the first image and the text according to a diffusion model; and a video generation module configured to generate the target video based on the reconstruction feature according to the decoder model.

[0121] According to an exemplary implementation of the present disclosure, the receiving module is further configured to: receive a second image used to generate a target video; and the reconstruction module is further configured to: generate a reconstructed feature of the target video based on the second image.

[0122] According to an example implementation of the present disclosure, a reconstruction module includes: a setting module configured to use a noise feature as a first feature of a target video; an encoding module configured to generate a second feature of the target video based on a first image and a second image according to an encoder model; and a feature acquisition module configured to obtain a reconstructed feature of the target video according to a diffusion model using the first feature, the second feature, and text.

[0123] According to an example implementation of the present disclosure, a feature acquisition module includes: a first reconstruction module, configured to obtain a reconstruction feature of a target video using a first feature, a second feature, and a text in a first step among multiple steps; a setting module, configured to set the first feature as a reconstruction feature of the target video in a second step after the first step among the multiple steps; and a second reconstruction module, configured to obtain a reconstruction feature of the target video based on the first feature, the second feature, and the text according to a diffusion model.

[0124] According to an exemplary implementation of the present disclosure, the apparatus further includes: a removal module configured to set a portion of the second feature corresponding to the second image to empty in a set of steps among the multiple steps.

[0125] According to an exemplary implementation of the present disclosure, the second image includes at least any one of the following: an image for specifying an end image of the target video, and a sketch for specifying content of the end image of the target video.

[0126] According to an example implementation of the present disclosure, the device further includes: an image setting module, configured to set the end image of the target video to the first image of another target video; the receiving module is further configured to receive a second image for generating another target video, and another text describing the content of the other target video; and the generating module is further configured to generate another target video based on the first image and the second image of the other target video, and the other text according to the generating model.

[0127] According to an exemplary implementation of the present disclosure, the apparatus further includes: a connection module configured to connect the target video and another target video.

[0128] Fig.13 1 shows a block diagram of a device 1300 capable of implementing multiple implementations of the present disclosure. It should be understood that Fig.13 The computing device 1300 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the implementations described herein. Fig.13 The computing device 1300 shown may be used to implement the methods described above.

[0129] like Fig.13 As shown, computing device 1300 is in the form of a general-purpose computing device. Components of computing device 1300 may include, but are not limited to, one or more processors or processing units 1310, memory 1320, storage device 1330, one or more communication units 1340, one or more input devices 1350, and one or more output devices 1360. Processing unit 1310 may be an actual or virtual processor and is capable of performing various processes according to a program stored in memory 1320. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to increase the parallel processing capabilities of computing device 1300.

[0130] The computing device 1300 typically includes a plurality of computer storage media. Such media may be any available media accessible to the computing device 1300, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 1320 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 1330 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data (e.g., training data for training) and may be accessed within the computing device 1300.

[0131] The computing device 1300 may further include additional removable / non-removable, volatile / non-volatile storage media. Fig.13 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. Memory 1320 may include a computer program product 1325 having one or more program modules that are configured to perform various methods or actions of various implementations of the present disclosure.

[0132] The communication unit 1340 enables communication with other computing devices via a communication medium. Additionally, the functions of the components of the computing device 1300 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 1300 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0133] Input device 1350 may be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 1360 may be one or more output devices, such as a display, speaker, printer, etc. Computing device 1300 may also communicate with one or more external devices (not shown) through communication unit 1340 as needed, such as storage devices, display devices, etc., communicate with one or more devices that allow a user to interact with computing device 1300, or communicate with any device that allows computing device 1300 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0134] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, the computer program product is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0135] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0136] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0137] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0138] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0139] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for generating a video, comprising: determining a first reference image and a second reference image from a plurality of reference images in a reference video; receiving a reference text for describing the reference video; as well as Based on the first reference image, the second reference image and the reference text, a generation model is acquired, where the generation model is used to generate a target video based on the first image, the second image and the text, including: Determining a first reference feature of the reference video, the first reference feature comprising a plurality of reference image features of the plurality of reference images; determining a second reference feature of the reference video, the second reference feature comprising a first reference image feature of the first reference image and a second reference image feature of the second reference image, wherein a first position of the first reference image feature in the second reference feature corresponds to a position of the first reference image in the reference video, and a second position of the second reference image feature in the second reference feature corresponds to a position of the second reference image in the reference video; and The generative model is determined based on the first reference feature, the second reference feature, and the reference text.

2. The method according to claim 1, wherein the first reference feature and the second reference feature are determined using an encoder model in the generative model, and determining the generative model comprises: A diffusion model in the generation model is determined based on the first reference feature, the second reference feature, and the reference text.

3. The method according to claim 1, wherein: Determining the first reference image includes: determining a reference image located at a head of the reference video as the first reference image; and Determining the second reference image includes: determining the second reference image from a group of reference images located within a predetermined range at the end of the reference video.

4. The method according to claim 1, wherein the dimension of the second reference feature is equal to the dimension of the first reference feature, and the features at other positions in the second reference feature other than the first position and the second position are set to empty.

5. The method according to claim 1, further comprising: The second reference image feature is set to empty according to a predetermined condition.

6. The method of claim 2, wherein determining the diffusion model comprises: Performing noise processing on the first reference feature and the second reference feature respectively to generate a first noise reference feature and a second noise reference feature of the reference video; linking the first noise reference signature and the second noise reference signature to generate a noise reference signature for the reference video; Determining, using the diffusion model, reconstruction features of the reference video based on the noise reference features and the reference text; and The diffusion model is updated based on a difference between the reconstructed signature and the first reference signature.

7. The method according to claim 2, further comprising: Generating a target video according to the obtained generation model includes: According to the generation model, receiving a first image used to generate the target video and text describing the content of the target video; Generate a reconstruction feature of the target video based on the first image and the text according to the diffusion model in the generation model; and The target video is generated based on the reconstructed features according to a decoder model in the generation model.

8. The method according to claim 2, further comprising: Generating a target video according to the obtained generation model includes: According to the generation model, receiving a first image and a second image used to generate the target video and text describing the content of the target video; Generate a reconstruction feature of the target video based on the first image, the second image, and the text according to the diffusion model in the generation model; and The target video is generated based on the reconstructed features according to a decoder model in the generation model.

9. The method according to claim 8, wherein generating the reconstructed features of the target video comprises: Using a noise feature as a first feature of the target video; generating, according to the encoder model, a second feature of the target video based on the first image and the second image; as well as According to the diffusion model, the first feature, the second feature and the text are used to obtain a reconstructed feature of the target video.

10. The method according to claim 9, wherein obtaining the reconstruction features of the target video according to the diffusion model comprises: In a first step of the multiple steps, a reconstructed feature of the target video is obtained using the first feature, the second feature, and the text; In a second step after the first step in the plurality of steps, replacing the first feature with the reconstructed feature; and According to the diffusion model, another reconstructed feature of the target video is obtained based on the first feature, the second feature and the text.

11. The method according to claim 10, further comprising: In one set of steps among the plurality of steps, a portion of the second feature corresponding to the second image is set to empty. 12 . The method according to claim 8 , wherein the second image comprises at least any one of the following: an image for specifying an end image of the target video, and a sketch for specifying content of the end image of the target video.

13. The method according to claim 8, further comprising: Setting the end image of the target video as the first image of another target video; receiving, according to the generation model, a second image for generating the other target video and another text describing the content of the other target video; as well as According to the generation model, the other target video is generated based on the first image and the second image of the other target video and the other text.

14. The method according to claim 13, further comprising: The target video and the another target video are connected.

15. An apparatus for generating a video, comprising: an image determination module, configured to determine a first reference image and a second reference image from a plurality of reference images in a reference video, wherein the first reference image is located at a head of the reference video, and the second reference image is located within a predetermined range of a tail of the reference video; A text determination module, configured to receive a reference text for describing the reference video; as well as An acquisition module is configured to acquire a generation model based on the first reference image, the second reference image and the reference text, wherein the generation model is used to generate a target video based on the first image, the second image and the text, including: A first encoding module is configured to determine a first reference feature of the reference video, wherein the first reference feature includes a plurality of reference image features of the plurality of reference images; a second encoding module configured to determine a second reference feature of the reference video, the second reference feature comprising a first reference image feature of the first reference image and a second reference image feature of the second reference image, wherein a first position of the first reference image feature in the second reference feature corresponds to a position of the first reference image in the reference video, and a second position of the second reference image feature in the second reference feature corresponds to a position of the second reference image in the reference video; and A determination module is configured to determine the generation model based on the first reference feature, the second reference feature, and the reference text.

16. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 14 when executed by the at least one processing unit.

17. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method according to any one of claims 1 to 14.

Images