Method and device for generating video, equipment and medium

Abstract

The invention provides a method, a device, equipment and a medium for generating a video. In one method, a plurality of images are received for describing a plurality of target images in a target video, respectively. Text for describing content of a target video is received. And generating the target video based on the plurality of images and texts according to the generation model. According to the embodiment of the invention, a plurality of received images can be used as guide data to determine the development direction of stories in the video, so that a richer and vivid dynamic video can be generated.

Description

[0001] This application is a divisional application of the Chinese invention patent application with the application date of November 17, 2023, application number 202311543966.5, and invention name “Methods, devices, equipment and media for generating videos”. Technical Field

[0002] Exemplary implementations of the present disclosure generally relate to computer vision, and in particular, to methods, devices, apparatuses, and computer-readable storage media for generating videos using machine learning models. Background Art

[0003] Machine learning technology has been widely used in many technical fields. In the field of computer vision, a variety of technical solutions have been proposed for automatically generating videos using machine learning models. For example, a corresponding video can be generated based on a pre-specified image and a text used to describe the video content. However, the images in the currently generated videos are usually less dynamic, for example, the objects in the video lack obvious movements and dynamic effects, etc., making it difficult to achieve a realistic motion visual effect. At this point, it is expected that a dynamic video including the desired content can be generated in a more convenient and effective way. Summary of the invention

[0004] In a first aspect of the present disclosure, a method for generating a video is provided. In the method, a plurality of images respectively used to describe a plurality of target images in a target video are received. A text for describing the content of the target video is received. According to a generation model, a target video is generated based on the plurality of images and the text.

[0005] In a second aspect of the present disclosure, a device for generating a video is provided. The device includes: an image receiving module configured to receive an image for describing at least one of a head image and a tail image of a target video; a text receiving module configured to receive text for describing the content of the target video; and a generation module configured to generate the target video based on the image and the text according to a generation model.

[0006] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, and when the instructions are executed by the at least one processing unit, the electronic device executes the method according to the first aspect of the present disclosure.

[0007] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor implements the method according to the first aspect of the present disclosure.

[0008] It should be understood that the content described in this content section is not intended to limit the key features or important features of the implementation of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In the following, in conjunction with the accompanying drawings and with reference to the following detailed description, the above and other features, advantages and aspects of various implementations of the present disclosure will become more apparent. In the accompanying drawings, the same or similar figure labels represent the same or similar elements, wherein:

[0010] Figure 1 A block diagram of a technical solution for generating a video is shown;

[0011] Figure 2 A block diagram illustrating a process for generating a video according to some implementations of the present disclosure is shown;

[0012] Figure 3A , Figure 3B and Figure 3C Block diagrams for acquiring a second image according to some implementations of the present disclosure are respectively shown;

[0013] Figure 4 A block diagram illustrating a process for generating a video according to some implementations of the present disclosure is shown;

[0014] Figure 5 A block diagram of a page for receiving input data for generating a subsequent video according to some implementations of the present disclosure is shown;

[0015] Figure 6 A block diagram illustrating a process for generating a subsequent video according to some implementations of the present disclosure;

[0016] Figure 7 A block diagram illustrating a process for generating a video based on a sketch according to some implementations of the present disclosure;

[0017] Figure 8 A block diagram showing a process of obtaining candidate inputs using a machine learning model according to some implementations of the present disclosure;

[0018] Fig. 9 A block diagram illustrating multiple candidate inputs according to some implementations of the present disclosure;

[0019] Fig.10 A block diagram showing a process of using a machine learning model to obtain candidate inputs based on summary text according to some implementations of the present disclosure;

[0020] Fig.11A block diagram showing a process of obtaining prompt information based on a machine learning model according to some implementations of the present disclosure;

[0021] Fig.12 A block diagram illustrating an overview process of training a generative model according to some implementations of the present disclosure;

[0022] Fig.13 A block diagram showing a detailed process for training a generative model according to some implementations of the present disclosure;

[0023] Fig.14 A block diagram showing a process of determining reference features of a reference video according to some implementations of the present disclosure;

[0024] Fig.15 A block diagram showing a process for generating a target video according to some implementations of the present disclosure;

[0025] Fig.16 A flowchart of a method for generating a video according to some implementations of the present disclosure is shown;

[0026] Fig.17 A block diagram showing an apparatus for generating a video according to some implementations of the present disclosure; and

[0027] Fig.18 A block diagram of a device capable of implementing multiple implementations of the present disclosure is shown. DETAILED DESCRIPTION

[0028] The implementation of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain implementations of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be interpreted as being limited to the implementations described herein. On the contrary, these implementations are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and implementations of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0029] In the description of the implementation methods of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "an implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". The following may also include other explicit and implicit definitions. As used herein, the term "model" can represent the association relationship between various data. For example, the above-mentioned association relationship can be obtained based on a variety of technical solutions currently known and / or to be developed in the future.

[0030] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0031] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0032] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0033] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0034] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0035] The term "in response to" as used herein refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of executing a subsequent action executed in response to the event or condition is not necessarily strongly related to the time when the event occurs or the condition is satisfied. For example, in some cases, the subsequent action may be executed immediately when the event occurs or the condition is satisfied; while in other cases, the subsequent action may be executed some time after the event occurs or the condition is satisfied.

[0036] Example Environment

[0037] Machine learning technology has been widely used in many technical fields. In the field of computer vision, it has been proposed to use machine learning models to automatically generate videos. Existing video generation methods mainly focus on text-to-video generation or generation based on a single image. Although the generated video can involve object motion, the action duration is short and only has short-term dynamics, poor dynamics, and cannot present the desired information.

[0038] See also Figure 1 Describe a way to generate Figure 1 A block diagram 100 is shown according to a technical solution for generating a video. Figure 1 As shown, a machine learning model 120 can be obtained, where the machine learning model 120 can be generated based on reference data in a pre-constructed training data set. The text 110 can be used to specify the content of the video to be generated, and the image 112 can be used to specify the environment of the video (for example, as the first frame image, or an image at another position). In this example, the text 110 can, for example, indicate "A cat is walking in the street". At this point, the machine learning model 120 can generate a video 130 based on the text 110 and the image 112, and the video content is a cat walking along the street.

[0039] However, the pictures in the currently generated videos are usually difficult to achieve real dynamic effects. Creating highly dynamic action videos, complex camera movements, visual effects, facial expressions or lens transitions faces a difficult challenge. Current video generation methods are mainly focused on the generation of text to video, tend to produce videos with minimal motion amplitude, can only generate video clips of shorter lengths, and the morphology of the object (e.g., cat) in the video may be similar to that in image 112, and the cat's movements may be relatively stiff and the amplitude of movement is very small. At this point, it is expected that a dynamic video including desired content can be generated in a more convenient and effective way.

[0040] Generate a summary of the video

[0041] Since videos require an additional time dimension and consist of many keyframes, it is difficult to accurately describe each keyframe using simple language. In addition, the types of motion in videos are very diverse. Existing technical solutions are not only resource-intensive, but also pose considerable challenges to the generation model. In order to understand complex text descriptions and generate matching videos, the size of the model will increase dramatically, and the amount of annotation data required will also increase significantly.

[0042] In order to at least partially address the deficiencies in the prior art, according to an exemplary implementation of the present disclosure, a technical solution for video generation based on a diffusion model (e.g., called PixelDance) is proposed. In summary, a machine learning architecture based on a diffusion model is proposed, which can combine the image instructions of the first and last frames in a video with the text instructions for video generation. Comprehensive experimental results show that the video generated using the technical solution of the present disclosure exhibits excellent visual effects in synthesizing videos with complex scenes and complex motions.

[0043] It should be understood that the generation model herein may be a pre-trained model, and the target video may be generated based on the model. Figure 2 Describes an overview of an example implementation according to the present disclosure, Figure 2 A block diagram 200 of a process for generating a video according to some implementations of the present disclosure is shown. Figure 2 As shown, a page 210 may be provided, and a plurality of controls for describing a plurality of target images in a target video may be provided in an area 230. It should be understood that the plurality of target images herein may be a plurality of target images having a time sequence in the target video. For example, a control 232 for describing a preceding target image may be provided, and a control 234 for describing a succeeding target image may be provided. Further, a control 236 for receiving text describing the content of the target video may be provided.

[0044] According to an exemplary implementation of the present disclosure, the video generation process can be implemented in a multi-language environment. Figure 2 The reference text in is expressed in English, and alternatively and / or additionally, the text can be expressed based on other languages, such as Chinese, French, Japanese, etc. Further, the user can call the generation model 220 via the control 242 to generate the target video 250. Alternatively and / or additionally, the user can return to the previous page via the control 240.

[0045] It should be understood that the generation model 220 here is obtained by combining the image instructions of the first frame and the last frame with the text instructions for video generation. By specifying multiple images in the target video, the generation model 220 can more comprehensively understand the change trends of each image in the video, thereby also being able to exhibit excellent visual effects in synthesizing videos with complex scenes and complex motions.

[0046] According to an example implementation of the present disclosure, the multiple images herein include a first image for describing a head image in the multiple target images and a second image for describing a tail image in the multiple target images. At this time, the generation model 220 can generate a target video 250 based on the received first image, second image and text. Compared with the existing technical solution of generating a video based only on a single image and text, multiple images can provide richer change information. In particular, since the head image and the tail image are respectively located at the head and tail ends of the target video, more guiding information about visual changes can be provided for the video generation process, so that the target video 250 can have a more excellent dynamic visual effect.

[0047] It should be understood that the head image herein may represent one or more images within a predetermined range at the beginning of the target video, for example, the first frame, the second frame, the third frame, etc. in the target video. Similarly, the tail image may represent one or more images within a predetermined range at the end of the target video, for example, the last frame, the second to last frame, the third to last frame, etc. in the target video.

[0048] Detailed process of generating videos

[0049] Having described the outline of generating a video, more details about generating a video will be described below. It should be understood that Figure 2 The process of receiving the first image and the second image is described only by way of example, and the above-mentioned images can be received based on a variety of methods. For example, the image address input by the user can be received at the control 232 and / or 234, or the user can drag the image to the control 232 and / or 234, and so on. According to an example implementation of the present disclosure, the second image can be received in a more flexible manner. Specifically, the second image can be received via at least any one of the following: an import control for importing the address of the second image; a drawing control for drawing the second image; and an editing control for editing the first image to generate the second image.

[0050] See also Figure 3A , 3B and 3C describe more details about acquiring the second image. Figure 3A A block diagram 300A for acquiring a second image according to some implementations of the present disclosure is shown. Figure 3A As shown, the page 340 for receiving the second image may include multiple controls. Specifically, a drawing control 310 for drawing the second image, an import control 320 for importing the address of the second image, and an editing control 330 for editing an image (e.g., the first image or other imported image) to generate the second image may be provided. The user may interact with the drawing control 310, and at this time, a drawing tool may be called to draw a sketch 312 of the second image.

[0051] Figure 3B A block diagram 300B for acquiring a second image according to some implementations of the present disclosure is shown. As shown, a user may interact with an import control 320, at which point an image file may be imported as the second image. Figure 3CA block diagram 300C for obtaining a second image according to some implementations of the present disclosure is shown. As shown, a user can interact with an editing control 330 to obtain a second image. In the context of the present disclosure, a user can edit a first image to generate a second image. For another example, a user can edit other images that have been imported to obtain a second image. For example, a user can move a polar bear at position 332 to position 334, and so on. In this way, a user can specify a tail image of a target video in a flexible and effective manner.

[0052] Figure 4 A block diagram 400 of a process for generating a video according to some implementations of the present disclosure is shown. A user may specify input data 410 for generating a model in the manner described above. For example, a user may utilize an import control to import a first image 412 and a second image 414, respectively, and may input text 416. The first image 412 may guide the generation of the first frame in a target video 420, and the second image 414 may guide the generation of the last frame in the target video 420. At this point, the first frame of the target video 420 corresponds to the first image 412, the last frame corresponds to the second image 414, and the content of the video is "A polar bear is walking on the ice surface".

[0053] According to an example implementation of the present disclosure, multiple target videos may be generated in a continuous manner. For example, a certain image (e.g., the last frame or other image) in the currently generated video may be used as the head image of the next video, thereby generating more videos in a cumulative manner. Figure 5 Describe in more detail the Figure 5 A block diagram 500 of a page for receiving input data for generating a subsequent video according to some implementations of the present disclosure is shown. Figure 5 As shown, a plurality of images in a target video (eg, the target video 420 that has been currently generated) may be provided in page 510 .

[0054] The user can interact with the desired image to select a head image for generating a subsequent video from a plurality of images. For example, the user can click on the third image 512 in the target video 420 to specify a head image for describing another target video after the target video. The user can specify the tail image as the fourth image 520 and input text 530 at the text box. At this point, another text for describing the content of another target video and the fourth image for describing the tail image of another target video can be obtained. Further, the generation model can generate another target video based on the third image 512, the fourth image 520 and another text 530.

[0055] like Figure 5 As shown, the user can press control 242 to generate another target video. Figure 6 Describe in more detail the Figure 6 A block diagram 600 of a process for generating a subsequent video according to some implementations of the present disclosure is shown. Figure 6 As shown, the input data 610 may include a third image 512, a fourth image 520, and text 530. At this time, the third image 512 may guide the generation of the first frame in the target video 520, and the fourth image 520 may guide the generation of the last frame in the target video 520. For example, the last frame may be the same as or different from the fourth image 520, and the content of the video is "a polar bear is walking on the ice surface, fireworks are bursting open, an astronaut is following the bear".

[0056] According to an example implementation of the present disclosure, a target video and another target video can be connected. The tail image of the current video can be used as the head image of the next video to be generated in a step-by-step cumulative manner. In this way, subsequent videos can be generated in a continuous manner, and long videos including richer plots can be gradually generated. Using the example implementation of the present disclosure, the complexity of video creation can be greatly simplified. For example, shorter videos can be constructed in the manner of movie storyboards, and these videos can then be edited to generate longer videos.

[0057] The above has described specifying the second image by importing an image. Alternatively and / or additionally, the second image can be a sketch of the content of the tail image of the specified target video. In some cases, it is difficult to obtain the end screen of the target video to be generated, and a sketch (e.g., hand-drawn) can specify the content of the tail image in a simpler and more effective manner.

[0058] Figure 7 A block diagram 700 of a process for generating a video based on a sketch according to some implementations of the present disclosure is shown. Figure 7 As shown, the input data 710 may include a first image 712, a second image 714, and text 716. At this time, the second image 714 may guide the generation of the last frame in the target video 720.

[0059] At this time, the polar bear in the last frame can wave in the posture specified by the second image 714. At this time, the first frame of the target video 720 corresponds to the first image 712, the last frame corresponds to the second image 714, and the content of the video is "a polar bear walking on the ice surface gradually stands up and waves". By using the example implementation of the present disclosure, the posture and position of the object in the video can be specified in a more convenient and effective way, thereby facilitating the generation of a dynamic video including richer visual content.

[0060] Although the above shows an example of specifying input data for generating a single video in a page, alternatively and / or additionally, the page may further include a control for receiving multiple images and a long text. At this time, the user can input multiple images for describing multiple key frames in a long video, and a long text for describing the content of the long video. Then, the machine learning model can determine the image and text for generating the image segment based on the received multiple images and long text.

[0061] See also Figure 8 Describe in more detail the Figure 8 A block diagram 800 is shown of a process of using a machine learning model to obtain candidate inputs according to some implementations of the present disclosure. Figure 8 As shown, the user can interact with control 820 to input multiple images 812, ..., 814, ..., 816 sorted by time, and further the user can enter a long text 830. The user can press control 842 to call the machine learning model 850 to perform further processing; alternatively and / or additionally, the user can press control 840 to return to the previous page.

[0062] It should be understood that the machine learning model 850 herein can be a pre-trained model with text processing capabilities and image processing capabilities. For example, the machine learning model 850 can be implemented based on a large language model. After receiving multiple images and a long text 830, the machine learning model 850 can split the long text 830 into multiple shorter texts, and select a head image and a tail image that match each shorter text from the multiple images.

[0063] See also Fig. 9 Describes the output of the machine learning model 850, which Fig. 9 A block diagram 900 is shown of multiple candidate inputs according to some implementations of the present disclosure. Fig. 9As shown, page 910 may include multiple candidate inputs, and each candidate input may correspond to a shorter video clip. Candidate inputs include, for example, images describing the first frame and the last frame of the video clip, and text describing the content of the video clip. In page 910, candidate input 920 may include images 812, 814, and text 922; ...; and candidate input 930 may include images 814, 816, and text 932. Using the example implementation of the present disclosure, the powerful processing capabilities of the machine learning model 850 can be fully utilized to assist the video generation process. For example, in film and television creation, a script fragment represented by text can be input to the machine learning model 850, and the machine learning model 850 can automatically output the relevant images and text descriptions of the multiple split shots. In this way, the model generation performance can be greatly improved.

[0064] According to an example implementation of the present disclosure, a summary text for describing a long video may be received, and then multiple texts for describing the contents of multiple video clips in the long video may be determined based on the machine learning model and the summary text. It should be understood that the summary text herein may describe the protagonist, environment, and story outline of the video to be generated, etc. Specifically, the summary text may require the generation of a story in which the protagonists are polar bears and astronauts, the story takes place in the Arctic, and the story outline is that the astronaut's spacecraft accidentally lands in the Arctic, encounters a polar bear, and slowly becomes friends.

[0065] The summary text can be input into the machine learning model, and then the powerful language ability of the machine learning model can be used to assist in the creation and generation of rich storylines. The machine learning model can also automatically split these storylines into multiple shots and generate corresponding description texts. Furthermore, the machine learning model can generate target images corresponding to multiple texts respectively, that is, specify the first frame image and the last frame image for each shot video clip. At this point, the generated target texts and corresponding images can be input into the generation model to obtain a longer dynamic video with a rich storyline.

[0066] Fig.10 A block diagram 1000 is shown of a process of using a machine learning model to obtain candidate inputs based on summary text according to some implementations of the present disclosure. Fig.10 As shown, a summary text 1010 may be input to the machine learning model 850, where the summary text 1010 may only limit the desire to create a video about a story between a polar bear and an astronaut, etc. The machine learning model 850 may generate a plurality of candidate inputs based on the summary text 1010. For example, the candidate input 1020 may include images 1022, 1024 and text 1026; ...; the candidate input 1030 may include images 1032, 1034 and text 1036.

[0067] It should be understood that the text 1026 and 1036 herein can be generated based on the summary text 1010, the images 1022 and 1024 can be generated based on the text 1026, and the images 1032 and 1034 can be generated based on the text 1036. In this process, the user only needs to input the summary text to obtain multiple candidate inputs without complicated manual labor. Further, the generation model can generate corresponding videos based on these candidate inputs. In this way, the powerful text processing ability and text-to-image conversion ability of the machine learning model 850 can be utilized to assist in generating longer videos including richer dynamic content.

[0068] According to an example implementation of the present disclosure, a machine learning model can be used to assist in subsequent video creation. For example, a video that has been generated can be input to the machine learning model, and the machine learning model is required to provide a subsequent content description of the video. The machine learning model can determine prompt information associated with another target video after the target video based on the target video, and the prompt information includes, for example, at least one image and another text. Subsequently, the generation model can generate another target video based on the at least one image and another text.

[0069] See also Fig.11 Describe in more detail the Fig.11 A block diagram 1100 is shown of a process for obtaining prompt information based on a machine learning model according to some implementations of the present disclosure. Fig.11 As shown, the generated target video 250 can be input to the machine learning model 850. At this time, the machine learning model 850 can output prompt information 1110, which may include an image 1112 for describing the first frame of the subsequent video, an image 1114 for describing the last frame of the subsequent video, and text 1116 for describing the content of the subsequent video. The user can interact with the prompt information 1110, and when confirming the prompt information 1110, the prompt information 1110 can be directly input into the generation model.

[0070] Alternatively and / or additionally, an editing control for editing the prompt information 1110 may be provided, for example, the user may modify the images 1112, 1114 and the text, and then the modified prompt information 1110 may be input to the generation model. In this way, in the process of automatically generating the story script with the help of the powerful processing capability of the machine learning model 850, the user is allowed to make adjustments according to his or her own expectations so as to obtain a video that better meets the user's expectations. In this way, the powerful reasoning capability of the machine learning model can be fully utilized to write subsequent storyboard scripts, thereby generating subsequent videos.

[0071] It should be understood that although the above describes an example in which the user specifies the first frame image, the last frame image and the text, and then generates a model. Alternatively and / or additionally, two images can be selected from an existing video and a new video can be created using a text description that is different from the content of the existing video. For example, if the user is not satisfied with the generated video of a polar bear walking on the ice, the text can be adjusted to "a polar bear is running on the ice surface". At this time, the content of the generated new video will be different from the existing video, for example, a video of a polar bear running will be generated. In this way, if the user is not satisfied with the existing video, the user can adjust the content of the video by simply modifying the text description, thereby achieving higher video generation performance. For example, in film and television production, the user can generate different versions of a certain storyboard and then select the version that best meets expectations.

[0072] The interactive process of using a pre-acquired generative model to generate a target video has been described above. More details about acquiring the generative model will be provided below. According to an exemplary implementation of the present disclosure, a large number of reference videos can be used to train the generative model. Fig.12 Describes more details about the training process. Fig.12 A block diagram 1200 is shown of a process for determining a generative model according to some implementations of the present disclosure. Fig.12 As shown, we can construct Fig.12 The generation model 220 shown in the figure can describe the association relationship between input data and output data. The input data can include, for example: text for describing the video content, a head image (for example, can be called a first image) for representing the head image in the video, and a leading image (for example, can be called a second image) for representing the tail image in the video. The output data can include, for example, a video.

[0073] According to an example implementation of the present disclosure, the generative model 220 may be established based on a variety of architectures currently known and / or to be developed in the future. In the training phase, sample data may be extracted from a reference video 1230 to train the generative model 220. A reference video 1230 including a plurality of reference images may be obtained, and a first reference image 1212 and a second reference image 1214 may be obtained from the plurality of reference images. Here, the first reference image 1212 (e.g., the first frame image) may be obtained from the head position of the reference video 1230, and the second reference image 1214 (e.g., the tail frame image, or an image within a predetermined range before the tail frame image) may be obtained from the tail position of the reference video 1230. Further, a reference text 1210 for describing the content of the reference video may be received.

[0074] In one example, the reference video 1230 may include a cat, and the cats in the first reference image 1212 and the second reference image 1214 may be located at different positions in the picture. At this time, the reference text 1210 may indicate "A cat is walking in the street". Further, the generation model 220 may be trained based on the first reference image 1212, the second reference image 1214, and the reference text 1210. At this time, the trained generation model 220 may generate a target video based on any one of the first image and the second image, and the text.

[0075] In the process of processing the last frame instruction, a variety of strategies can be used to regulate the influence of the last frame. During training, the last frame instruction can be randomly selected from the last three true value frames of the reference video. Noise can be introduced into the instruction to improve the robustness of the model to instruction dependence. During the training phase, the last frame instruction can be randomly discarded with a certain probability. Accordingly, a simple and effective inference strategy is proposed. For example, in the first τ denoising steps of the reverse process of the diffusion model, the last frame instruction can be used to guide the video generation towards the desired end state. Subsequently, the last frame instruction can be discarded in the remaining steps, allowing the model to generate more temporally coherent video content. The strength of the last frame guidance can be adjusted by τ.

[0076] It should be understood that Fig.12 The process of obtaining the generation model 220 is only schematically shown. Alternatively and / or additionally, multiple reference videos may be obtained and corresponding data may be extracted from the multiple reference videos, thereby continuously updating the generation model 220 in an iterative manner. For example, reference videos of different objects performing different actions may be obtained, thereby enabling the generation model 220 to master more abundant video generation knowledge.

[0077] Using the exemplary implementation of the present disclosure, in the process of determining the generation model 220, the first reference image 1212 can be used as the head image of the video, and the second reference image 1214 can be used as guiding data to determine the development direction of the story in the video. For example, the generated video can use the second reference image 1214 as the last frame. In this way, the generation model 220 can clearly grasp the changes in the content of each image in the video, which helps to generate a richer and more realistic dynamic video.

[0078] According to an example implementation of the present disclosure, the first frame image instruction can set the scene of video generation (and establish a character). The first frame also enables the model to generate a continuous video. In this case, the model uses the last frame of the previous video as the first frame instruction for the subsequent video to generate the subsequent video. Further, the last frame instruction describing the end state of the video can serve as an additional control mechanism. In this way, the alignment between the user's expectation and the text can be enhanced (that is, making the generated video more in line with the user's expectation), and the model can be enabled to construct complex shots, ultimately generating rich video content, thereby ensuring diversity and coherence.

[0079] Generally speaking, <text (文本), first frame (首帧), last frame (末帧)> instructions can be used as conditions. Given the above three instructions, the model can focus on learning the dynamics of people, animals, objects, and other entities in the world during the training phase; and in inference, the model can "generalize" the learned laws of motion in the physical world to areas not covered during training, such as realizing the movement of anime characters, special effects shots, and so on.

[0080] Specifically, the above information can be integrated into the diffusion model. For example, the text information is encoded by a pre-trained text encoder and then embedded into the diffusion model using a cross-attention mechanism. The image instruction is encoded by a pre-trained variational autoencoder (VAE) encoder and connected to the perturbed video content or Gaussian noise as the input of the diffusion model. During the training process, the first frame image of the video can be directly used as the first frame instruction to force the model to strictly comply with the instruction, thereby maintaining the continuity between consecutive videos. During the inference process, the instruction can be conveniently obtained from the text-to-image model or directly provided by the user.

[0081] According to an example implementation of the present disclosure, to determine the first reference image, the reference image located at the head of the reference video can be determined as the first reference image. In other words, the first frame image in the reference video can be used as the first reference image. In this way, the process of constructing training data can be simplified, and training data can be obtained quickly and effectively. The second reference image can be determined based on various methods. For example, the last frame image in the reference video can be used as the second reference image. Alternatively and / or additionally, to introduce a perturbation factor into the training data, the second reference image can be determined from a set of reference images within a predetermined range located at the tail of the reference video.

[0082] Assuming that the reference video 230 includes multiple reference images, the predetermined range may represent the last 3 frames (or other number) at the end of the reference video. Assuming that the reference video includes N image frames, any one of the Nth frame, the N-1th frame, and the N-2th frame in the reference video may be used as the second reference image. At this time, the second reference image may include any one of the last three reference images. In this way, the influence of the last frame on the training process may be adjusted to a certain extent, thereby obtaining a more coherent video.

[0083] See also Fig.13 More information describing the generative model, Fig.13 A block diagram 1300 of a process for training a generative model according to some implementations of the present disclosure is shown. Fig.13 As shown, the generation model 220 may include: an encoder model 1310, a connection model 1320, a diffusion model 1330, and a decoder model 1340. Here, the encoder model 1310 and the decoder model 1340 may have a predetermined structure and parameters. During the subsequent update of the parameters of the generation model 220, the parameters of the encoder model 1310 and the decoder model 1340 will not be updated. According to an example implementation of the present disclosure, a VAE and a corresponding decoder may be used.

[0084] exist Fig.13 In the example, the generation model 220 can be obtained based on the first reference image 1212, the second reference image 1214, and the reference text 1210. Specifically, the encoder model 1310 can be used to determine the first reference feature 1312 (for example, represented as Z) of the reference video 1230, and the first reference feature 1312 can include multiple reference image features of multiple reference images. Further, the encoder model 1310 can be used to determine the second reference feature 1314 (for example, represented as c image ), the second reference feature includes a first reference image feature of the first reference image and a second reference image feature of the second reference image.

[0085] According to an example implementation of the present disclosure, the first reference feature 1312 may include multiple dimensions, and each dimension may correspond to a reference image, that is, a feature from a reference image (having a predetermined dimension, for example, Where C represents the number of channels, H represents the image height, and W represents the image width). The first reference feature and the second reference feature may have the same dimension. In this case, the number of dimensions is equal to the number of images in the reference video. For example, the reference video may be specified to include 16 (or other number of) images.

[0086] See also Fig.14 Describes more details about the identified features. Fig.14 A block diagram 1400 is shown of a process for determining reference features of a reference video according to some implementations of the present disclosure. Fig.14 As shown, each reference image in the reference video 1230 may be processed one by one. For example, the encoder model may be used to extract features of the first reference image 1212 so as to generate a first reference image feature. The first reference image feature may be placed at the first position (i.e., position 1410) of the first reference feature 1312. Subsequently, the encoder model may be used to extract reference image features of another reference image after the first reference image 1212, and the reference image feature may be placed at the second position of the first reference feature 1312, and so on. Each reference image in the reference video 1230 may be continuously processed until the second reference feature of the last reference image (i.e., the second reference image 1214) in the reference video 1230 is placed at the last position (i.e., position 1412) of the first reference feature 1312.

[0087] Similarly, the encoder model 1310 may be used to determine the second reference feature of the reference video. The encoder model 1310 may be used to extract the features of the first reference image 1212 to generate the first reference image feature. The first reference image feature may be placed at the first position (i.e., position 1420) of the second reference feature 1314. The encoder model 1310 may be used to place the second reference image feature of the last reference image in the reference video 1230 (i.e., the second reference image 1214) at the last position 1422 of the second reference feature 1314.

[0088] like Fig.14 As shown, the position of the first reference image feature in the first reference feature 1312 (i.e., the first position) corresponds to the position of the first reference image in the reference video (i.e., the first position), and the position of the second reference image feature in the second reference feature 1314 (i.e., the last position) corresponds to the position of the second reference image in the reference video (i.e., the last position). In this way, the first reference feature and the second reference feature can be easily aligned in the subsequent connection process, thereby facilitating the improvement of the accuracy of the diffusion model.

[0089] According to an exemplary implementation of the present disclosure, features at other positions other than the first position and the second position in the second reference feature may be set to empty. Specifically, Fig.14 Position 1430 in the middle may be set to be empty, for example, it may be filled with data "0". In this way, the second reference feature may adjust the influence of the first reference image and the second reference image on the video generation process, so that the generation model may learn more about the conversion relationship between the first reference image and the second reference image.

[0090] Specifically, VAE can be used to encode the reference video and obtain the first reference feature 1312, that is, each frame in the reference video is encoded using VAE. VAE can be used to encode the first reference image and the second reference image, and then the position of the intermediate frame is padded with 0 to obtain the second reference feature 1314 (the dimension is the same as the first reference feature 1312).

[0091] return Fig.13 , a diffusion model 1330 may be determined based on the first reference feature 1312, the second reference feature 1314, and the reference text 1210. Specifically, noise may be added to the first reference feature 1312 and the second reference feature 1314, respectively, based on the principle of the diffusion model (for example, different degrees of noise may be added to the two features in different ways), thereby generating a first noise reference feature 1312' and a second noise reference feature 1314'. Further, a connection model 1320 may be used to connect the first noise reference feature 1312' and the second noise reference feature 1314', and generate a noise reference feature 1322. Further, corresponding training data may be constructed based on the principle of the diffusion model, thereby updating the parameters of the diffusion model 1330.

[0092] According to an example implementation of the present disclosure, the diffusion model 1330 can be implemented based on a variety of structures currently known and / or to be developed in the future. The diffusion process of the diffusion model 1330 involves a forward noise process and a reverse denoising process. In the forward noise process, noise data can be added to the features in multiple steps. For example, the initial feature can be represented as Z_0. In each step, noise data can be continuously added, the noise feature of step t can be represented as Z_t, and the noise feature of step t+1 can be represented as Z_(t+1). For example, noise data can be added to Z_t at step t. Then, random Gaussian data can be obtained at step T.

[0093] In the reverse denoising process, the inverse process of the above-mentioned denoising process can be performed in multiple steps to gradually obtain the initial features. Specifically, the random Gaussian data and the reference text and the corresponding second reference features can be input into the diffusion model so that the denoising process can be performed step by step in multiple steps. For example, in the t-th step, a portion of the noise in the noise feature Z_t can be removed to form a noise feature Z_(t-1) that is relatively clean relative to the noise feature Z_t. Each denoising process can be performed in an iterative manner to reversely derive the initial features (i.e., the video features without the addition of noise).

[0094] According to an exemplary implementation of the present disclosure, in the process of determining the diffusion model, the diffusion model 1330 is used to determine the reconstruction feature 1332 (for example, represented as ). Further, a loss function is constructed based on the difference between the reconstructed features and the reference features, and then the diffusion model 1330 is updated using the loss function.

[0095] According to an example implementation of the present disclosure, a 2D UNet (U-type network) can be used to implement a diffusion model. The model can be constructed by spatial downsampling and then spatial upsampling by inserting jump connections. Specifically, the model may include two basic blocks, namely a 2D convolution block and a 2D attention. The 2D UNet is extended to 3D by inserting a temporal layer, where a 1D convolution layer follows the 2D convolution layer along the temporal dimension, and a 1D attention layer follows the 2D attention layer along the temporal dimension. The model can be trained with images and videos to maintain high-fidelity spatial generation capabilities and disable 1D temporal operations on image inputs. Bidirectional self-attention can be used in all attention layers. The text instructions are encoded using a pre-trained text encoder, and the embedded c text By injecting the cross-attention layer in UNet, the hidden state is used as the query, c text as key and value.

[0096] Regarding image command injection, the image commands of the first and last frames can be combined with text guidance. Given the image commands on the first and last frames, for example, represented as {I first ,I last}, the above representation can be encoded into the same latent space as the input space of the diffusion model, resulting in {f first ,f last},in To insert instructions without losing temporal position information, the final image condition can then be constructed as in Condition c image Then, with the noised latent features z along the channel dimension t connections, which are used as input to the diffusion model.

[0097] According to an exemplary implementation of the present disclosure, a first reference image, a second reference image, and a reference text can be determined from a large number of reference videos, and then the first reference feature Z, the second reference feature c of each reference video can be determined using the method described above. image And the corresponding reconstruction features Here, the reconstruction feature is determined by the diffusion model 1330 and includes the influence of the forward noise process. At this time, the reconstruction feature Z can be based on the first reference feature Z and the corresponding reconstruction feature The loss function is constructed by taking the difference between .

[0098] A large number of reference videos can be continuously used to iteratively update the diffusion model 1330 until the expected stopping condition is met. It should be understood that in this process, no manual annotation data is required, but the first frame, the last frame and the corresponding text description of the reference video can be directly used as training data. In this way, the workload of obtaining manual annotation data can be eliminated, thereby improving the efficiency of obtaining the generation model 220.

[0099] It should be understood that although the second reference feature c is shown above image An example of simultaneously including relevant feature information of the first reference image and the second reference image, alternatively and / or additionally, in some cases the second reference feature c image Only the feature information of the first reference image may be included. Specifically, the second reference image feature in the second reference feature may be set to null according to a predetermined condition. In other words, Fig.14 The feature at position 1422 can be removed, for example, the value at this position can be set to 0.

[0100] By using the example implementation of the present disclosure, the second reference image features can be removed according to a predetermined ratio (e.g., 25% or other values). In this way, the application scenario of the second reference image may not be considered in the process of updating the generated model, thereby making the generated model suitable for more application scenarios, thereby improving the accuracy of the generated model in various situations.

[0101] According to an example implementation of the present disclosure, the generative model 220 may be trained based on a variety of methods currently known and / or to be developed in the future, so that the generative model 220 may grasp the association relationship between the first image, the second image, the text describing the video content, and the corresponding video.

[0102] According to an example implementation of the present disclosure, the generation model 230 further includes a decoder model 1340. After the generation model 220 has been obtained, the requirements for generating a video can be input to the generation model 220 to generate a corresponding target video. According to an example implementation of the present disclosure, the first image and text can be input to the generation model, and then the generation model is used to generate a corresponding target video. According to an example implementation of the present disclosure, the first image, the second image and text can be input to the generation model, and then the generation model is used to generate a corresponding target video. Alternatively and / or additionally, the text can even be empty, and the target video can still be generated at this time.

[0103] In order to obtain a target video that is more in line with the expectation, a first image and a second image for generating the target video, and a text describing the content of the target video can be received. Then, the reconstruction features of the target video can be generated based on the first image, the second image, and the text according to the trained diffusion model. Further, the target video can be generated based on the reconstruction features according to the decoder model. In this way, the diffusion model 1330 can be fully utilized to perform the reverse denoising process, so as to obtain the reconstruction features of the target video in a more accurate manner.

[0104] See also Fig.15 Describes more details about the generated video. Fig.15 A block diagram 1500 of a process for generating a target video according to some implementations of the present disclosure is shown. Fig.15 As shown, a first image 1512, a second image 1514, and a text 1510 for describing the image content can be received. Here, the first image 1512 may include, for example, an ice surface, so as to be used as the first frame of the target video 1530 to be generated. The second image 1514 may include, for example, an ice surface and a polar bear on the ice surface. The second image 1514 may be used as guiding information to generate the last frame of the target video 1530. At this time, the target video 1530 generated by the generation model 220 under the condition of the text 1510, and the content of the video is "A polar bear is walking on the ice surface".

[0105] The first image 1512, the second image 1514, and the text 1510 may be input to the generation model 220. At this point, the generation model 220 will output a target video 1530, and the first frame of the target video 1530 corresponds to the first image 1512, and the last frame corresponds to the second image 1514. It should be understood that since the target video 1530 has not yet been generated, a noise feature is used as the first feature 1520 of the target video 1530. According to an example implementation of the present disclosure, the first feature 1520 may be determined using Gaussian noise.

[0106] Further, the second feature 1522 of the target video 1530 may be generated based on the first image 1512 and the second image 1514 according to the encoder model 1310. For example, the encoder model 1310 may be used to respectively determine the first image feature of the first image 1512 and the second image feature of the second image 1514. Then, the first image feature may be placed at the first position of the second feature 1522, and the second image feature may be placed at the last position of the second feature 1522. It should be understood that the process of generating the second feature 1522 here is similar to that in the training stage, and thus will not be described in detail.

[0107] In the case where the first feature 1520 and the second feature 1522 have been determined, the reconstructed feature 1526 of the target video can be obtained according to the diffusion model using the first feature 1520, the second feature 1522 and the text 1510. Similar to the process of determining the reconstructed feature in the training phase, the first noise feature 1520' and the second noise feature 1522' can be obtained, and the connection model 1320 is used to connect the first noise feature 1520' and the second noise feature 1522' to form the noise feature 1524. Further, the inverse denoising phase of the diffusion model 1330 can be used to generate the reconstructed feature 1526.

[0108] It should be understood that, since the diffusion model 1330 has been trained based on the above loss function, the reconstructed features 1526 generated by the diffusion model 1330 can accurately describe various aspects of the target video 1530. Further, the reconstructed features 1526 can be input to the decoder model 1340, and the decoder model 1340 will output the corresponding target video 1530, and the target video 1530 will accurately match the first image 1512, the second image 1514 and the text 1510 in the input data.

[0109] It should be understood that in the process of generating a video, the reverse denoising process of the diffusion model 1330 can be performed in an iterative manner, and the reverse denoising process can be performed iteratively in multiple steps. Assuming that the total number of multiple steps is T, in the first step (t=1), the first feature 1520, the second feature 1522 and the text 1510 can be used to obtain the reconstructed feature 1526 of the target video.

[0110] Then, in a second step (t=2) after the first step in the multiple steps, the first feature can be set as the reconstruction feature of the target video, that is, using To replace Z, and then perform subsequent processing. In the second step, the diffusion model can use the first feature, the second feature and the text to obtain new reconstructed features. In the subsequent third step (t=3), similar processing can be performed until the predetermined Tth step is reached. At this point, the reverse denoising process ends, and the output reconstructed features are the features that describe the target video 1530 to be generated. Using the example implementation of the present disclosure, noise can be gradually removed in multiple reverse denoising steps to generate more accurate reconstructed features.

[0111] According to an example implementation of the present disclosure, in the process of generating a target video, a user may be allowed to specify the degree of influence of the second image. For example, a control for inputting an influence factor may be provided in a page, and the influence factor may be expressed, for example, as a percentage. The multiple steps for calling a diffusion model in a generation model may be divided into a first stage and a second stage based on the influence factor. In the first stage and the second stage, the diffusion model may determine the reconstruction features of the target video based on the first image and the second image, and then the decoder model in the generation model may generate the target video based on the reconstruction features. Using the example implementation of the present disclosure, the influence factor may be used to control the degree of influence of the second image 1514.

[0112] Specifically, in each step in the first stage, the guiding effect of the second image 1514 may be considered; and in each step in the second stage, the guiding effect of the second image 1514 may not be considered. In the process of determining the reconstructed features of the target video, in the first stage, the diffusion model determines the intermediate reconstructed features associated with the target video based on the first image and the second image; and in the second stage, the diffusion model determines the reconstructed features based on the first image and the intermediate reconstructed features. In other words, in the second stage, the influence of the second image 1514 may be removed from the second feature, that is, the portion of the second feature corresponding to the second image may be set to empty. Fig.15 In the example, the feature at the last position in the second feature 1522 can be set to 0.

[0113] Specifically, in the first τ steps of all T denoising steps, the last frame condition can be applied to guide the video generation towards the desired end state, and discarded in subsequent steps to generate a more credible and temporally consistent video:

[0114]

[0115] In the above formula, τ determines the degree of model dependence on the last frame instruction, and adjusting τ can adapt to various applications. For example, the model can generate high-quality videos without the last frame condition (i.e., τ = 0). In addition, classifier-free guidance can be applied, which mixes the model score estimates under text prompts and without text prompts.

[0116] According to an example implementation of the present disclosure, the number of steps in which the second image 1514 is not considered may be determined according to a predetermined ratio. For example, in the step of τ=T*40% (or other values) among all T steps, the influence of the second image 1514 may be ignored. Assuming T=50, in the first 30 (50*(1-40%)) steps, the complete second feature 1522 is used; further, in the last 20 (50*40%) steps, the value at the last position in the second feature 1522 may be set to 0. In this way, the role of the first image 1512 and the text 1510 in generating the target video may be adjusted, thereby making the generated target video 1530 more in line with expectations.

[0117] In summary, according to an example implementation of the present disclosure, a new video generation architecture PixelDance based on a diffusion model is proposed, which combines image instructions of the first frame and the last frame with text guidance. The corresponding training process and reasoning process can be implemented based on the PixelDance architecture. Using the example implementation of the present disclosure, flexible control of the video generation process and last frame guidance of different strengths can be provided.

[0118] The proposed technical solution can be implemented on a variety of datasets, and experiments show that the videos generated by PixelDance have strong advantages in synthesizing complex scenes and / or actions. Specifically, the generative model can be trained using various available datasets, where each video in the dataset is associated with a paired text that usually provides a rough description and shows a weak correlation with the video content. Since image instructions help learn complex video distributions, PixelDance can make full use of various datasets without the need for labeled data, demonstrating its excellent ability in generating videos with complex scenes and motion.

[0119] Example Process

[0120] Fig.16A flow chart of a method 1600 for generating a video according to some implementations of the present disclosure is shown. At block 1610, an image for describing at least one of a head image and a tail image of a target video is received. At block 1620, text for describing the content of the target video is received. At block 1630, a target video is generated based on the image and the text according to a generation model.

[0121] According to an example implementation of the present disclosure, the plurality of images include a first image for describing a head image among the plurality of target images and a second image for describing a tail image among the plurality of target images.

[0122] According to an example implementation of the present disclosure, the second image is received via at least any one of: an import control for importing an address of the second image; a drawing control for drawing the second image; and an editing control for editing the first image to generate the second image.

[0123] According to an example implementation of the present disclosure, the method further includes: providing multiple images in a target video; in response to receiving an interaction with a third image among the multiple images, determining a head image for describing another target video after the target video; obtaining another text for describing the content of the other target video and a fourth image for describing the tail image of the other target video; and generating another target video based on the third image, the fourth image and the another text according to a generation model.

[0124] According to an example implementation of the present disclosure, images and text are determined based on: receiving multiple images, where the multiple images are respectively used to describe multiple key frames in a long video; receiving a long text for describing the content of the long video; and determining the image and text based on the multiple images and the long text according to a machine learning model.

[0125] According to an example implementation of the present disclosure, an image and text are determined based on: receiving a summary text for describing a long video; determining, based on the summary text, multiple texts respectively used to describe the contents of multiple video clips in the long video according to a machine learning model; generating a target image based on a target text among the multiple texts according to the machine learning model; and taking the target image as an image and the target text as text.

[0126] According to an example implementation of the present disclosure, the method further includes: determining, based on the target video according to a machine learning model, prompt information associated with another target video after the target video, the prompt information including at least one image and another text; and generating, based on the generation model, another target video based on the at least one image and another text.

[0127] According to an example implementation of the present disclosure, generating a target video further includes: receiving an influence factor for specifying the degree of influence of a second image; and dividing a plurality of steps for calling a diffusion model in a generation model into a first stage and a second stage based on the influence factor; in the first stage and the second stage, determining reconstruction features of the target video according to the diffusion model and based on the first image and the second image; and generating the target video based on the reconstruction features using a decoder model in the generation model.

[0128] According to an example implementation of the present disclosure, determining reconstruction features of a target video includes: in a first stage, determining intermediate reconstruction features associated with the target video according to a diffusion model and based on a first image and a second image; and in a second stage, determining reconstruction features according to the diffusion model and based on the first image and the intermediate reconstruction features.

[0129] According to an example implementation of the present disclosure, a generative model is determined based on: determining a first reference image and a second reference image from a plurality of reference images in a reference video; receiving a reference text for describing the reference video; and based on the first reference image, the second reference image and the reference text, obtaining a generative model, the generative model being used to generate a target video based on any one of the first image and the second image and the text.

[0130] According to an exemplary implementation of the present disclosure, the first reference image is located at the head of the reference video, and the second reference image is located within a predetermined range at the tail of the reference video.

[0131] According to an example implementation of the present disclosure, a generation model includes an encoder model and a diffusion model, and obtaining the generation model based on a first reference image, a second reference image, and a reference text includes: using the encoder model to determine a first reference feature of a reference video, the first reference feature including multiple reference image features of multiple reference images; using the encoder model to determine a second reference feature of the reference video, the second reference feature including a first reference image feature of the first reference image and a second reference image feature of the second reference image; and determining the diffusion model based on the first reference feature, the second reference feature, and the reference text.

[0132] According to an example implementation of the present disclosure, a first position of the first reference image feature in the second reference feature corresponds to a position of the first reference image in the reference video, and a second position of the second reference image feature in the second reference feature corresponds to a position of the second reference image in the reference video.

[0133] According to an example implementation of the present disclosure, the dimension of the second reference feature is equal to the dimension of the first reference feature, and features at positions other than the first position and the second position in the second reference feature are set to empty.

[0134] According to an exemplary implementation of the present disclosure, the method further includes: setting the second reference image feature to empty according to a predetermined condition.

[0135] According to an example implementation of the present disclosure, determining a diffusion model based on a first reference feature, a second reference feature, and a reference text includes: performing noise processing on the first reference feature and the second reference feature to generate a first noise reference feature and a second noise reference feature, respectively; connecting the first noise reference feature and the second noise reference feature to generate a noise reference feature of a reference video; using the diffusion model, determining a reconstructed feature of the reference video based on the noise reference feature and the reference text; and updating the diffusion model based on the difference between the reconstructed feature and the reference feature.

[0136] According to an example implementation of the present disclosure, the head image is the first image frame in the target video, and the tail image is the last image frame in the target video.

[0137] Example devices and equipment

[0138] Fig.17 A block diagram of a device 1700 for generating a video according to some implementations of the present disclosure is shown. The device 1700 includes: an image receiving module 1710 configured to receive a plurality of images respectively used to describe a plurality of target images in a target video; a text receiving module 1720 configured to receive text used to describe the content of the target video; and a generating module 1730 configured to generate a target video based on the plurality of images and the text according to a generating model.

[0139] According to an example implementation of the present disclosure, the plurality of images include a first image for describing a head image among the plurality of target images and a second image for describing a tail image among the plurality of target images.

[0140] According to an example implementation of the present disclosure, the second image is received via at least any one of: an import control for importing an address of the second image; a drawing control for drawing the second image; and an editing control for editing the first image to generate the second image.

[0141] According to an example implementation of the present disclosure, the device further includes: a providing module configured to provide multiple images in a target video; a determining module configured to determine, in response to receiving an interaction with a third image among the multiple images, a head image for describing another target video after the target video; an acquiring module configured to acquire another text for describing the content of the other target video and a fourth image for describing the tail image of the other target video; and a generating module further configured to generate another target video based on the third image, the fourth image and the another text according to a generating model.

[0142] According to an example implementation of the present disclosure, the device further includes: a key frame receiving module configured to receive multiple images, where the multiple images are respectively used to describe multiple key frames in a long video; a long text receiving module configured to receive a long text for describing the content of the long video; and a first calling module configured to determine images and texts based on the multiple images and the long text according to a machine learning model.

[0143] According to an example implementation of the present disclosure, the device further includes: a summary receiving module configured to receive a summary text for describing a long video; a second calling module configured to determine, based on the summary text according to a machine learning model, multiple texts respectively used to describe the contents of multiple video clips in the long video; a third calling module configured to generate a target image based on a target text among the multiple texts according to the machine learning model; and an identification module configured to take the target image as an image and the target text as text.

[0144] According to an example implementation of the present disclosure, the device further includes: a fourth calling module, configured to determine prompt information associated with another target video after the target video based on the target video according to a machine learning model, the prompt information including at least one image and another text; and the generation module is further configured to generate another target video based on the at least one image and another text according to the generation model.

[0145] According to an example implementation of the present disclosure, the generation module further includes: an influence factor receiving module, configured to receive an influence factor for specifying the degree of influence of the second image; and a division module, configured to divide a plurality of steps for calling a diffusion model in the generation model into a first stage and a second stage based on the influence factor; a reconstruction module, configured to determine, in the first stage and the second stage, a reconstruction feature of a target video according to the diffusion model and based on the first image and the second image; and a video generation module, configured to generate a target video based on the reconstruction feature using a decoder model in the generation model.

[0146] According to an example implementation of the present disclosure, a reconstruction module includes: a first determination module configured to determine, in a first stage, an intermediate reconstruction feature associated with a target video according to a diffusion model and based on a first image and a second image; and a second determination module configured to determine, in a second stage, a reconstruction feature according to the diffusion model and based on the first image and the intermediate reconstruction feature.

[0147] According to an example implementation of the present disclosure, a model determination module is further configured to determine a generation model, including: an image determination module, configured to determine a first reference image and a second reference image from a plurality of reference images in a reference video; a text receiving module, configured to receive a reference text for describing the reference video; and a model acquisition module, configured to acquire a generation model based on the first reference image, the second reference image and the reference text, the generation model being used to generate a target video based on any one of the first image and the second image and the text.

[0148] According to an exemplary implementation of the present disclosure, the first reference image is located at the head of the reference video, and the second reference image is located within a predetermined range at the tail of the reference video.

[0149] According to an example implementation of the present disclosure, a generation model includes an encoder model and a diffusion model, and a model acquisition module includes: a first feature determination module, configured to determine a first reference feature of a reference video using the encoder model, the first reference feature including multiple reference image features of multiple reference images; a second feature determination module, configured to determine a second reference feature of the reference video using the encoder model, the second reference feature including a first reference image feature of the first reference image and a second reference image feature of the second reference image; and a diffusion model determination module, configured to determine the diffusion model based on the first reference feature, the second reference feature, and a reference text.

[0150] According to an example implementation of the present disclosure, a first position of the first reference image feature in the second reference feature corresponds to a position of the first reference image in the reference video, and a second position of the second reference image feature in the second reference feature corresponds to a position of the second reference image in the reference video.

[0151] According to an example implementation of the present disclosure, the dimension of the second reference feature is equal to the dimension of the first reference feature, and features at positions other than the first position and the second position in the second reference feature are set to empty.

[0152] According to an exemplary implementation of the present disclosure, the apparatus further includes: a setting module configured to set the second reference image feature to empty according to a predetermined condition.

[0153] According to an example implementation of the present disclosure, a diffusion model determination module includes: a noiseization module, configured to perform noiseization processing on a first reference feature and a second reference feature, respectively, to generate a first noise reference feature and a second noise reference feature; a connection module, configured to connect the first noise reference feature and the second noise reference feature to generate a noise reference feature of a reference video; a reconstruction module, configured to determine a reconstruction feature of a reference video based on the noise reference feature and a reference text using a diffusion model; and an update module, configured to update the diffusion model based on the difference between the reconstructed feature and the reference feature.

[0154] According to an example implementation of the present disclosure, the head image is the first image frame in the target video, and the tail image is the last image frame in the target video.

[0155] Fig.18 1 shows a block diagram of a device 1800 capable of implementing multiple implementations of the present disclosure. It should be understood that Fig.18 The computing device 1800 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the implementations described herein. Fig.18 The computing device 1800 shown may be used to implement the methods described above.

[0156] like Fig.18 As shown, computing device 1800 is in the form of a general-purpose computing device. Components of computing device 1800 may include, but are not limited to, one or more processors or processing units 1810, memory 1820, storage device 1830, one or more communication units 1840, one or more input devices 1850, and one or more output devices 1860. Processing unit 1810 may be an actual or virtual processor and is capable of performing various processes according to a program stored in memory 1820. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to increase the parallel processing capabilities of computing device 1800.

[0157] The computing device 1800 typically includes a plurality of computer storage media. Such media may be any available media accessible to the computing device 1800, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 1820 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 1830 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data (e.g., training data for training) and may be accessed within the computing device 1800.

[0158] The computing device 1800 may further include additional removable / non-removable, volatile / non-volatile storage media. Fig.18 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. Memory 1820 may include a computer program product 1825 having one or more program modules that are configured to perform various methods or actions of various implementations of the present disclosure.

[0159] The communication unit 1840 enables communication with other computing devices via a communication medium. Additionally, the functions of the components of the computing device 1800 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 1800 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0160] Input device 1850 may be one or more input devices, such as a mouse, keyboard, tracking ball, etc. Output device 1860 may be one or more output devices, such as a display, a speaker, a printer, etc. Computing device 1800 may also communicate with one or more external devices (not shown) through communication unit 1840 as needed, such as storage devices, display devices, etc., communicate with one or more devices that allow a user to interact with computing device 1800, or communicate with any device that allows computing device 1800 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0161] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, the computer program product is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0162] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0163] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0164] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0165] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0166] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for generating a video, comprising: receiving a plurality of images respectively used to describe a plurality of target images in a target video, the plurality of images comprising a first image and a second image; receiving text describing the content of the target video; Generate the target video based on the multiple images and the text according to the generation model; In response to receiving an interaction with a third image in the target video, determining a head image for describing another target video after the target video; acquiring another text for describing the content of the another target video and a fourth image for describing the tail image of the another target video; and According to the generation model, the another target video is generated based on the third image, the fourth image and the another text. 2 . The method according to claim 1 , wherein the plurality of images include the first image for describing a head image among the plurality of target images and the second image for describing a tail image among the plurality of target images.

3. The method of claim 2, wherein the second image is received via at least any one of: an import control for importing the address of the second image; a drawing control for drawing the second image; and An editing control for editing the first image to generate the second image.

4. The method of claim 1, wherein the image and the text are determined based on: Receiving a plurality of images, wherein the plurality of images are respectively used to describe a plurality of key frames in a long video; receiving a long text for describing the content of the long video; as well as According to a machine learning model, the image and the text are determined based on the multiple images and the long text.

5. The method of claim 1, wherein the image and the text are determined based on: Receive summary text describing the long video; Determine, based on the summary text and according to the machine learning model, a plurality of texts respectively used to describe the contents of a plurality of video clips in the long video; generating a target image based on a target text among the plurality of texts according to a machine learning model; as well as The target image is taken as the image and the target text is taken as the text.

6. The method according to claim 1, further comprising: Determining, based on the target video, prompt information associated with another target video after the target video according to the machine learning model, wherein the prompt information includes at least one image and another text; as well as According to the generation model, the another target video is generated based on the at least one image and the another text.

7. The method according to claim 2, wherein generating the target video further comprises: receiving an influence factor for specifying a degree of influence of the second image; as well as dividing a plurality of steps for calling a diffusion model in the generation model into a first stage and a second stage based on the impact factor; In the first stage and the second stage, determining the reconstruction features of the target video according to the diffusion model and based on the first image and the second image; as well as The target video is generated based on the reconstructed features using a decoder model in the generative model.

8. The method of claim 7, wherein determining the reconstructed features of the target video comprises: In the first stage, determining intermediate reconstructed features associated with the target video according to the diffusion model and based on the first image and the second image; as well as In the second stage, the reconstruction features are determined according to the diffusion model and based on the first image and the intermediate reconstruction features.

9. The method of claim 1, wherein the generative model is determined based on: determining a first reference image and a second reference image from a plurality of reference images in a reference video; receiving a reference text for describing the reference video; and Based on the first reference image, the second reference image and the reference text, a generation model is acquired, where the generation model is used to generate a target video based on any one of the first image and the second image and the text. 10 . The method according to claim 9 , wherein the first reference image is located at a head of the reference video, and the second reference image is located within a predetermined range of a tail of the reference video.

11. The method of claim 9, wherein the generative model comprises an encoder model and a diffusion model, and acquiring the generative model based on the first reference image, the second reference image, and the reference text comprises: Determining, using the encoder model, a first reference feature of the reference video, the first reference feature comprising a plurality of reference image features of the plurality of reference images; Determine a second reference feature of the reference video using the encoder model, the second reference feature comprising a first reference image feature of the first reference image and a second reference image feature of the second reference image; as well as The diffusion model is determined based on the first reference feature, the second reference feature, and the reference text.

12. The method of claim 11, wherein a first position of the first reference image feature in the second reference feature corresponds to a position of the first reference image in the reference video, and a second position of the second reference image feature in the second reference feature corresponds to a position of the second reference image in the reference video.

13. The method according to claim 12, wherein the dimension of the second reference feature is equal to the dimension of the first reference feature, and the features at other positions in the second reference feature other than the first position and the second position are set to empty.

14. The method according to claim 13, further comprising: The second reference image feature is set to empty according to a predetermined condition.

15. The method of claim 11, wherein determining the diffusion model based on the first reference feature, the second reference feature, and the reference text comprises: Performing noise processing on the first reference feature and the second reference feature respectively to generate a first noise reference feature and a second noise reference feature; connecting the first noise reference signature and the second noise reference signature to generate a noise reference signature of the reference video; Determining, using the diffusion model, reconstruction features of the reference video based on the noise reference features and the reference text; and The diffusion model is updated based on the difference between the reconstructed signature and the reference signature.

16. The method of claim 2, wherein the head image is a first image frame in the target video, and the tail image is a last image frame in the target video.

17. An apparatus for generating a video, comprising: An image receiving module is configured to receive a plurality of images respectively used to describe a plurality of target images in a target video, wherein the plurality of images include a first image and a second image; A text receiving module, configured to receive text for describing the content of the target video; A generation module, configured to generate the target video based on the multiple images and the text according to a generation model; a determination module configured to determine, in response to receiving an interaction with a third image in the target video, a head image for describing another target video subsequent to the target video; An acquisition module configured to acquire another text for describing the content of the another target video and a fourth image for describing the tail image of the another target video; as well as The generation module is further configured to generate the another target video based on the third image, the fourth image and the another text according to the generation model.

18. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 16 when executed by the at least one processing unit.

19. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method according to any one of claims 1 to 16.

20. A computer program product comprising a computer program, wherein the computer program implements the method according to any one of claims 1 to 16 when executed by a processor.

Images