Method, apparatus, device and medium for generating video

Abstract

The present disclosure provides methods, apparatus, devices, and media for generating video. In one embodiment, the method includes receiving a plurality of images for respectively describing a plurality of target images in a target video; receiving text for describing the content of the target video; and generating a target video based on the plurality of images and the text according to a generative model. Using exemplary embodiments of the present disclosure, the received plurality of images can be used as guide data to determine a story direction in the video, which can further help generate a richer, more compelling, dynamic video.

Description

[Technical field]

[0001] Exemplary embodiments of the present disclosure relate generally to computer vision, and more particularly to methods, apparatus, devices, and computer-readable storage media for generating videos utilizing machine learning models. [Background technology]

[0002] Machine learning technology has been widely used in many technical fields, and in the field of computer vision, many technical proposals have been proposed to automatically generate videos using machine learning models. For example, a corresponding video can be generated based on a pre-specified image and text for describing the content of the video. However, the screen in the currently generated video usually has poor dynamics, for example, the object in the video lacks obvious movement and dynamic effects, making it difficult to achieve a realistic motion visual effect. Therefore, it is expected to generate a dynamic video containing desired content in a simpler and more effective way. Summary of the Invention

[0003] In a first aspect of the present disclosure, a method for generating a video is provided, comprising: receiving a plurality of images for respectively describing a plurality of target images in a target video; receiving text for describing content of the target video; and generating the target video based on the plurality of images and the text according to a generative model.

[0004] In a second aspect of the present disclosure, an apparatus for generating a video is provided, the apparatus comprising: an image receiving module arranged to receive an image for describing at least one of a head image and a tail image of a target video, a text receiving module arranged to receive a text for describing a content of the target video, and a generating module arranged to generate the target video based on the plurality of images and the text according to a generative model.

[0005] In a third aspect of the present disclosure, an electronic device is provided, comprising at least one processing unit and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform the method of the first aspect of the present disclosure.

[0006] In a fourth aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, causes the processor to perform a method according to the first aspect of the present disclosure.

[0007] It should be understood that the contents described in this section are not intended to limit the main or important features of the embodiments of the present disclosure, and are not intended to limit the scope of the present disclosure. Other features of the present disclosure will be easily understood from the following description. [Brief description of the drawings]

[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent hereinafter by reference to the following detailed description taken in conjunction with the accompanying drawings, in which the same or similar reference numerals indicate the same or similar elements. [Figure 1] 1 shows a block diagram of a technical solution for generating a video. [Diagram 2] FIG. 2 shows a block diagram of a process for generating a video according to some embodiments of the present disclosure. [Figure 3A] FIG. 1 shows a block diagram for acquiring a second image according to some embodiments of the present disclosure. [Figure 3B] FIG. 1 shows a block diagram for acquiring a second image according to some embodiments of the present disclosure. [Figure 3C] FIG. 1 shows a block diagram for acquiring a second image according to some embodiments of the present disclosure. [Figure 4]FIG. 2 shows a block diagram of a process for generating a video according to some embodiments of the present disclosure. [Diagram 5] 1 illustrates a block diagram of a page for receiving input data for generating a subsequent video according to some embodiments of the present disclosure. [Figure 6] FIG. 2 illustrates a block diagram of a process for generating a follow-on video according to some embodiments of the present disclosure. [Figure 7] FIG. 1 illustrates a block diagram of a process for generating video based on a sketch in accordance with some embodiments of the present disclosure. [Figure 8] FIG. 1 illustrates a block diagram of a process for obtaining candidate inputs utilizing a machine learning model according to some embodiments of the present disclosure. [Figure 9] FIG. 2 illustrates a block diagram of multiple candidate inputs according to some embodiments of the present disclosure. [Figure 10] FIG. 1 illustrates a block diagram of a process for obtaining candidate inputs based on summary text utilizing a machine learning model according to some embodiments of the present disclosure. [Figure 11] FIG. 1 illustrates a block diagram of a process for obtaining a prompt message based on a machine learning model according to some embodiments of the present disclosure. [Figure 12] FIG. 1 illustrates a block diagram of an overview process for training a generative model according to some embodiments of the present disclosure. [Figure 13] FIG. 1 illustrates a block diagram of a detailed process for training a generative model according to some embodiments of the present disclosure. [Figure 14] 1 illustrates a block diagram of a process for determining reference features of a reference video according to some embodiments of the present disclosure. [Figure 15] FIG. 2 illustrates a block diagram of a process for generating a target video according to some embodiments of the present disclosure. [Figure 16] 1 illustrates a flowchart of a method for generating a video according to some embodiments of the present disclosure. [Figure 17] FIG. 2 shows a block diagram of an apparatus for generating video according to some embodiments of the present disclosure. [Figure 18] 1 shows a block diagram of a device in which numerous embodiments of the present disclosure can be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Although the accompanying drawings show several embodiments of the present disclosure, it should be understood that the present disclosure can be realized in various forms and should not be interpreted as being limited to the embodiments described herein, but rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the accompanying drawings and embodiments of the present disclosure are intended to play an exemplary role and are not intended to limit the protection scope of the present disclosure.

[0010] In describing the embodiments of the present disclosure, the term "including" and similar terms should be understood as non-exclusive inclusion, i.e., "including but not limited to." The term "based on" should be understood as "based at least in part on." The term "one embodiment" or "this embodiment" should be understood as "at least one embodiment." The term "some embodiments" should be understood as "at least some embodiments." Other explicit and implicit definitions may be included below. As used herein, the term "model" may indicate a relationship between each piece of data. For example, the above relationship may be obtained based on a number of currently known and / or future developed technical proposals.

[0011] It should be understood that any data related to the present technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of corresponding laws and regulations and related regulations.

[0012] It should be understood that before using the technical solutions disclosed in each embodiment of the present disclosure, the types, scope of use, usage scenarios, etc. of personal information related to the present disclosure should be notified to users by appropriate means in accordance with relevant laws and regulations, and user approval should be obtained.

[0013] For example, in response to receiving an unsolicited request from a user, a prompt message is sent to the user to explicitly prompt the user that the requested operation requires access to and use of the user's personal information, so that the user can autonomously choose whether or not to provide personal information to software or hardware, such as an electronic device, an application, a server, or a storage medium, that executes the operation of the technical solution of the present disclosure, based on the prompt message.

[0014] As an optional, but non-limiting embodiment, the manner in which the prompt message is sent to the user in response to receiving an unsolicited request from the user may be, for example, by a pop-up window in which the prompt message may be presented in text, and the pop-up window may further include a selection control for the user to select "agree" or "disagree" to providing personal information to the electronic device.

[0015] It should be understood that the above process of notification and obtaining user authorization is merely exemplary and is not intended to limit the embodiments of the present disclosure, and other methods that satisfy relevant laws and regulations may also be applied to the embodiments of the present disclosure.

[0016] The term "responsive to" as used herein refers to a state in which a corresponding event has occurred or a condition has been satisfied. It should be understood that there is not necessarily a strong correlation between the timing of the execution of a subsequent action executed in response to the event or condition and the time when the event has occurred or the condition has been satisfied. For example, in some cases, the subsequent action is executed immediately when the event occurs or the condition is satisfied, while in other cases, the subsequent action is executed a certain time after the event has occurred or the condition has been satisfied.

[0017] Example Environment

[0018] Machine learning technology has been widely used in many technical fields, and in the field of computer vision, it has been proposed to automatically generate videos using machine learning models. Traditional video generation methods focus on text-to-video generation or generation based on a single image. The generated video may contain object movements, but the movements are short-lasting and only have short-term dynamicity, so they lack dynamicity and cannot present the desired information.

[0019] Please refer to FIG. 1 for describing the generating method, which shows a block diagram 100 of a technical proposal for generating a video. As shown in FIG. 1, a machine learning model 120 is obtained, where the machine learning model 120 may be generated based on reference data in a pre-constructed training dataset. A text 110 may be used to specify the content of the video to be generated, and an image 112 may be used to specify the environment of the video (e.g., as an image of the first frame, or an image of another location). In this example, the text 110 may indicate, for example, "A cat is walking in the street". At this time, the machine learning model 120 may generate a video 130 having the content that a cat is walking on the street based on the text 110 and 112.

[0020] However, currently generated video screens often have difficulty in achieving realistic dynamic effects. Creating videos with high dynamic motion, complex imaging actions, visual effects, facial close-ups, or shot conversions faces formidable challenges. Conventional video generation methods focus on text-to-video generation, tend to generate videos with minimal motion range, and can only generate video segments with short length, and although the morphology of an object (e.g., a cat) in the video is similar to that of the image 112, the cat's motion may be stiff and the motion range may be small. Therefore, it is expected that dynamic videos containing desired contents can be generated more simply and efficiently.

[0021] Overview of generating videos

[0022] Videos require an additional time dimension and consist of many keyframes, making it difficult to accurately describe each keyframe in a simple language. In addition, the types of motion in videos are very diverse. Conventional approaches are not only source intensive, but also pose a great challenge for generative models. In order to understand complex text descriptions and generate matching videos, the size of the model grows dramatically and the amount of annotation data required also increases significantly.

[0023] In order to at least partially address the shortcomings of the prior art, according to an exemplary embodiment of the present disclosure, a technical proposal for video generation based on a diffusion model (e.g., called PixelDance) is proposed. In general, a machine learning architecture based on a diffusion model is proposed, which combines image instructions of the first and last frames in a video with text instructions for video generation. Experimental results can be summarized that the video generated using the technical proposal of the present disclosure exhibits excellent visual effects in synthesizing videos with complex scenes and complex movements.

[0024] It should be understood that the generative model here may be a pre-trained model, and the target video may be generated based on the model. Referring to FIG. 2 for explaining an overview according to an exemplary embodiment of the present disclosure, FIG. 2 shows a block diagram 200 of a process for generating a video according to some embodiments of the present disclosure. As shown in FIG. 2, a page 210 may be provided, and a controller may be provided in an area 230 for describing a plurality of images of a plurality of target images in a target video. It should be understood that the plurality of target images may be a plurality of target images in a target video having a time order. For example, a controller 232 may be provided for describing a previous target image, and a controller 234 may be provided for describing a subsequent target image. Furthermore, a controller 236 may be provided for receiving a text describing the content of the target video.

[0025] According to an exemplary embodiment of the present disclosure, the video generation process may be performed in a multilingual environment. Although the reference text in FIG. 2 is expressed in English, alternatively and / or additionally, the text may be expressed based on other languages ​​(e.g., other languages ​​such as Chinese, French, Japanese, etc.). Furthermore, the user may invoke the generative model 220 via the controller 242 to further generate the target video 250. Alternatively and / or additionally, the user may return to a previous page via the controller 240.

[0026] It should be understood that the generative model 220 here is obtained based on the combination of image instructions for the first and final frames and text instructions for video generation. By specifying multiple images in the target video, the generative model 220 can more fully understand the change trends of individual images in the video, and as a result, can show excellent visual effects even in the synthesis of videos with complex scenes and complex movements.

[0027] According to an exemplary embodiment of the present disclosure, the multiple images here include a first image for describing a head image in the multiple target images and a second image for describing a tail image in the multiple target images. At this time, the generative model 220 can generate a target video 250 based on the received first image, second image, and text. Compared with the known technical solution of generating a video based only on a single image and text, the multiple images can provide richer change information. In particular, the head image and the tail image are located at the head end and the tail end of the target video respectively, which provides more guide information about visual changes to the video generation process, so that the target video 250 can have better dynamic visual effects.

[0028] It should be understood that the head image herein may represent one or more images within a predetermined range from the beginning of the target video, such as the first frame, the second frame, the third frame, etc. in the target video. Similarly, the tail image may represent one or more images within a predetermined range from the end of the target video, such as the last frame, the penultimate frame, the third penultimate frame, etc. in the target video.

[0029] The detailed process of generating a video

[0030] Having described an overview of generating a video, further details regarding the generation of a video are described below. It should be understood that FIG. 2 merely exemplarily illustrates the process of receiving a first image and a second image, and the images may be received based on various methods. For example, an image address entered by a user may be received by controller 232 and / or 234, or a user may drag and drop an image onto controller 232 and / or 234. According to an exemplary embodiment of the present disclosure, the second image may be received in a more flexible manner. Specifically, the second image may be received via at least one of an import controller for importing an address of the second image, a drawing controller for drawing the second image, and an editing controller for editing the first image to generate the second image.

[0031] Further details regarding the acquisition of the second image will be described with reference to Figures 3A, 3B, and 3C. Figure 3A shows a block diagram 300A for acquiring a second image according to some embodiments of the present disclosure. As shown in Figure 3A, a page 340 for receiving a second image may include multiple controllers. Specifically, a drawing controller 310 for drawing the second image, an import controller 320 for importing an address of the second image, and an editing controller 330 for editing an image (e.g., the first image or another imported image) to generate the second image are provided. A user interacts with the drawing controller 310, and can then invoke a drawing tool to draw a sketch 312 of the second image.

[0032] FIG. 3B illustrates a block diagram 300B for acquiring a second image according to some embodiments of the present disclosure. As shown, a user may interact with an import controller 320 to import an image file as the second image. FIG. 3C illustrates a block diagram 300C for acquiring a second image according to some embodiments of the present disclosure. As shown, a user may interact with an edit controller 330 to acquire the second image. In the context of the present disclosure, a user may edit a first image to generate a second image. As another example, a user may edit other imported images to acquire a second image. For example, a user may move a polar bear at position 332 to position 334. In this manner, a user can specify a tail image of a target video in a flexible and effective manner.

[0033] FIG. 4 illustrates a block diagram 400 of a process for generating a video according to some embodiments of the present disclosure. A user may specify input data 410 for generating a model as described above. For example, a user may use an import controller to import a first image 412 and a second image 414, respectively, and input text 416. The first image 412 may indicate the generation of a first frame in a target video 420, and the second image 414 may indicate the generation of a last frame in the target video 420. In this case, the first frame of the target video 420 corresponds to the first image 412, the last frame corresponds to the second image 414, and the content of the video is "A polar bear is walking on the ice surface."

[0034] According to an exemplary embodiment of the present disclosure, multiple target videos may be generated continuously. For example, an image (e.g., a final frame or other image) in a currently generated video may be used as a head image of a next video, and more videos may be generated cumulatively. For further details, refer to FIG. 5, which shows a block diagram 500 of a page for receiving input data for generating a subsequent video according to some embodiments of the present disclosure. As shown in FIG. 5, multiple images in a target video (e.g., a currently generated target video 420) may be provided on a page 510.

[0035] A user may interact with a desired image to select a head image for generating a subsequent video from multiple images. For example, a user may specify a head image for describing another target video following the target video by clicking the third image 512 of the target video 420. The user may specify a tail image as the fourth image 520 and input text 530 in the text box. At this time, another text for describing the content of the other target video and a fourth image for describing the tail image of the other target video are obtained. Furthermore, the generative model may generate another target video based on the third image 512, the fourth image 520, and the another text 530.

[0036] As shown in FIG. 5, the user may press the controller 242 to generate another target video. For further details, reference is made to FIG. 6, which illustrates a block diagram 600 of a process for generating a subsequent video according to some embodiments of the present disclosure. As shown in FIG. 6, the input data 610 may include a third image 512, a fourth image 520, and text 530. Here, the third image 512 may indicate the generation of a first frame in the target video 520, and the fourth image 520 may indicate the generation of a final frame in the target video 520. For example, the final frame may be the same or different from the fourth image 520, and the content of the video is "a polar bear is walking on the ice surface, fireworks are bursting open, an astronaut is following the bear."

[0037] According to an exemplary embodiment of the present disclosure, a target video may be combined with another target video. The tail image of a current video may be used as the head image of the next video to be generated step by step and cumulatively. In this way, subsequent videos may be generated successively, gradually generating a long video with a richer plot. The exemplary embodiment of the present disclosure may be utilized to greatly simplify the complexity of video creation, for example, by constructing shorter videos in the manner of sub-shots in a movie, and then editing these videos to generate a longer video.

[0038] Although the above describes the designation of the second image by importing an image, alternatively and / or additionally, the second image may be a sketch that designates the content of the tail image of the target video. In some cases, it is difficult to obtain the end screen of the target video to be generated, and the content of the tail image can be designated in a simpler and more effective way by sketching (e.g., hand-drawing).

[0039] 7 illustrates a block diagram 700 of a process for generating a video based on a sketch according to some embodiments of the present disclosure. As shown in FIG. 7, input data 710 may include a first image 712, a second image 714, and text 716, where the second image 714 may dictate the generation of a final frame in a target video 720.

[0040] Then, the polar bear in the final frame may wave with the pose specified in the second image 714, where the first frame of the target video 720 corresponds to the first image 712, the final frame corresponds to the second image 714, and the content of the video is "a polar bear walking on the ice surface gradually stands up and waves." Using the exemplary embodiments of the present disclosure, the poses and positions of objects in a video can be specified in a more convenient and efficient manner, and further facilitates the generation of dynamic videos with richer visual content.

[0041] Although an example of specifying input data for generating a single video in the page is given above, alternatively and / or additionally, the page may further include a controller for receiving a plurality of images and a long text, where the user may input a plurality of images for describing a plurality of keyframes in the long video and a long text for describing the content of the long video, respectively. Subsequently, the machine learning model may determine images and text for generating image segments based on the received plurality of images and the long text.

[0042] For further details, reference is made to FIG. 8 , which illustrates a block diagram 800 of a process for obtaining candidate inputs utilizing a machine learning model according to some embodiments of the present disclosure. As shown in FIG. 8 , a user may interact with a controller 820 to input a number of images 812, ..., 814, ..., 816 sorted in chronological order, and may also input a long text 830. The user may press a controller 842 to invoke a machine learning model 850 for further processing, or alternatively and / or additionally, the user may press a controller 840 to return to a previous page.

[0043] It should be understood that the machine learning model 850 here may be a pre-trained model with text processing and image processing capabilities. For example, the machine learning model 850 may be implemented based on a large language model. Upon receiving the multiple images and long text 830, the machine learning model 850 may divide the long text 830 into multiple short texts, and select head images and tail images that match each of the short texts from the multiple images.

[0044] Reference is now made to FIG. 9, which illustrates the output results of the machine learning model 850, and which shows a block diagram 900 of multiple candidate inputs according to some embodiments of the present disclosure. As shown in FIG. 9, a page 910 includes multiple candidate inputs, each of which may correspond to a short video segment. The candidate inputs may include, for example, images to describe the first and last frames of the video segment, and text describing the content of the video segment. In the page 910, the candidate input 920 may include images 812, 814, and text 922..., and the candidate input 930 may include images 814, 816, and text 932. Using the exemplary embodiments of the present disclosure, the powerful processing capabilities of the machine learning model 850 can be fully utilized to assist in the video generation process. For example, in film and television production, a textual script segment may be input to the machine learning model 850, which may automatically output associated images and text descriptions of multiple segmented sub-shots. In this manner, the model generation performance may be significantly improved.

[0045] According to an exemplary embodiment of the present disclosure, a summary text for describing a long video may be received, and a plurality of texts for describing the contents of a plurality of video segments in the long video may be determined based on the summary text according to a machine learning model, respectively. It should be understood that the summary text here may describe the main characters, setting, story outline, etc. of the desired generated video. Specifically, the summary text may request the generation of a story featuring a polar bear and an astronaut, the story takes place in the North Pole, and the story outline is that the astronaut's spaceship accidentally lands in the North Pole, and the astronaut meets the polar bear and gradually becomes friends with him.

[0046] The summary text can be input to a machine learning model to utilize the powerful linguistic ability of the machine learning model to assist in the creation and generation of rich storylines. The machine learning model can also automatically divide these storylines into multiple subplots and generate corresponding description texts. Furthermore, the machine learning model can generate target images corresponding to the multiple texts, i.e., specify an image of the first frame and an image of the last frame for each sub-shot video segment. At this time, a long dynamic video with a rich storyline can be obtained by inputting each generated target text and the corresponding image into the generative model.

[0047] FIG. 10 illustrates a block diagram 1000 of a process for obtaining candidate inputs based on summary text using a machine learning model according to some embodiments of the present disclosure. As shown in FIG. 10, a machine learning model 850 may be input with summary text 1010, where the summary text 1010 may only limit that it is desirable to create a video about a story between a polar bear and an astronaut. The machine learning model 850 may generate multiple candidate inputs based on the summary text 1010. For example, the candidate input 1020 may include images 1022, 1024, and text 1026... and the candidate input 1030 may include images 1032, 1034, and text 1036.

[0048] It should be understood that the text 1026 and 1036 here may be generated based on the summary text 1010, the images 1022 and 1024 may be generated based on the text 1026, and the images 1032 and 1034 may be generated based on the text 1036. In this process, the user can only input the summary text and obtain multiple candidate inputs without complex manual operations. Furthermore, the generative model may generate corresponding videos based on these candidate inputs. In this way, the powerful text processing and text-to-image conversion capabilities of the machine learning model 850 can be utilized to assist in generating full-length videos with richer dynamic content.

[0049] According to an exemplary embodiment of the present disclosure, a machine learning model may be utilized to assist in the creation of a subsequent video. For example, a generated video may be input to a machine learning model and the machine learning model may be requested to provide a description of the subsequent content of the video. The machine learning model may determine a prompt message related to another target video following the target video based on the target video, the prompt message including, for example, at least one image and another text. Furthermore, the generative model may generate another target video based on the at least one image and another text.

[0050] For further details, refer to FIG. 11, which shows a block diagram 1100 of a process for obtaining a prompt message based on a machine learning model according to some embodiments of the present disclosure. As shown in FIG. 11, the generated target video 250 is input to the machine learning model 850, and the machine learning model 850 outputs a prompt message 1110, which may include an image 1112 for describing the first frame of the subsequent video, an image 1114 for describing the last frame of the subsequent video, and a text 1116 for describing the content of the subsequent video. When a user interacts with the prompt message 1110 and confirms the prompt message 1110, the user can directly input the prompt message 1110 to the generative model.

[0051] Alternatively and / or additionally, an editing controller may be provided for editing the prompt message 1110, for example, a user can modify the images 1112, 1114 and text, and then input the modified prompt message 1110 into the generative model. In this way, in the process of automatically generating a story script with the powerful processing ability of the machine learning model 850, the user can make adjustments according to his / her own expectations to obtain a video that is more in line with the user's expectations. In this way, the powerful inference ability of the machine learning model can be fully utilized to create subsequent sub-shot scripts and generate subsequent videos.

[0052] In the above, an example was described in which a user specifies an image of the first frame, an image of the final frame, and text to generate a model. Alternatively and / or additionally, a new video may be created by selecting two images from an existing video and utilizing a text description with content different from that of the existing video. For example, if a user is not satisfied with a generated video of a polar bear walking on an ice surface, the text can be adjusted to "a polar bear is running on the ice surface." In this case, the content of the new video to be generated is different from that of the existing video, for example, a video of a polar bear running is generated. In this way, if a user is not satisfied with an existing video, the user can adjust the content of the video in a simple manner of modifying the text description, and a higher video generation performance can be achieved. For example, in film and television production, a user can generate different versions of a certain sub-shot and further select the one that best meets his / her expectations.

[0053] The iterative process of generating a target video using a pre-obtained generative model has been described above, and further details regarding the acquisition of the generative model will be described below. According to an exemplary embodiment of the present disclosure, a number of reference videos can be used to train the generative model, and further details regarding the training process will be described with reference to FIG. 12. FIG. 12 illustrates a block diagram 1200 of a process for determining a generative model according to some embodiments of the present disclosure. As shown in FIG. 12, a generative model 220 shown in FIG. 12 may be constructed, which may describe an association relationship between input data and output data. The input data may include, for example, text to describe the content of the video, a head image (e.g., may be referred to as a first image) to indicate a head image in the video, and a guide image (e.g., may be referred to as a second image) to indicate a tail image in the video. The output data may include, for example, a video.

[0054] According to an exemplary embodiment of the present disclosure, the generative model 220 may be constructed based on various architectures currently known and / or developed in the future. In the training stage, sample data may be extracted from a reference video 1230 to further train the generative model 220. A reference video 1230 including a plurality of reference images may be acquired, and a first reference image 1212 and a second reference image 1214 may be acquired from the plurality of reference images. Here, the first reference image 1212 (e.g., an image of a first frame) may be acquired from a head position of the reference video 1230, and the second reference image 1214 (e.g., an image of a final frame, or an image within a predetermined range before the image of the final frame) may be acquired from a tail position of the reference video 1230. Furthermore, a reference text 1210 for describing the content of the reference video may be received.

[0055] In one example, the reference video 1230 includes a cat, and the cat in the first reference image 1212 and the second reference image 1214 may be located at different positions in the screen. In this case, the reference text 1210 may indicate "A cat is walking in the street." Further, the generative model 220 may be trained based on the first reference image 1212, the second reference image 1214, and the reference text 1210. Then, the trained generative model 220 may generate a target video based on either the first image or the second image, and the text.

[0056] In the process of processing the final frame instruction, multiple strategies may be adopted to adjust the influence of the final frame. During the training period, the final frame instruction may be randomly selected from the last three true frames of the reference video. To improve the robustness of the model against instruction dependency, noise may be introduced into the instruction. During the training stage, the final frame instruction may be randomly discarded with a certain probability. Correspondingly, a simple and effective inference strategy is proposed. For example, in the τ denoising steps before the inverse process of the diffusion model, the final frame instruction may be utilized to guide the video generation to a desired end state. Following this, the remaining steps discard this final frame instruction, which can generate a more consistent video content over the model generation time. The strength of the final frame guide can be adjusted by τ.

[0057] 12 only diagrammatically illustrates the process of obtaining the generative model 220, and it should be understood that alternatively and / or additionally, multiple reference videos may be obtained and corresponding data may be extracted from the multiple reference videos to iteratively and continuously update the generative model 220. For example, reference videos in which different objects perform different actions may be obtained, and the generative model 220 may grasp richer knowledge about video generation.

[0058] Using exemplary embodiments of the present disclosure, in the process of determining the generation model 220, the first reference image 1212 may be used as the head image of the video, and the second reference image 1214 may be used as guide data to determine the development direction of the story in the video. For example, the generated video may use the second reference image 214 as the final frame. In this way, the generation model 220 can clearly understand the changes in the content of individual images in the video and can help generate a more rich and compelling dynamic video.

[0059] According to an exemplary embodiment of the present disclosure, the image instruction of the first frame sets the scene (and establishes the characters) for video generation. Also, the first frame enables the model to generate a continuous video, in which case the model uses the final frame of the previous video as the instruction for the first frame of the subsequent video to generate the subsequent video. Further, the instruction of the final frame describing the end state of the video can function as an additional control mechanism. In this way, the consistency between the user's expectation and the text is strengthened (that is, the generated video conforms to the user's expectation), the model constructs complex shots, and finally generates rich video content, and as a result, diversity and consistency can be ensured.

[0060] Generally, <text (text), first frame (first frame), last frame (last frame)> instructions can be used conditionally. When these three instructions are given, the model can intensively learn the dynamics of people, animals, objects, and other entities in the world during the training stage. In inference, the model can "generalize" the learned motion rules of the physical world to areas not covered during training, such as realizing the movement of anime characters or special effect shots.

[0061] Specifically, the above information can be integrated into the diffusion model. For example, text information is encoded by a pre-trained text encoder and embedded into the diffusion model using a cross-attention mechanism. Image instructions are encoded by a pre-trained variational autoencoder (VAE) and combined with disturbing video content or Gaussian noise as input to the diffusion model. In the training process, the image of the first frame of the video can be directly used as the instruction of the first frame to force the model to strictly follow the instruction, thereby maintaining the continuity between consecutive videos. In the inference process, the instruction can be conveniently obtained from the text-to-image model or can be directly provided by the user.

[0062] According to an exemplary embodiment of the present disclosure, to determine the first reference image, a reference image located at the head of the reference video may be determined as the first reference image. In other words, an image of a first frame in the reference video may be taken as the first reference image. In this way, the process of constructing training data is simplified, and training data can be obtained in a fast and efficient manner. The second reference image may be determined based on various methods. For example, an image of a last frame in the reference video may be taken as the second reference image. Alternatively and / or additionally, in order to introduce a disturbance factor into the training data, the second reference image may be determined from a set of reference images located within a predetermined range from the tail of the reference video.

[0063] Assuming that the reference video 230 includes multiple reference images, the predetermined range may represent the last three frames (or other number) of the tail of the reference video. Assuming that the reference video includes N image frames, any one of frame N, frame N-1, and frame N-2 of the reference video may be the second reference image. At this time, the second reference image may include any one of the reference images of the last three frames. In this way, the influence of the final frame on the training process can be adjusted to a certain extent, so that a more consistent video can be obtained.

[0064] Please refer to FIG. 13 for describing more information of the generative model, which shows a block diagram 1300 of a process for training a generative model according to some embodiments of the present disclosure. As shown in FIG. 13, the generative model 220 may include an encoder model 1310, a combination model 1320, a diffusion model 1330, and a decoder model 1340. Here, the encoder model 1310 and the decoder model 1340 may have a predetermined configuration and parameters. The parameters of the encoder model 1310 and the decoder model 1340 are not updated during a subsequent update of the parameters of the generative model 220. According to an exemplary embodiment of the present disclosure, a VAE and a corresponding decoder may be used.

[0065] 13, the generative model 220 may be obtained based on the first reference image 1212, the second reference image 1214, and the reference text 1210. Specifically, the encoder model 1310 may be used to determine a first reference feature 1312 (e.g., denoted as Z) of the reference video 1230, where the first reference feature 1312 may include multiple reference image features of multiple reference images. Further, the encoder model 1310 may be used to determine a second reference feature 1314 (e.g., c image , where the second reference features may include a first reference image feature of the first reference image and a second reference image feature of the second reference image.

[0066] According to an exemplary embodiment of the present disclosure, the first reference feature 1312 includes multiple dimensions, each dimension corresponding to one reference image, i.e., in each dimension, a feature from one reference image (a predetermined dimension, e.g., R C*H*W where C represents the number of channels, H represents the height of the image, and W represents the width of the image) may be stored. The first reference feature and the second reference feature may have the same dimension, where the number of dimensions is equal to the number of images in the reference video. For example, it may be specified that the reference video includes 16 images (or other number).

[0067] For further details regarding feature determination, reference is made to FIG. 14, which illustrates a block diagram 1400 of a process for determining reference features for a reference video according to some embodiments of the present disclosure. As illustrated in FIG. 14, each reference image in the reference video 1230 may be processed one by one. For example, an encoder model may be utilized to extract features of the first reference image 1212 to generate a first reference image feature. The first reference image feature may be placed at a first position (i.e., position 1410) of the first reference feature 1312. Following this, the encoder model may be utilized to extract a reference image feature of another reference image after the first reference image 1212 and place the reference image feature at a second position of the first reference feature 1312. Each reference image in the reference model 1230 may be processed successively until the second reference image feature of the last reference image (i.e., second reference image 1214) in the reference model 1230 is placed at the last position (i.e., position 1412) of the first reference feature 1312.

[0068] Similarly, the encoder model 1310 may be used to determine a second reference feature of the reference video. To generate the first reference image feature, the encoder model 1310 may be used to extract features of the first reference image 1212. The first reference image feature may be placed at a first position (i.e., position 1420) of the second reference feature 1314. The encoder model 1310 may be used to place a second reference image feature of the last reference image (i.e., second reference image 1214) in the reference video 1230 at a last position 1422 of the second reference feature 1314.

[0069] 14, the position (i.e., the first position) of the first reference image feature in the first reference feature 1312 corresponds to the position (i.e., the first position) of the first reference image in the reference video, and the position (i.e., the last position) of the second reference image feature in the second reference feature 1314 corresponds to the position (i.e., the last position) of the second reference image in the reference video. In this way, it is convenient to align the first and second reference features in the subsequent combining process, which can facilitate improving the accuracy of the diffusion model.

[0070] According to an exemplary embodiment of the present disclosure, features at positions other than the first and second positions in the second reference feature may be set to NULL. Specifically, position 1430 in FIG. 14 may be set to NULL, e.g., filled with data "0". In this way, the second reference feature can adjust the influence of the first and second reference images on the video generation process, so that the generative model can learn more knowledge about the transformation relationship between the first and second reference images.

[0071] Specifically, the reference video can be encoded using a VAE to obtain a first reference feature 1312, that is, each frame of the reference video can be encoded by a VAE respectively. The first reference image and the second reference image can be encoded using a VAE, and then the positions of the intermediate frames are filled with 0, and a second reference feature 1314 (whose dimensions are the same as those of the first reference feature 1312) can be obtained.

[0072] Returning to FIG. 13, the diffusion model 1330 may be determined based on the first reference feature 1312, the second reference feature 1314, and the reference text 1210. Specifically, based on the principle of the diffusion model, noise may be added to the first reference feature 1312 and the second reference feature 1314 respectively (for example, different levels of noise may be added to the two features in different ways), and the first noisy reference feature 1312' and the second noisy reference feature 1314' may be further generated. Furthermore, the combination model 1320 may be used to combine the first noisy reference feature 1312' and the second noisy reference feature 1314' to generate the noisy reference feature 1322. Furthermore, corresponding training data may be constructed based on the principle of the diffusion model, and the parameters of the diffusion model 1330 may be updated.

[0073] According to an exemplary embodiment of the present disclosure, the diffusion model 1330 may be realized based on various currently known and / or future developed structures. The diffusion process of the diffusion model 1330 involves a forward noise process and a backward denoising process. In the forward noise process, noise data may be added to the features separately in multiple steps. For example, the initial feature may be denoted as Z_0. Noise data may be added continuously in each step, and the noise feature in step t may be denoted as Z_t, and the noise feature in step t+1 may be denoted as Z_(t+1). For example, in step t, noise data may be added to Z_t. This may be followed by random Gaussian data in step T.

[0074] In the backward denoising process, the inverse process of the above-mentioned noise generation process may be performed in multiple steps to obtain the initial features step by step. Specifically, the random Gaussian data, the reference text, and the corresponding second reference features may be input to the diffusion model to perform the denoising process step by step in multiple steps. For example, some noise of the noise feature Z_t may be removed in the t-th step to form a noise feature Z_(t-1) that is cleaner for the noise feature Z_t. Each denoising process may be performed iteratively to infer the initial features (i.e., the video features without added noise).

[0075] According to an exemplary embodiment of the present disclosure, in the process of determining the diffusion model, the diffusion model 1330 is utilized to calculate the reconstruction features 1332 (e.g.,

number

[0076] According to an exemplary embodiment of the present disclosure, a 2D UNet (U-shaped network) may be used to realize the diffusion model. The model may be constructed by spatially downsampling and then spatially upsampling with the insertion of skip connections. Specifically, the model includes two basic blocks: a 2D convolutional block and 2D attention. We extend the 2D UNet to 3D by inserting a temporal layer, where a 1D convolutional layer is located after the 2D convolutional layer along the time dimension, and a 1D attention layer is located after the 2D attention layer along the time dimension. The model can be trained with images and videos to maintain high-fidelity spatial generation capabilities and disable 1D temporal manipulation of image inputs. Bidirectional self-attention can be used in all attention layers. Text instructions are encoded using a pre-trained text encoder and embedded c text is injected through the cross-attention layer of UNet, hiding the state as a query, and text Let be the key and value.

[0077] Regarding image command injection, the image command of the first and last frame can be combined with a text instruction. For example, first ,I last Given image instructions on the first and last frames, expressed as {f first ,f last} can be obtained, where:

[0078]

number

[0079]

number

[0080]

number

[0081] According to an exemplary embodiment of the present disclosure, the first reference image, the second reference image, and the reference text are determined from multiple reference videos, and then the first reference feature Z, the second reference feature c, and the like of each reference video are determined using the above-described aspects. image , and the corresponding reconstruction features

[0082]

number

[0083]

number

[0084] The diffusion model 1330 may be iteratively and continuously updated using multiple reference videos until a desired stopping condition is met. It should be understood that in this process, no manual annotation data is required, rather the first and last frames of the reference video and the corresponding text descriptions can be directly used as training data. In this way, the workload of obtaining manual annotation data can be eliminated, thereby improving the efficiency of obtaining the generative model 220.

[0085] The above is the second reference feature c image Although an example has been given in which the first reference image and the second reference image simultaneously include related feature information, alternatively and / or additionally, in some cases, the second reference feature c imageIt should be understood that may include feature information of only the first reference image. Specifically, the features of the second reference image in the second reference features may be set to NULL according to a predetermined condition. In other words, in FIG. 14, the feature at position 1422 may be removed, for example, the value at that position may be set to 0.

[0086] Using the exemplary embodiments of the present disclosure, the second reference image features may be removed at a predetermined rate (e.g., 25% or other value). In this way, the process of updating the generative model includes application scenarios that do not consider the second reference image, so that the generative model is suitable for a wider range of application scenarios and the accuracy of the generative model in various situations can be improved.

[0087] According to an exemplary embodiment of the present disclosure, the generative model 220 may be trained based on various methods now known and / or later developed to capture associative relationships between a first image, a second image, text describing the content of the video, and the corresponding video.

[0088] According to an exemplary embodiment of the present disclosure, the generative model 230 further comprises a decoder model 1340. After the generative model 220 is obtained, the generative model 220 may be input with a need for generating a video to generate a corresponding target video. According to an exemplary embodiment of the present disclosure, after the first image and text are input into the generative model, the generative model may be utilized to generate a corresponding target video. According to an exemplary embodiment of the present disclosure, after the first image, the second image, and the text are input into the generative model, the generative model may be utilized to generate a corresponding target video. Alternatively and / or additionally, the text may be NULL, and the target video may still be generated.

[0089] In order to obtain a more expected target video, a first image and a second image for generating the target video and a text describing the content of the target video may be received. Following this, a reconstructed feature of the target video may be generated based on the first image, the second image, and the text according to the trained diffusion model. Furthermore, the target video may be generated based on the reconstructed feature according to the decoder model. In this way, the diffusion model 1330 can be fully utilized to perform a backward denoising process, thereby obtaining the reconstructed feature of the target video in a more accurate manner.

[0090] For further details regarding the generation of the video, please refer to FIG. 15, which shows a block diagram 1500 of a process for generating a target video according to some embodiments of the present disclosure. As shown in FIG. 15, a first image 1512, a second image 1514, and text 1510 for describing the content of the image may be received. Here, the first image 1512 may be, for example, a first frame of a target video 1530 to be generated, including an ice surface. The second image 1514 may be, for example, an ice surface and a polar bear on the ice surface. The second image 1514 may be used as guide information to generate a final frame of the target video 1530. At this time, the generative model 220 will generate the target video 1530 under the condition of the text 1510, and the content of the video will be "A polar bear is walking on the ice surface".

[0091] The first image 1512, the second image 1514, and the text 1510 may be input to the generative model 220. The generative model 220 then outputs a target video 1530, where the first frame of the target video 1530 corresponds to the first image 1512 and the last frame corresponds to the second image 1514. It should be appreciated that at this point, the target video 1530 has not yet been generated, so noise features are utilized as the first features 1520 of the target video 1530. According to an exemplary embodiment of the present disclosure, Gaussian noise may be utilized to determine the first features 1520.

[0092] Further, the second feature 1522 of the target video 1530 may be generated based on the first image 1512 and the second image 1514 according to the encoder model 1310. For example, the encoder model 1310 may be used to determine the first image feature of the first image 1512 and the second image feature of the second image 1514, respectively. Then, the first image feature may be placed at the first position of the second feature 1522, and the second image feature may be placed at the last position of the second feature 1522. It should be understood that the process of generating the second feature 1522 here is similar to the training stage, and therefore a detailed description will be omitted.

[0093] If the first feature 1520 and the second feature 1522 have already been determined, the reconstructed feature 1526 of the target video may be obtained using the first feature 1520, the second feature 1522, and the text 1510 based on the diffusion model. Similar to the process of determining the reconstructed feature in the training stage, the first noise feature 1520′ and the second noise feature 1522′ may be obtained, and the noise feature 1524 may be formed by combining the first noise feature 1520′ and the second noise feature 1522′ using the combination model 1320. Furthermore, the backward denoising stage of the diffusion model 1330 may be used to generate the reconstructed feature 1526.

[0094] It should be appreciated that since the diffusion model 1330 is trained based on the loss function described above, the reconstructed features 1526 generated by the diffusion model 1330 can then accurately describe various aspects of the information in the target video 1530. Furthermore, the reconstructed features 1526 can be input to the decoder model 1340, which then outputs a corresponding target video 1530 that accurately matches the first image 1512, the second image 1514, and the text 1510 in the input data.

[0095] It should be understood that in the process of generating a video, the backward denoising process of the diffusion model 1330 may be performed iteratively, and the backward denoising process may be performed iteratively in multiple steps. Assuming that the total number of multiple steps is T, in the first step (t=1), the first feature 1520, the second feature 1522, and the text 1510 may be utilized to obtain the reconstructed feature 1526 of the target video.

[0096] Following this, in the second step (t=2) after the first step in the multiple steps, the first feature is set as the reconstructed feature of the target video, i.e., Z is

[0097]

number

[0098] According to an exemplary embodiment of the present disclosure, in the process of generating the target video, the user may specify the influence degree of the second image. For example, a controller for inputting an influence factor (e.g., expressed as a percentage) may be provided on the page. The steps for invoking the diffusion model in the generative model may be divided into a first stage and a second stage based on the influence factor. In the first stage and the second stage, the diffusion model may determine the reconstruction features of the target video based on the first image and the second image, and then the decoder model in the generative model may generate the target video based on the reconstruction features. Using the exemplary embodiment of the present disclosure, the influence factor may be used to control the influence degree of the second image 1514.

[0099] Specifically, in each step in the first stage, the guide effect of the second image 1514 may be taken into account, and in each step in the second stage, the guide effect of the second image 1514 may not be taken into account. In the process of determining the reconstruction feature of the target video, in the first stage, the diffusion model determines an intermediate reconstruction feature related to the target video based on the first image and the second image, and in the second stage, the diffusion model determines the reconstruction feature based on the first image and the intermediate reconstruction feature. In other words, the influence of the second image 1514 may be removed from the second feature in the second stage, that is, the part of the second feature corresponding to the second image may be set to NULL. In FIG. 15, the feature located at the last position of the second feature 1522 may be set to 0.

[0100] Specifically, in τ steps prior to all T denoising steps, a final frame condition is applied to guide the generation of the video towards a desired end state, and the final frame condition is discarded in subsequent steps to generate a more plausible and temporally consistent video.

[0101]

number

[0102] According to an exemplary embodiment of the present disclosure, the number of steps in which the second image 1514 is not considered may be determined in a predetermined ratio. For example, the influence of the second image 1514 may not be considered in τ=T*40% (or other value) of all T steps. For T=50, the complete second features 1522 may be used in the first 30 (50*(1-40%)) steps, and the value of the last position of the second features 1522 may be set to 0 in the last 20 (50*40%) steps. In this way, the roles of the first image 1512 and the text 1510 in generating the target video may be adjusted, so that the generated target video 1530 may be more in line with expectations.

[0103] To summarize the above, according to an exemplary embodiment of the present disclosure, a novel video generation architecture based on a diffusion model, PixelDance, is proposed, which combines image instructions and text indications of the first and last frames. The corresponding training and inference processes can be implemented based on the PixelDance architecture. The exemplary embodiment of the present disclosure can be utilized to provide flexible control of the video generation process and a final frame indication with varying strength.

[0104] The proposed technique can be implemented on a large number of datasets, and experiments have shown that videos generated using PixelDance have strong advantages in synthesizing those with complex scenes and / or motions, etc. In particular, a generative model can be trained using various available datasets, and each video in the dataset is associated with a paired text, which usually provides a rough description and represents a weak correlation with the content of the video. Since image instructions are useful for learning complex video distributions, PixelDance can make full use of various datasets without annotating the data and excel in generating videos with complex scenes and motions.

[0105] Example Process

[0106] 16 shows a flowchart of a method 1600 for generating a video according to some embodiments of the present disclosure. In box 1610, at least one of a head image and a tail image for describing a target video is received. In box 1620, text for describing the content of the target video is received. In box 1630, the target video is generated based on the image and the text according to a generative model.

[0107] According to an exemplary embodiment of the present disclosure, the plurality of images includes a first image for describing a head image in the plurality of target images and a second image for describing a tail image in the plurality of target images.

[0108] According to an exemplary embodiment of the present disclosure, the second image is received through at least one of an import controller for importing an address of the second image, a drawing controller for drawing the second image, and an edit controller for editing the first image to generate the second image.

[0109] According to an exemplary embodiment of the present disclosure, the method further includes providing a plurality of images in the target video; in response to receiving an interaction on a third image in the plurality of images, determining a head image for describing another target video after the target video; obtaining another text for describing content of the another target video and a fourth image for describing a tail image of the another target video; and generating the another target video based on the third image, the fourth image, and the another text according to a generative model.

[0110] According to an exemplary embodiment of the present disclosure, the image and text are determined by receiving a plurality of images for respectively describing a plurality of keyframes in a long video, receiving a long text for describing content of the long video, and determining the image and text based on the plurality of images and the long text according to a machine learning model.

[0111] According to an exemplary embodiment of the present disclosure, the image and the text are determined by receiving a summary text for describing a long video, determining a plurality of texts for respectively describing content of a plurality of video segments in the long video based on the summary text according to a machine learning model, generating a target image based on a target text among the plurality of texts according to the machine learning model, and determining the image and the text by taking the target image as an image and the target text as a text.

[0112] According to an exemplary embodiment of the present disclosure, the method further includes determining a prompt message related to another target video after the target video based on the target video according to a machine learning model, the prompt message including at least one image and another text, and generating the other target video based on the at least one image and another text according to a generative model.

[0113] According to an exemplary embodiment of the present disclosure, generating the target video further includes receiving an influence factor for specifying an influence degree of the second image, dividing multiple steps for invoking a diffusion model in the generative model into a first stage and a second stage based on the influence factor, determining reconstruction features of the target video based on the first image and the second image according to the diffusion model in the first stage and the second stage, and generating the target video based on the reconstruction features using a decoder model in the generative model.

[0114] According to an exemplary embodiment of the present disclosure, determining reconstruction features of a target video includes, in a first stage, determining intermediate reconstruction features related to the target video based on a first image and a second image according to a diffusion model, and, in a second stage, determining reconstruction features based on the first image and the intermediate reconstruction features according to the diffusion model.

[0115] According to an exemplary embodiment of the present disclosure, a generative model is determined by determining a first reference image and a second reference image from a plurality of reference images in a reference video, receiving reference text for describing the reference video, and obtaining a generative model based on the first reference image, the second reference image and the reference text, and the generative model is utilized to generate a target video based on one of the first image and the second image and the text.

[0116] According to an exemplary embodiment of the present disclosure, the first reference image is located at the head of the reference video, and the second reference image is located within a predetermined range from the tail of the reference video.

[0117] According to an exemplary embodiment of the present disclosure, the generative model includes an encoder model and a diffusion model, and obtaining the generative model based on the first reference image, the second reference image, and the reference text includes: using the encoder model to determine a first reference feature of the reference video, the first reference feature including a plurality of reference image features of the plurality of reference images; using the encoder model to determine a second reference feature of the reference video, the second reference feature including the first reference image feature of the first reference image and the second reference image feature of the second reference image; and determining the diffusion model based on the first reference feature, the second reference feature, and the reference text.

[0118] According to an exemplary embodiment of the present disclosure, a first position of the first reference image feature in the second reference feature corresponds to a position of the first reference image in the reference video, and a second position of the second reference image feature in the second reference feature corresponds to a position of the second reference image in the reference video.

[0119] According to an exemplary embodiment of the present disclosure, the dimension of the second reference feature is equal to the dimension of the first reference feature, and features at positions other than the first position and the second position in the second reference feature are set to NULL.

[0120] According to an exemplary embodiment of the present disclosure, the method further includes setting the second reference image feature to NULL according to a predetermined condition.

[0121] According to an exemplary embodiment of the present disclosure, determining a diffusion model based on the first reference feature, the second reference feature, and the reference text includes performing a noiseification process on the first reference feature and the second reference feature, respectively, to generate a first noisy reference feature and a second noisy reference feature, combining the first noisy reference feature and the second noisy reference feature to generate a noisy reference feature of the reference video, determining a reconstructed feature of the reference video based on the noisy reference feature and the reference text using the diffusion model, and updating the diffusion model based on a difference between the reconstructed feature and the reference feature.

[0122] According to an exemplary embodiment of the present disclosure, the head image is the first image frame in the target video and the tail image is the last image frame in the target video.

[0123] Exemplary Apparatus and Equipment

[0124] 17 shows a block diagram of an apparatus 1700 for generating a video according to some embodiments of the present disclosure. The apparatus 1700 includes an image receiving module 1710 arranged to receive a plurality of images for respectively describing a plurality of target images in a target video, a text receiving module 1720 arranged to receive text for describing the content of the target video, and a generating module 1730 arranged to generate the target video based on the plurality of images and the text according to a generative model.

[0125] According to an exemplary embodiment of the present disclosure, the plurality of images includes a first image for describing a head image in the plurality of target images and a second image for describing a tail image in the plurality of target images.

[0126] According to an exemplary embodiment of the present disclosure, the second image is received through at least one of an import controller for importing an address of the second image, a drawing controller for drawing the second image, and an edit controller for editing the first image to generate the second image.

[0127] According to an exemplary embodiment of the present disclosure, the apparatus further includes a providing module configured to provide a plurality of images in the target video, a determining module configured to determine a head image for describing another target video after the target video in response to receiving an interaction on a third image in the plurality of images, and an acquiring module configured to acquire another text for describing content of the another target video and a fourth image for describing a tail image of the another target video, wherein the generating module is further configured to generate another target video based on the third image, the fourth image and the another text according to the generative model.

[0128] According to an exemplary embodiment of the present disclosure, the apparatus further includes a keyframe receiving module arranged to receive a plurality of images for respectively describing a plurality of keyframes in the long video, a long text receiving module arranged to receive a long text for describing content of the long video, and a first calling module arranged to determine an image and a text based on the plurality of images and the long text according to a machine learning model.

[0129] According to an exemplary embodiment of the present disclosure, the apparatus further includes a summary receiving module arranged to receive a summary text for describing the long video; a second calling module arranged to determine a plurality of texts for respectively describing content of a plurality of video segments in the long video based on the summary text according to a machine learning model; a third calling module arranged to generate a target image based on a target text among the plurality of texts according to the machine learning model; and an identification module arranged to determine the target image as an image and the target text as a text.

[0130] According to an exemplary embodiment of the present disclosure, the apparatus further includes a fourth calling module configured to determine, based on the target video according to the machine learning model, a prompt message related to another target video after the target video, the prompt message including at least one image and another text, and the generation module is further configured to generate the other target video based on the at least one image and another text according to the generative model.

[0131] According to an exemplary embodiment of the present disclosure, the generation module further includes an influence factor receiving module arranged to receive an influence factor for specifying the influence degree of the second image, a division module arranged to divide multiple steps for invoking a diffusion model in the generative model into a first stage and a second stage based on the influence factor, a reconstruction module arranged to determine reconstruction features of the target video based on the first image and the second image according to the diffusion model in the first stage and the second stage, and a video generation module arranged to generate the target video based on the reconstruction features using a decoder model in the generative model.

[0132] According to an exemplary embodiment of the present disclosure, the reconstruction module comprises, in a first stage, a first determination module arranged to determine intermediate reconstruction features related to the target video based on the first image and the second image in accordance with a diffusion model, and in a second stage, a second determination module arranged to determine reconstruction features based on the first image and the intermediate reconstruction features in accordance with the diffusion model.

[0133] According to an exemplary embodiment of the present disclosure, the video processing apparatus further includes a model determination module arranged to determine a generative model, the model determination module including an image determination module arranged to determine a first reference image and a second reference image from a plurality of reference images in a reference video, a text receiving module arranged to receive reference text for describing the reference video, and a model acquisition module arranged to obtain a generative model based on the first reference image, the second reference image and the reference text, and the generative model is utilized to generate a target video based on one of the first image and the second image and the text.

[0134] According to an exemplary embodiment of the present disclosure, the first reference image is located at the head of the reference video, and the second reference image is located within a predetermined range from the tail of the reference video.

[0135] According to an exemplary embodiment of the present disclosure, the generative model includes an encoder model and a diffusion model, and the model acquisition module comprises a first feature determination module arranged to determine a first reference feature of the reference video using the encoder model, the first reference feature including a plurality of reference image features of a plurality of reference images, a second feature determination module arranged to determine a second reference feature of the reference video using the encoder model, the second reference feature including a first reference image feature of the first reference image and a second reference image feature of the second reference image, and a diffusion model determination module arranged to determine a diffusion model based on the first reference feature, the second reference feature and the reference text.

[0136] According to an exemplary embodiment of the present disclosure, a first position of a first reference image feature in the second reference feature corresponds to a position of the first reference image in the reference video, and a second position of a second reference image feature in the second reference feature corresponds to a position of the second reference image in the reference video.

[0137] According to an exemplary embodiment of the present disclosure, the dimension of the second reference feature is equal to the dimension of the first reference feature, and features at positions other than the first position and the second position in the second reference feature are set to NULL.

[0138] According to an exemplary embodiment of the present disclosure, the apparatus further comprises a setting module arranged to set the second reference image feature to NULL according to a predetermined condition.

[0139] According to an exemplary embodiment of the present disclosure, the diffusion model determination module includes a noiseization module arranged to perform a noiseization process on the first reference feature and the second reference feature, respectively, to generate a first noisy reference feature and a second noisy reference feature; a combination module arranged to combine the first noisy reference feature and the second noisy reference feature to generate a noisy reference feature of the reference video; a reconstruction module arranged to determine a reconstructed feature of the reference video based on the noisy reference feature and the reference text using a diffusion model; and an update module arranged to update the diffusion model based on a difference between the reconstructed feature and the reference feature.

[0140] According to an exemplary embodiment of the present disclosure, the head image is the first image frame in the target video and the tail image is the last image frame in the target video.

[0141] Figure 18 illustrates a block diagram of an apparatus 1800 capable of implementing embodiments of the present disclosure. It should be understood that the computing device 1800 illustrated in Figure 18 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. The computing device 1800 illustrated in Figure 18 can be used to implement the methods described above.

[0142] As shown in Fig. 18, the computing device 1800 is in the form of a general-purpose computing device. The components of the computing device 1800 may include, but are not limited to, one or more processors or processing units 1810, memory 1820, storage device 1830, one or more communication units 1840, one or more input devices 1850, and one or more output devices 1860. The processing unit 1810 may be a real processor or a virtual processor, and can perform various processes based on a program stored in the memory 1820. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capability of the computing device 1800.

[0143] Computing device 1800 typically includes a number of computer storage media. Such media may be any available media accessible by computing device 1800, including but not limited to volatile and nonvolatile media, removable and non-removable media. Memory 1820 may be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or a combination thereof. Storage device 1830 may be removable or non-removable media and may include machine-readable media, such as a flash drive, a disk, or any other medium that may be used to store information and / or data (e.g., training data for training) and that may be accessible within computing device 1800.

[0144] The computing device 1800 may further include other removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 18, a disk drive for reading from and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading from and writing to a removable non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data medium interfaces. The memory 1820 may include a computer program product 1825 having one or more program modules arranged to perform various methods or operations of various embodiments of the present disclosure.

[0145] The communications unit 1840 enables communication with other computing devices over a communications medium. Additionally, the functionality of the components of the computing device 1800 may be implemented as a single computing cluster or multiple computing machines capable of communicating over a communications connection. Thus, the computing device 1800 may use logical connections to one or more other servers, networked personal computers (PCs), or other network nodes to operate in a networked environment.

[0146] The input device(s) 1850 may be one or more input devices, such as, for example, a mouse, a keyboard, a tracking ball, etc. The output device(s) 1860 may be one or more output devices, such as, for example, a monitor, a speaker, a printer, etc. The computing device 1800 may communicate, as necessary, via the communication unit 1840 with one or more external devices (not shown), such as a storage device, a display device, one or more devices that allow a user to interact with the computing device 1800, or any device (e.g., a network card, a modem, etc.) that allows the computing device 1800 to communicate with one or more other computing devices. Such communication may be performed via an input / output (I / O) interface (not shown).

[0147] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium having computer-executable instructions stored thereon is provided, the computer-executable instructions being executed by a processor to perform the above-mentioned method. According to an exemplary embodiment of the present disclosure, a computer program product is also provided, the computer program product being tangibly stored on a non-transitory computer-readable medium and having computer-executable instructions, the computer-executable instructions being executed by a processor to perform the above-mentioned method. According to an exemplary embodiment of the present disclosure, a computer program product is provided, the computer program product having a computer program stored thereon, the program being executed by a processor to perform the above-mentioned method.

[0148] Aspects of the present disclosure have been described herein with reference to flowcharts and / or block diagrams of methods, apparatus, devices, and computer program products implemented in accordance with the present disclosure. It will be understood that each box in the flowcharts and / or block diagrams, and combinations of boxes in the flowcharts and / or block diagrams, can be implemented by computer readable program instructions.

[0149] These computer readable program instructions can be provided to a processing unit of a general purpose computer, special purpose computer, or other programmable data processing apparatus to create a machine such that the instructions, when executed by the processing unit of the computer or other programmable data processing apparatus, create an apparatus that performs the functions / operations specified in one or more boxes of the flowcharts and / or block diagrams. Also, by storing these computer readable program instructions, which cause a computer, programmable data processing apparatus, and / or other device to function in a particular manner, on a computer readable storage medium, the computer readable medium having the instructions stored thereon has an article of manufacture including instructions that perform each aspect of the functions / operations specified in one or more boxes in the flowcharts and / or block diagrams.

[0150] The computer readable program instructions, when loaded onto a computer, other programmable data processing apparatus, or other device, may cause a series of operational steps to be executed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions executing on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more boxes of the flowcharts and / or block diagrams.

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate architecture, functionality, and operation that may be implemented in the systems, methods, and computer program products implemented according to the present disclosure. In this regard, each box in the flowcharts or block diagrams may represent a module, program segment, or part of an instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the boxes may occur in a different order than those marked in the accompanying drawings. For example, two consecutive boxes may actually be executed substantially in parallel, or may be executed in a reverse order depending on the functions involved. It should also be noted that each box in the block diagrams and / or flowcharts, and combinations of boxes in the block diagrams and / or flowcharts, may be implemented in a dedicated hardware-based system that executes a given function or operation, or may be implemented in a combination of dedicated hardware and computer instructions.

[0152] Although each implementation of the present disclosure has been described above, the above description is illustrative, not exhaustive, and not limited to each of the disclosed implementations. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of each of the described implementations. The selection of terms used in this specification is intended to best explain the principles of each implementation, practical application or improvement of the technology in the market, or to allow other ordinary skilled in the art to understand the various embodiments disclosed herein.

Claims

1. A method for generating a video, the method being performed by an electronic device, the method comprising: receiving a plurality of images for respectively representing a plurality of target images in a target video, the plurality of images including a first image for representing a head image in the plurality of target images and a second image for representing a tail image in the plurality of target images; receiving text describing content of the target video; generating the target video based on the plurality of images and the text according to a generative model; determining features for generating the target video, where a first image feature of the first image is positioned at the beginning of the features and a second image feature of the second image is positioned at the end of the features; generating the target video based on the features and the text to generate the target video. Generating, 13. A method for generating a video, comprising:

2. 2. The method of claim 1, further comprising receiving the second image through at least one of an import controller for importing an address of the second image, a drawing controller for drawing the second image, and an edit controller for editing the first image to generate the second image.

3. providing a plurality of images in the target video; determining a head image to represent another target video after the target video in response to receiving an interaction on a third image in the plurality of images; obtaining another text for describing a content of the other target video and a fourth image for representing a tail image of the other target video; generating the another target video based on the third image, the fourth image and the another text according to the generative model; The method of claim 1 further comprising:

4. Receiving a plurality of images each representing a plurality of keyframes in a first video including a plurality of sub-videos each having different content; receiving a first text describing content of the first video; determining the images and the text based on the plurality of images and the first text according to a machine learning model; The method of claim 1 , wherein the image and the text are determined by:

5. Receiving summary text describing a first video that includes multiple sub-videos, each sub-video having different content; determining a plurality of texts for respectively describing content of a plurality of video segments in the first video based on the summary text according to a machine learning model; generating a target image based on a target text in the plurality of texts according to a machine learning model; said target image being said image and said target text being said text; The method of claim 1 , wherein the image and the text are determined by:

6. determining, based on the target video according to a machine learning model, a prompt message related to another target video after the target video, the prompt message including at least one image and another text; The method of claim 1 , further comprising: generating the other target video based on the at least one image and the other text according to the generative model.

7. generating the target video receiving an influence factor for designating an influence degree of the second image; Dividing a plurality of steps for invoking a diffusion model in the generative model into a first stage and a second stage based on the influencing factors; determining, in the first and second steps, a reconstructed feature of the target video based on the first and second images according to the diffusion model; generating the target video based on the reconstructed features utilizing a decoder model in the generative model; The method of claim 1 further comprising:

8. Determining the reconstruction features of the target video includes: determining, in the first stage, intermediate reconstruction features associated with the target video based on the first image and the second image according to the diffusion model; determining, in the second step, the reconstruction features based on the first image and the intermediate reconstruction features according to the diffusion model; The method of claim 7, comprising:

9. determining a first reference image and a second reference image from a plurality of reference images in a reference video; receiving a reference text for describing the reference video; obtaining a generative model based on the first reference image, the second reference image and the reference text; determining the generative model by The method of claim 1 , wherein the generative model is used to generate the target video based on one of the first image and the second image and the text.

10. 10. The method of claim 9, wherein the first reference image is located at a head of the reference video and the second reference image is located within a predetermined range from a tail of the reference video.

11. The generative model includes an encoder model and a diffusion model, Obtaining the generative model based on the first reference image, the second reference image, and the reference text includes: determining a first reference feature of the reference video utilizing the encoder model, the first reference feature comprising a plurality of reference image features of the plurality of reference images; determining second reference features of the reference video utilizing the encoder model, the second reference features including a first reference image feature of the first reference image and a second reference image feature of the second reference image; determining the diffusion model based on the first reference feature, the second reference feature and the reference text; 10. The method of claim 9, comprising:

12. The method of claim 11 , wherein a first position of the first reference image feature in the second reference feature corresponds to a position of the first reference image in the reference video, and a second position of the second reference image feature in the second reference feature corresponds to a position of the second reference image in the reference video.

13. The method of claim 12 , wherein a dimension of the second reference feature is equal to a dimension of the first reference feature, and features at other positions in the second reference feature than the first position and the second position are set to NULL.

14. The method of claim 13 , further comprising setting the second reference image feature to NULL according to a predetermined condition.

15. Determining the diffusion model based on the first reference feature, the second reference feature, and the reference text includes: performing a noise generation process on the first reference feature and the second reference feature to generate a first noisy reference feature and a second noisy reference feature; combining the first and second noisy reference features to generate a noisy reference feature of the reference video; and determining a reconstruction feature of the reference video based on the noise reference feature and the reference text utilizing the diffusion model; and updating the diffusion model based on a difference between the reconstructed features and the first reference features.

16. The method of claim 1 , wherein the head image is the first image frame in the target video and the tail image is the last image frame in the target video.

17. an image receiving module arranged to receive a plurality of images for respectively representing a plurality of target images in a target video, the plurality of images including a first image for representing a head image in the plurality of target images and a second image for representing a tail image in the plurality of target images; a text receiving module arranged to receive text describing a content of the target video; determining features for generating the target video, where a first image feature of the first image is positioned at the beginning of the features and a second image feature of the second image is positioned at the end of the features; and generating the target video based on the features and the text to generate the target video; a generation module configured to generate the target video based on the plurality of images and the text according to a generative model by 16. An apparatus for generating a video, comprising:

18. At least one processing unit; at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit; An electronic device for performing the method of any one of claims 1 to 15, wherein the instructions, when executed by the at least one processing unit,

19. A computer readable storage medium having stored thereon a computer program which, when executed by a processor, causes said processor to perform the method according to any one of claims 1 to 15.

Images