Video generation method, and server

By using a dual-branch video generation model that combines an image reference branch and a video generation branch, the problem of low-quality text-generated videos is solved, and video generation with higher semantic matching to the descriptive text is achieved.

WO2024228676A1PCT designated stage expired Publication Date: 2024-11-07ALIBABA INNOVATION PRIVATE LIMITED

Patent Information

Application Number
PCT/SG2024/050284
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-04
Filing Date
2024-04-30
Publication Date
2024-11-07

AI Technical Summary

Technical Problem

Existing text-to-video methods produce low-quality videos, especially those with poor semantic matching to the descriptive text.

Method used

A dual-branch video generation model is adopted, including an image reference branch and a video generation branch. The image reference branch extracts reference image features and combines them with the text features of the descriptive text to generate a sequence of frames of the target video. Based on the reference image features and text features, diffusion processing is performed to generate a target video that semantically matches the descriptive text.

Benefits of technology

The quality of the generated videos has been improved, resulting in a higher semantic match between the generated videos and the descriptive text, and better content continuity and dynamic consistency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure SG2024050284_07112024_PF_FP_ABST
    Figure SG2024050284_07112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided in the present disclosure are a video generation method, and a server. The method in the present disclosure comprises: acquiring description text and a reference image, which are used for generating a target video; inputting the description text and the reference image into a double-branch video generation model, and by means of the video generation model, generating, on the basis of the reference image, sequence frames of the target video which semantically match the description text, wherein the video generation model comprises an image reference branch and a video generation branch, the image reference branch is used for extracting a reference image feature of the reference image, and the video generation branch is used for generating the sequence frames of the target video on the basis of the reference image feature and text features of the description text; and further generating the target video according to the sequence frames of the target video. In this way, image features of a given reference image can be introduced into a video generation branch, and the given reference image is diffused on the basis of text features of description text so as to generate a target video, thereby improving the quality of the generated video.
Need to check novelty before this filing date? Find Prior Art

Description

[0001]Cross-Reference This disclosure claims priority to Chinese patent application No. 202310487811.8, filed with the China Patent Office on May 4, 2023, entitled "Method and Server for Video Generation," the entire contents of which are incorporated herein by reference. Technical Field This disclosure relates to computer technology, and more particularly to a method and server for video generation. Background: Text-to-video (T2V) aims to generate a coherent video that semantically matches the input text from a description. This is a cross-modal conversion task that requires a model to capture the spatiotemporal motion patterns of the video while also incorporating textual information, posing significant challenges. Traditional methods for generating video from text are mostly extended from methods for generating images from text. By adding a temporal attention module, the text-to-image conversion is expanded into a text-to-video frame sequence to achieve the goal of generating video. However, traditional methods for generating video from text suffer from low video quality. SUMMARY This disclosure provides a method and server for generating video to address the low video quality issues associated with these methods. In a first aspect, the present disclosure provides a method for video generation, comprising: obtaining descriptive text and a reference image configured to generate a target video; inputting the descriptive text and the reference image into a video generation model, and generating, by the video generation model and based on the reference image, a sequence of frames of a target video that semantically matches the descriptive text, wherein the video generation model includes an image reference branch and a video generation branch, the image reference branch being configured to extract reference image features of the reference image, and the video generation branch being configured to generate sequence frames of the target video based on the reference image features and text features of the descriptive text; and generating a target video based on the sequence frames of the target video. In a second aspect, the present disclosure provides a video generation method, characterized in that it includes: receiving a description text input by a user; generating a reference image that semantically matches the description text based on the description text; inputting the description text and the reference image into a video generation model, and generating a sequence frame of a target video that semantically matches the description text based on the reference image through the video generation model, wherein the video generation model includes an image reference branch and a video generation branch, the image reference branch is configured to extract reference image features of the reference image, and the video generation branch is configured to generate a sequence frame of the target video based on the reference image features and text features of the description text; generating a target video based on the sequence frames of the target video; and outputting the target video.In a third aspect, the present disclosure provides a server comprising: a processor and a memory communicatively connected to the processor; the memory storing computer-executable instructions; and the processor executing the computer-executable instructions stored in the memory to implement the method of the first or second aspect. The video generation method and server provided herein obtain descriptive text and a reference image configured to generate a target video, input the descriptive text and the reference image into a two-branch video generation model, and generate a sequence of target video frames semantically matching the descriptive text based on the reference image using the video generation model. The video generation model includes an image reference branch and a video generation branch. The image reference branch is configured to extract reference image features of the reference image, and the video generation branch generates a sequence of target video frames based on the reference image features and text features of the descriptive text. Furthermore, the target video is generated based on the sequence of target video frames. The image features of a given reference image are introduced into the video generation branch, and the given reference image is diffused to generate the target video based on the text features of the descriptive text, thereby improving the quality of the generated video. BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings herein are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present disclosure, and together with the specification, serve to explain the principles of the present disclosure. Figure 1 is an architecture diagram of an exemplary video generation system applicable to the present disclosure; Figure 2 is a flow chart of a video generation method provided by an exemplary embodiment of the present disclosure; Figure 3 is an architecture diagram of a dual-branch video generation model provided by an exemplary embodiment of the present disclosure; Figure 4 is a structural schematic diagram of an attention layer in a video generation model provided by an exemplary embodiment of the present disclosure; Figure 5 is a structural diagram of a hidden layer temporal attention module provided by an exemplary embodiment of the present disclosure; Figure 6 is an architecture diagram of a dual-branch video generation model configured for image dynamics provided by an exemplary embodiment of the present disclosure; Figure 7 is a flow chart of a method for image dynamics provided by an exemplary embodiment of the present disclosure; Figure 8 is an architecture diagram of a dual-branch video generation model configured for frame interpolation provided by an exemplary embodiment of the present disclosure; Figure 9 is a flow chart of a method for frame interpolation provided by an exemplary embodiment of the present disclosure; Figure 10 is a flow chart of a method for generating video from text provided by an exemplary embodiment of the present disclosure; Figure 11 is an overall framework diagram of generating video from text provided by an exemplary embodiment of the present disclosure; Figure 12 is an interactive flow chart of video generation provided by an exemplary embodiment of the present disclosure; Figure 13 is a flow chart of a method for generating video in a question-and-answer scenario provided by an exemplary embodiment of the present disclosure; Figure 14 is an example diagram of an interactive interface for a question-and-answer scenario provided by an exemplary embodiment of the present disclosure; FIG15 is a schematic diagram of the structure of a server provided in an embodiment of the present disclosure. The above figures have illustrated clear embodiments of the present disclosure, which will be described in more detail below.These figures and descriptions are not intended to limit the scope of the present disclosure in any way, but rather to illustrate the concepts of the present disclosure to those skilled in the art by reference to specific embodiments. The exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, identical numerals in different drawings represent identical or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are examples of apparatuses and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims. First, the terms used in the present disclosure are explained: Text to Image (T2I): Also known as Text to Image, this refers to the conversion of text descriptions into corresponding images. Text to Video (T2V): Also known as Text to Video, this refers to the conversion of text descriptions into corresponding videos. Diffusion models are generative models that generate images or videos by gradually diffusing noise samples. Latent Diffusion Models (LDMs) utilize latent space for generation, reducing computational complexity. Stable Diffusion (SD) models are generative models based on latent diffusion models. They obtain the image of interest by iteratively denoising and sampling an initialized noisy image. By iterating in a low-dimensional latent space rather than the original pixel space, SD models significantly reduce memory and computational requirements. Image animation, also known as image animation, is a technique for converting static images into dynamic ones. Multi-scale Cross-attention is a multi-scale cross-attention method based on the attention mechanism, designed to extract features and fuse information across multiple layers. Image super-resolution (SR) is the process of restoring an image from low resolution to high resolution. Global motion information is extracted based on the features of multiple image frames to be generated, including the overall motion information of multiple image frames. Local motion information is extracted based on the features of a single image frame to be generated, including the motion information of a single image frame.To address the issue of low video quality in text-based video generation methods, such as a low degree of semantic match with a given descriptive text, the present disclosure provides a video generation method. This method obtains descriptive text and a reference image configured to generate a target video. These are then input into a two-branch video generation model. The video generation model then generates a sequence of target video frames that semantically match the descriptive text based on the reference image. The video generation model comprises an image reference branch and a video generation branch. The image reference branch is configured to extract reference image features from the reference image. The video generation branch generates a sequence of target video frames based on the reference image features and text features of the descriptive text. The target video is then generated based on the sequence of target video frames. This method incorporates image features from the given reference image into the video generation branch, allowing the given reference image to be diffused into the target video based on the text features of the descriptive text, thereby improving the quality of the generated video. The video generation method provided in this disclosure aims to generate a target video based on the reference image and matching the descriptive text, based on the descriptive text and reference image. Specifically, the method can be used in various video generation scenarios, such as generating an introduction video for a given item or generating video material. Figure 1 is a diagram of an exemplary video generation system architecture applicable to the present disclosure. As shown in Figure 1 , the system architecture may specifically include a server and end-side devices. The server may be a server cluster located in the cloud or a locally deployed server. A communication link exists between the server and each end-side device, enabling communication between the server and each end-side device. Figure 1 illustrates the example of a cloud server located in the cloud. End-side devices may be devices used by users, specifically hardware devices with network communication, computing, and information display functions, including but not limited to smartphones, tablets, desktop computers, IoT devices, and servers. In one usage scenario, a user sends a given descriptive text and reference image to a server via a end-side device, configured to generate a target video. The end-side device can trigger the server to generate the target video based on the given descriptive text and reference image by sending a video generation request / instruction to the server or by invoking an application program interface (API) configured to generate a video provided by the server and providing the server with the descriptive text and reference image configured to generate the target video.In response to a video generation request / instruction, or invoking a provided application programming interface (API) configured to generate a video, the server obtains descriptive text and reference images provided by the end-user device for generating a target video. The descriptive text and reference images are input into a video generation model. The video generation model then generates a sequence of target video frames that semantically matches the descriptive text based on the reference images. The video generation model includes an image reference branch and a video generation branch. The image reference branch is configured to extract reference image features from the reference image, and the video generation branch is configured to generate a sequence of target video frames based on the reference image features and text features of the descriptive text. The target video is then generated based on the sequence of target video frames. In another usage scenario, as shown in Figure 1, a user sends a given descriptive text configured to generate a target video to the server via the end-user device. The end-user device can trigger the server to generate the target video based on the given descriptive text by sending a video generation request / instruction to the server, or by invoking an application programming interface (API) configured to generate a video provided by the server, and providing the server with the descriptive text configured to generate the target video. In response to a video generation request / instruction, or invoking an application program interface configured to generate a video, the server obtains descriptive text provided by the end-side device and configured to generate a target video. Using a text-to-image model, the server generates a reference image that matches the descriptive text based on textual features of the descriptive text. The descriptive text and the reference image are input into the video generation model, and the video generation model generates a sequence of target video frames that semantically matches the descriptive text based on the reference image. The video generation model includes an image reference branch and a video generation branch. The image reference branch is configured to extract reference image features from the reference image, and the video generation branch is configured to generate a sequence of target video frames based on the reference image features and textual features of the descriptive text. The target video is generated based on the sequence of target video frames. Furthermore, after generating the target video, the server may output the target video to the end-side device, or the server may output the target video, the descriptive text, and / or the reference image to the end-side device. Optionally, the server may also send the sequence of target video frames to the end-side device. The client device receives a sequence of frames from a target video and allows the user to adjust (delete or replace) one or more frames in the sequence. The server generates a video file based on the adjusted sequence of frames and outputs the video file to the user. Optionally, the server can also generate a video file based on the sequence of frames from the target video and send the video file or video file download information to the client device. The client device receives the video file or downloads the video file based on the received video file download information and outputs the video file to the user.For example, in the case of generating video material, a user enters the descriptive text "a dog is running" through a client-side device. The client-side device then sends a sequence of frames of a target video matching the descriptive text "a dog is running" to a server. The server sends the sequence of frames of the target video to the client-side device, which then outputs the sequence of frames for the user to browse and adjust. The target video is then generated and played based on the adjusted sequence of frames. The server generates a video file based on the sequence of frames of the target video and sends the video file to the client-side device. The client-side device then plays the video file for the user to view and download. For example, in a question-and-answer scenario, the server provides an interactive interface. The user enters a descriptive text, such as "a dog is running," through the interactive interface and sends a video generation instruction to the server. Based on the user-entered descriptive text, the server generates a reference image that semantically matches the descriptive text and, based on the reference image, generates a target video that semantically matches the descriptive text. The server then outputs the generated target video through the interactive interface. In addition, the user can edit the target video outputted by the interactive interface. The server will synchronously modify the target video based on the user's editing operations, generate the modified video, and output the modified video on the interactive interface. Furthermore, the server can output a sequence of frames of the target video. The user can specify any frame in the sequence as a reference image, or edit any frame in the specified sequence as a reference image. The server will regenerate a new target video based on the newly specified reference image and output the newly generated target video through the interactive interface. The following detailed embodiments will explain the technical solution of the present disclosure and how it solves the aforementioned technical problems. The following specific embodiments may be combined with each other, and identical or similar concepts or processes may not be described in detail in some embodiments. The embodiments of the present disclosure will be described below with reference to the accompanying figures. Figure 2 is a flow chart of a video generation method provided by an exemplary embodiment of the present disclosure. The execution entity of this embodiment is the server in the aforementioned video generation system architecture, which can be a cloud server or a locally deployed server. As shown in Figure 2, the method comprises the following steps: Step S21: Obtain descriptive text and reference images configured to generate the target video. The video generation method provided in this embodiment can be based on given descriptive text and a reference image. The reference image is diffused according to the descriptive text to generate a sequence of frames that semantically matches the descriptive text. Based on the generated sequence of frames, a target video that semantically matches the descriptive text can be generated at a certain frame rate. When applied to different video generation scenarios, the obtained descriptive text and reference image can be different.Exemplarily, when implementing image animation, a given reference image is used. In this embodiment, a dual-branch video generation model is used to animate the given reference image into multiple image frames based on the text features of a given description text. The multiple image frames are played back at a certain frame rate to form a moving image. Exemplarily, when implementing frame interpolation, the reference image includes a given starting frame and ending frame. In this embodiment, a dual-branch video generation model is used to generate multiple image frames inserted between the starting frame and the ending frame based on the text features of the given description text. Exemplarily, when implementing video editing, a given reference image is used. In this embodiment, a dual-branch video generation model is used to re-edit original video frames in the video to be edited into new video frames based on the text features of the given description text and the reference image. The original video frame in the video to be edited is used as the initial image for the multiple image frames to be generated. Based on the text features of the given description text and the given reference image, multiple new image frames are generated as the edited video frames. The specific processing is similar to that used to dynamically generate multiple image frames, except that the original video frames in the video to be edited are used as the initial images for the multiple image frames to be generated. For example, when implementing virtual fitting, the reference image is a user-provided facial image and a given descriptive text (e.g., "How I look in this dress"). Multiple images of the target garment the user wants to try on (e.g., two-dimensional photos of the target garment from multiple different angles) are used as the initial images for the multiple image frames to be generated. In this embodiment, based on a dual-branch video generation model, the multiple images of the target garment are used as the initial images for the multiple image frames to be generated. Based on the textual features of the given descriptive text and the given user's facial image, multiple new image frames are generated as the user's fitting images. The specific processing is similar to that used to dynamically generate multiple image frames, except that the multiple images of the target garment are used as the initial images for the multiple image frames to be generated. In an optional implementation of this step, a descriptive text set for generating the target video can be obtained, and based on this descriptive text, a pre-trained text-to-image generation model is used to generate reference images that semantically match the descriptive text. This implementation can be applied to scenarios where video is generated based on text, and the process of generating video from text can be converted into two sub-processes: generating an image from text and generating a video based on text and a reference image.By using a pretrained text-to-image model to generate a reference image that matches the descriptive text, the highly accurate pretrained text-to-image model can be fully utilized to obtain a high-quality reference image that closely matches the semantics of the descriptive text. A target video is then generated based on the descriptive text and the high-quality reference image. The generated target video's sequence frames closely match the semantics of the descriptive text, improving the quality of the generated video. Furthermore, when obtaining the descriptive text for generating the target video, user input can be received, allowing the user to directly provide the descriptive text. Furthermore, when obtaining the descriptive text for generating the target video, input content data can be received, which can be any of the following: images, audio, video, or documents. The input content data is converted into text information to obtain the descriptive text for generating the target video. This allows users to input data in various modalities, such as images, audio, video, and documents, automatically converting the user-input content data into text, improving the convenience and efficiency of human-computer interaction. In another optional implementation of this step, the user-provided reference image can be directly obtained, that is, allowing the user to provide both the descriptive text and the reference image. For example, the user can directly input a given reference image, or the user can specify any frame in a sequence of frames in an already generated video as a reference image for generating the target video. Alternatively, the user can edit any frame in a sequence of frames in an already generated video to generate a new frame as the reference image for generating the target video. Furthermore, this optional embodiment can employ the same method for obtaining the description text as in the previous optional embodiment of this step, which will not be further described here. Step S22: The description text and the reference image are input into a video generation model. The video generation model generates a sequence of frames of the target video based on the reference image that semantically matches the description text. The video generation model includes an image reference branch and a video generation branch. The image reference branch is configured to extract reference image features from the reference image, while the video generation branch is configured to generate a sequence of frames of the target video based on the reference image features and text features of the description text. In this embodiment, a dual-branch video generation model is used to generate a sequence of frames based on the input description text and reference image, with the content of the reference image as the basis and semantically matching the description text. The video generation model includes an image reference branch and a video generation branch. Among them, the image reference branch is set to extract reference image features of the reference image.The video generation branch uses a cross-attention mechanism to fuse the reference image features and text features of the descriptive text extracted by the image reference branch with the video frame features of the video frame to be generated. The video frame features are then diffused to generate multiple video frames arranged in a fixed order. These frames are then arranged to form a sequence of frames for the target video. Step S23: Generate a target video based on the sequence of frames of the target video. After generating the sequence of frames of the target video, a video is generated based on the sequence of frames at a certain frame rate, resulting in a target video that uses the content of the reference image as its base content and semantically matches the descriptive text. In this embodiment, descriptive text and a reference image are obtained for generating the target video. These are then input into a two-branch video generation model. The video generation model then generates a sequence of frames of the target video based on the reference image, semantically matching the descriptive text. The video generation model includes an image reference branch and a video generation branch. The image reference branch is configured to extract reference image features from a reference image, while the video generation branch generates a target video sequence frame based on the reference image features and text features of the descriptive text. This allows the image features of a given reference image to be incorporated into the video generation branch. Based on the text features of the descriptive text, the given reference image is diffused to generate a target video sequence frame. The target video is generated based on the target video sequence frame, thereby improving the quality of the generated video. In practical applications, videos generated by traditional video generation models are typically short and unsuitable for scenarios requiring longer video generation. In an optional embodiment, after obtaining the target video sequence frame in step S22, the last frame of the sequence frame can be used as a reference image. Based on the descriptive text and the last frame of the generated sequence frame, the process of step S22 is repeatedly executed to generate a new sequence frame. All sequence frames are then sequentially organized to obtain a longer sequence frame. Specifically, after obtaining a sequence of target video frames, the duration of the target video can be determined based on the sequence of frames and a preset frame rate, and a determination can be made as to whether the duration of the target video is greater than or equal to the preset duration. If the duration of the target video is determined to be insufficient (less than the preset duration), a new sequence of frames is generated based on the description text and a new reference image. The newly generated sequence of frames is then spliced ​​with the existing sequence of frames to obtain a longer sequence of frames. Based on the spliced ​​sequence of frames, a determination is made as to whether the duration of the generated target video is greater than or equal to the preset duration, until a sequence of frames is obtained that satisfies the preset duration requirement. Optionally, the minimum number of frames required for the target video to be generated can be calculated based on the preset frame rate and preset duration.Based on the minimum number of frames and the length of the sequence frames generated by executing S22 once, the number of times S22 should be looped is determined. S22 is looped a corresponding number of times to obtain multiple sequence frames. These multiple sequence frames are then spliced ​​in sequence to obtain the sequence frames of the target video that meets the required duration. The method of this embodiment can generate videos of long duration (unlimited length) with continuous content, meeting various video generation requirements. In an optional embodiment, the video generation solution disclosed herein can be applied in a question-and-answer scenario. Through multiple rounds of dialogue with the user, the system receives video generation requests / instructions, receives description text / input content data, and outputs the generated target video. The system also supports user modification / editing of the generated target video and outputs the modified / edited video. For example, after generating the target video from the target video's sequence frames, in response to a modification of the description text and / or reference image, the server obtains the modified description text and / or reference image and generates a new target video based on the modified description text and / or reference image. Furthermore, a new target video is output, or the target video and modified description text are output, or the target video, modified description text, and reference image are output. The modification operation on the description text may specifically involve modifying part of the description text or replacing the description text with new description text. The modification operation on the reference image may involve designating any image frame in the output target video as a new reference image; editing any image frame in the output target video and designating the edited image as a new reference image; or inputting a new reference image. For example, a user can designate a key image frame in the target video as a new reference image to provide a reference image that better meets user requirements and thus generate a target video that better meets user requirements. For example, a user can designate a region in a key image frame in the target video and use the image within the region as a new reference image to provide a reference image that better meets user requirements and thus generate a target video that better meets user requirements. Optionally, the server can output a sequence of target video frames, allowing the user to specify any frame in the sequence, or a region within any frame, as a new reference image, thereby improving the convenience and efficiency of human-computer interaction. Furthermore, in response to an export operation on at least one frame in the output sequence of target video frames, the selected frame is stored in a designated path. The designated path can be a preset default storage path or a path entered by the user.In an optional embodiment, after obtaining a sequence of target video frames, an image super-resolution model can be used to increase the resolution of the target video frame sequence to obtain high-definition video frames, thereby improving the resolution of the video frames included in the target video and enhancing the visual quality and resolution of the target video. Specifically, each frame in the obtained sequence of frames is input into the image super-resolution model for image super-resolution processing to obtain corresponding high-resolution frames, thereby forming a high-resolution sequence of frames. The image super-resolution model can be implemented using any existing model for image super-resolution processing, such as an interpolation-based image super-resolution model, a reconstruction-based image super-resolution model, or a deep learning-based image super-resolution model, without specific limitation herein. Figure 3 is an architecture diagram of a dual-branch video generation model provided in an exemplary embodiment of the present disclosure. As shown in Figure 3, the dual-branch video generation model provided in the present disclosure includes an image reference branch and a video generation branch. The image reference branch includes multiple first diffusion processing layers, and the video generation branch includes multiple second diffusion processing layers. The first diffusion processing layers of the image reference branch correspond to the second diffusion processing layers of the video generation branch. The input to the image reference branch is the image features of the reference image. The image reference branch iteratively diffuses the features of the reference image through multiple first diffusion processing layers to extract reference image features at multiple scales. The sequence of frames includes multiple image frames to be generated. The input to the video generation branch is the initialization image of the image frame to be generated. The video generation branch iteratively diffuses the features of the initialization image of the image frame to be generated through multiple second diffusion processing layers to extract image frame features at multiple scales. As shown in Figure 3, the video generation model also includes an attention layer, which is connected after each second diffusion processing layer. The attention layer connects the corresponding diffusion processing layers in the two branches and fuses the text features of the descriptive text and the reference image features extracted by the connected first diffusion processing layer into the image frame features extracted by the corresponding second diffusion processing layer. The resulting fused image frame features serve as the input to the next second diffusion processing layer. Furthermore, the decoder decodes the fused image frame features ultimately output by the attention layer into an image, thereby obtaining the sequence of frames of the target video. Based on the dual-branch video generation model provided by the present invention, reference image features at multiple scales of the reference image are extracted through the reference image branch and connected to the video generation branch. The target video is generated through diffusion through the video generation branch. The video generation branch is connected with the reference image prior knowledge of the image reference branch through multi-scale cross attention, which improves the accuracy of the generated video.In an optional embodiment, Figure 4 is a schematic diagram of the structure of the attention layer in a video generation model provided by an exemplary embodiment of the present disclosure. Based on the model architecture shown in Figure 3, as shown in Figure 4, the attention layer includes a spatial attention module, a hidden layer temporal attention module, and a cross-attention module. The attention layer includes spatial attention modules corresponding to a reference image and each image frame to be generated. The spatial attention module corresponding to the reference image is connected to the first diffusion processing layer and is configured to perform spatial attention calculations on the reference image features extracted by the first diffusion processing layer. The inputs of the spatial attention module corresponding to the image frame to be generated are the reference image features extracted by the first diffusion processing layer and the image frame features of the corresponding image frame extracted by the second diffusion processing layer. The module is configured to perform a spatial attention calculation on the image frame features of the corresponding image frame extracted by the connected second diffusion processing layer and the reference image features extracted by the corresponding first diffusion processing layer, and obtain a first intermediate feature of the corresponding image frame by integrating the reference image features into the image frame features of the corresponding image frame. Optionally, the spatial attention module corresponding to the image frame to be generated may specifically use the image frame features of the image frame extracted by the connected second diffusion processing layer as query features, concatenate the reference image features extracted by the corresponding first diffusion processing layer and the query features as key features and value features, and perform a spatial attention calculation to obtain the first intermediate feature of the image frame. Optionally, the spatial attention module corresponding to the image frame to be generated may use the image frame features of the image frame extracted by the connected second diffusion processing layer as query features, concatenate the reference image features extracted by the corresponding first diffusion processing layer as key features and value features, and perform a spatial attention calculation to obtain the first intermediate feature of the image. The input to the hidden layer temporal attention module is the first intermediate features of each image frame output by the spatial attention module. The hidden layer temporal attention module is configured to extract global motion information from the first intermediate features of each image frame and integrate this global motion information into the first intermediate features of each image frame to obtain the second intermediate features of each image frame. The attention layer includes a cross-attention module corresponding to each image frame to be generated. The input to the cross-attention module is the text features describing the text and the second intermediate features of the corresponding image frame. The cross-attention module is configured to perform a cross-attention calculation on the second intermediate features of the corresponding image frame and the text features describing the text to obtain a fused feature for the corresponding image frame. This fused feature is then input into the next second diffusion processing layer.Exemplarily, the cross-attention module is specifically configured to use the second intermediate features of each image frame as query features (Q), and the text features describing the text as key features (K) and value features (V), perform cross-attention calculations, and obtain fused features for the image frame. The obtained fused features are then input into the next diffusion processing layer. Figure 4 illustrates a possible implementation of the attention layer, using the attention layer connected after the first diffusion processing layer as an example. The structure of the attention layers connected after the subsequent diffusion processing layers is similar (not shown in Figure 4). In the example structure shown in Figure 4, the attention layer can be composed of multiple layers of stacked attention units. The portion within the dashed box in Figure 4 represents a layer of attention units. Each layer of attention units includes a spatial attention module, a hidden layer temporal attention module, and a cross-attention module. The fused features of multiple image frames output by the attention units in the previous layer serve as the image frame features of multiple image frames input to the attention units in the next layer. The fused features of each image frame output by the attention units in the last layer of the attention layer serve as the final fused features of the image frames output by the attention units in the attention layer. In another optional embodiment, in the first-layer attention units of the attention layer, the spatial attention module may perform spatial attention calculations on the image frame features of each input image frame separately, without incorporating the reference image features of the reference image into the image frame features of each image frame. That is, in the first-layer attention units of the attention layer, the image frame features of the image frame extracted by the connected second diffusion processing layer are used as query features, key features, and value features to perform spatial attention calculations and obtain the first intermediate features of the image frame. In subsequent layers of attention units, the reference image features of the reference image are then incorporated into the image frame features of each image frame. That is, the image frame features of the input image frame are used as query features, and the reference image features of the corresponding layer are concatenated with the query features as key features and value features to perform spatial attention calculations and obtain the first intermediate features of the image frame. By stacking multiple layers of stacking units in the attention layer, reference image features and text features can be fully integrated with the image frame features of the image frame to be generated. Global motion information can also be fully incorporated into the image frame features of each image frame, thereby helping to improve the consistency of the generated image frames, ensuring a high degree of consistency in content and dynamics. In an optional embodiment, FIG5 is a structural diagram of a latent temporal attention module provided in an exemplary embodiment of the present disclosure. As shown in FIG5 , the latent temporal attention module may specifically include an attention pooling unit, a self-attention unit, and a cross-attention unit.The attention pooling unit is configured to perform an attention pooling operation on the first intermediate features of the input image frames to obtain local motion information for each frame. The self-attention unit is configured to perform a self-attention calculation on the concatenated features of the local motion information of each frame to obtain global motion information for the frame. The cross-attention unit is configured to perform a cross-attention calculation using the first intermediate features of each frame as query features and the global motion information of the frame as key and value features to obtain the second intermediate features of each frame. The hidden layer temporal attention module integrates the global motion information of each frame into the frame features of each frame, helping to improve the consistency of the detailed content of each generated frame, ensuring a high degree of consistency in content and dynamics. It should be noted that during model training, the spatial attention modules in the attention layer, the attention pooling units in the hidden layer temporal attention module, and the cross-attention units in the hidden layer temporal attention module share parameters, which can reduce training complexity and improve model training efficiency. In an optional embodiment, the image reference branch in the dual-branch video generation model can employ a diffusion model based on a two-dimensional U-net. When training the video generation model, the image reference branch is initialized using the parameters of a pretrained text-to-image model, and the image reference branch parameters are fixed during training. This allows the reuse of prior knowledge from the pretrained text-to-image model, reducing the difficulty and complexity of model training. Alternatively, the image reference branch can employ a stable diffusion model based on a two-dimensional U-net, which can reduce the computational complexity of the first diffusion processing layer in the image reference branch and improve video generation efficiency. Alternatively, in the image reference branch, the reference image features extracted by the previous first diffusion processing layer can be used as input for the next first diffusion processing layer. Alternatively, in the image reference branch, the results of spatial attention calculations on the reference image features extracted by the previous first diffusion processing layer can be used as input for the next first diffusion processing layer, thereby improving the model's expressiveness and, consequently, the quality of video generation. The parameters of the spatial attention module in the image reference branch, which performs spatial attention calculations on the reference image features extracted by the previous first diffusion processing layer, are fixed during model training. This can reduce the training load and improve training efficiency. In an optional embodiment, the video generation branch in the dual-branch video generation model employs a diffusion model based on a three-dimensional U-net (3D U-Net).When training the video generation model, the parameters of the pretrained text-to-image model are used to initialize the video generation branch. The parameters of the video generation branch and the attention layer are updated during training. Optionally, the video generation branch can employ a stable diffusion model based on a three-dimensional U-shaped network. This can reduce the computational complexity of the second diffusion processing layer in the video generation branch and improve video generation efficiency. The dual-branch video generation model provided in this embodiment uses the pretrained text-to-image model to initialize the parameters of both branches. During training, the parameters of the image reference branch are fixed, and the parameters of the video generation branch and the attention layer are updated. This fully leverages the prior knowledge of the pretrained text-to-image model, reducing the difficulty and complexity of model training. Furthermore, multi-scale cross-attention is used to connect the prior image knowledge of the video generation branch with the image reference branch. Multi-scale cross-attention is then used to dynamically diffuse the reference image into a video. This combines the advantages of both the text-to-image model and video generation. By first generating a high-quality reference image using the text-to-image model and then connecting the reference image to the video generation branch using multi-scale cross-attention, the training difficulty and computational complexity can be significantly reduced while ensuring video quality, thereby lowering computing resource requirements. The dual-branch video generation model provided in this embodiment can be applied to tasks such as text-to-video generation, frame interpolation, image animation generation, video editing, and virtual fitting. In an optional embodiment, Figure 6 shows the architecture of a dual-branch video generation model configured for image animation, provided in an exemplary embodiment of the present disclosure. In an image animation (image animation) scenario, a given reference image is animated based on a given descriptive text to generate multiple image frames. These multiple image frames can be organized into an animation or video, achieving image animation / image animation of the reference image. Based on the model architecture shown in Figure 4, when applied to an image animation (image animation) scenario, as shown in Figure 6, the spatial attention module corresponding to the image frame to be generated can specifically use the image frame features extracted by the connected second diffusion processing layer as query features (Q), concatenate the reference image features extracted by the corresponding first diffusion processing layer with the query features to form key features (K) and value features (V), perform spatial attention calculations, and obtain the first intermediate features of the image frame. As shown in Figure 6, each cross-attention module is specifically configured to use the second intermediate feature of each image frame as the query feature (Q), the text features describing the text as the key feature (K) and the value feature (V), perform cross-attention calculations, and obtain a fused feature for the image frame. This fused feature is then input into the next second diffusion processing layer. The rest of the structure is the same as in Figure 4 and will not be further described here.FIG7 is a flow chart of a method for image animation provided by an exemplary embodiment of the present disclosure. As shown in FIG7 , based on the dual-branch video generation model architecture shown in FIG6 , the specific steps of image animation are as follows: Step S71: In response to an image animation request, obtain a given description text and a reference image to be animated. When implementing image animation, the reference image is a given image. In this embodiment, based on the dual-branch diffusion model, the given reference image is animated into multiple image frames based on the text features of the given description text. The multiple image frames are played back at a certain frame rate to form an animated image. For example, the given image may be an image of a dog on an outdoor lawn, and the given description text may be "a dog is running." Based on the description text, the given reference image is animated to obtain multiple image frames of a dog running on an outdoor lawn. The multiple image frames are played back continuously to form an animated image of a dog running on an outdoor lawn. The specific implementation of this step is similar to that of step S21. Please refer to the relevant content of the aforementioned embodiment for details and will not be repeated here. In step S72, the description text and reference image are input into a video generation model. The video generation model then generates a sequence of target video frames semantically matching the description text based on the reference image. The video generation model includes an image reference branch and a video generation branch. The image reference branch is configured to extract reference image features from the reference image, while the video generation branch is configured to generate multiple image frames based on the reference image features and text features of the description text. The specific implementation of this step is similar to that of step S22. For details, please refer to the relevant content of the previous embodiment and will not be repeated here. In step S73, a GIF file is generated based on the multiple image frames and output. In this embodiment, the dual-branch video generation model implements image animation of the reference image based on the description text, which can improve the quality of the image animation. In an optional embodiment, FIG8 is an architecture diagram of a dual-branch video generation model configured for frame interpolation, provided in an exemplary embodiment of the present disclosure. In the frame interpolation scenario, based on the given description text and two given reference images, a start frame and an end frame, multiple image frames are generated to be inserted between the start frame and the end frame. Based on the model architecture shown in Figure 4, when applied to the frame interpolation scenario, as shown in Figure 8, the spatial attention module corresponding to the image frame to be generated can specifically use the image frame features of the corresponding image frame extracted by the connected second diffusion processing layer as the query feature (Q), and the concatenation of the reference image features extracted by the corresponding first diffusion processing layer as the key feature (K) and value feature (V), perform spatial attention calculation, and obtain the first intermediate feature of the image.As shown in Figure 8 , each cross-attention module is specifically configured to use the second intermediate features of each image frame as the query feature (Q), the text features of the description text as the key feature (K) and the value feature (V), perform cross-attention calculations, obtain fused features for the image frame, and input the obtained fused features for the image frame into the next second diffusion processing layer. The remaining parts are the same as those in Figure 4 and are not further described here. Figure 9 is a flow chart of a frame interpolation method provided by an exemplary embodiment of the present disclosure. As shown in Figure 9 , based on the dual-branch video generation model architecture shown in Figure 8 , the specific steps of frame interpolation are as follows: Step S91: In response to a frame interpolation request, obtain given description text and a reference image for the frame to be interpolated. The reference image includes a starting frame and an ending frame. When implementing frame interpolation, the reference image includes a given starting frame and an ending frame. In this embodiment, based on the dual-branch video generation model, multiple image frames to be interpolated between the starting frame and the ending frame are generated based on the text features of the given description text. For example, given two images of a person wearing astronaut clothing riding a horse (with different postures), one as the starting frame and the other as the ending frame, the given description text may be "An astronaut riding a horse." Based on the description text, multiple image frames of the astronaut riding a horse are inserted between the starting frame and the ending frame, so that the actions in the multiple image frames connect the actions in the starting frame and the ending frame, and are played continuously to form a GIF / video of the astronaut riding a horse. The specific implementation of this step is similar to the implementation of the aforementioned step S21. For details, please refer to the relevant content of the aforementioned embodiment and will not be repeated here. Step S92: The description text and reference image are input into a video generation model. The video generation model generates a sequence of target video frames that semantically match the description text based on the reference image. The video generation model includes an image reference branch and a video generation branch. The image reference branch is configured to extract reference image features of the reference image, and the video generation branch is configured to generate multiple image frames based on the reference image features and text features of the description text. The specific implementation of this step is similar to the implementation of the aforementioned step S22. For details, please refer to the relevant content of the aforementioned embodiment and will not be repeated here. Step S93: Insert the generated image frame to be inserted between the start and end frames to obtain a sequence of interpolated frames. In this embodiment, the dual-branch video generation model allows for the insertion of multiple image frames between a given start and end frame based on descriptive text to form a sequence of frames, thereby improving the quality of frame interpolation.Figure 10 is a flow chart of a method for generating video from text, provided in an exemplary embodiment of the present disclosure. As shown in Figure 10 , based on a dual-branch video generation model, generating video from text can be implemented using the following steps: Step S101: Obtaining description text for generating a target video. In this embodiment, based on the given description text, a pre-trained text-to-image model is used to generate a reference image that semantically matches the description text. The dual-branch video generation model is then used to generate a sequence of frames that semantically match the description text. Based on the generated sequence of frames, a target video that semantically matches the description text can be generated at a certain frame rate. Step S102: Based on the text features of the description text, a reference image that matches the description text is generated. In this step, the description text is first encoded using a text encoding model to generate text features of the description text. Specifically, any existing text encoding model that encodes text into feature vectors can be used for implementation, such as a text encoder from a Contrastive Language-Image Pre-training (CLIP) model, though this is not specifically limited here. After obtaining the text features of the description text, the text features are input into a pre-trained text-to-image (T2I) model. The pre-trained T2I model generates a high-quality reference image that matches the description text. The pre-trained T2I model can be implemented using a Stable Diffusion (SD) model, or an existing, effective pre-trained T2I model, which is not specifically limited here. In step S103, based on the text features of the description text, a dual-branch video generation model is used to perform image animation processing on the reference image, generating multiple keyframes that semantically match the description text. After obtaining a reference image that matches the description text, this step uses the dual-branch video generation model configured for image animation, as shown in FIG6 , to perform image animation processing on the reference image based on the text features of the description text, generating multiple keyframes that semantically match the description text. The number of keyframes generated in this step can be set based on actual needs and is not specifically limited here. The specific implementation of this step is described in the image animation method shown in FIG. 7 . For details, refer to the relevant content of the previous embodiment and will not be repeated here. In step S104, based on the text features of the descriptive text, a dual-branch video generation model is used to insert at least one video frame between adjacent frames in the keyframes to obtain a sequence of frames of the target video.After obtaining multiple keyframes that semantically match the description text, this step uses the dual-branch video generation model configured for frame interpolation, as shown in FIG8 . Based on the textual features of the description text, at least one video frame is inserted between adjacent keyframes. The multiple keyframes and the video frames inserted between adjacent keyframes constitute the target video frame sequence. The number of video frames inserted between adjacent frames in this step can be set based on actual needs and is not specifically limited here. The specific implementation of this step is similar to the implementation of the frame interpolation method shown in FIG9 . For details, please refer to the relevant content of the previous embodiment and will not be repeated here. The method of this embodiment converts the process of generating video from text into a process of generating images from text, and performing image animation and frame interpolation on reference images based on the text. A pretrained text-to-image model is used to generate reference images that match the description text based on the description text. This highly accurate pretrained text-to-image model is fully utilized to obtain high-quality reference images that closely match the semantics of the description text. Furthermore, the reference images are animated based on the textual features of the description text to generate multiple key frames that semantically match the description text. Based on the textual features of the description text, at least one video frame is inserted between adjacent frames in the key frames to obtain a sequence of frames of the target video. Thus, based on the description text and the high-quality reference images, a sequence of frames of the target video is generated through image animation and frame interpolation. The generated sequence of frames of the target video closely matches the semantics of the description text, thereby improving the quality of the generated video. In an optional embodiment, after generating a sequence of frames for a target video, the last frame of the sequence can be used as a reference image. Based on the description text and the last frame of the generated sequence, steps S92-S94 are repeatedly executed to generate a new sequence of frames. All the sequence frames are sequentially organized to obtain a longer sequence of frames. Specifically, after obtaining the sequence of frames for the target video, the duration of the target video can be determined based on the sequence of frames and a preset frame rate, and a determination is made as to whether the duration of the target video is greater than or equal to the preset duration. If the duration of the target video is determined to be insufficient (less than the preset duration), a new sequence of frames is generated based on the description text and a new reference image. The newly generated sequence of frames is then spliced ​​onto the existing sequence of frames to obtain a longer sequence of frames. Based on the spliced ​​sequence of frames, a determination is made as to whether the duration of the generated target video is greater than or equal to the preset duration, until a sequence of frames is obtained that satisfies the preset duration requirement for the target video. Optionally, the minimum number of frames required for the target video to be generated can be calculated based on the preset frame rate and preset duration.Based on the minimum number of frames and the length of the sequence frames generated by executing S92-S94 once, the number of iterations of S22 is determined. S92-S94 is then executed a corresponding number of times to obtain multiple sequence frames. These multiple sequence frames are then spliced ​​together in sequence to obtain the sequence frames of the target video that meets the required duration. The method of this embodiment can generate videos of long duration (unlimited length) with continuous content, adapting to various video generation requirements. For example, Figure 11 illustrates the overall framework of text-to-video generation according to an exemplary embodiment of the present disclosure. As shown in Figure 11, the overall framework of text-to-video generation provided by the present disclosure includes the following four components: text-to-image (T2I), image animation (I2V), frame interpolation (FR), and image super-resolution (SR). Text-to-image (T2I) refers to generating a reference image based on text features describing the text. Image animation (I2V) involves performing image animation processing on a reference image based on the textual features of the descriptive text using the dual-branch video generation model shown in Figure 6, generating multiple keyframes that semantically match the descriptive text. Frame interpolation (FR) involves inserting at least one video frame between adjacent keyframes based on the textual features of the descriptive text using the dual-branch video generation model shown in Figure 8, to obtain a sequence of frames for the target video. Image super-resolution (SR) involves performing image super-resolution processing on the obtained sequence of frames to obtain a sequence of frames with higher resolution. In an optional embodiment, the text-to-video solution provided in this disclosure can be applied in question-answering scenarios. Through multi-round conversations with the user, it receives video generation requests / instructions (including descriptive text), receives the descriptive text / input content data, and outputs the generated target video. It also supports user modification / editing of the generated target video and outputs the modified / edited video. Exemplarily, after generating a target video from a sequence of frames of the target video, in response to a modification operation on the description text and / or reference image, the server obtains the modified description text and / or reference image; and generates a new target video based on the modified description text and / or reference image. Furthermore, the server outputs the new target video, or outputs the target video and the modified description text, or outputs the target video, the modified description text, and the reference image. The modification operation on the description text may specifically involve modifying part of the description text or replacing the description text with new description text.Modifying a reference image can involve designating any frame in the output target video as a new reference image; editing any frame in the output target video and designating the resulting image as a new reference image; or inputting a new reference image. For example, a user can designate a key frame in the target video as a new reference image, providing a reference image that better meets user needs and helps generate a target video that better meets user needs. For example, a user can designate a region within a key frame in the target video and use the image within that region as a new reference image, providing a reference image that better meets user needs and helps generate a target video that better meets user needs. Alternatively, the server can output a sequence of target video frames, allowing the user to designate any frame in the sequence, or an image within a region within any frame, as a new reference image, thereby improving the convenience and efficiency of human-computer interaction. Furthermore, in response to an export operation on at least one frame in the output target video sequence, the selected at least one frame is stored in a designated path. The specified path can be a preset default storage path or a path entered by the user. Figure 12 is an interactive flow chart for video generation provided by an exemplary embodiment of the present disclosure. As shown in Figure 12, during the video generation process, the interaction process between the terminal device and the server is as follows: Step S1201: The terminal device receives input description text, and the description text is set to generate the target video. Step S1202: The terminal device sends a video generation request to the server, and the video generation request includes the description text. Step S1203: The server obtains the description text sent by the terminal device, which is set to generate the target video. In this step, the server receives the video generation request sent by the terminal device and obtains the description text set to generate the target video from the video generation request. In step S1204, the server generates a reference image that matches the description text based on the text features of the description text. In step S1205, the server performs image dynamic processing on the reference image based on the text features of the description text to generate multiple key frames that semantically match the description text. In step S1206, the server inserts at least one video frame between adjacent frames in the key frames based on the text features of the description text to obtain a sequence of frames of the target video. In this embodiment, the specific implementation of steps S1204-S1206 is the same as that of steps S102-S104 above. For details, please refer to the relevant content of the above embodiment and will not be repeated here. In step S1207, the server sends the sequence of frames of the target video to the end-side device. In step S1208, the end-side device receives the sequence of frames of the target video sent by the server.Step S1209: The client device outputs the target video's sequence of frames. Step S1210: In response to an adjustment operation on at least one frame in the target video's sequence of frames, the client device updates the target video's sequence of frames. In this embodiment, the server returns the generated sequence of frames to the client device, which can display the multiple frames contained in the sequence of frames for the user to view and adjust. The user can perform adjustment operations such as deleting, replacing, or modifying any frame in the sequence of frames. Based on the adjustment operation on at least one frame in the target video's sequence of frames, the client device adjusts the corresponding frame and updates the sequence of frames, obtaining an updated target video's sequence of frames. Step S1211: A video file is generated based on the updated target video's sequence of frames and outputs the video file. After obtaining the updated target video's sequence of frames, the client device generates a video file for the target video at a preset frame rate and outputs the video file. In another embodiment, the server can generate the video file based on the target video's sequence of frames and the preset frame rate, and the server sends the generated video file to the client device. The end-side device receives and outputs the video file. The method provided in this disclosure can be applied to question-and-answer scenarios. During a multi-round conversation with a user, it can generate a video based on a given descriptive text; generate a video based on the descriptive text and a reference image; dynamically process a given reference image into an animated image / video based on the descriptive text; insert multiple frames between the starting and ending frames based on the descriptive text to form a continuous animated image / video, etc. The method also supports user modification / editing of the generated video and outputs the modified / edited video. Figure 13 is a flow chart of a method for generating a video in a question-and-answer scenario, provided by an exemplary embodiment of this disclosure. As shown in Figure 13, the method comprises the following steps: Step S131: Receive descriptive text input by the user. In a question-and-answer scenario, the user can enter descriptive text for generating a target video in the text input area provided in the interactive interface; alternatively, the user can enter a voice description of the content of the video to be generated, and the server will convert the voice description into text; alternatively, the user can enter or specify content such as a video, image, or document as the description content of the target video, and the server will convert the specified input content such as the video, image, or document into text as the descriptive text for generating the target video. In this embodiment, the specific implementation method for the user-entered description text is not specifically limited. Step S132: Generate a reference image semantically matching the description text based on the description text. In this step, the server uses a pre-trained text-to-image model based on the user-entered description text to generate a reference image semantically matching the description text. Specifically, the server encodes the description text using a text encoding model to generate text features for the description text.Specifically, this can be implemented using any existing text encoding model that encodes text into feature vectors, such as a text encoder using the Contrastive Language-Image Pre-training (CLIP) model. This is not a specific limitation here. After obtaining text features describing the text, these features are input into a pre-trained Text-to-Image (T2I) model. The pre-trained T2I model generates a high-quality reference image that matches the text. The pre-trained T2I model can be implemented using a Stable Diffusion (SD) model, or using an existing, effective pre-trained T2I model. This is not a specific limitation here. In step S133, the description text and reference image are input into a video generation model. The video generation model then generates a target video frame sequence based on the reference image, semantically matching the description text. The video generation model includes an image reference branch and a video generation branch. The image reference branch is configured to extract reference image features from the reference image, while the video generation branch is configured to generate the target video frame sequence based on the reference image features and text features of the description text. In this embodiment, this step can be implemented using the aforementioned step S22. For details, please refer to the aforementioned embodiment and will not be repeated here. In step S134, a target video is generated based on the target video frame sequence. After generating the target video frame sequence, a video is generated based on the frame sequence and a certain frame rate, resulting in a target video based on the reference image content and semantically matching the description text. In step S135, the target video is output. For example, Figure 14 illustrates an example of an interactive interface for a question-and-answer scenario provided in this embodiment. This example uses the user entering a text description as an example. As shown in Figure 14, after enabling the video generation function, the Q&A system displays a prompt message, "Please provide a text description of the video you wish to generate," prompting the user to enter the text description. After the user enters the text "a dog is running," the Q&A system generates a target video based on the user-entered description and outputs the target video. Optionally, this step also outputs the generated target video along with the description text and / or a reference image matching the description text for the user to review, modify, or edit. Furthermore, the user can modify / edit the generated target video, and the modified / edited video is output. For example, the user can modify the description text and / or reference image used to generate the target video.The server can regenerate a new target video based on the modified description text and / or reference image and output the new target video, thereby achieving the effect of synchronously modifying the target video based on the modification operations on the description text and / or reference image. Furthermore, the server can also output the modified description text and / or reference image simultaneously with the new target video. The user's modification of the description text can specifically involve modifying part of the description text or replacing the description text with new description text. The user's modification of the reference image can involve designating any frame in the output target video as a new reference image; editing any frame in the output target video and designating the resulting edited image as a new reference image; or inputting a new reference image. For example, the user can designate a key frame in the target video as a new reference image to provide a reference image that better meets the user's needs and generate a target video that better meets the user's needs. For example, the user can designate a region in a key frame in the target video and use the image within that region as a new reference image to provide a reference image that better meets the user's needs and generate a target video that better meets the user's needs. Optionally, the server can output a sequence of frames of a target video, allowing the user to specify any frame in the sequence, or an image within a region within any frame, as a new reference image, thereby improving the convenience and efficiency of human-computer interaction. Furthermore, in response to an export operation on at least one frame in the output sequence of frames of the target video, the selected frame is stored in a designated path. The designated path can be a preset default storage path or a path entered by the user. The method of this embodiment can be applied to question-and-answer scenarios, implementing human-computer interaction including video generation. Based on one or more modal information input by the user, the server determines descriptive text and reference images, generates the target video, and provides the target video to the user. The server also supports user modification / editing of the generated target video, enriching the interactive content and methods of question-and-answer scenarios. Figure 15 is a schematic diagram of the server structure provided by an embodiment of the present disclosure. As shown in FIG15 , the server includes a memory 1501 and a processor 1502. The memory 1501 is configured to store computer-executable instructions and can be configured to store various other data to support operations on the server. The processor 1502 is in communication with the memory 1501 and is configured to execute the computer-executable instructions stored in the memory 1501 to implement the technical solution provided by any of the above method embodiments. The specific functions and technical effects achieved by these embodiments are similar and will not be further described here.Optionally, as shown in Figure 15 , the server further includes other components, such as a firewall 1503, a load balancer 1504, a communication component 1505, and a power supply component 1506. The schematic illustration of some components in Figure 15 does not imply that the server only includes the components shown in Figure 15 . The present disclosure also provides a computer-readable storage medium storing computer-executable instructions. When executed by a processor, the computer-executable instructions are configured to implement the solution provided by any of the aforementioned method embodiments. The specific functions and technical effects achieved are not detailed here. The present disclosure also provides a computer program product, comprising a computer program stored in a readable storage medium. At least one processor of the server can read the computer program from the readable storage medium. Execution of the computer program by the at least one processor causes the server to implement the solution provided by any of the aforementioned method embodiments. The specific functions and technical effects achieved are not detailed here. The present disclosure also provides a chip, comprising a processing module and a communication interface. The processing module is capable of implementing the technical solution provided by the server in the aforementioned method embodiments. Optionally, the chip further includes a storage module (e.g., memory), configured to store instructions, and a processing module configured to execute the instructions stored in the storage module. Execution of the instructions stored in the storage module causes the processing module to implement the technical solution provided by any of the aforementioned method embodiments. The memory may be an object storage service (OSS). The memory may be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The communication component is configured to facilitate wired or wireless communication between the device in which the communication component is located and other devices. The device containing the communication component can access wireless networks based on communication standards, such as mobile hotspots (WiFi), second-generation (2G), third-generation (3G), fourth-generation (4G) / long-term evolution (LTE), fifth-generation (5G), or other mobile communication networks, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component also includes a near-field communication (NFC) module to facilitate short-range communication.For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies. The aforementioned power supply assembly provides power to various components of the device in which the power supply assembly resides. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device in which the power supply assembly resides. Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, such that the instructions, when executed by the processor of the computer or other programmable data processing device, produce a device configured to implement the functions specified in one or more processes in the flowcharts and / or one or more blocks in the block diagrams. These computer program instructions can also be stored in a computer-readable memory capable of directing the computer or other programmable data processing device to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device that implements the functions specified in one or more processes in the flowcharts and / or one or more blocks in the block diagrams. These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing the computer or other programmable device to execute a series of operational steps to produce a computer-implemented process. The instructions executed on the computer or other programmable device provide steps configured to implement the functions specified in one or more flows in a flowchart and / or one or more blocks in a block diagram. In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.Memory may include non-permanent storage in computer-readable media, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media. Computer-readable media include both permanent and non-permanent, removable and non-removable media, and may be implemented using any method or technology for information storage. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmitting medium configured to store information accessible by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves. It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, storage, and display) referred to in this disclosure are all authorized by the user or fully authorized by all parties. The collection, use, and processing of relevant data must comply with relevant laws, regulations, and standards, and corresponding operation portals are provided for users to choose to authorize or deny. Furthermore, some processes described in the above embodiments and accompanying figures include multiple operations that appear in a specific order. However, it should be understood that these operations may be executed in a different order than those presented herein or in parallel. Sequence numbers are used to distinguish between different operations, and do not represent any specific execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the terms "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not indicate a sequential order or limit the "first" and "second" to different types. "Multiple" means more than two, unless otherwise specified.It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, storage, and display) referred to in this disclosure are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of the relevant data must comply with relevant laws, regulations, and standards, and corresponding operation portals are provided for users to choose to authorize or reject. Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be regarded as exemplary; the true scope and spirit of the present disclosure are indicated by the following claims. It should be understood that the present disclosure is not limited to the precise structure described above and illustrated in the drawings, and various modifications and changes may be made without departing from its scope. The scope of the present disclosure is defined by the appended claims. Industrial Applicability: The video generation method and server provided by the embodiments of the present disclosure obtain descriptive text and a reference image configured to generate a target video, input the descriptive text and the reference image into a two-branch video generation model, and then generate a sequence of target video frames that semantically match the descriptive text based on the reference image. The video generation model includes an image reference branch and a video generation branch. The image reference branch is configured to extract reference image features from the reference image, and the video generation branch generates a sequence of target video frames based on the reference image features and text features of the descriptive text. Furthermore, the target video is generated based on the sequence of target video frames. The video generation branch can incorporate image features of the given reference image into the video generation branch, and diffuse the given reference image into the target video based on text features of the descriptive text, thereby improving the quality of the generated video.

Claims

Claims 1. A method for generating a video, comprising: Get the description text and reference image set to generate the target video; The description text and the reference image are input into a video generation model, and a sequence frame of a target video semantically matching the description text is generated based on the reference image by the video generation model, wherein the video generation model includes an image reference branch and a video generation branch, wherein the image reference branch is configured to extract reference image features of the reference image, and the video generation branch is configured to generate a sequence frame of the target video based on the reference image features and text features of the description text; and a target video is generated according to the sequence frame of the target video.

2. The method according to claim 1, wherein: The image reference branch includes a plurality of first diffusion processing layers, the video generation branch includes a plurality of second diffusion processing layers, the first diffusion processing layers and the second diffusion processing layers are arranged correspondingly, the video generation model further includes an attention layer, and the attention layer connects the diffusion processing layers arranged correspondingly in the two branches; The features of the reference image are iteratively diffused through the multiple first diffusion processing layers of the image reference branch to extract reference image features of multiple scales of the reference image; the features of the initialization image of the image frame to be generated are iteratively diffused through the multiple second diffusion processing layers of the video generation branch to extract image frame features of multiple scales of the image frame, and the sequence frame includes multiple image frames to be generated; the text features of the descriptive text and the reference image features extracted by the connected first diffusion processing layer are fused into the image frame features extracted by the corresponding second diffusion processing layer through the attention layer, and the obtained fused features of the image frame are used as the input of the next second diffusion processing layer; the sequence frame of the target video is obtained by decoding the fused features of the image frame finally output by the attention layer.

3. The method according to claim 2, wherein: The attention layer includes a spatial attention module, a hidden layer temporal attention module and a cross attention module, wherein the hidden layer temporal attention module includes: an attention pooling unit, a self-attention unit and a cross attention unit; the spatial attention module is configured to: perform spatial attention calculation on the image frame features extracted by the connected second diffusion processing layer and the reference image features extracted by the corresponding first diffusion processing layer to obtain the first intermediate features of the image; the hidden layer temporal attention module is configured to: perform attention pooling operations on the first intermediate features of the input image frames respectively through the attention pooling unit to obtain the local motion information of each image frame; perform self-attention calculation on the splicing features of the local motion information of each image frame through the self-attention unit to obtain the global motion information of the image frame; perform cross-attention calculation on the first intermediate features of each image frame respectively as query features through the cross-attention unit, and use the global motion information of the image frame as the key feature and value feature to obtain the second intermediate features of each image frame; the cross-attention module is configured to: use the second intermediate features of each image frame as the query feature, and the text features of the description text as the key feature and value feature, Perform cross attention calculation to obtain fusion features of the image frame, and input the obtained fusion features of the image frame into the next second diffusion processing layer.

4. The method according to claim 3, wherein: The spatial attention module performs spatial attention calculation on the image frame features of the image frame extracted by the connected second diffusion processing layer and the reference image features extracted by the corresponding first diffusion processing layer to obtain the first intermediate features of the image, including: the spatial attention module uses the image frame features of the image frame extracted by the connected second diffusion processing layer as query features, concatenates the reference image features extracted by the corresponding first diffusion processing layer and the query features as key features and value features, performs spatial attention calculation, and obtains the first intermediate features of the image frame; or The spatial attention module uses the image frame features of the image frame extracted by the connected second diffusion processing layer as query features, concatenates the reference image features extracted by the corresponding first diffusion processing layer as key features and value features, performs spatial attention calculation, and obtains the first intermediate features of the image.

5. The method according to claim 2, wherein: The image reference branch adopts a diffusion model based on a two-dimensional U-shaped network. When training the video generation model, the image reference branch is initialized using parameters of a pre-trained text generation model, and the parameters of the image reference branch are fixed during the training process.

6. The method according to claim 2, wherein: The video generation branch adopts a diffusion model based on a three-dimensional U-shaped network. When training the video generation model, the video generation branch is initialized using parameters of a pre-trained text-to-image model. During the training process, the parameters of the video generation branch and the parameters of the attention layer are updated.

7. The method according to claim 1, wherein: After the sequence frames of the target video are generated, the method further includes: based on the sequence frames of the target video, when it is determined that the duration of the target video is insufficient, taking the last frame in the sequence frames as a new reference image, generating a new sequence frame based on the description text and the new reference image, and splicing the newly generated sequence frame to the existing sequence frame until a sequence frame that makes the target video meet a preset duration condition is obtained.

8. The method according to any one of claims 1 to 7, wherein: The obtaining of description text and reference image configured to generate a target video includes: obtaining description text configured to generate a target video; and generating a reference image semantically matching the description text using a pre-trained text-to-image model according to the description text.

9. The method according to claim 8, wherein: The obtaining of the description text set to generate the target video includes: receiving input content data, where the input content data is any one of the following: picture, audio, video, document; converting the input content data into text information to obtain the description text set to generate the target video.

10. The method according to any one of claims 1 to 7, wherein: After generating the target video according to the sequence frames of the target video, the method further includes: in response to a modification operation on the description text and / or the reference image, acquiring a modified description text and / or a reference image; generating a new target video according to the modified description text and / or the reference image; and outputting the new target video, or outputting the target video and the description text.

11. The method according to claim 10, wherein: The modification operation on the reference image includes: designating any image frame in the target video as a new reference image; or, editing any image frame in the target video and designating the image obtained after editing as a new reference image; or, inputting a new reference image.

12. A video generation method, comprising: Receive description text entered by the user; According to the description text, generating a reference image that semantically matches the description text; Inputting the description text and the reference image into a video generation model, generating a sequence frame of a target video semantically matching the description text based on the reference image through the video generation model, wherein the video generation model includes an image reference branch and a video generation branch, the image reference branch is configured to extract reference image features of the reference image, and the video generation branch is configured to generate a sequence frame of the target video based on the reference image features and text features of the description text; Generate a target video according to the sequence frames of the target video; and output the target video.

13. The method according to claim 12, wherein: After outputting the target video, the method further includes: in response to a modification operation on the description text and / or the reference image, acquiring a modified description text and / or a reference image; generating a new target video according to the modified description text and / or the reference image; and outputting the new target video, or outputting the target video and the description text.

14. A server, comprising: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 13.

Citation Information

Patent Citations

  • Video generation method and device, medium and equipment

    CN109889849A

  • Video generation method and device and electronic equipment

    CN116012479A

  • System and method for synthesizing photo-realistic video of a speech

    US20220084273A1

Cited By

  • Video generation method and device, electronic equipment, storage medium and program product

    CN119421016A

  • Video frame insertion method, electronic equipment, chip system and readable storage medium

    CN119478614A

  • Video generation method and device, equipment, storage medium and program product

    CN119629439A

  • Method and device for generating video from text and storage medium

    CN119697456A

  • Method for generating small-sample high-quality carrier target detection data set

    CN119919751A