Video generation method, electronic device and computer-readable storage medium

Abstract

The present disclosure relates to the technical fields of video processing technology and computers. Disclosed are a video generation method, an electronic device and a computer-readable storage medium. The method comprises: generating a first video on the basis of prompt text and noise data, wherein the prompt text is used for describing video content of a target video to be generated, and the data dimension of the noise data is the same as the video data dimension of the target video; and performing video generation processing on the first video on the basis of a target video generation model, so as to obtain the target video, wherein a video diffusion model and an image diffusion model are integrated into the target video generation model, and the initial time step of the video diffusion model is the same as the initial time step of the image diffusion model. The present disclosure solves the technical problem of the quality of videos generated by video generation models in the related art being relatively poor.

Description

Video generation methods, electronic devices and computer-readable storage media Technical Field

[0001] This disclosure relates to video processing technology and computer technology, and more specifically, to a video generation method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the rapid development of generative artificial intelligence, video generation models have become one of the research hotspots in the field of artificial intelligence. However, video generation models are often limited by the quality of training data during the training process, resulting in poor quality generated videos.

[0003] Currently, some methods only fine-tune the temporal layers during video generation model training and then recombine them with the image generation model. However, the low-quality factors in the temporal layers cannot be eliminated, resulting in poor video quality. Other methods fine-tune all parameters during video generation model training to generate videos with more natural movements, but the quality of the synthesized videos is still poor, and it is impossible to introduce a high-quality image generation model for improvement during testing.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This disclosure provides a video generation method, an electronic device, and a computer-readable storage medium to at least solve the technical problem of poor video quality generated by video generation models in related technologies.

[0006] According to one aspect of the present disclosure, a video generation method is provided, comprising: generating a first video based on prompt text and noise data, wherein the prompt text is used to describe the video content of the target video to be generated, and the data dimension of the noise data is the same as the video data dimension of the target video; performing video generation processing on the first video based on a target video generation model to obtain a target video, wherein the target video generation model integrates a video diffusion model and an image diffusion model, and the initial time step of the video diffusion model is the same as the initial time step of the image diffusion model.

[0007] According to another aspect of the embodiments of this disclosure, a video generation method is also provided, comprising: generating an initial animated video based on animated prompt text and noise data, wherein the animated prompt text is used to describe the video content of the target animated video to be generated, and the data dimension of the noise data is the same as the video data dimension of the target animated video; performing video generation processing on the initial animated video based on a target animated video generation model to obtain a target animated video, wherein the target animated video generation model integrates a video diffusion model and an image diffusion model, and the initial time step of the video diffusion model is the same as the initial time step of the image diffusion model.

[0008] According to another aspect of the embodiments of this disclosure, a video generation method is also provided, comprising: obtaining a video generation request through a first application programming interface (API), wherein the request data carried in the video generation request includes: prompt text and noise data, the prompt text being used to describe the video content of the target video to be generated, and the data dimension of the noise data being the same as the video data dimension of the target video; and returning a video generation response through a second API, wherein the response data carried in the video generation response includes: a target video, the target video being obtained by performing video generation processing on a first video based on a target video generation model, the target video generation model integrating a video diffusion model and an image diffusion model, the initial time step of the video diffusion model being the same as the initial time step of the image diffusion model, and the first video being generated based on the prompt text and noise data.

[0009] According to another aspect of the embodiments of this disclosure, a video generation method is also provided, comprising: acquiring a currently input video generation dialogue request, wherein the request data carried in the video generation dialogue request includes: prompt text and noise data, the prompt text being used to describe the video content of the target video to be generated, and the data dimension of the noise data being the same as the video data dimension of the target video; responding to the video generation dialogue request and returning a video generation dialogue response, wherein the information carried in the video generation dialogue response includes: a target video, the target video being obtained by performing video generation processing on a first video based on a target video generation model, the target video generation model integrating a video diffusion model and an image diffusion model, the initial time step of the video diffusion model being the same as the initial time step of the image diffusion model, and the first video being generated based on the prompt text and noise data; and displaying the target video in a graphical user interface.

[0010] According to another aspect of the present disclosure, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes any of the above-described video generation methods during runtime.

[0011] According to another aspect of the present disclosure, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform any of the above-described video generation methods.

[0012] According to another aspect of the present disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements any of the above-described video generation methods.

[0013] In this embodiment, a first video is generated using prompt text describing the video content of the target video to be generated, and noise data with the same video data dimension as the target video. Then, the first video is processed by a target video generation model that integrates a video diffusion model and an image diffusion model to obtain the target video. The initial time step of the video diffusion model is the same as that of the image diffusion model, thereby achieving the goal of combining the video diffusion model with a high-quality image generation model to generate a high-quality target video. This improves the video quality of the generated video, enriches the video style of the generated video, and enhances the consistency between the generated video and the text description, thus solving the technical problem of poor video quality generated by video generation models in related technologies.

[0014] It is worth noting that the above general description and the following detailed description are merely for illustrative and explanatory purposes and do not constitute a limitation thereof. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0016] Figure 1 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a video generation method according to Embodiment 1 of this disclosure;

[0017] Figure 2 is a flowchart of a video generation method according to Embodiment 1 of this disclosure;

[0018] Figure 3 is a flowchart of another video generation method according to Embodiment 1 of this disclosure;

[0019] Figure 4 is a flowchart of a video generation method according to Embodiment 2 of this disclosure;

[0020] Figure 5 is a flowchart of a video generation method according to Embodiment 3 of this disclosure;

[0021] Figure 6 is a flowchart of a video generation method according to Embodiment 4 of this disclosure;

[0022] Figure 7 is a schematic diagram of a video generation device according to Embodiment 5 of the present disclosure;

[0023] Figure 8 is a schematic diagram of another video generation apparatus according to Embodiment 5 of this disclosure;

[0024] Figure 9 is a schematic diagram of another video generation apparatus according to Embodiment 5 of this disclosure;

[0025] Figure 10 is a schematic diagram of the structure of another video generation device according to Embodiment 5 of this disclosure;

[0026] Figure 11 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] First, some nouns or terms that appear in the description of the embodiments of this disclosure shall be interpreted as follows:

[0030] Diffusion model: It can be understood as a Markov chain trained with variational estimation. The goal of the diffusion model is to learn the latent structure of the dataset by modeling how data points diffuse in the latent space.

[0031] Video generation: This usually refers to text-to-video generation, where the model automatically generates a corresponding video based on a text input by the user.

[0032] Video Diffusion Model: A deep learning model for generating video content. Based on the ideas of Denoising Diffusion Probabilistic Models (DDPMs), the video diffusion model generates video by progressively adding and removing noise.

[0033] Image Diffusion Probabilistic Models (DDPMs) are generative models used to generate high-quality images. They belong to a class of models in deep learning, particularly excelling in areas such as image synthesis, denoising, and style transfer. The core idea of ​​DDPMs is to generate images by simulating a random diffusion process and then progressively reversing this process.

[0034] Diffusion models have facilitated the rapid development of generative artificial intelligence and achieved significant success in both image and video generation tasks. The excellent generative capabilities of diffusion models largely depend on sufficient and high-quality data. For example, text-image generation models require hundreds of millions of training images to achieve decent results. However, due to the lack of sufficient and high-quality training videos, video generation models still lag far behind image generation models.

[0035] Currently, video generation models improve their performance by incorporating image generation models. Leveraging the spatial modeling capabilities of image generation models, video generation models can focus on learning temporal dynamic changes. However, as training progresses, video generation models are inevitably affected by low-quality factors in the training videos, such as motion blur and low resolution. Even with the introduction of high-quality image generation models, these low-quality factors cannot be corrected, resulting in poor video quality. Other methods fine-tune all parameters of the video generation model during training to generate more natural-looking videos, but the synthesized videos remain of poor quality and cannot be improved by introducing high-quality image generation models during testing.

[0036] The video generation models of related technologies have the following shortcomings.

[0037] Defect 1: During training, the video generation model only fine-tunes the temporal layers. However, low-quality factors in the temporal layers cannot be eliminated. The video generation model is still affected by these low-quality factors in the training video, resulting in poor video quality. Since the video generation model fine-tunes all parameters during training, it cannot be combined with image generation models for improvement, leading to poor video quality.

[0038] To address the aforementioned deficiencies, no effective solution has been proposed prior to this disclosure.

[0039] Example 1

[0040] According to embodiments of this disclosure, a video generation method is provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0041] The method embodiment provided in Embodiment 1 of this disclosure can be executed in a mobile terminal, computer terminal, or similar computing device. Figure 1 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a video generation method according to Embodiment 1 of this disclosure. As shown in Figure 1, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) (processor 102 may include, but is not limited to, a microprocessor (MCU) or a programmable gate array (FPGA) or the like), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of a bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that the structure shown in Figure 1 is merely illustrative and does not limit the structure of the above-described electronic device. For example, computer terminal 10 may also include more or fewer components than shown in FIG1, or have a different configuration than shown in FIG1.

[0042] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuitry are generally referred to herein as "data processing circuitry". This data processing circuitry may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in embodiments of this disclosure, the data processing circuitry serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0043] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the video generation method in this embodiment of the present disclosure. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-described video generation method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0044] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0045] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0046] Under the above operating environment, this disclosure provides a video generation method as shown in Figure 2. Figure 2 is a flowchart of a video generation method according to Embodiment 1 of this disclosure. As shown in Figure 2, the method may include the following steps:

[0047] Step S21: Generate a first video based on the prompt text and noise data, wherein the prompt text is used to describe the video content of the target video to be generated, and the data dimension of the noise data is the same as the video data dimension of the target video;

[0048] Step S22: Perform video generation processing on the first video based on the target video generation model to obtain the target video. The target video generation model integrates a video diffusion model and an image diffusion model. The initial time step of the video diffusion model is the same as that of the image diffusion model.

[0049] In this embodiment of the disclosure, the prompt text is used to describe the video content of the target video to be generated. This can be understood as the prompt text guiding the target video generation model to generate a video with a specific theme or content, that is, to generate a video containing the content described by the prompt text. For example, the prompt text can be text input by the user, thereby helping the target video generation model understand the user's needs and intentions, enabling the target video generation model to generate the target video that the user expects.

[0050] Optionally, the prompt text can be text directly entered by the user using an input method, or text obtained by converting the user's voice audio into text; there is no restriction here. Furthermore, the prompt text can be in various languages, including but not limited to Chinese, English, and Japanese; there is no restriction here either.

[0051] Noise data can be understood as random values ​​introduced during video generation to increase randomness and variability; that is, random noise. By introducing random noise during video generation, the generated video content can be made more diverse and more realistic.

[0052] In this embodiment of the disclosure, the data dimension of the noise data is the same as the video data dimension of the target video to be generated. Therefore, the target video generation model can better learn the features and structure of the target video to be generated, thereby helping to improve the training effect of the target video generation model, that is, to improve the video quality and video content accuracy of the generated target video.

[0053] In this embodiment of the disclosure, prompt text and noise data can be input into video generation tools such as video generators, video editing software, and video generation models, so as to generate a first video through the video generation tools. No limitation is imposed here.

[0054] For example, taking a video generation model as an example, the video generation model can choose a video diffusion model. The prompt text and noise data are input into the video diffusion model, thereby generating a first video that matches the video content described by the prompt text. It should be noted that the video quality of the first video generated here needs improvement.

[0055] As can be understood, video diffusion models refer to the use of diffusion models to model and learn information in video data. In video diffusion models, by modeling the relationships between video frames, the underlying structure in the video data can be learned, thereby enabling the analysis, recognition, and understanding of video content.

[0056] Image diffusion models refer to the use of diffusion models to model and learn information in image data. By modeling the relationships between image pixels, the latent structure in the image data can be learned, thereby enabling the analysis, recognition, and processing of image content.

[0057] In this embodiment, the target video generation model is used to generate a target video that matches the video content described in the prompt text. The target video generation model integrates a video diffusion model and an image diffusion model, with the initial time step of the video diffusion model being the same as that of the image diffusion model. This allows the target video generation model to generate high-quality target videos based on the capabilities of both models. Furthermore, because the initial time step of the video diffusion model is the same as that of the image diffusion model, consistency and continuity are maintained during training and actual inference, avoiding disjointed target videos and further improving the video quality of the target videos generated by the target video generation model.

[0058] Optionally, the video diffusion model integrated in the target video generation model can be a video generation model based on a variational autoencoder (VAE), a video generation model based on a generative adversarial network (GAN), or a video generation model based on a recurrent neural network (RNN), etc., and there is no limitation here. It is understood that if a video diffusion model is used to generate the first video, the video diffusion model integrated in the target video generation model can be the same as or different from the video diffusion model used to generate the first video, and there is no limitation here.

[0059] Optionally, the image diffusion model integrated in the target video generation model can be a high-quality image diffusion model, such as a stable diffusion model, a personalized image diffusion model, or a distilled lightweight image diffusion model; no restrictions are imposed here.

[0060] The target video is the video output by the target video generation model. The target video generation model outputs a high-quality target video by making multiple adjustments and optimizations to the first video.

[0061] It is understandable that the target video's size, format, and other attributes can be predetermined, or the target video's attributes can be included in the prompt text; there are no restrictions on this.

[0062] In this embodiment, a first video is generated using prompt text describing the video content of the target video to be generated, and noise data with the same video data dimension as the target video. Then, a target video is processed using a target video generation model integrating a video diffusion model and an image diffusion model to obtain the target video. The initial time step of the video diffusion model is the same as that of the image diffusion model. It can be seen that the video generation method proposed in this embodiment effectively improves the quality of the target video generated by the target video generation model by combining a high-quality image diffusion model with a video diffusion model, enriching the style of the target video. It also effectively solves the problem of misalignment between the generated target video and the text description, enhancing the consistency between the generated target video and the prompt text. Furthermore, the video generation method proposed in this embodiment is plug-and-play without training, meaning it can use various image generation models to enhance the video quality of the target video to be generated.

[0063] The video generation method provided in this disclosure can be applied, but is not limited to, to application scenarios involving generating corresponding videos based on prompt text in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, it can generate corresponding videos for e-commerce services, educational services, and legal services based on prompt text. No limitation is imposed here.

[0064] By employing the embodiments of this disclosure, a first video is generated using prompt text describing the video content of the target video to be generated, and noise data with the same video data dimension as the target video. Then, a target video is processed by a target video generation model integrating a video diffusion model and an image diffusion model to obtain the target video. The initial time step of the video diffusion model is the same as that of the image diffusion model, thereby achieving the goal of combining the video diffusion model with a high-quality image generation model to generate a high-quality target video. This achieves the technical effects of improving the video quality of the generated video, enriching the video style of the generated video, and enhancing the consistency between the generated video and the text description, thus solving the technical problem of poor video quality generated by video generation models in related technologies.

[0065] In an optional embodiment, in step S21, generating a first video based on the prompt text and noise data includes the following method steps:

[0066] Step S211: Noise data is obtained by sampling from Gaussian distributed random numbers based on the video data dimension of the target video;

[0067] Step S212: Diffusion processing is performed on the prompt text and noise data to obtain the first video.

[0068] In this embodiment of the disclosure, when generating the first video based on the prompt text and noise data, noise data can be sampled from Gaussian distributed random numbers based on the video data dimension of the target video. That is, it can be understood as sampling random noise with the same dimension as the video data from a Gaussian distribution.

[0069] After sampling the noise data, the prompt text and noise data are diffused to obtain the first video. This can be understood as using the principle of the diffusion model to predict the first video by mapping noise data for a given prompt text.

[0070] Understandably, the first video is a relatively clear video initially generated based on the prompt text. In order to improve the video quality, the first video still needs to be continuously optimized and adjusted.

[0071] In an optional embodiment, in step S22, the first video is processed based on the target video generation model to obtain the target video, including the following method steps:

[0072] Step S221: Generate a second video based on the video diffusion model and the first video, wherein the video content of the second video has a higher fidelity than the video content of the first video;

[0073] Step S222: Generate multiple first video frame images based on the image diffusion model and the second video, wherein the image granularity of any first video frame image is greater than the image granularity of the video frame images in the second video.

[0074] Step S223: Generate the target video based on multiple first video frame images.

[0075] In this embodiment, since the target video generation model integrates a video diffusion model and an image diffusion model, when the first video is processed based on the target video generation model to obtain the target video, a second video can be generated based on the video diffusion model and the first video. This can be understood as processing the first video using the video diffusion model to obtain the second video. Furthermore, the realism of the obtained second video will be higher than that of the first video. In other words, processing the first video using the video diffusion model can improve the realism of the first video's content, resulting in a second video with higher realism and more natural content.

[0076] After obtaining the second video, multiple first video frame images can be generated based on the image diffusion model and the second video. This can be understood as processing the second video using the image diffusion model to generate multiple first video frame images. Furthermore, the image granularity of any one of the resulting multiple first video frame images is greater than the image granularity of the video frame images in the second video. In other words, by processing the second video using the image diffusion model, multiple first video frame images with richer image details can be predicted based on the video frame images in the second video.

[0077] After obtaining multiple first video frame images, the target video can be generated based on the multiple first video frame images with richer image details.

[0078] As can be seen, this disclosure combines video diffusion models and image diffusion models to generate target videos with more natural and richer content, thereby improving the video quality of the generated target videos.

[0079] In an optional embodiment, in step S221, a second video is generated based on the video diffusion model and the first video, including the following method steps:

[0080] Step S2211: The first video is denoised using a low-pass filter to obtain the third video. The low-pass filter is constructed based on the fast Fourier transform, and the smoothness of the third video is higher than that of the first video.

[0081] Step S2212: Add Gaussian noise to the third video to obtain the fourth video, wherein the fourth video conforms to the domain of the video diffusion model at the time step.

[0082] Step S2213: Iteratively process the fourth video based on the video diffusion model to obtain the second video.

[0083] In this embodiment, the low-pass filter is constructed based on the Fast Fourier Transform. Using this low-pass filter to perform noise reduction on the video effectively reduces high-frequency jitter, thereby obtaining a stable video. Exemplarily, the low-pass filter can be a Gaussian low-pass filter, an ideal low-pass filter, etc., and is not limited thereto.

[0084] The video diffusion model generates video in an iterative process, and the time step of the video diffusion model is used to represent the total number of iterations of the video diffusion model.

[0085] In this embodiment, when generating the second video from the video diffusion model and the first video, a low-pass filter can be used to denoise the first video to obtain the third video. This can be understood as inputting the first video into a low-pass filter and using the low-pass filter to remove high-frequency jitter from the first video, thereby obtaining the third video. Furthermore, the smoothness of the third video is higher than that of the first video; that is, processing the first video with a low-pass filter can improve the smoothness of the first video, resulting in a third video with higher smoothness and greater stability.

[0086] Since the input to the video diffusion model is always video with Gaussian noise, after obtaining the third video, Gaussian noise can be added to it to obtain the fourth video. This step can be understood as, based on the forward process of the diffusion model, adding Gaussian noise to the third video to obtain the fourth video, thus ensuring that the obtained fourth video conforms to the domain defined by the video diffusion model at each time step. In other words, it ensures that the changes in the obtained fourth video conform to the characteristics of the video diffusion model. This guarantees that the video generated by the video diffusion model maintains reasonable changes and characteristics at each time step during the video generation process, thereby improving the quality of the generated video and making it more consistent with the dynamic laws of the real world.

[0087] After obtaining the fourth video, iterative processing can be performed on it based on the video diffusion model to obtain the second video. In each iteration, the video diffusion model predicts the corresponding video frame in the previous time step based on each video frame in the fourth video. By performing multiple iterations on the fourth video according to the time steps of the video diffusion model, a second video with more realistic and natural content can be predicted from the fourth video.

[0088] As can be seen, by introducing a low-pass filter, this disclosure can significantly reduce high-frequency jitter in the generated video and improve the stability of the generated video.

[0089] In an optional embodiment, in step S222, multiple first video frame images are generated based on the image diffusion model and the second video, including the following method steps:

[0090] Step S2221: Perform diffusion processing on the second video to obtain the fifth video, wherein the clarity of the fifth video is higher than that of the second video;

[0091] Step S2222: The fifth video is mapped based on the differential function to obtain multiple second video frame images, wherein the multiple second video frame images conform to the Gaussian noise distribution of the image diffusion model at the time step;

[0092] Step S2223: Based on the image diffusion model, perform prediction processing on multiple second video frame images to obtain multiple first video frame images.

[0093] In this embodiment of the disclosure, the differential function can be an ordinary differential equation (ODE). When applying the ODE to the forward process of the diffusion model, there are various ways, such as using the inversion of the denoising diffusion implicit model (DDIM) and the diffusion probabilistic model (DDPM), etc., which are not limited here.

[0094] In this embodiment of the disclosure, when generating multiple first video frame images based on the image diffusion model and the second video, the second video can be diffused to obtain the fifth video. That is, it can be understood that the second video is diffused according to the principle of the diffusion model, so as to predict a fifth video with clearer video content based on the second video.

[0095] After obtaining the fifth video, it can be mapped using a differential function to obtain multiple second video frame images. This can be understood as mapping the fifth video to multiple second video frame images using ordinary differential equations (ODEs) based on the forward process of the image diffusion model. This ensures that the resulting multiple second video frame images conform to the Gaussian noise distribution of the image diffusion model at the time step. In other words, the video content of the fifth video is processed and adjusted, converting it into static images that conform to the Gaussian noise distribution and characteristics of the image diffusion model, facilitating subsequent processing and prediction.

[0096] After obtaining multiple second video frame images, the multiple second video frame images can be predicted based on the image diffusion model to obtain multiple first video frame images. This can be understood as inputting multiple second video frame images into the image diffusion model, using the image diffusion model to perform iterative prediction processing on each second video frame in the multiple second video frame images, to obtain the video frame corresponding to each second video frame in the previous time step. After multiple iterative predictions, multiple first video frame images with more detailed content are obtained.

[0097] Understandably, the image diffusion model can learn the latent distribution and features of images to generate new image content. In step S2223, the image diffusion model learns the latent distribution and features of multiple second video frame images, thereby performing prediction processing on these images. This allows it to predict the video frame corresponding to each video frame in the previous time step, i.e., predicting video frames with more details and features. Therefore, through multiple iterative processes of the multiple second video frame images, the image diffusion model ultimately obtains multiple first video frame images with more detailed content.

[0098] In an optional embodiment, step S223, generating a target video based on multiple first video frame images, includes the following method steps:

[0099] Step S2231: Determine the current target time step;

[0100] Step S2232: In response to the target time step being greater than zero, update the target time step;

[0101] Step S2233: At the updated target time step, multiple first video frame images are updated based on a low-pass filter, a video diffusion model, and an image diffusion model to obtain updated multiple first video frame images. Then, it is determined whether the updated target time step is greater than zero until the updated target time step is equal to zero. The target video is generated based on the updated multiple first video frame images. The image granularity of the updated multiple first video frame images is greater than the image granularity of the original multiple first video frame images.

[0102] In step S2234, in response to the target time step being zero, a target video is generated based on multiple first video frame images.

[0103] In this embodiment of the disclosure, when generating a target video based on multiple first video frame images, the current target time step can be determined, which can be understood as determining the time step of the video diffusion model or the time step of the image diffusion model in the current iteration process, so as to determine whether it is necessary to continue iterative processing according to the size of the target time step.

[0104] After determining the target time step, if the target time step is greater than zero, it indicates that the iteration is not complete. Therefore, the target time step needs to be updated and the iteration process continues. For example, the target time step t can be updated to t-1, and then a new round of iteration processing can be performed. Under the updated target time step, that is, under the new round of iteration processing, multiple first video frame images are updated again based on the low-pass filter, video diffusion model, and image diffusion model, thereby obtaining updated multiple first video frame images.

[0105] It is understandable that during the iteration process, multiple first video frame images will be continuously adjusted and optimized to obtain updated first video frame images. Therefore, the image granularity of the updated first video frame images is greater than that of the original first video frame images.

[0106] After obtaining multiple updated first video frame images through one round of iterative processing, it is necessary to further determine whether the current target time step is greater than zero, that is, whether t-1 is greater than zero. If t-1 is still greater than zero, the target time step is updated again, that is, t-1 is updated to t-2 and a new round of iterative processing is continued until the updated target time step is equal to zero. This will not be elaborated here.

[0107] After multiple iterations to make the target time step equal to zero, the target video is generated based on the updated multiple first video frame images, that is, the target video is generated based on the multiple first video frame images finally obtained from multiple iterations.

[0108] Alternatively, after determining the target time step, if the target time step is equal to zero, then the target video is generated based on multiple first video frame images. This can be understood as the target time step being equal to zero indicating that multiple iterations have been performed, resulting in multiple first video frame images with more details. Therefore, a high-quality target video can be generated directly from multiple first video frame images.

[0109] Understandably, the generated target video can be displayed to provide feedback to the user.

[0110] In an optional embodiment, in step S2233, multiple first video frame images are updated based on a low-pass filter, a video diffusion model, and an image diffusion model to obtain updated multiple first video frame images, including the following method steps:

[0111] Step S22331: Determine the sixth video based on multiple first video frame images;

[0112] Step S22332: Use a low-pass filter to perform noise reduction on the sixth video to obtain the seventh video. The smoothness of the seventh video is higher than that of the sixth video.

[0113] Step S22333: Iteratively process the seventh video based on the video diffusion model to obtain the eighth video, wherein the video content of the eighth video has a higher fidelity than that of the seventh video.

[0114] Step S22334: Based on the image diffusion model, perform prediction processing on multiple video frame images of the eighth video to obtain updated multiple first video frame images.

[0115] In this embodiment, when updating multiple first video frame images based on a low-pass filter, a video diffusion model, and an image diffusion model to obtain updated multiple first video frame images, a sixth video can be generated based on the multiple first video frame images obtained in the previous iteration. Then, a low-pass filter is used to perform noise reduction processing on the sixth video to obtain a seventh video with higher video smoothness. It is understood that before obtaining the seventh video, the sixth video can also undergo mapping processing based on noise data, which will not be elaborated here.

[0116] After obtaining the seventh video, iterative processing is performed on it based on the video diffusion model to obtain the eighth video, which has higher realism and more natural and vivid content. Understandably, Gaussian noise can be added to the seventh video before obtaining the eighth video, but this will not be elaborated upon here.

[0117] After obtaining the eighth video, multiple video frame images of the eighth video are predicted using an image diffusion model to obtain updated multiple first video frame images with more details. It is understandable that before obtaining the updated multiple first video frame images, the eighth video can also undergo diffusion processing to predict a clearer eighth video. Then, based on a differential function, the clearer eighth video is mapped to convert its content into static images, resulting in multiple video frame images of the eighth video. This will not be elaborated upon here.

[0118] Figure 3 is a flowchart of another video generation method according to Embodiment 1 of this disclosure. As shown in Figure 3, before video generation processing, a low-pass filter based on Fast Fourier Transform is first constructed, a video generation model based on a diffusion network is constructed, and an image generation model based on the diffusion model is constructed. Then, prompt text describing the video content of the target video to be generated, noise data z_t^I with the same video data dimension as the target video, and the same initial time step t are set for the video diffusion model and the image diffusion model. Then, according to the principle of the diffusion model, for the given prompt text, a clear first video z_(t→0)^I is directly predicted by mapping the noise data. The first video is then input into the low-pass filter to remove high-frequency jitter in the first video to obtain a stable third video. Next, based on the forward propagation process of the diffusion model, a certain amount of Gaussian noise is added to the third video to obtain the fourth video. The fourth video is made to fit the domain of the video diffusion model at time step t. Then, the fourth video is input into the video diffusion model and iterated multiple times to obtain more natural motion, thus yielding the second video. Then, based on the principle of diffusion model, diffusion processing is performed on the second video, and a clear fifth video is directly predicted using the mapping of the second video. Then, based on the forward process of the diffusion model, the fifth video is mapped using ordinary differential equations (ODE) to obtain multiple second video frame images. The process involves ensuring that multiple second video frame images conform to the Gaussian noise distribution of the image diffusion model at time step t. These second video frame images are then input into the image diffusion model, which predicts multiple first video frame images z_(t-1)^I with more detail based on them. Finally, it checks if the current target time step t is greater than 0. If t is greater than 0, t is updated to t-1, and iterative processing continues using a low-pass filter, the video diffusion model, and the image diffusion model. If t is not greater than 0, a high-quality target video is generated based on the currently obtained multiple video frame images.

[0119] As can be seen, the embodiments of this disclosure integrate a high-quality image diffusion model into the sampling process of the video diffusion model. During sampling, both the video diffusion model and the image diffusion model are used to denoise the random noise in the input, thereby generating a high-quality video that conforms to the text description. This effectively improves the video quality of the generated video, enriches its style, and enhances the consistency between the generated video and the text description. Simultaneously, this disclosure introduces a low-pass filter, significantly reducing high-frequency jitter in the generated video and improving its temporal consistency and stability.

[0120] It is easy to understand that the beneficial effects of the video generation method provided in this disclosure include the following points.

[0121] Beneficial effects (1) This disclosure integrates a high-quality image diffusion model into the sampling process of a video diffusion model, which can generate high-quality videos that conform to the text description and enrich the video style of the generated videos.

[0122] Beneficial effect (2): The present disclosure introduces a low-pass filter, which significantly reduces high-frequency jitter in the generated video and improves the temporal consistency and stability of the generated video.

[0123] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0124] Furthermore, it should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0125] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solutions of this disclosure, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0126] Example 2

[0127] In the operating environment as described in Embodiment 1, this disclosure provides a video generation method as shown in Figure 4. Figure 4 is a flowchart of a video generation method according to Embodiment 2 of this disclosure. As shown in Figure 4, the method includes:

[0128] Step S41: Generate a first animation video based on the animation prompt text and noise data, wherein the animation prompt text is used to describe the video content of the target animation video to be generated, and the data dimension of the noise data is the same as the video data dimension of the target animation video;

[0129] Step S42: Perform video generation processing on the first animation video based on the target animation video generation model to obtain the target animation video. The target animation video generation model integrates a video diffusion model and an image diffusion model. The initial time step of the video diffusion model is the same as that of the image diffusion model.

[0130] In this embodiment of the disclosure, the animation prompt text is used to describe the video content of the target animated video to be generated. This can be understood as the animation prompt text guiding the target animated video generation model to generate an animated video with a specific theme or content, that is, generating an animated video containing the video content described by the animation prompt text. For example, the animation prompt text can be text input by the user, thereby helping the target animated video generation model understand the user's needs and intentions, enabling the target animated video generation model to generate the target animated video that the user expects.

[0131] Noise data can be understood as random values ​​introduced during video generation to increase randomness and variability; that is, random noise. By introducing random noise during video generation, the generated video content can be made more diverse and more realistic.

[0132] In this embodiment of the disclosure, the data dimension of the noise data is the same as the video data dimension of the target animation video to be generated. Therefore, the target animation video generation model can better learn the features and structure of the target animation video to be generated, thereby helping to improve the training effect of the target animation video generation model, that is, to improve the video quality and video content accuracy of the generated target animation video.

[0133] In this embodiment of the disclosure, animated prompt text and noise data can be input into animation video generation tools such as animation video generators, animation video editing software, and animation video generation models, so as to generate a first animation video through the animation video generation tools. No limitation is imposed here.

[0134] In this embodiment, the target animation video generation model is used to generate a target animation video that is consistent with the video content described in the animation prompt text. The target animation video generation model integrates a video diffusion model and an image diffusion model, and the initial time step of the video diffusion model is the same as that of the image diffusion model. Therefore, the target animation video generation model can generate high-quality target animation videos based on the capabilities of both models. Furthermore, because the initial time step of the video diffusion model is the same as that of the image diffusion model, consistency and continuity can be maintained during training and actual inference, avoiding discontinuous generated target animation videos and further improving the video quality of the target animation videos generated by the target animation video generation model.

[0135] The target animation video is the video output by the target animation video generation model. The target animation video generation model outputs a high-quality target animation video by making multiple adjustments and optimizations to the first animation video.

[0136] It is understandable that the target animation video's video size, video format, and other attribute information can be predetermined, or the target animation video's attribute information can be included in the animation prompt text; there are no restrictions here.

[0137] In this embodiment, a first animation video is generated using animated prompt text describing the video content of the target animation video to be generated, and noise data with the same video data dimension as the target animation video. Then, a target animation video generation model integrating a video diffusion model and an image diffusion model is used to generate the first animation video, thereby obtaining the target animation video. The initial time step of the video diffusion model is the same as that of the image diffusion model. It can be seen that the video generation method proposed in this embodiment, by combining a high-quality image diffusion model with a video diffusion model, can effectively improve the quality of the target animation video generated by the target animation video generation model, enrich the style of the target animation video, and effectively solve the problem of misalignment between the generated target animation video and the text description, enhancing the consistency between the generated target animation video and the animated prompt text. Furthermore, the video generation method proposed in this embodiment is plug-and-play without training, meaning it can use various image generation models to enhance the video quality of the target animation video to be generated.

[0138] The video generation method provided in this disclosure can be applied, but is not limited to, to application scenarios involving generating corresponding videos based on prompt text in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, it can generate corresponding videos for e-commerce services, educational services, and legal services based on prompt text. No limitation is imposed here.

[0139] By employing the embodiments of this disclosure, a first animation video is generated using animated prompt text describing the video content of the target animation video to be generated, and noise data with the same video data dimension as the target animation video to be generated. Then, a target animation video generation model integrating a video diffusion model and an image diffusion model is used to perform video generation processing on the first animation video to obtain the target animation video. The initial time step of the video diffusion model is the same as the initial time step of the image diffusion model, thereby achieving the goal of combining the video diffusion model with a high-quality image generation model to generate a high-quality target video. This achieves the technical effects of improving the video quality of the generated video, enriching the video style of the generated video, and enhancing the consistency between the generated video and the text description, thus solving the technical problem of poor video quality generated by video generation models in related technologies.

[0140] In an optional embodiment, in step S42, the first animation video is processed based on the target animation video generation model to obtain the target animation video, including the following method steps:

[0141] Step S421: Generate a second animation video based on the video diffusion model and the first animation video, wherein the video content of the second animation video has a higher fidelity than that of the first animation video.

[0142] Step S422: Generate multiple first video frame images based on the image diffusion model and the second animation video, wherein the image granularity of any first video frame image is greater than the image granularity of the video frame images in the second animation video;

[0143] Step S423: Generate the target animation video based on multiple first video frame images.

[0144] In this embodiment, since the target animation video generation model integrates a video diffusion model and an image diffusion model, when the first animation video is processed based on the target animation video generation model to obtain the target animation video, a second animation video can be generated based on the video diffusion model and the first animation video. This can be understood as processing the first animation video using the video diffusion model to obtain the second animation video. Furthermore, the realism of the obtained second animation video will be higher than that of the first animation video. In other words, processing the first animation video using the video diffusion model can improve the realism of the first animation video, resulting in a second animation video with higher realism and more natural content.

[0145] After obtaining the second animated video, multiple first animated video frame images can be generated based on the image diffusion model and the second animated video. This can be understood as processing the second animated video using the image diffusion model to generate multiple first animated video frame images. Furthermore, the image granularity of any one of the resulting multiple first animated video frame images is greater than the image granularity of the video frame images in the second animated video. In other words, by processing the second animated video using the image diffusion model, multiple first animated video frame images with richer image details can be predicted based on the video frame images in the second animated video.

[0146] After obtaining multiple first animation video frame images, the target animation video can be generated based on the multiple first animation video frame images with richer image details.

[0147] As can be seen, this disclosure combines video diffusion models and image diffusion models to generate target animation videos with more natural and richer video content, thereby improving the video quality of the generated target animation videos.

[0148] It should be noted that the preferred implementation of this embodiment can be found in the relevant description in Embodiment 1, and will not be repeated here.

[0149] Example 3

[0150] In the operating environment as described in Embodiment 1, this disclosure provides a video generation method as shown in Figure 5. Figure 5 is a flowchart of a video generation method according to Embodiment 3 of this disclosure. As shown in Figure 5, the method includes:

[0151] Step S51: Obtain a video generation request through the first application programming interface. The request data carried in the video generation request includes: prompt text and noise data. The prompt text is used to describe the video content of the target video to be generated, and the data dimension of the noise data is the same as the video data dimension of the target video.

[0152] Step S52: Return a video generation response through the second application programming interface. The response data carried in the video generation response includes: a target video, which is obtained by performing video generation processing on the first video based on the target video generation model. The target video generation model integrates a video diffusion model and an image diffusion model. The initial time step of the video diffusion model is the same as the initial time step of the image diffusion model. The first video is generated based on prompt text and noise data.

[0153] The first and second application programming interfaces (APIs) mentioned above can be the same or different APIs. In one optional embodiment, the interface parameters in the first and second APIs may include, but are not limited to: a global interface identifier, an interface signing key, an interface timestamp, an interface request identifier, and a system call credential identifier. The first API can use GET or POST as the interface request method to obtain the file processing request. The second API can use JSON format to return the file processing response.

[0154] In this embodiment of the disclosure, a video generation request can be understood as a video generation request used to request the generation of corresponding video content. The request data carried in the video generation request includes: prompt text and noise data. A video generation response can be understood as the response content to the video generation request. The response data carried in the video generation response includes: the target video.

[0155] For further details, please refer to the description of Example 1, which will not be repeated here.

[0156] In this embodiment of the disclosure, a video generation request carrying prompt text and noise data is obtained through a first application programming interface (API). The prompt text describes the video content of the target video to be generated, and the data dimension of the noise data is the same as that of the target video. Then, a video generation response carrying the target video is returned through a second API. The target video is obtained by performing video generation processing on a first video based on a target video generation model. The target video generation model integrates a video diffusion model and an image diffusion model. The initial time step of the video diffusion model is the same as that of the image diffusion model. The first video is generated based on the prompt text and noise data.

[0157] As can be seen, the video generation method proposed in this disclosure combines a high-quality image diffusion model with a video diffusion model, thereby effectively improving the quality of the target video generated by the target video generation model, enriching the style of the target video, and effectively solving the problem of misalignment between the generated target video and the text description, enhancing the consistency between the generated target video and the prompt text. Furthermore, the video generation method proposed in this disclosure is plug-and-play without training, meaning it can use various image generation models to enhance the video quality of the target video to be generated.

[0158] The video generation method provided in this disclosure can be applied, but is not limited to, to application scenarios involving generating corresponding videos based on prompt text in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, it can generate corresponding videos for e-commerce services, educational services, and legal services based on prompt text. No limitation is imposed here.

[0159] In this embodiment, a video generation request carrying prompt text and noise data is obtained through a first application programming interface (API). The prompt text describes the video content of the target video to be generated, and the data dimension of the noise data is the same as that of the target video. Then, a video generation response carrying the target video is returned through a second API. The target video is obtained by performing video generation processing on a first video based on a target video generation model. The target video generation model integrates a video diffusion model and an image diffusion model. The initial time step of the video diffusion model is the same as that of the image diffusion model. The first video is generated based on the prompt text and noise data. This achieves the goal of combining the video diffusion model with a high-quality image generation model to generate a high-quality target video. This improves the video quality of the generated video, enriches the video style of the generated video, and enhances the consistency between the generated video and the text description. This solves the technical problem of poor video quality generated by video generation models in related technologies.

[0160] It should be noted that the preferred implementation of this embodiment can be found in the relevant description in Embodiment 1, and will not be repeated here.

[0161] Example 4

[0162] In the operating environment as described in Embodiment 1, this disclosure provides a video generation method as shown in FIG6. FIG6 is a flowchart of a video generation method according to Embodiment 4 of this disclosure. As shown in FIG6, the method includes:

[0163] Step S61: Obtain the currently input video generation dialogue request. The request data carried in the video generation dialogue request includes: prompt text and noise data. The prompt text is used to describe the video content of the target video to be generated. The data dimension of the noise data is the same as the video data dimension of the target video.

[0164] Step S62, in response to the video generation dialogue request, return a video generation dialogue response, wherein the information carried in the video generation dialogue response includes: target video, the target video is obtained by performing video generation processing on the first video based on the target video generation model, the target video generation model integrates a video diffusion model and an image diffusion model, the initial time step of the video diffusion model is the same as the initial time step of the image diffusion model, and the first video is generated based on prompt text and noise data;

[0165] Step S63: Display the target video within the graphical user interface.

[0166] In this embodiment of the disclosure, the video generation dialogue request can be understood as a dialogue request initiated by a user when engaging in a dialogue with a smart device to request the generation of a video. The request data carried in the video generation dialogue request includes: prompt text and noise data. The video generation dialogue response can be understood as the content of the reply to the video generation dialogue request. The information carried in the video generation dialogue response includes: the target video.

[0167] For further details, please refer to the description of Example 1, which will not be repeated here.

[0168] In this embodiment of the disclosure, a video generation dialogue request is obtained by acquiring the currently input video generation dialogue request. The request data carried in the video generation dialogue request includes: prompt text and noise data. The prompt text is used to describe the video content of the target video to be generated. The data dimension of the noise data is the same as the video data dimension of the target video. Then, in response to the video generation dialogue request, a video generation dialogue response is returned. The information carried in the video generation dialogue response includes: the target video, which is obtained by performing video generation processing on the first video based on the target video generation model. The target video generation model integrates a video diffusion model and an image diffusion model. The initial time step of the video diffusion model is the same as the initial time step of the image diffusion model. The first video is generated based on the prompt text and noise data. After obtaining the target video, the target video is displayed in the graphical user interface to provide feedback to the user.

[0169] As can be seen, the video generation method proposed in this disclosure combines a high-quality image diffusion model with a video diffusion model, thereby effectively improving the quality of the target video generated by the target video generation model, enriching the style of the target video, and effectively solving the problem of misalignment between the generated target video and the text description, enhancing the consistency between the generated target video and the prompt text. Furthermore, the video generation method proposed in this disclosure is plug-and-play without training, meaning it can use various image generation models to enhance the video quality of the target video to be generated.

[0170] The video generation method provided in this disclosure can be applied, but is not limited to, to application scenarios involving generating corresponding videos based on prompt text in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, it can generate corresponding videos for e-commerce services, educational services, and legal services based on prompt text. No limitation is imposed here.

[0171] By employing the embodiments of this disclosure, a video generation dialogue request is obtained from the current input. The request data carried in the video generation dialogue request includes: prompt text and noise data. The prompt text describes the video content of the target video to be generated, and the data dimension of the noise data is the same as the video data dimension of the target video. Then, in response to the video generation dialogue request, a video generation dialogue response is returned. The information carried in the video generation dialogue response includes: the target video, which is obtained by processing a first video based on a target video generation model. The target video generation model integrates a video diffusion model and an image diffusion model. The initial time step of the video diffusion model is the same as the initial time step of the image diffusion model. The first video is generated based on the prompt text and noise data. After obtaining the target video, it is displayed in the graphical user interface to provide feedback to the user. This achieves the goal of combining the video diffusion model with a high-quality image generation model to generate a high-quality target video, thereby improving the video quality of the generated video, enriching the video style of the generated video, and enhancing the consistency between the generated video and the text description. This solves the technical problem of poor video quality generated by video generation models in related technologies.

[0172] It should be noted that the preferred implementation of this embodiment can be found in the relevant description in Embodiment 1, and will not be repeated here.

[0173] Example 5

[0174] According to embodiments of this disclosure, an apparatus embodiment for implementing the above-described video generation method is also provided. Figure 7 is a schematic structural diagram of a video generation apparatus according to embodiment 5 of this disclosure. As shown in Figure 7, the apparatus includes:

[0175] The first generation module 701 is configured to generate a first video based on prompt text and noise data, wherein the prompt text is used to describe the video content of the target video to be generated, and the data dimension of the noise data is the same as the video data dimension of the target video.

[0176] The first processing module 702 is configured to perform video generation processing on the first video based on the target video generation model to obtain the target video. The target video generation model integrates a video diffusion model and an image diffusion model, and the initial time step of the video diffusion model is the same as that of the image diffusion model.

[0177] Optionally, the first processing module 702 is further configured to: generate a second video based on a video diffusion model and a first video, wherein the video content realism of the second video is higher than that of the first video; generate multiple first video frame images based on an image diffusion model and the second video, wherein the image granularity of any first video frame image is greater than the image granularity of the video frame images in the second video; and generate a target video based on the multiple first video frame images.

[0178] Optionally, the first processing module 702 is further configured to: perform noise reduction processing on the first video using a low-pass filter to obtain a third video, wherein the low-pass filter is constructed based on the fast Fourier transform, and the smoothness of the video frame of the third video is higher than that of the video frame of the first video; add Gaussian noise to the third video to obtain a fourth video, wherein the fourth video conforms to the domain of the video diffusion model at the time step; and perform iterative processing on the fourth video based on the video diffusion model to obtain a second video.

[0179] Optionally, the first processing module 702 is further configured to: perform diffusion processing on the second video to obtain a fifth video, wherein the clarity of the fifth video is higher than that of the second video; perform mapping processing on the fifth video based on a differential function to obtain multiple second video frame images, wherein the multiple second video frame images conform to the Gaussian noise distribution of the image diffusion model at the time step; and perform prediction processing on the multiple second video frame images based on the image diffusion model to obtain multiple first video frame images.

[0180] Optionally, the first processing module 702 is further configured to: determine the current target time step; update the target time step in response to the target time step being greater than zero; update multiple first video frame images based on a low-pass filter, a video diffusion model, and an image diffusion model at the updated target time step to obtain updated multiple first video frame images, and continue to determine whether the updated target time step is greater than zero until the updated target time step is equal to zero; generate a target video based on the updated multiple first video frame images, wherein the image granularity of the updated multiple first video frame images is greater than the image granularity of the original multiple first video frame images; and generate a target video based on the multiple first video frame images in response to the target time step being equal to zero.

[0181] Optionally, the first processing module 702 is further configured to: determine a sixth video based on multiple first video frame images; perform noise reduction processing on the sixth video using a low-pass filter to obtain a seventh video, wherein the smoothness of the video frame of the seventh video is higher than that of the sixth video; perform iterative processing on the seventh video based on a video diffusion model to obtain an eighth video, wherein the video content realism of the eighth video is higher than that of the seventh video; and perform prediction processing on multiple video frame images of the eighth video based on an image diffusion model to obtain updated multiple first video frame images.

[0182] Optionally, the first generation module 701 is further configured to: sample noise data from Gaussian distributed random numbers based on the video data dimension of the target video; and perform diffusion processing on the prompt text and noise data to obtain the first video.

[0183] By employing the embodiments of this disclosure, a first video is generated using prompt text describing the video content of the target video to be generated, and noise data with the same video data dimension as the target video. Then, a target video is processed by a target video generation model integrating a video diffusion model and an image diffusion model to obtain the target video. The initial time step of the video diffusion model is the same as that of the image diffusion model, thereby achieving the goal of combining the video diffusion model with a high-quality image generation model to generate a high-quality target video. This achieves the technical effects of improving the video quality of the generated video, enriching the video style of the generated video, and enhancing the consistency between the generated video and the text description, thus solving the technical problem of poor video quality generated by video generation models in related technologies.

[0184] It should be noted that the first generation module 701 and the first processing module 702 mentioned above correspond to steps S21 and S22 in Embodiment 1. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0185] According to an embodiment of this disclosure, another apparatus embodiment for implementing the above-described video generation method is also provided. Figure 8 is a schematic structural diagram of another video generation apparatus according to embodiment 5 of this disclosure. As shown in Figure 8, the apparatus includes:

[0186] The second generation module 801 is configured to generate an initial animation video based on animation prompt text and noise data. The animation prompt text is used to describe the video content of the target animation video to be generated, and the data dimension of the noise data is the same as the video data dimension of the target animation video.

[0187] The second processing module 802 is configured to perform video generation processing on the initial animation video based on the target animation video generation model to obtain the target animation video. The target animation video generation model integrates a video diffusion model and an image diffusion model, and the initial time step of the video diffusion model is the same as that of the image diffusion model.

[0188] Optionally, the second processing module 802 is further configured to: generate a second animation video based on a video diffusion model and a first animation video, wherein the video content realism of the second animation video is higher than that of the first animation video; generate multiple first video frame images based on an image diffusion model and the second animation video, wherein the image granularity of any first video frame image is greater than the image granularity of the video frame images in the second animation video; and generate a target animation video based on the multiple first video frame images.

[0189] By employing the embodiments of this disclosure, a first animation video is generated using animated prompt text describing the video content of the target animation video to be generated, and noise data with the same video data dimension as the target animation video to be generated. Then, a target animation video generation model integrating a video diffusion model and an image diffusion model is used to perform video generation processing on the first animation video to obtain the target animation video. The initial time step of the video diffusion model is the same as the initial time step of the image diffusion model, thereby achieving the goal of combining the video diffusion model with a high-quality image generation model to generate a high-quality target video. This achieves the technical effects of improving the video quality of the generated video, enriching the video style of the generated video, and enhancing the consistency between the generated video and the text description, thus solving the technical problem of poor video quality generated by video generation models in related technologies.

[0190] It should be noted that the second generation module 801 and the second processing module 802 mentioned above correspond to steps S41 and S42 in Embodiment 2. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 2. It should be noted that the above modules or units can be hardware components or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0191] According to embodiments of this disclosure, another embodiment of an apparatus for implementing the above-described video generation method is also provided. Figure 9 is a schematic structural diagram of another video generation apparatus according to embodiment 5 of this disclosure. As shown in Figure 9, the apparatus includes:

[0192] The first acquisition module 901 is configured to acquire a video generation request through a first application programming interface. The request data carried in the video generation request includes: prompt text and noise data. The prompt text is used to describe the video content of the target video to be generated, and the data dimension of the noise data is the same as the video data dimension of the target video.

[0193] The first return module 902 is configured to return a video generation response through a second application programming interface. The response data carried in the video generation response includes: a target video, which is obtained by performing video generation processing on the first video based on a target video generation model. The target video generation model integrates a video diffusion model and an image diffusion model. The initial time step of the video diffusion model is the same as the initial time step of the image diffusion model. The first video is generated based on prompt text and noise data.

[0194] In this embodiment, a video generation request carrying prompt text and noise data is obtained through a first application programming interface (API). The prompt text describes the video content of the target video to be generated, and the data dimension of the noise data is the same as that of the target video. Then, a video generation response carrying the target video is returned through a second API. The target video is obtained by performing video generation processing on a first video based on a target video generation model. The target video generation model integrates a video diffusion model and an image diffusion model. The initial time step of the video diffusion model is the same as that of the image diffusion model. The first video is generated based on the prompt text and noise data. This achieves the goal of combining the video diffusion model with a high-quality image generation model to generate a high-quality target video. This improves the video quality of the generated video, enriches the video style of the generated video, and enhances the consistency between the generated video and the text description. This solves the technical problem of poor video quality generated by video generation models in related technologies.

[0195] It should be noted that the first acquisition module 901 and the first return module 902 mentioned above correspond to steps S51 and S52 in Embodiment 3. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 3. It should be noted that the above modules or units can be hardware components or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of the device and run in the computer terminal 10 provided in Embodiment 1.

[0196] According to embodiments of this disclosure, another apparatus embodiment for implementing the above-described video generation method is also provided. Figure 10 is a schematic structural diagram of another video generation apparatus according to embodiment 5 of this disclosure. As shown in Figure 10, the apparatus includes:

[0197] The second acquisition module 1001 is configured to acquire the currently input video generation dialogue request, wherein the request data carried in the video generation dialogue request includes: prompt text and noise data. The prompt text is used to describe the video content of the target video to be generated, and the data dimension of the noise data is the same as the video data dimension of the target video.

[0198] The second return module 1002 is configured to return a video generation dialogue response in response to a video generation dialogue request. The video generation dialogue response carries information including: a target video, which is obtained by performing video generation processing on a first video based on a target video generation model. The target video generation model integrates a video diffusion model and an image diffusion model. The initial time step of the video diffusion model is the same as the initial time step of the image diffusion model. The first video is generated based on prompt text and noise data.

[0199] Display module 1003 is configured to display the target video within a graphical user interface.

[0200] By employing the embodiments of this disclosure, a video generation dialogue request is obtained from the current input. The request data carried in the video generation dialogue request includes: prompt text and noise data. The prompt text describes the video content of the target video to be generated, and the data dimension of the noise data is the same as the video data dimension of the target video. Then, in response to the video generation dialogue request, a video generation dialogue response is returned. The information carried in the video generation dialogue response includes: the target video, which is obtained by processing a first video based on a target video generation model. The target video generation model integrates a video diffusion model and an image diffusion model. The initial time step of the video diffusion model is the same as the initial time step of the image diffusion model. The first video is generated based on the prompt text and noise data. After obtaining the target video, it is displayed in the graphical user interface to provide feedback to the user. This achieves the goal of combining the video diffusion model with a high-quality image generation model to generate a high-quality target video, thereby improving the video quality of the generated video, enriching the video style of the generated video, and enhancing the consistency between the generated video and the text description. This solves the technical problem of poor video quality generated by video generation models in related technologies.

[0201] It should be noted that the second acquisition module 1001, the second return module 1002, and the display module 1003 mentioned above correspond to steps S61 to S63 in Embodiment 4. The three modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 4. It should be noted that the above modules or units can be hardware components or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of the device and run in the computer terminal 10 provided in Embodiment 1.

[0202] It should be noted that the preferred implementation schemes involved in the above embodiments of this disclosure are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0203] Example 6

[0204] The embodiments of this disclosure can provide a computer terminal (or electronic device), which can be any one of a group of computer terminal devices. Optionally, in this embodiment, the computer terminal can also be replaced by a mobile terminal or other terminal device.

[0205] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0206] In this embodiment, the computer terminal described above can execute the program code for the following steps in the video generation method: generating a first video based on prompt text and noise data, wherein the prompt text is used to describe the video content of the target video to be generated, and the data dimension of the noise data is the same as the video data dimension of the target video; performing video generation processing on the first video based on the target video generation model to obtain the target video, wherein the target video generation model integrates a video diffusion model and an image diffusion model, and the initial time step of the video diffusion model is the same as the initial time step of the image diffusion model.

[0207] Optionally, FIG11 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. As shown in FIG11, taking electronic device A as an example, electronic device A may include: one or more (only one is shown in the figure) processors 1102, memory 1104, memory controller, and peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0208] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the video generation method and apparatus in this embodiment. The processor executes the stored software programs and modules to perform various functional applications and data processing, thereby realizing the aforementioned video generation method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to electronic device A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0209] The processor can invoke information and applications stored in the memory through the transmission device to perform the following steps: generating a first video based on prompt text and noise data, wherein the prompt text is used to describe the video content of the target video to be generated, and the data dimension of the noise data is the same as the video data dimension of the target video; performing video generation processing on the first video based on the target video generation model to obtain the target video, wherein the target video generation model integrates a video diffusion model and an image diffusion model, and the initial time step of the video diffusion model is the same as the initial time step of the image diffusion model.

[0210] Optionally, the processor may also execute program code for the following steps: generating a second video based on a video diffusion model and a first video, wherein the video content of the second video has a higher fidelity than the video content of the first video; generating multiple first video frame images based on an image diffusion model and the second video, wherein the image granularity of any first video frame image is greater than the image granularity of the video frame images in the second video; and generating a target video based on the multiple first video frame images.

[0211] Optionally, the processor may also execute program code for the following steps: applying a low-pass filter to the first video to reduce noise, resulting in a third video, wherein the low-pass filter is constructed based on the Fast Fourier Transform, and the smoothness of the third video is higher than that of the first video; adding Gaussian noise to the third video to obtain a fourth video, wherein the fourth video conforms to the domain of the video diffusion model at the time step; and iteratively processing the fourth video based on the video diffusion model to obtain a second video.

[0212] Optionally, the processor may also execute program code for the following steps: performing diffusion processing on the second video to obtain a fifth video, wherein the clarity of the fifth video is higher than that of the second video; performing mapping processing on the fifth video based on a differential function to obtain multiple second video frame images, wherein the multiple second video frame images conform to the Gaussian noise distribution of the image diffusion model at the time step; and performing prediction processing on the multiple second video frame images based on the image diffusion model to obtain multiple first video frame images.

[0213] Optionally, the processor may also execute program code that performs the following steps: determining the current target time step; updating the target time step in response to the target time step being greater than zero; updating multiple first video frame images based on a low-pass filter, a video diffusion model, and an image diffusion model at the updated target time step to obtain updated multiple first video frame images, and continuing to determine whether the updated target time step is greater than zero until the updated target time step is equal to zero; generating a target video based on the updated multiple first video frame images, wherein the image granularity of the updated multiple first video frame images is greater than the image granularity of the original multiple first video frame images; and generating a target video based on the multiple first video frame images in response to the target time step being equal to zero.

[0214] Optionally, the processor may also execute program code for the following steps: determining a sixth video based on multiple first video frame images; performing noise reduction processing on the sixth video using a low-pass filter to obtain a seventh video, wherein the smoothness of the video frame of the seventh video is higher than that of the sixth video; performing iterative processing on the seventh video based on a video diffusion model to obtain an eighth video, wherein the video content fidelity of the eighth video is higher than that of the seventh video; and performing prediction processing on multiple video frame images of the eighth video based on an image diffusion model to obtain updated multiple first video frame images.

[0215] Optionally, the processor may also execute program code that performs the following steps: sampling noise data from Gaussian distributed random numbers based on the video data dimension of the target video; performing diffusion processing on the prompt text and noise data to obtain the first video.

[0216] By employing the embodiments of this disclosure, a first video is generated using prompt text describing the video content of the target video to be generated, and noise data with the same video data dimension as the target video. Then, a target video is processed by a target video generation model integrating a video diffusion model and an image diffusion model to obtain the target video. The initial time step of the video diffusion model is the same as that of the image diffusion model, thereby achieving the goal of combining the video diffusion model with a high-quality image generation model to generate a high-quality target video. This achieves the technical effects of improving the video quality of the generated video, enriching the video style of the generated video, and enhancing the consistency between the generated video and the text description, thus solving the technical problem of poor video quality generated by video generation models in related technologies.

[0217] It will be understood by those skilled in the art that the structure shown in Figure 11 is merely illustrative, and electronic device A may also be a smartphone (such as an Android phone, iOS phone, etc.), tablet computer, PDA, mobile internet device (MID), PAD, or other terminal device. Figure 11 does not limit the structure of the aforementioned electronic device. For example, electronic device A may also include more or fewer components (such as network interfaces, display devices, etc.) than shown in Figure 11, or have a different configuration than shown in Figure 11.

[0218] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0219] Example 7

[0220] Embodiments of this disclosure also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the video generation method provided in Embodiment 1.

[0221] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0222] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: generating a first video based on prompt text and noise data, wherein the prompt text is used to describe the video content of the target video to be generated, and the data dimension of the noise data is the same as the video data dimension of the target video; performing video generation processing on the first video based on the target video generation model to obtain the target video, wherein the target video generation model integrates a video diffusion model and an image diffusion model, and the initial time step of the video diffusion model is the same as the initial time step of the image diffusion model.

[0223] Embodiments of this disclosure also provide a computer program product comprising a computer program that, when executed by a processor, implements any of the above-described video generation methods.

[0224] Optionally, in this embodiment, when the computer program product is executed by the processor, the program code is used to perform the following steps: generating a first video based on prompt text and noise data, wherein the prompt text is used to describe the video content of the target video to be generated, and the data dimension of the noise data is the same as the video data dimension of the target video; performing video generation processing on the first video based on the target video generation model to obtain the target video, wherein the target video generation model integrates a video diffusion model and an image diffusion model, and the initial time step of the video diffusion model is the same as the initial time step of the image diffusion model.

[0225] The sequence numbers of the embodiments disclosed above are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0226] In the above embodiments of this disclosure, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0227] In the several embodiments provided in this disclosure, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0228] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0229] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0230] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0231] The above description is only a preferred embodiment of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of this disclosure, and these improvements and modifications should also be considered within the scope of protection of this disclosure.

Claims

1. A video generation method, comprising: A first video is generated based on prompt text and noise data, wherein the prompt text is used to describe the video content of the target video to be generated, and the data dimension of the noise data is the same as the video data dimension of the target video; The first video is processed by a target video generation model to obtain the target video. The target video generation model integrates a video diffusion model and an image diffusion model. The initial time step of the video diffusion model is the same as that of the image diffusion model.

2. The video generation method according to claim 1, wherein, The process of generating the target video from the first video using the target video generation model includes: A second video is generated based on the video diffusion model and the first video, wherein the video content of the second video has a higher fidelity than the video content of the first video. Multiple first video frame images are generated based on the image diffusion model and the second video, wherein the image granularity of any first video frame image is greater than the image granularity of the video frame images in the second video. The target video is generated based on the multiple first video frame images.

3. The video generation method according to claim 2, wherein, The generation of the second video based on the video diffusion model and the first video includes: The first video is denoised using a low-pass filter to obtain a third video. The low-pass filter is constructed based on the fast Fourier transform, and the smoothness of the third video is higher than that of the first video. Gaussian noise is added to the third video to obtain a fourth video, wherein the fourth video conforms to the domain of the video diffusion model at the time step; The fourth video is iteratively processed based on the video diffusion model to obtain the second video.

4. The video generation method according to claim 3, wherein, The step of generating multiple first video frame images based on the image diffusion model and the second video includes: The second video is diffused to obtain a fifth video, wherein the clarity of the fifth video is higher than that of the second video; The fifth video is mapped based on the differential function to obtain multiple second video frame images, wherein the multiple second video frame images conform to the Gaussian noise distribution of the image diffusion model at the time step; Based on the image diffusion model, the multiple second video frame images are predicted to obtain the multiple first video frame images.

5. The video generation method according to claim 4, wherein, The process of generating the target video based on the plurality of first video frame images includes: Determine the current target time step; In response to the target time step being greater than zero, update the target time step; At the updated target time step, the multiple first video frame images are updated based on the low-pass filter, the video diffusion model, and the image diffusion model to obtain the updated multiple first video frame images. Then, it is determined whether the updated target time step is greater than zero until the updated target time step is equal to zero. The target video is generated based on the updated multiple first video frame images, wherein the image granularity of the updated multiple first video frame images is greater than the image granularity of the original multiple first video frame images. In response to the target time step being zero, the target video is generated based on the plurality of first video frame images.

6. The video generation method according to claim 5, wherein, The step of updating the plurality of first video frame images based on the low-pass filter, the video diffusion model, and the image diffusion model to obtain the updated plurality of first video frame images includes: The sixth video is determined based on the multiple first video frame images; The sixth video is denoised using the low-pass filter to obtain the seventh video, wherein the smoothness of the seventh video is higher than that of the sixth video. The seventh video is iteratively processed based on the video diffusion model to obtain the eighth video, wherein the video content of the eighth video has a higher fidelity than that of the seventh video. Based on the image diffusion model, multiple video frame images of the eighth video are predicted to obtain updated multiple first video frame images.

7. The video generation method according to claim 1, wherein, The generation of the first video based on the prompt text and noise data includes: The noise data is obtained by sampling from Gaussian distributed random numbers based on the video data dimension of the target video; The prompt text and the noise data are diffused to obtain the first video.

8. A video generation method, comprising: A first animated video is generated based on animated prompt text and noise data, wherein the animated prompt text is used to describe the video content of the target animated video to be generated, and the data dimension of the noise data is the same as the video data dimension of the target animated video; The first animation video is processed by a target animation video generation model to obtain the target animation video. The target animation video generation model integrates a video diffusion model and an image diffusion model. The initial time step of the video diffusion model is the same as that of the image diffusion model.

9. The video generation method according to claim 8, wherein, The process of generating the target animation video by performing video generation processing on the first animation video based on the target animation video generation model includes: A second animated video is generated based on the video diffusion model and the first animated video, wherein the video content of the second animated video has a higher realism than the video content of the first animated video. Multiple first video frame images are generated based on the image diffusion model and the second animation video, wherein the image granularity of any first video frame image is greater than the image granularity of the video frame images in the second animation video. The target animation video is generated based on the multiple first video frame images.

10. A video generation method, comprising: A video generation request is obtained through a first application programming interface, wherein the request data carried in the video generation request includes: prompt text and noise data, the prompt text is used to describe the video content of the target video to be generated, and the data dimension of the noise data is the same as the video data dimension of the target video; A video generation response is returned through a second application programming interface, wherein the response data carried in the video generation response includes: the target video, which is obtained by performing video generation processing on the first video based on a target video generation model, the target video generation model integrating a video diffusion model and an image diffusion model, the initial time step of the video diffusion model being the same as the initial time step of the image diffusion model, and the first video being generated based on the prompt text and the noise data.

11. A video generation method, comprising: Obtain the currently input video generation dialogue request, wherein the request data carried in the video generation dialogue request includes: prompt text and noise data, wherein the prompt text is used to describe the video content of the target video to be generated, and the data dimension of the noise data is the same as the video data dimension of the target video; In response to the video generation dialogue request, a video generation dialogue response is returned, wherein the information carried in the video generation dialogue response includes: the target video, the target video being obtained by performing video generation processing on the first video based on the target video generation model, the target video generation model integrating a video diffusion model and an image diffusion model, the initial time step of the video diffusion model being the same as the initial time step of the image diffusion model, and the first video being generated based on the prompt text and the noise data; The target video is displayed within a graphical user interface.

12. An electronic device, comprising: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the video generation method according to any one of claims 1 to 11.

13. A computer-readable storage medium comprising a stored executable program, wherein, When the executable program is executed, it controls the device containing the computer-readable storage medium to perform the video generation method according to any one of claims 1 to 11.

14. A computer program product comprising a computer program that, when executed by a processor, implements the video generation method according to any one of claims 1 to 11.

Images