Video generation methods, electronic device, and computer-readable storage medium

Abstract

The present disclosure relates to the technical field of computers and video processing. Disclosed are video generation methods, an electronic device, and a computer-readable storage medium. A method comprises: acquiring a generation condition and a control condition, the generation condition being used for providing a video material for a target video to be generated, the control condition being used for guiding generation of video content matched with the generation condition, and the duration of the target video being greater than a preset duration; and, on the basis of the generation condition, the control condition and target noise corresponding to the target video, generating the target video, the target noise being the same initial noise added to a plurality of target image frames contained in the generated target video, and the target noise being used for controlling the smooth transition of the target video between the different image frames. The present disclosure solves the technical problems of bad video rationality, video continuity and video transition of long videos generated by long video generation methods in the prior art.

Description

Video generation methods, electronic devices and computer-readable storage media Technical Field

[0001] This disclosure relates to the fields of computer technology and video processing technology, and more specifically, to a video generation method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the continuous development of text-to-image technology, video generation has attracted much attention. Current video generation methods are usually based on diffusion models to generate short videos of corresponding lengths. However, many problems still exist when generating long videos.

[0003] One existing long video generation method fixes the parameters of the basic text-to-image model during training. This can lead to unreasonable motion in a set of generated image frames. Furthermore, when generating multiple sets of image frames, the method may cause background jumps, resulting in poor video continuity and transitions in the generated long videos.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This disclosure provides a video generation method, an electronic device, and a computer-readable storage medium to at least solve the technical problems of poor video rationality, video continuity, and video transition in long videos generated by related art long video generation methods.

[0006] According to one aspect of the present disclosure, a video generation method is provided, comprising: acquiring generation conditions and control conditions, wherein the generation conditions are used to provide video material for a target video to be generated, and the control conditions are used to guide the generation of video content adapted to the generation conditions, wherein the video duration of the target video is greater than a preset duration; generating a target video based on the generation conditions, the control conditions, and target noise corresponding to the target video, wherein the target noise is the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise is used to control the smooth transition between different image frames of the target video.

[0007] According to another aspect of the embodiments of this disclosure, a video generation method is also provided, comprising: obtaining a video generation request through a first application programming interface; and returning a video generation response through a second application programming interface; wherein the request data carried in the video generation request includes: generation conditions and control conditions, and the response data carried in the video generation response includes: a target video, the generation conditions being used to provide video material for the target video to be generated, the control conditions being used to guide the generation of video content adapted to the generation conditions, the video duration of the target video being greater than a preset duration, the target video being generated based on the generation conditions, the control conditions, and the target noise corresponding to the target video, the target noise being the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise being used to control the smooth transition of the target video between different image frames.

[0008] According to another aspect of the embodiments of this disclosure, a video generation method is also provided, comprising: acquiring a currently input video generation dialogue request; responding to the video generation dialogue request and returning a video generation dialogue response; wherein the request data carried in the video generation dialogue request includes: generation conditions and control conditions, and the information carried in the video generation dialogue response includes: a target video, the generation conditions being used to provide video material for the target video to be generated, the control conditions being used to guide the generation of video content adapted to the generation conditions, the video duration of the target video being greater than a preset duration, the target video being generated based on the generation conditions, the control conditions, and the target noise corresponding to the target video, the target noise being the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise being used to control the smooth transition of the target video between different image frames; and displaying the target video in a graphical user interface.

[0009] According to another aspect of the present disclosure, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes any of the above-described video generation methods during runtime.

[0010] According to another aspect of the present disclosure, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform any of the above-described video generation methods.

[0011] According to another aspect of the present disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements any of the above-described video generation methods.

[0012] In this embodiment, by acquiring generation conditions for providing video material for the target video to be generated and control conditions for guiding the generation of video content adapted to the generation conditions, and generating the target video based on the generation conditions, control conditions, and the same initialization noise added to multiple target image frames of the target video to be generated, the target video can be generated stably with a video duration longer than a preset duration. By controlling the smooth transition of the target video between different image frames, a stable long video can be generated. This achieves the goal of generating a stable, reasonable, continuous, and well-transitioned long video, thereby improving the video reasonableness, video continuity, and video transition of the generated long video. This solves the technical problem that the long video generation methods in related technologies have poor video reasonableness, video continuity, and video transition.

[0013] It is worth noting that the above general description and the following detailed description are merely for illustrative and explanatory purposes and do not constitute a limitation thereof. Attached Figure Description

[0014] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0015] Figure 1 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a video generation method according to Embodiment 1 of this disclosure;

[0016] Figure 2 is a flowchart of a video generation method according to Embodiment 1 of this disclosure;

[0017] Figure 3 is a schematic diagram of inference frame grouping according to Embodiment 1 of this disclosure;

[0018] Figure 4 is a schematic diagram of model training according to Embodiment 1 of this disclosure;

[0019] Figure 5 is a flowchart of a video generation method according to Embodiment 2 of this disclosure;

[0020] Figure 6 is a flowchart of a video generation method according to Embodiment 3 of this disclosure;

[0021] Figure 7 is a schematic diagram of a video generation device according to Embodiment 4 of the present disclosure;

[0022] Figure 8 is a schematic diagram of another video generation apparatus according to Embodiment 4 of this disclosure;

[0023] Figure 9 is a schematic diagram of another video generation apparatus according to Embodiment 4 of this disclosure;

[0024] Figure 10 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] First, some nouns or terms that appear in the description of the embodiments of this disclosure shall be interpreted as follows:

[0028] A diffusion model is a mathematical model used to describe and predict the propagation and diffusion of matter or information in space. Diffusion models are typically based on diffusion equations, which describe the speed and direction of propagation of diffusing matter or information in space. Diffusion equations usually include parameters such as diffusion coefficients and initial conditions, which can be used to model and predict the diffusion process. In this embodiment, the diffusion model learns the transformation from noise to the image latent space by modeling the diffusion pattern of data in the latent space.

[0029] Stable diffusion is a mathematical model used in statistics and stochastic process theory to describe stochastic processes with long-tailed distributions. A stable diffusion process is a special type of stochastic process characterized by the stability of its distribution under certain conditions, meaning that the distribution of the random variable remains unchanged under scaling and translation transformations. In this embodiment, Stable Diffusion refers to a common textural graph diffusion model.

[0030] animediff: A time series model for Stable Diffusion that can provide some stability in the time dimension.

[0031] ControlNet: A neural network that controls a pre-trained image diffusion model (such as Stable Diffusion). ControlNet allows input of a modulated image, which is then used to manipulate the generated image. There are many types of modulated images, such as doodles, edge maps, pose keypoints, depth maps, segmentation maps, normal maps, etc., all of which can be used as conditional inputs to guide the content of the generated image.

[0032] Stable long video: In this embodiment of the disclosure, stable long video refers to a long video that has been processed and has relatively stable image quality and smoothness, and is not prone to stuttering or discontinuous images.

[0033] Latent space, in machine learning and deep learning, refers to the low-dimensional space used to represent data. In this space, each dimension represents a feature or attribute of the data, thus capturing its key characteristics. The concept of latent space is often used to represent the data representation after processing by an encoder, which maps high-dimensional input data into the latent space. In deep learning, latent space is commonly used to represent compressed representations of data, better capturing its essential features and thus enabling more efficient use in dimensionality reduction or feature extraction. Furthermore, latent space can also be used in generative models, such as by using a decoder to map the representation in the latent space back to the original data space, thereby achieving data generation or reconstruction.

[0034] Variational Autoencoder (VAE): A generative model that combines the ideas of autoencoders and variational inference. It encodes input data by learning the latent distribution of the data, and can also sample from the latent space to generate new data. In the embodiments of this disclosure, VAE can map images or videos to the latent space, or decode latent space features into images or videos.

[0035] Currently, there are two types of long video generation methods. One type fixes the parameters of the basic text-to-image model during training, which can lead to unreasonable motion in a set of generated image frames. Furthermore, when generating multiple sets of image frames, this type of method may experience background jumps, resulting in poor video continuity and transitions. The other type of long video generation method integrates different groups of latent space features. This method is limited by GPU memory and requires a longer cross-inference time.

[0036] The long video generation methods of related technologies have the following drawbacks.

[0037] Defect 1: The generated long videos have poor video fidelity, and when generating multiple sets of image frames, there are background jumps, resulting in poor video continuity and transitions.

[0038] Defect 2: Limited by video memory and requiring a long cross-inference time, resulting in a long video generation time.

[0039] To address the aforementioned deficiencies, no effective solution has been proposed prior to this disclosure.

[0040] Example 1

[0041] According to embodiments of this disclosure, a video generation method is provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0042] The method embodiment provided in Embodiment 1 of this disclosure can be executed in a mobile terminal, computer terminal, or similar computing device. Figure 1 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a video generation method according to Embodiment 1 of this disclosure. As shown in Figure 1, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that the structure shown in Figure 1 is merely illustrative and does not limit the structure of the above-described electronic device. For example, the computer terminal 10 may also include more or fewer components than shown in Figure 1, or have a different configuration than shown in Figure 1.

[0043] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuitry are generally referred to herein as "data processing circuitry". This data processing circuitry may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in embodiments of this disclosure, the data processing circuitry serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0044] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the video generation method in this embodiment of the present disclosure. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-described video generation method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0045] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0046] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0047] Under the above operating environment, this disclosure provides a video generation method as shown in Figure 2. Figure 2 is a flowchart of a video generation method according to Embodiment 1 of this disclosure. As shown in Figure 2, the method may include the following steps:

[0048] Step S21: Obtain generation conditions and control conditions. The generation conditions are used to provide video materials for the target video to be generated, and the control conditions are used to guide the generation of video content that matches the generation conditions. The video length of the target video is longer than the preset length.

[0049] Step S22: Generate a target video based on the generation conditions, control conditions, and target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames contained in the generated target video. The target noise is used to control the smooth transition of the target video between different image frames.

[0050] In this embodiment of the disclosure, videos with a duration longer than a preset duration are defined as long videos, and correspondingly, videos with a duration less than or equal to the preset duration are defined as short videos. The preset duration can be set according to actual circumstances and is not limited here.

[0051] The target video to be generated has a duration longer than a preset duration, which can be understood as a long video. That is, the video generation method provided in this embodiment is used to generate long videos.

[0052] The generation conditions are used to provide video materials for the target video to be generated, that is, to determine the video content to be displayed in the target video. Generation conditions can be in the form of text, images, audio, etc., and are not limited here. For example, generation conditions can be prompt text entered by the user through an input method, a specific reference image uploaded by the user, or an audio file described by the user through voice, etc., and are not limited here.

[0053] Control conditions are used to guide the generation of video content that matches the generation conditions. In other words, they guide the generation of video content that matches the video content determined by the generation conditions. This can be understood as the control conditions manipulating the video content of the target video to be generated based on the provided video materials. Control conditions can be determined by selecting an appropriate control model; no restrictions are imposed here.

[0054] Target noise is the same initialization noise added to multiple target image frames contained in the generated target video. Target noise is used to control the smooth transition between different image frames in the target video.

[0055] It is understandable that the target video to be generated includes multiple target image frames. When generating the target video, the final video is often obtained by optimizing noise step by step. In the actual reasoning process of this disclosure, before each group of image frames begins optimization, i.e., during the initialization phase, the same noise is used to optimize each group of image frames. That is, the same initialization noise is added to multiple target image frames contained in the target video, thereby ensuring that the starting point of each group of image frames is consistent during the optimization process. This allows for the stable generation of multiple consecutive groups of images, avoiding jumps between groups, and thus controlling the smooth transition between different image frames in the generated target video, ensuring the reasonableness of the generated target video.

[0056] In this embodiment of the disclosure, by obtaining generation conditions for providing video material for the target video to be generated and control conditions for guiding the generation of video content that is compatible with the generation conditions, and generating the target video based on the generation conditions, control conditions, and the same initialization noise added to multiple target image frames of the target video to be generated, it is possible to stably generate target videos with a video duration longer than a preset duration, and control the smooth transition of the target video between different image frames, that is, to generate stable long videos.

[0057] As can be seen, this disclosure optimizes each group of image frames using the same initialization noise during inference, that is, it uses the same noise for initialization processing, thereby avoiding video jumps between different groups of image frames, ensuring that the generated long video is more stable and the video content is more reasonable, and improving the video continuity and video transition of the generated long video.

[0058] The video generation method provided in this disclosure can be applied, but is not limited to, to video generation application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, it can generate e-commerce related videos, education related videos, and legal related videos, etc., and is not limited here.

[0059] By employing the embodiments of this disclosure, generation conditions for providing video material for the target video to be generated and control conditions for guiding the generation of video content adapted to the generation conditions are obtained. Based on the generation conditions, control conditions, and the same initialization noise added to multiple target image frames of the target video to be generated, the target video is generated. This enables the stable generation of target videos with a duration longer than a preset duration and controls the smooth transition of the target video between different image frames, thus generating stable long videos. This achieves the goal of generating stable, reasonable, continuous, and well-transitioned long videos, thereby improving the video reasonableness, video continuity, and video transition of the generated long videos. This solves the technical problem that the long videos generated by related long video generation methods have poor video reasonableness, video continuity, and video transition.

[0060] In an optional embodiment, step S22 involves generating a target video based on generation conditions, control conditions, and target noise corresponding to the target video, including the following method steps:

[0061] Step S221: The target video generation model is used to process the generation conditions, control conditions and target noise to generate a target video. The target video generation model is used to perform stable diffusion based on the generation conditions, control conditions and target noise to obtain multiple target image frames and to perform temporal linking of the multiple target image frames obtained after diffusion to obtain the target video.

[0062] It is understandable that a stable diffusion model refers to a model in which the diffusion process in time and space can reach a stable state. That is, the diffusion rate and diffusion range of a stable diffusion model remain relatively stable within a certain range and will not change drastically.

[0063] Temporal linking refers to a connection or association between two temporally adjacent events or states. This connection can be causal, sequential, or simultaneous, etc. Temporal linking allows events or states to be connected and organized in chronological order. In this embodiment, temporal linking of multiple target image frames obtained after diffusion can be understood as stitching the multiple target image frames together in chronological order to obtain the target video.

[0064] A temporal linkage model is a mathematical model used to describe and analyze the temporal relationships between events or states. It can be used to establish the chronological order of events or states and the correlations between them. Temporal linkage models can be used to predict future event trends, analyze causal relationships between events, or model and predict time series data.

[0065] In this embodiment of the disclosure, the target video generation model can perform stable diffusion based on generation conditions, control conditions, and target noise to obtain multiple target image frames. That is, the target video generation model in this embodiment of the disclosure has the function of a stable diffusion model.

[0066] It is understood that a video can be viewed as being composed of multiple sets of consecutive images stitched together. The target video generation model can stitch together the generated images and perform temporal smoothing processing on multiple images. That is, the target video generation model can perform temporal linking on multiple target image frames to obtain the target video. In other words, the target video generation model in this embodiment of the present disclosure has the function of a temporal linking model, thereby accurately capturing the temporal dependency between multiple target image frames obtained after diffusion and accurately stitching the multiple target image frames obtained after diffusion into the target video.

[0067] In this embodiment of the disclosure, when generating a target video based on generation conditions, control conditions, and target noise corresponding to the target video, a target video generation model can be used to stably diffuse the generation conditions, control conditions, and target noise to obtain multiple target image frames. Then, the target video generation model is used to temporally link the multiple target image frames obtained after diffusion to obtain the target video.

[0068] In an optional embodiment, the target video generation model includes: a target stable diffusion model, a target control model, and a target temporal linking model. In step S221, the target video generation model is used to process the generation conditions, control conditions, and target noise to obtain the target video, including the following method steps:

[0069] Step S2211: Under the guidance of the control conditions input by the target control model, a stable diffusion model is used to perform image diffusion processing on the generation conditions and target noise to obtain multiple consecutive image frames, and a temporal linking model is used to perform temporal smoothing on the multiple consecutive image frames to obtain multiple target image frames.

[0070] Step S2212: Perform image stitching processing on multiple target image frames to obtain the target video.

[0071] In this embodiment, the target video generation model includes: a target stable diffusion model, a target control model, and a target temporal linking model. The target stable diffusion model can employ a basic stable diffusion model, which is not limited here, to ensure stable diffusion in both time and space. For example, the target stable diffusion model can use the stable diffusion structure commonly used in text-based image models, meaning that images in the training data domain can be generated through multi-step optimization.

[0072] The target control model can be a robust controller, model predictive control (MPC), or similar model that helps the system better control and regulate its behavior, thereby achieving a more stable, efficient, and reliable operating state. For example, the target control model can employ the ControlNet neural network, i.e., the ControlNet model, without limitation. Through ControlNet, the generation process of images in the target video can be better controlled, resulting in more specific and desirable images.

[0073] Temporal linking models can be models such as the animediff model, Long Short-Term Memory (LSTM) network, and Gated Recurrent Unit (GRU) used to capture temporal dependencies and sequence patterns in data. For example, the animediff model can be chosen as the temporal linking model; this is not a limitation. The animediff model is used to extend the text graph model into the animation generator without requiring adjustments for a specific model.

[0074] In this embodiment of the disclosure, when a target video is obtained by using a target video generation model to process the generation conditions, control conditions and target noise, a stable diffusion model (i.e., a basic stable diffusion model) can be used to process the generation conditions and target noise under the guidance of the control conditions input by the target control model (e.g., ControlNet model) to obtain multiple consecutive image frames, that is, multiple consecutive image frames corresponding to each group of images.

[0075] Simultaneously, a temporal linking model (such as the Animediff model) is used to perform temporal smoothing on the obtained multiple consecutive image frames to obtain multiple target image frames. That is, the temporal linking model is used to perform temporal smoothing on the features of multiple consecutive image frames corresponding to each group of images, and a smooth transition is performed between multiple consecutive image frames to obtain multiple target image frames corresponding to each group of images, which is to obtain the generated images of each group.

[0076] Finally, multiple target image frames are stitched together to obtain the target video. This involves stitching together the generated images and then using the same initial noise to smooth the transition between different groups of images, thus obtaining the target video.

[0077] In an optional embodiment, in step S2211, a stable diffusion model is used to perform image diffusion processing on the generation conditions and target noise to obtain multiple consecutive image frames, including the following method steps:

[0078] Step S22111: In the stable diffusion model, feature encoding is performed on the generation conditions to obtain the encoding results corresponding to the generation conditions;

[0079] Step S22112: Spatial mapping is performed on the encoding results to obtain spatial mapping results corresponding to multiple initial image frames;

[0080] Step S22113: Add the target noise to the spatial mapping results corresponding to multiple initial image frames according to the preset noise optimization steps to obtain the latent space features corresponding to multiple initial image frames.

[0081] Step S22114: Perform image diffusion processing on the latent space features corresponding to multiple initial image frames to obtain image features corresponding to multiple initial image frames;

[0082] Step S22115: Decode the image features corresponding to multiple initial image frames to obtain multiple consecutive image frames;

[0083] The number of initial image frames is the pre-set number of image frames to be included in the target video.

[0084] In this embodiment, the stable diffusion model includes a Variational Autoencoder (VAE) and a denoising diffusion model. The VAE is a deep learning model used for model generation and feature learning. The VAE maps input data to a latent space through an encoder, and then generates new data from the latent space through a decoder. The denoising diffusion model is a network used to learn the process of diffusing features from noise into the image's latent space.

[0085] In this embodiment, the stable diffusion model can not only be used to extract spatial or temporal features of video frames for video analysis, identification, or prediction, but it can also process time-series data to extract key temporal information. That is, the stable diffusion model can be used to focus on and integrate encoded features temporally, enabling more effective modeling and analysis of time-series data.

[0086] The initial image frame can be understood as the initial image frame before the target image frame of the target video is obtained through video generation processing, or it can be understood as the inference frame in the actual inference process.

[0087] The number of initial image frames is the pre-set number of image frames to be included in the target video, that is, the pre-set number of video frames to be generated, which is not limited here. For example, if you want to generate a 128-frame video, then the number of initial image frames is 128, that is, the number of inference frames is 128.

[0088] The preset noise optimization steps can be understood as the number or frequency of times noise is introduced during each parameter update in the model training process. For example, the preset noise optimization steps can be adjusted according to the task and the model, and there are no restrictions here.

[0089] In this embodiment of the disclosure, when using a stable diffusion model to perform image diffusion processing on the generation conditions and target noise to obtain image features corresponding to multiple initial image frames, a stable diffusion model can be used to encode the generation conditions to obtain the encoding results corresponding to the generation conditions.

[0090] After obtaining the encoding results corresponding to the generation conditions, a stable diffusion model is used to spatially map the encoding results to obtain spatial mapping results corresponding to multiple initial image frames. This can be understood as mapping the encoding results to the latent space through a VAE, thereby obtaining spatial mapping results corresponding to multiple initial image frames, that is, obtaining spatial mapping results corresponding to all inference image frames during the inference process.

[0091] After obtaining the spatial mapping results corresponding to multiple initial image frames, target noise is added to the spatial mapping results of the multiple initial image frames according to a preset noise optimization step number to obtain the latent spatial features corresponding to the multiple initial image frames. This can be understood as adding the same initial noise to the spatial mapping results of the multiple initial image frames according to a preset noise optimization step number to learn the noise optimization process, that is, learning how to recover the original image from the noisy image, thereby obtaining the latent spatial features corresponding to the multiple initial image frames.

[0092] After obtaining the latent space features corresponding to multiple initial image frames, image diffusion processing is performed on these latent space features to obtain the image features corresponding to the multiple initial image frames. Then, feature decoding is performed on the image features corresponding to the multiple initial image frames to obtain multiple consecutive image frames, i.e., multiple consecutive image frames corresponding to each group of images.

[0093] In one optional embodiment, the size of the noise map corresponding to the target noise is determined based on the image frame size of multiple target image frames, and the content of the noise map corresponding to the target noise is determined based on a random number within a preset range.

[0094] In this embodiment of the disclosure, the size of the noise map corresponding to the target noise is determined based on the image frame size of multiple target image frames, that is, by the size of the target video to be generated. For example, if each frame of the target video is 512*512 RGB (Red, Green, Blue) three-channel, the size converted to the latent space is 64*64*4, then the corresponding target noise will also be 64*64*4, and the noise size is determined by the size of the target video to be generated.

[0095] It is understandable that, regardless of whether the generation conditions are in text, image, or audio format, the size of the noise map corresponding to the target noise is determined by the size of the target video to be generated.

[0096] Understandably, target noise needs to be randomly generated before each video is generated, and the content of the noise map corresponding to the target noise is determined based on random numbers within a preset range. For example, if the target noise is 64*64*4 in size, the random numbers within the preset range are random numbers in the range of -1 to 1, that is, random numbers in the range of -1 to 1 are used to fill the 64*64*4 size, which is not limited here.

[0097] In one optional embodiment, multiple initial image frames are divided into multiple sets of image frame sequences according to a preset number and a preset ratio. The preset number determines the number of image frames contained in each set of image frame sequences, and the preset ratio determines the proportion of overlapping image frames between adjacent sets of image frame sequences. Each set of image frame sequences can use the same initialization noise to utilize target noise to control the smooth transition of the target video between different sets of image frame sequences.

[0098] In this embodiment of the disclosure, during the actual inference process, all inference frames can be grouped according to a preset number, and adjacent groups overlap according to a preset ratio. For example, if the preset number is 24 frames and the preset ratio is 1 / 4, then when grouping all inference frames, the first group of inference frames includes images from frame 1 to frame 24, the second group of inference frames includes images from frame 17 to frame 40, the third group of inference frames includes images from frame 33 to frame 56, and so on, and subsequent groups can be inferred in the same way, which will not be elaborated here.

[0099] In an optional embodiment, in step S22112, the target noise is added to the spatial mapping results corresponding to multiple initial image frames according to the noise optimization steps to obtain the latent space features corresponding to multiple initial image frames, including the following method steps:

[0100] Step S221121: Add the target noise to the spatial mapping results of two adjacent image frame sequences in multiple image frame sequences according to the preset noise optimization steps, obtain the latent space features corresponding to the two adjacent image frame sequences, and record the number of noise optimizations for the image frames that overlap between the two adjacent image frame sequences.

[0101] Step S221122: Overlay the latent space features corresponding to two adjacent sets of image frame sequences to obtain the overlapping region;

[0102] Step S221123: Obtain the average value of features within the overlapping region;

[0103] Step S221124: Based on the feature average value and the number of noise optimization steps, obtain the latent space features corresponding to the image frames that overlap between two adjacent image frame sequences, until all noise optimization steps are completed, and obtain the latent space features corresponding to multiple initial image frames.

[0104] In this embodiment of the disclosure, when adding target noise to the spatial mapping results corresponding to multiple initial image frames according to the noise optimization steps to obtain latent spatial features corresponding to multiple initial image frames, the target noise can be added to the spatial mapping results corresponding to two adjacent groups of image frame sequences according to a preset noise optimization step number, thereby obtaining latent spatial features corresponding to two adjacent groups of image frame sequences. It is understood that, since this disclosure groups multiple initial image frames, and there is overlap between adjacent groups, when the latent spatial features are continuously noise optimized, a certain overlapping area will appear with each noise optimization. Therefore, it is also necessary to record the number of noise optimizations corresponding to the image frames that overlap between adjacent groups of image frame sequences.

[0105] After obtaining the latent space features corresponding to two adjacent sets of image frame sequences, the latent space features of the two adjacent sets of image frame sequences are superimposed to obtain overlapping regions. Then, the feature average value is obtained within the overlapping region. Based on the feature average value and the number of noise optimization steps, the latent space features corresponding to the image frames that overlap between the two adjacent sets of image frame sequences are obtained. This process continues until all noise optimization steps are completed, resulting in latent space features corresponding to multiple initial image frames. For example, multiple initial image frames are divided into two groups: the first group of inference frames includes frames 1-24, and the second group of inference frames includes frames 17-40. Since frames 17-24 are optimized twice during the frame optimization process for both groups, the features of the two groups need to be superimposed after each frame optimization and then divided by the number of noise optimization steps. That is, the latent space features corresponding to frames 17-24 are averaged after each optimization. This ensures the stable and continuous generation of multiple sets of images, without jumps between groups, making the final generated long video more reasonable.

[0106] Figure 3 is a schematic diagram of inference frame grouping according to Embodiment 1 of this disclosure. As shown in Figure 3, the overlap of adjacent group M-1, group M, and group M+1 image frame sequences is illustrated. It can be seen that there are two overlapping regions in the group M image frame sequence that overlap with the preceding adjacent group M-1 image frame sequence and the following adjacent group M+1 image frame sequence. Furthermore, each group of image frame sequences is repeatedly optimized during the continuous optimization process, meaning that there are repeated optimized regions.

[0107] In an optional embodiment, the video generation method further includes the following method steps:

[0108] Step S222: Train the initial video generation model using sample videos, sample conditions, and sample noise to generate the target video generation model.

[0109] In this embodiment of the disclosure, the initial video generation model can be understood as the video generation model before training is completed, that is, the video generation model before the target video generation model is trained.

[0110] In this embodiment of the disclosure, when training the target video generation model, sample videos, sample conditions, and sample noise can be used to train the initial video generation model, thereby generating the target video generation model. Here, sample videos can be understood as sample videos providing video material content during model training, and sample noise can be understood as the initialization noise determined during model training.

[0111] Sample conditions can be understood as control conditions during model training, used to guide the generation of corresponding video content.

[0112] In an optional embodiment, the initial video generation model includes: an initial stable diffusion model, an initial control model, and an initial temporal linking model. In step S222, the initial video generation model is trained using sample videos, sample conditions, and sample noise to generate the target video generation model, including the following method steps:

[0113] Step S2221: Train the initial video generation model using sample videos, sample conditions, and sample noise to obtain the training results to be used.

[0114] Step S2222: Based on the training results to be used, the model parameters of the initial temporal linking model and the model parameters of the initial stable diffusion model are alternately updated to generate the target video generation model.

[0115] In this embodiment of the disclosure, the initial video generation model includes: an initial stable diffusion model, an initial control model, and an initial temporal linking model. For example, the initial stable diffusion model can use the stable diffusion structure commonly used in text-based graph models, the initial control model can adopt the ControlNet model, and the initial temporal linking model can be animatediff model. No restrictions are imposed here.

[0116] In this embodiment of the disclosure, when training the initial video generation model using sample videos, sample conditions, and sample noise to generate the target video generation model, the initial video generation model can be trained using sample videos, sample conditions, and sample noise to obtain the training results to be used. Then, based on the training results to be used, the model parameters of the initial temporal linking model and the model parameters of the initial stable diffusion model are alternately updated, that is, the initial temporal linking model and the initial stable diffusion model are trained alternately to generate the target video generation model.

[0117] In an optional embodiment, in step S2221, the initial video generation model is trained using sample videos, sample conditions, and sample noise to obtain the training results to be used, including the following method steps:

[0118] Step S2221: The sample video is split into multiple sub-videos;

[0119] Step S2222: Perform image frame sampling on multiple sub-videos according to a preset image frame sampling method to obtain sampling results. The preset image frame sampling method is used to adapt to videos with different change rates.

[0120] Step S2223: The initial video generation model is used to process the sampling results, sample conditions and sample noise to generate video, and the training results to be used are obtained.

[0121] In this embodiment of the disclosure, when training the initial video generation model using sample videos, sample conditions, and sample noise to obtain the training results to be used, the sample videos can be split into multiple sub-videos. This can be understood as splitting a long sample video into multiple short videos with shorter durations.

[0122] After obtaining multiple sub-videos, image frames are sampled from the multiple sub-videos according to a preset image frame sampling method to obtain the sampling results. In this embodiment of the disclosure, the preset image frame sampling method can be a random interval frame method to adapt to videos with different change rates. That is, the number of random intervals can be set based on the different change rates of the videos that need to be learned, and there is no limitation here. For example, the maximum number of intervals is usually set to 4 frames, and each time data is sampled, an integer value between 0 and 4 is randomly selected as the interval value, that is, the interval number is randomly selected between 0, 1, 2, 3, and 4.

[0123] After obtaining multiple short videos, image frames can be sampled from the split short videos at random intervals to obtain sampling results. Then, an initial video generation model is used to process the sampling results, sample conditions, and sample noise to generate videos, thereby obtaining the training results to be used.

[0124] For example, during the initial video generation model training process, the sample video can be split into short videos of approximately 200 frames each. Each training image frame sequence consists of approximately N frames, where N is a positive integer and can be 24 frames or higher, depending on the available video memory. These N frames are then sampled from the original short video at random intervals, allowing the initial video generation model to learn videos with different rates of change and obtain sampling results. The initial video generation model is then used to process the sampling results, sample conditions, and sample noise to generate the final training result.

[0125] Understandably, during training, noise of randomly initialized size N frames of latent space features can be used. Typically, the length and width of the latent space features will become 1 / 4 of the original image after VAE mapping. Then, the initialized noise is added to the latent space features of the training image sequence according to the number of noise optimization steps, thereby enabling the model to learn the noise optimization process.

[0126] In an optional embodiment, in step S2222, the model parameters of the initial temporal linking model and the initial stable diffusion model are alternately updated based on the training results to be used to generate the target video generation model, including the following method steps:

[0127] Step S22221: Under the premise that the model parameters of the initial stable diffusion model and the model parameters of the initial control model are fixed in advance, the model parameters of the initial temporal linking model are updated based on the training results to be used, and the first update result is obtained.

[0128] Step S22222: Under the premise that the model parameters of the initial time-series linked model are fixed in advance, the model parameters of the initial control model and the model parameters of the initial stable diffusion model are updated based on the first update result to obtain the second update result;

[0129] Step S22223: The first update result and the second update result are continuously updated alternately according to the preset number of training steps until the target video generation model is generated.

[0130] The preset number of training steps can be determined based on the actual training situation; there is no limit here.

[0131] In this embodiment of the disclosure, when adjusting the model parameters of the initial temporal linking model and the initial stable diffusion model alternately based on the training results to be used to generate the target video generation model, the model parameters of the initial temporal linking model can be updated based on the training results to be used, provided that the model parameters of the initial stable diffusion model and the initial control model are both fixed in advance, to obtain the first update result. This can be understood as follows: during the training of the target video generation model, the model parameters of the initial stable diffusion model and the initial control model are first fixed, only the initial temporal linking model is trained, and the model parameters of the initial temporal linking model are updated to obtain the first update result.

[0132] After training and updating the initial temporal linked model, and with the model parameters of the initial temporal linked model fixed in advance, the model parameters of the initial control model and the initial stable diffusion model are updated based on the first update result to obtain the second update result. This can be understood as follows: Given that the initial temporal linked model training is basically complete, the model parameters of the initial temporal linked model are fixed, the initial control model and the initial stable diffusion model are trained, and their model parameters are updated to obtain the second update result.

[0133] Then, following a preset number of training steps, the first and second update results are continuously and alternately updated. This involves alternately training the initial control model, the initial temporal link model, and the initial stable diffusion model until the initial control model, initial temporal link model, and initial stable diffusion model with multiple parameter updates are obtained, which in turn yield the target control model, target temporal link model, and target stable diffusion model. Finally, the target video generation model is generated based on the target temporal link model, target stable diffusion model, and target control model.

[0134] For example, during the initial training of the video generation model, the parameters of the original Stable Diffusion model and ControlNet model are initially fixed, and the parameters of the Animatediff model are trained and updated separately. Then, when the Animatediff model is basically complete in training, for example, after more than 50,000 steps using 8 GPUs, the Animatediff model is fixed, and the parameters of the Stable Diffusion model and ControlNet model are trained and updated, and this process is repeated. During the loop, it can be set to alternate training and parameter updates every X training steps, for example, every 10,000 training steps, ultimately resulting in the Stable Diffusion model and the Animatediff model with updated parameters.

[0135] Figure 4 is a schematic diagram of model training according to Embodiment 1 of this disclosure. As shown in Figure 4, the target video generation model of this disclosure includes a base model (i.e., a stable diffusion model), a temporal model (i.e., a temporal linking model), and a control model. During the training of the target video generation model, the model parameters of the base model and the control model are first fixed, and the temporal model is trained. After the temporal model training is basically complete, the model parameters of the temporal model are fixed again, and the base model and the control model are trained. After the base model and the control model training is basically complete, the model parameters of the trained base model and the control model are fixed again, and the previously trained temporal model is trained. This process is repeated cyclically for alternating training.

[0136] As can be seen, this disclosure provides a stable long video generation method based on a diffusion model. This method uses short videos for training and, by modifying the original stable diffusion model, can generate stable long videos. During training, the video is split into short videos, each containing approximately 200 frames. Then, samples are taken from the original short videos at random intervals, allowing the model to learn videos with different rates of change. In the actual long video generation process, control conditions such as skeleton points and depth maps can be used to manipulate the generation. Furthermore, noise synchronization initialization processing is required during inference to ensure the continuity and transition of the generated long videos, offering advantages such as small memory usage, short processing time, and good continuity.

[0137] It is easy to understand that the beneficial effects of the video generation method provided in this disclosure include the following points.

[0138] Beneficial effect (1): By sampling the original short video at random intervals, it is possible to learn videos with different rates of change.

[0139] Beneficial effect (2): During the training process, the basic model and the time series model will be trained alternately, which can make the two better integrated.

[0140] Beneficial effect (3): The use of synchronous initialization noise during inference can reduce video jumps between different groups, making long videos more stable and improving video rationality, video continuity and video transition.

[0141] Beneficial effects (4): It occupies less memory and has a shorter inference time.

[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0143] Furthermore, it should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0144] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solutions of this disclosure, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0145] Example 2

[0146] In the operating environment as described in Embodiment 1, this disclosure provides a video generation method as shown in Figure 5. Figure 5 is a flowchart of a video generation method according to Embodiment 2 of this disclosure. As shown in Figure 5, the method includes:

[0147] Step S51: Obtain a video generation request through the first application programming interface;

[0148] Step S52: Return a video generation response through the second application programming interface; wherein, the request data carried in the video generation request includes: generation conditions and control conditions, and the response data carried in the video generation response includes: target video, the generation conditions are used to provide video material for the target video to be generated, the control conditions are used to guide the generation of video content that matches the generation conditions, the video duration of the target video is greater than the preset duration, the target video is generated based on the generation conditions, the control conditions and the target noise corresponding to the target video, the target noise is the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise is used to control the smooth transition between different image frames of the target video.

[0149] The first and second application programming interfaces (APIs) mentioned above can be the same or different APIs. In one optional embodiment, the interface parameters in the first and second APIs may include, but are not limited to: a global interface identifier, an interface signing key, an interface timestamp, an interface request identifier, and a system call credential identifier. The first API can use GET or POST as the interface request method to obtain the file processing request. The second API can use JSON format to return the file processing response.

[0150] In this embodiment of the disclosure, a video generation request can be understood as a request to generate a video, and the request data carried in the video generation request includes: generation conditions and control conditions. A video generation response can be understood as a response to the video generation request, and the response data carried in the video generation response includes: the target video.

[0151] For a detailed description, please refer to the description in Example 1, which will not be repeated here.

[0152] In this embodiment of the disclosure, a video generation request carrying generation conditions and control conditions is obtained through a first application programming interface (API), and then a video generation response carrying the target video is returned through a second API. The generation conditions are used to provide video material for the target video to be generated, and the control conditions are used to guide the generation of video content that matches the generation conditions. The video duration of the target video is longer than a preset duration. The target video is generated based on the generation conditions, control conditions, and target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames contained in the generated target video. The target noise is used to control the smooth transition between different image frames of the target video.

[0153] As can be seen, this disclosure optimizes each group of image frames using the same initialization noise during inference, that is, it uses the same noise for initialization processing, thereby avoiding video jumps between different groups of image frames, ensuring that the generated long video is more stable and the video content is more reasonable, and improving the video continuity and video transition of the generated long video.

[0154] The video generation method provided in this disclosure can be applied, but is not limited to, to video generation application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, it can generate e-commerce related videos, education related videos, and legal related videos, etc., and is not limited here.

[0155] By employing the embodiments of this disclosure, a video generation request carrying generation conditions and control conditions is obtained through a first application programming interface (API), and then a video generation response carrying the target video is returned through a second API. The generation conditions provide video material for the target video to be generated, the control conditions guide the generation of video content adapted to the generation conditions, the target video's duration is greater than a preset duration, and the target video is generated based on the generation conditions, control conditions, and target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise is used to control the smooth transition between different image frames of the target video. This achieves the goal of generating a stable, reasonable, continuous, and well-transitioned long video, thereby improving the video reasonableness, video continuity, and video transition of the generated long video. This solves the technical problem of poor video reasonableness, video continuity, and video transition in related long video generation methods.

[0156] It should be noted that the preferred implementation of this embodiment can be found in the relevant description in Embodiment 1, and will not be repeated here.

[0157] Example 3

[0158] In the operating environment as described in Embodiment 1, this disclosure provides a video generation method as shown in Figure 6. Figure 6 is a flowchart of a video generation method according to Embodiment 3 of this disclosure. As shown in Figure 6, the method includes:

[0159] Step S61: Obtain the currently input video to generate a dialogue request;

[0160] Step S62: In response to the video generation dialogue request, return a video generation dialogue reply; wherein, the request data carried in the video generation dialogue request includes: generation conditions and control conditions, and the information carried in the video generation dialogue reply includes: target video, the generation conditions are used to provide video material for the target video to be generated, the control conditions are used to guide the generation of video content that matches the generation conditions, the video duration of the target video is greater than the preset duration, the target video is generated based on the generation conditions, the control conditions and the target noise corresponding to the target video, the target noise is the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise is used to control the smooth transition between different image frames of the target video;

[0161] Step S63: Display the target video within the graphical user interface.

[0162] In this embodiment of the disclosure, the video generation dialogue request can be understood as a video generation dialogue request initiated by a user when conversing with a smart device, requesting the generation of a video. The request data carried in the video generation dialogue request includes: generation conditions and control conditions. The video generation dialogue response can be understood as the response made by the smart device in response to the video generation dialogue request initiated by the user. The information carried in the video generation dialogue response includes: the target video.

[0163] For a detailed description, please refer to the description in Example 1, which will not be repeated here.

[0164] In this embodiment of the disclosure, a video generation dialog request carrying generation and control conditions is obtained from the current input. Then, in response to the video generation dialog request, a video generation dialog response carrying the target video is returned. The generation conditions are used to provide video materials for the target video to be generated, and the control conditions are used to guide the generation of video content that matches the generation conditions. The video duration of the target video is longer than a preset duration. The target video is generated based on the generation conditions, control conditions, and target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames contained in the generated target video. The target noise is used to control the smooth transition between different image frames of the target video. After obtaining the target video, the target video is displayed in the graphical user interface to provide feedback to the user.

[0165] As can be seen, this disclosure optimizes each group of image frames using the same initialization noise during inference, that is, it uses the same noise for initialization processing, thereby avoiding video jumps between different groups of image frames, ensuring that the generated long video is more stable and the video content is more reasonable, and improving the video continuity and video transition of the generated long video.

[0166] The video generation method provided in this disclosure can be applied, but is not limited to, to video generation application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, it can generate e-commerce related videos, education related videos, and legal related videos, etc., and is not limited here.

[0167] By employing the embodiments of this disclosure, a video generation dialog request carrying generation and control conditions is obtained from the current input. Then, in response to the video generation dialog request, a video generation dialog response carrying the target video is returned. The generation conditions provide video material for the target video to be generated, and the control conditions guide the generation of video content adapted to the generation conditions. The target video's duration is greater than a preset duration. The target video is generated based on the generation conditions, control conditions, and target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames contained in the generated target video. The target noise is used to control the smooth transition between different image frames of the target video. After obtaining the target video, it is displayed in the graphical user interface to provide feedback to the user. This achieves the goal of generating a stable, reasonable, continuous, and well-transitioned long video, thereby improving the video reasonableness, video continuity, and video transition of the generated long video. This solves the technical problem of poor video reasonableness, video continuity, and video transition in related long video generation methods.

[0168] It should be noted that the preferred implementation of this embodiment can be found in the relevant description in Embodiment 1, and will not be repeated here.

[0169] Example 4

[0170] According to embodiments of this disclosure, an apparatus embodiment for implementing the above-described video generation method is also provided. Figure 7 is a schematic structural diagram of a video generation apparatus according to embodiment 4 of this disclosure. As shown in Figure 7, the apparatus includes:

[0171] The first acquisition module 701 is set to acquire generation conditions and control conditions. The generation conditions are used to provide video materials for the target video to be generated, and the control conditions are used to guide the generation of video content that matches the generation conditions. The video duration of the target video is longer than the preset duration.

[0172] The generation module 702 is configured to generate a target video based on generation conditions, control conditions, and target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames contained in the generated target video. The target noise is used to control the smooth transition of the target video between different image frames.

[0173] Optionally, the first acquisition module 701 is further configured to: use a target video generation model to perform video generation processing on the generation conditions, control conditions and target noise to obtain a target video, wherein the target video generation model is used to perform stable diffusion based on the generation conditions, control conditions and target noise to obtain multiple target image frames and to perform temporal linking on the multiple target image frames obtained after diffusion to obtain a target video.

[0174] Optionally, the target video generation model includes: a target stable diffusion model, a target control model, and a target temporal linking model. The first acquisition module 701 is further configured to: under the guidance of the control conditions input through the target control model, use the stable diffusion model to perform image diffusion processing on the generation conditions and target noise to obtain multiple consecutive image frames, and use the temporal linking model to perform temporal smoothing on the multiple consecutive image frames to obtain multiple target image frames; and perform image stitching processing on the multiple target image frames to obtain the target video.

[0175] Optionally, the first acquisition module 701 is further configured to: in a stable diffusion model, perform feature encoding on the generation conditions to obtain the encoding result corresponding to the generation conditions; perform spatial mapping on the encoding result to obtain spatial mapping results corresponding to multiple initial image frames; add target noise to the spatial mapping results corresponding to multiple initial image frames according to a preset noise optimization step number to obtain latent space features corresponding to multiple initial image frames; perform image diffusion processing on the latent space features corresponding to multiple initial image frames to obtain image features corresponding to multiple initial image frames; and perform feature decoding on the image features corresponding to multiple initial image frames to obtain multiple consecutive image frames, wherein the number of multiple initial image frames is the preset number of image frames to be included in the target video.

[0176] Optionally, the size of the noise map corresponding to the target noise is determined based on the image frame size of multiple target image frames, and the content of the noise map corresponding to the target noise is determined based on random numbers within a preset range.

[0177] Optionally, multiple initial image frames are divided into multiple sets of image frame sequences according to a preset number and a preset ratio. The preset number is used to determine the number of image frames contained in each set of image frame sequences, and the preset ratio is used to determine the proportion of image frames that overlap between two adjacent sets of image frame sequences.

[0178] Optionally, the first acquisition module 701 is further configured to: add target noise to the spatial mapping results corresponding to two adjacent image frame sequences in multiple image frame sequences according to a preset noise optimization step number, obtain the latent space features corresponding to the two adjacent image frame sequences, and record the noise optimization number corresponding to the image frames that overlap between the two adjacent image frame sequences; perform feature superposition on the current latent space features corresponding to the two adjacent image frame sequences to obtain the overlapping region; obtain the feature average value in the overlapping region; obtain the latent space features corresponding to the image frames that overlap between the two adjacent image frame sequences based on the feature average value and the noise optimization number, until all noise optimization steps are completed, and obtain the latent space features corresponding to multiple initial image frames.

[0179] Optionally, it also includes a training module, configured to train an initial video generation model using sample videos, sample conditions, and sample noise to generate a target video generation model.

[0180] Optionally, the initial video generation model includes: an initial stable diffusion model, an initial control model, and an initial temporal linking model. The training module is further configured to: train the initial video generation model using sample videos, sample conditions, and sample noise to obtain the training results to be used; and alternately update the model parameters of the initial temporal linking model and the model parameters of the initial stable diffusion model based on the training results to be used, thereby generating the target video generation model.

[0181] Optionally, the training module is further configured to: update the model parameters of the initial stable diffusion model and the initial control model based on the training results to be used, given that the model parameters of the initial temporal linking model and the initial control model are both fixed in advance, to obtain a first update result; update the model parameters of the initial control model and the initial stable diffusion model based on the first update result, given that the model parameters of the initial temporal linking model are fixed in advance, to obtain a second update result; and continuously update the first update result and the second update result alternately according to a preset number of training steps until the target video generation model is generated.

[0182] Optionally, the above training module is further configured to: split the sample video into multiple sub-videos; sample the multiple sub-videos according to a preset image frame sampling method to obtain sampling results, wherein the preset image frame sampling method is used to adapt to videos with different change rates; and use an initial video generation model to perform video generation processing on the sampling results, sample conditions and sample noise to obtain training results to be used.

[0183] By employing the embodiments of this disclosure, generation conditions for providing video material for the target video to be generated and control conditions for guiding the generation of video content adapted to the generation conditions are obtained. Based on the generation conditions, control conditions, and the same initialization noise added to multiple target image frames of the target video to be generated, the target video is generated. This enables the stable generation of target videos with a duration longer than a preset duration and controls the smooth transition of the target video between different image frames, thus generating stable long videos. This achieves the goal of generating stable, reasonable, continuous, and well-transitioned long videos, thereby improving the video reasonableness, video continuity, and video transition of the generated long videos. This solves the technical problem that the long videos generated by related long video generation methods have poor video reasonableness, video continuity, and video transition.

[0184] It should be noted that the first acquisition module 701 and the generation module 702 mentioned above correspond to steps S21 and S22 in Embodiment 1. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware components or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of the device and run in the computer terminal 10 provided in Embodiment 1.

[0185] According to embodiments of this disclosure, another apparatus embodiment for implementing the above-described video generation method is also provided. Figure 8 is a schematic structural diagram of another video generation method apparatus according to Embodiment 4 of this disclosure. As shown in Figure 8, the apparatus includes:

[0186] The second acquisition module 801 is configured to acquire a video generation request through the first application programming interface;

[0187] The first return module 802 is configured to return a video generation response via a second application programming interface. The video generation request carries request data including generation conditions and control conditions. The video generation response carries response data including a target video. The generation conditions provide video material for the target video to be generated. The control conditions guide the generation of video content that matches the generation conditions. The target video's duration is greater than a preset duration. The target video is generated based on the generation conditions, control conditions, and target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames contained in the generated target video. The target noise is used to control the smooth transition between different image frames in the target video.

[0188] By employing the embodiments of this disclosure, a video generation request carrying generation conditions and control conditions is obtained through a first application programming interface (API), and then a video generation response carrying the target video is returned through a second API. The generation conditions provide video material for the target video to be generated, the control conditions guide the generation of video content adapted to the generation conditions, the target video's duration is greater than a preset duration, and the target video is generated based on the generation conditions, control conditions, and target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise is used to control the smooth transition between different image frames of the target video. This achieves the goal of generating a stable, reasonable, continuous, and well-transitioned long video, thereby improving the video reasonableness, video continuity, and video transition of the generated long video. This solves the technical problem of poor video reasonableness, video continuity, and video transition in related long video generation methods.

[0189] It should be noted that the second acquisition module 801 and the first return module 802 mentioned above correspond to steps S51 and S52 in Embodiment 2. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware components or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of the device and run in the computer terminal 10 provided in Embodiment 1.

[0190] According to embodiments of this disclosure, another embodiment of an apparatus for implementing the above-described video generation method is also provided. Figure 9 is a schematic structural diagram of another video generation apparatus according to embodiment 4 of this disclosure. As shown in Figure 9, the apparatus includes:

[0191] The third acquisition module 901 is configured to acquire the currently input video to generate a dialogue request;

[0192] The second return module 902 is configured to respond to a video generation dialog request and return a video generation dialog response. The request data carried in the video generation dialog request includes: generation conditions and control conditions. The information carried in the video generation dialog response includes: target video, generation conditions are used to provide video material for the target video to be generated, control conditions are used to guide the generation of video content that matches the generation conditions, the video duration of the target video is longer than a preset duration, the target video is generated based on the generation conditions, control conditions and target noise corresponding to the target video, the target noise is the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise is used to control the smooth transition between different image frames of the target video.

[0193] Display module 903 is configured to display the target video within a graphical user interface.

[0194] By employing the embodiments of this disclosure, a video generation dialog request carrying generation and control conditions is obtained from the current input. Then, in response to the video generation dialog request, a video generation dialog response carrying the target video is returned. The generation conditions provide video material for the target video to be generated, and the control conditions guide the generation of video content adapted to the generation conditions. The target video's duration is greater than a preset duration. The target video is generated based on the generation conditions, control conditions, and target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames contained in the generated target video. The target noise is used to control the smooth transition between different image frames of the target video. After obtaining the target video, it is displayed in the graphical user interface to provide feedback to the user. This achieves the goal of generating a stable, reasonable, continuous, and well-transitioned long video, thereby improving the video reasonableness, video continuity, and video transition of the generated long video. This solves the technical problem of poor video reasonableness, video continuity, and video transition in related long video generation methods.

[0195] It should be noted that the third acquisition module 901, the second return module 902, and the display module 903 mentioned above correspond to steps S61 to S63 in Embodiment 3. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0196] It should be noted that the preferred implementation schemes involved in the above embodiments of this disclosure are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0197] Example 5

[0198] The embodiments of this disclosure can provide a computer terminal (or electronic device), which can be any one of a group of computer terminal devices. Optionally, in this embodiment, the computer terminal can also be replaced by a mobile terminal or other terminal device.

[0199] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0200] In this embodiment, the computer terminal described above can execute the program code for the following steps in the video generation method: obtaining generation conditions and control conditions, wherein the generation conditions are used to provide video material for the target video to be generated, and the control conditions are used to guide the generation of video content that matches the generation conditions, and the video duration of the target video is greater than a preset duration; generating the target video based on the generation conditions, control conditions, and target noise corresponding to the target video, wherein the target noise is the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise is used to control the smooth transition between different image frames of the target video.

[0201] Optionally, FIG10 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. As shown in FIG10, taking computer terminal A as an example, the electronic device may include: one or more (only one is shown in the figure) processors 1002, memory 1004, memory controller, and peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0202] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the video generation method and apparatus in this embodiment. The processor executes the stored software programs and modules to perform various functional applications and data processing, thereby realizing the aforementioned video generation method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to computer terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0203] The processor can invoke information and application programs stored in the memory through the transmission device to perform the following steps: obtaining generation conditions and control conditions, wherein the generation conditions are used to provide video material for the target video to be generated, and the control conditions are used to guide the generation of video content that is compatible with the generation conditions, and the video duration of the target video is longer than a preset duration; generating the target video based on the generation conditions, control conditions, and target noise corresponding to the target video, wherein the target noise is the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise is used to control the smooth transition between different image frames of the target video.

[0204] Optionally, the processor may also execute program code for the following steps: using a target video generation model to perform video generation processing on generation conditions, control conditions, and target noise to obtain a target video, wherein the target video generation model is used to perform stable diffusion based on generation conditions, control conditions, and target noise to obtain multiple target image frames and to perform temporal linking of the multiple target image frames obtained after diffusion to obtain a target video.

[0205] Optionally, the target video generation model includes: a target stable diffusion model, a target control model, and a target temporal linking model. The processor can also execute program code with the following steps: under the guidance of control conditions input through the target control model, the stable diffusion model is used to perform image diffusion processing on the generation conditions and target noise to obtain multiple consecutive image frames, and the temporal linking model is used to perform temporal smoothing on the multiple consecutive image frames to obtain multiple target image frames; the multiple target image frames are then image stitched together to obtain the target video.

[0206] Optionally, the processor may also execute program code for the following steps: in a stable diffusion model, feature encoding is performed on the generation conditions to obtain the encoding results corresponding to the generation conditions; spatial mapping is performed on the encoding results to obtain spatial mapping results corresponding to multiple initial image frames; target noise is added to the spatial mapping results corresponding to multiple initial image frames according to a preset noise optimization step number to obtain latent space features corresponding to multiple initial image frames; image diffusion processing is performed on the latent space features corresponding to multiple initial image frames to obtain image features corresponding to multiple initial image frames; feature decoding is performed on the image features corresponding to multiple initial image frames to obtain multiple consecutive image frames; wherein, the number of multiple initial image frames is the preset number of image frames to be included in the target video.

[0207] Optionally, the size of the noise map corresponding to the target noise is determined based on the image frame size of multiple target image frames, and the content of the noise map corresponding to the target noise is determined based on random numbers within a preset range.

[0208] Optionally, multiple initial image frames are divided into multiple sets of image frame sequences according to a preset number and a preset ratio. The preset number is used to determine the number of image frames contained in each set of image frame sequences, and the preset ratio is used to determine the proportion of image frames that overlap between two adjacent sets of image frame sequences.

[0209] Optionally, the processor may also execute program code for the following steps: adding target noise to the spatial mapping results of adjacent two sets of image frame sequences according to a preset number of noise optimization steps, obtaining latent space features corresponding to adjacent two sets of image frame sequences, and recording the number of noise optimization steps for image frames that overlap between adjacent two sets of image frame sequences; performing feature superposition on the current latent space features corresponding to adjacent two sets of image frame sequences to obtain overlapping regions; obtaining the feature average value within the overlapping regions; obtaining the latent space features corresponding to image frames that overlap between adjacent two sets of image frame sequences based on the feature average value and the number of noise optimization steps, until all noise optimization steps have been executed, and obtaining latent space features corresponding to multiple initial image frames.

[0210] Optionally, the processor may also execute program code that performs the following steps: training the initial video generation model using sample videos, sample conditions, and sample noise to generate the target video generation model.

[0211] Optionally, the initial video generation model includes: an initial stable diffusion model, an initial control model, and an initial temporal linking model. The processor can also execute program code for the following steps: training the initial video generation model using sample videos, sample conditions, and sample noise to obtain the training results to be used; and alternately updating the model parameters of the initial temporal linking model and the model parameters of the initial stable diffusion model based on the training results to be used, thereby generating the target video generation model.

[0212] Optionally, the processor may also execute program code with the following steps: Given that the model parameters of the initial stable diffusion model and the initial control model are both fixed in advance, update the model parameters of the initial temporal linking model based on the training results to be used, to obtain a first update result; given that the model parameters of the initial temporal linking model are fixed in advance, update the model parameters of the initial control model and the initial stable diffusion model based on the first update result, to obtain a second update result; and continuously update the first update result and the second update result alternately according to a preset number of training steps until the target video generation model is generated.

[0213] Optionally, the processor may also execute program code for the following steps: splitting the sample video into multiple sub-videos; sampling the multiple sub-videos according to a preset image frame sampling method to obtain sampling results, wherein the preset image frame sampling method is used to adapt to videos with different change rates; and using an initial video generation model to perform video generation processing on the sampling results, sample conditions, and sample noise to obtain training results to be used.

[0214] By employing the embodiments of this disclosure, generation conditions for providing video material for the target video to be generated and control conditions for guiding the generation of video content adapted to the generation conditions are obtained. Based on the generation conditions, control conditions, and the same initialization noise added to multiple target image frames of the target video to be generated, the target video is generated. This enables the stable generation of target videos with a duration longer than a preset duration and controls the smooth transition of the target video between different image frames, thus generating stable long videos. This achieves the goal of generating stable, reasonable, continuous, and well-transitioned long videos, thereby improving the video reasonableness, video continuity, and video transition of the generated long videos. This solves the technical problem that the long videos generated by related long video generation methods have poor video reasonableness, video continuity, and video transition.

[0215] It will be understood by those skilled in the art that the structure shown in Figure 10 is merely illustrative, and computer terminal A may also be a smartphone (such as an Android phone, iOS phone, etc.), tablet computer, PDA, mobile internet device (MID), PAD, or other terminal device. Figure 10 does not limit the structure of the aforementioned electronic device. For example, computer terminal A may include more or fewer components (such as network interface, display device, etc.) than shown in Figure 10, or have a different configuration than shown in Figure 10.

[0216] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0217] Example 6

[0218] Embodiments of this disclosure also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the video generation method provided in Embodiment 1.

[0219] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0220] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining generation conditions and control conditions, wherein the generation conditions are used to provide video material for the target video to be generated, and the control conditions are used to guide the generation of video content that is compatible with the generation conditions, and the video duration of the target video is greater than a preset duration; generating the target video based on the generation conditions, control conditions, and target noise corresponding to the target video, wherein the target noise is the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise is used to control the smooth transition of the target video between different image frames.

[0221] Embodiments of this disclosure also provide a computer program product comprising a computer program that, when executed by a processor, implements any of the above-described video generation methods.

[0222] Optionally, in this embodiment, when the computer program product is executed by the processor, the program code is used to perform the following steps: obtaining generation conditions and control conditions, wherein the generation conditions are used to provide video material for the target video to be generated, the control conditions are used to guide the generation of video content that is compatible with the generation conditions, and the video duration of the target video is greater than a preset duration; generating the target video based on the generation conditions, control conditions, and target noise corresponding to the target video, wherein the target noise is the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise is used to control the smooth transition between different image frames of the target video.

[0223] The sequence numbers of the embodiments disclosed above are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0224] In the above embodiments of this disclosure, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0225] In the several embodiments provided in this disclosure, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0226] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0227] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0228] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0229] The above description is only a preferred embodiment of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of this disclosure, and these improvements and modifications should also be considered within the scope of protection of this disclosure.

Claims

1. A video generation method, comprising: The generation conditions and control conditions are obtained, wherein the generation conditions are used to provide video materials for the target video to be generated, and the control conditions are used to guide the generation of video content that is compatible with the generation conditions, and the video duration of the target video is greater than a preset duration; The target video is generated based on the generation conditions, the control conditions, and the target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames included in the generation of the target video. The target noise is used to control the smooth transition of the target video between different image frames.

2. The video generation method according to claim 1, wherein, Generating the target video based on the generation conditions, the control conditions, and the target noise corresponding to the target video includes: A target video generation model is used to perform video generation processing on the generation conditions, the control conditions, and the target noise to obtain the target video. The target video generation model is used to perform stable diffusion based on the generation conditions, the control conditions, and the target noise to obtain the plurality of target image frames, and to perform temporal linking on the plurality of target image frames obtained after diffusion to obtain the target video.

3. The video generation method according to claim 2, wherein, The target video generation model includes: a target stable diffusion model, a target control model, and a target temporal linking model. The target video is generated by applying the generation conditions, the control conditions, and the target noise using the target video generation model, resulting in the following target video: Guided by the control conditions input through the target control model, the stable diffusion model is used to perform image diffusion processing on the generation conditions and the target noise to obtain multiple consecutive image frames, and the temporal linking model is used to perform temporal smoothing on the multiple consecutive image frames to obtain the multiple target image frames. The target video is obtained by stitching together the multiple target image frames.

4. The video generation method according to claim 3, wherein, The image diffusion process, which uses the stable diffusion model to perform image diffusion processing on the generation conditions and the target noise, yields the plurality of consecutive image frames, including: In the stable diffusion model, the generation conditions are feature-encoded to obtain the encoding results corresponding to the generation conditions; Spatial mapping is performed on the encoding results to obtain spatial mapping results corresponding to multiple initial image frames; The target noise is added to the spatial mapping result corresponding to the multiple initial image frames according to the preset noise optimization steps to obtain the latent space features corresponding to the multiple initial image frames; Image diffusion processing is performed on the latent space features corresponding to the plurality of initial image frames to obtain the image features corresponding to the plurality of initial image frames; The image features corresponding to the plurality of initial image frames are decoded to obtain the plurality of consecutive image frames; The number of the plurality of initial image frames is the number of image frames to be included in the target video as preset.

5. The video generation method according to claim 4, wherein, The size of the noise map corresponding to the target noise is determined based on the image frame size of the plurality of target image frames, and the content of the noise map corresponding to the target noise is determined based on random numbers within a preset range.

6. The video generation method according to claim 4, wherein, The initial image frames are divided into multiple image frame sequences according to a preset number and a preset ratio. The preset number is used to determine the number of image frames contained in each image frame sequence, and the preset ratio is used to determine the proportion of image frames that overlap between two adjacent image frame sequences.

7. The video generation method according to claim 6, wherein, The target noise is added to the spatial mapping results corresponding to the plurality of initial image frames according to the noise optimization steps, and the latent spatial features corresponding to the plurality of initial image frames are obtained as follows: According to the preset noise optimization steps, the target noise is added to the spatial mapping results corresponding to two adjacent image frame sequences in the multiple image frame sequences to obtain the latent spatial features corresponding to the two adjacent image frame sequences, and the number of noise optimizations corresponding to the image frames that overlap between the two adjacent image frame sequences is recorded. The latent space features corresponding to the two adjacent sets of image frame sequences are superimposed to obtain the overlapping region; The average value of the features is obtained within the overlapping region; Based on the average value of the features and the number of noise optimization steps, the latent space features corresponding to the image frames that overlap between the two adjacent image frame sequences are obtained, until all the noise optimization steps are completed, and the latent space features corresponding to the multiple initial image frames are obtained.

8. The video generation method according to claim 2, wherein, The video generation method further includes: The initial video generation model is trained using sample videos, sample conditions, and sample noise to generate the target video generation model.

9. The video generation method according to claim 8, wherein, The initial video generation model includes: an initial stable diffusion model, an initial control model, and an initial temporal linking model. The initial video generation model is trained using the sample video, the sample conditions, and the sample noise. Generating the target video generation model includes: The initial video generation model is trained using the sample video, the sample conditions, and the sample noise to obtain the training results to be used. Based on the training results to be used, the model parameters of the initial temporal linking model and the model parameters of the initial stable diffusion model are alternately updated to generate the target video generation model.

10. The video generation method according to claim 9, wherein, Based on the training results to be used, the model parameters of the initial temporal linking model and the initial stable diffusion model are alternately updated to generate the target video generation model, including: With the model parameters of the initial stable diffusion model and the initial control model both fixed in advance, the model parameters of the initial temporal linking model are updated based on the training results to be used, to obtain the first update result; With the model parameters of the initial time-series linkage model fixed in advance, the model parameters of the initial control model and the model parameters of the initial stable diffusion model are updated based on the first update result to obtain the second update result; The first update result and the second update result are continuously updated alternately according to a preset number of training steps until the target video generation model is generated.

11. The video generation method according to claim 9, wherein, The initial video generation model is trained using the sample video, the sample conditions, and the sample noise to obtain the training results to be used, including: The sample video is split into multiple sub-videos; The multiple sub-videos are sampled according to a preset image frame sampling method to obtain sampling results. The preset image frame sampling method is used to adapt to videos with different change rates. The initial video generation model is used to process the sampling results, the sample conditions, and the sample noise to generate a video, thereby obtaining the training results to be used.

12. A video generation method, comprising: The video generation request is obtained through the first application programming interface; The video generation response is returned via the second application programming interface; The video generation request carries request data including generation conditions and control conditions. The video generation response carries response data including a target video. The generation conditions are used to provide video material for the target video to be generated. The control conditions are used to guide the generation of video content that matches the generation conditions. The video duration of the target video is greater than a preset duration. The target video is generated based on the generation conditions, the control conditions, and the target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames included in the generation of the target video. The target noise is used to control the smooth transition of the target video between different image frames.

13. A video generation method, comprising: Get the currently input video and generate a dialogue request; In response to the video generation dialogue request, a video generation dialogue reply is returned; The video generation dialogue request carries request data including generation conditions and control conditions. The video generation dialogue response carries information including a target video. The generation conditions are used to provide video material for the target video to be generated. The control conditions are used to guide the generation of video content that matches the generation conditions. The video duration of the target video is longer than a preset duration. The target video is generated based on the generation conditions, the control conditions, and the target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames contained in the target video. The target noise is used to control the smooth transition of the target video between different image frames. The target video is displayed within a graphical user interface.

14. An electronic device comprising: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the video generation method according to any one of claims 1 to 13.

15. A computer-readable storage medium comprising a stored executable program, wherein, When the executable program is executed, it controls the device containing the computer-readable storage medium to perform the video generation method according to any one of claims 1 to 13.

16. A computer program product comprising a computer program that, when executed by a processor, implements the video generation method according to any one of claims 1 to 13.

Images