Video generation method, electronic equipment and computer readable storage medium

Abstract

The invention discloses a video generation method, electronic equipment and a computer readable storage medium, and relates to the technical field of computer technology and video processing. The method comprises the steps that a generation condition and a control condition are acquired, the generation condition is used for providing video materials for a to-be-generated target video, the control condition is used for guiding generation of video content matched with the generation condition, and the video duration of the target video is larger than a preset duration; the target video is generated based on the generation condition, the control condition and target noise corresponding to the target video, the target noise is the same initialization noise added for generating multiple target image frames contained in the target video, and the target noise is used for controlling smooth transition of the target video among different image frames. According to the invention, the technical problems of poor video rationality, video continuity and video transition of a long video generated by a long video generation method in related technologies are solved.

Description

Technical Field

[0001] This application relates to the fields of computer technology and video processing technology, and more specifically, to a video generation method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the continuous development of text-to-image technology, video generation has attracted much attention. Current video generation methods are usually based on diffusion models to generate short videos of corresponding lengths. However, many problems still exist when generating long videos.

[0003] One existing long video generation method fixes the parameters of the basic text-to-image model during training. This can lead to unreasonable motion in a set of generated image frames. Furthermore, when generating multiple sets of image frames, the method may cause background jumps, resulting in poor video continuity and transitions in the generated long videos.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a video generation method, an electronic device, and a computer-readable storage medium to at least solve the technical problems of poor video rationality, video continuity, and video transition in long videos generated by related art long video generation methods.

[0006] According to one aspect of the embodiments of this application, a video generation method is provided, comprising: acquiring generation conditions and control conditions, wherein the generation conditions are used to provide video material for a target video to be generated, and the control conditions are used to guide the generation of video content that is compatible with the generation conditions, and the video duration of the target video is greater than a preset duration; generating a target video based on the generation conditions, the control conditions, and target noise corresponding to the target video, wherein the target noise is the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise is used to control the smooth transition between different image frames of the target video.

[0007] According to another aspect of the embodiments of this application, a video generation method is also provided, comprising: obtaining a video generation request through a first application programming interface; and returning a video generation response through a second application programming interface; wherein the request data carried in the video generation request includes: generation conditions and control conditions, and the response data carried in the video generation response includes: a target video, the generation conditions are used to provide video material for the target video to be generated, the control conditions are used to guide the generation of video content that is compatible with the generation conditions, the video duration of the target video is greater than a preset duration, the target video is generated based on the generation conditions, the control conditions, and the target noise corresponding to the target video, the target noise is the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise is used to control the smooth transition of the target video between different image frames.

[0008] According to another aspect of the embodiments of this application, a video generation method is also provided, comprising: acquiring a currently input video generation dialogue request; responding to the video generation dialogue request and returning a video generation dialogue response; wherein the request data carried in the video generation dialogue request includes: generation conditions and control conditions, and the information carried in the video generation dialogue response includes: a target video, the generation conditions being used to provide video material for the target video to be generated, the control conditions being used to guide the generation of video content adapted to the generation conditions, the video duration of the target video being greater than a preset duration, the target video being generated based on the generation conditions, the control conditions, and the target noise corresponding to the target video, the target noise being the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise being used to control the smooth transition of the target video between different image frames; and displaying the target video in a graphical user interface.

[0009] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes any one of the video generation methods described above when it runs.

[0010] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform any of the above-described video generation methods.

[0011] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements any of the above-described video generation methods.

[0012] In this embodiment, by obtaining generation conditions for providing video material for the target video to be generated and control conditions for guiding the generation of video content adapted to the generation conditions, and generating the target video based on the generation conditions, control conditions, and the same initialization noise added to multiple target image frames of the target video to be generated, the target video can be generated stably with a video duration longer than a preset duration. By controlling the smooth transition of the target video between different image frames, a stable long video can be generated. This achieves the goal of generating a stable, reasonable, continuous, and well-transitioned long video, thereby improving the video reasonableness, video continuity, and video transition of the generated long video. This solves the technical problem that the long video generation methods in related technologies have poor video reasonableness, video continuity, and video transition.

[0013] It is worth noting that the general description above and the detailed description that follow are merely for illustrative purposes and do not constitute a limitation on this application. Attached Figure Description

[0014] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0015] Figure 1 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a video generation method according to Embodiment 1 of this application;

[0016] Figure 2 This is a flowchart of a video generation method according to Embodiment 1 of this application;

[0017] Figure 3 This is a schematic diagram of inference frame grouping according to Embodiment 1 of this application;

[0018] Figure 4 This is a schematic diagram of model training according to Embodiment 1 of this application;

[0019] Figure 5 This is a flowchart of a video generation method according to Embodiment 2 of this application;

[0020] Figure 6 This is a flowchart of a video generation method according to Embodiment 3 of this application;

[0021] Figure 7 This is a schematic diagram of the structure of a video generation device according to Embodiment 4 of this application;

[0022] Figure 8 This is a schematic diagram of another video generation apparatus according to Embodiment 4 of this application;

[0023] Figure 9 A schematic diagram of the structure of another video generation apparatus according to Embodiment 4 of this application;

[0024] Figure 10 A structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0028] A diffusion model is a mathematical model used to describe and predict the propagation and diffusion of matter or information in space. Diffusion models are typically based on diffusion equations, which describe the speed and direction of propagation of diffusing matter or information in space. Diffusion equations usually include parameters such as diffusion coefficients and initial conditions, which can be used to model and predict the diffusion process. In this embodiment, the diffusion model learns the transformation from noise to the image latent space by modeling the diffusion pattern of data in the latent space.

[0029] Stable diffusion is a mathematical model used in statistics and stochastic process theory to describe stochastic processes with long-tailed distributions. A stable diffusion process is a special type of stochastic process characterized by the stability of its distribution under certain conditions, meaning that the distribution of the random variable remains unchanged under scaling and translation transformations. In this application, Stable Diffusion refers to a common textural graph diffusion model.

[0030] animediff: A time series model for Stable Diffusion that can provide some stability in the time dimension.

[0031] ControlNet: A neural network that controls a pre-trained image diffusion model (such as Stable Diffusion). ControlNet allows input of a modulated image, which is then used to manipulate the generated image. There are many types of modulated images, such as doodles, edge maps, pose keypoints, depth maps, segmentation maps, normal maps, etc., all of which can be used as conditional inputs to guide the content of the generated image.

[0032] Stable long video: In this embodiment of the application, stable long video refers to a long video that has been processed and has relatively stable picture quality and smoothness, and is not prone to stuttering or discontinuous picture.

[0033] Latent space, in machine learning and deep learning, refers to the low-dimensional space used to represent data. In this space, each dimension represents a feature or attribute of the data, thus capturing its key characteristics. The concept of latent space is often used to represent the data representation after processing by an encoder, which maps high-dimensional input data into the latent space. In deep learning, latent space is commonly used to represent compressed representations of data, better capturing its essential features and thus enabling more efficient use in dimensionality reduction or feature extraction. Furthermore, latent space can also be used in generative models, such as by using a decoder to map the representation in the latent space back to the original data space, thereby achieving data generation or reconstruction.

[0034] Variational Autoencoder (VAE): A generative model that combines the ideas of autoencoders and variational inference. It encodes input data by learning the latent distribution of the data, and can also sample from the latent space to generate new data. In the embodiments of this application, VAE can map images or videos to the latent space, or decode latent space features into images or videos.

[0035] Currently, there are two types of long video generation methods. One type fixes the parameters of the basic text-to-image model during training, which can lead to unreasonable motion in a set of generated image frames. Furthermore, when generating multiple sets of image frames, this type of method may experience background jumps, resulting in poor video continuity and transitions. The other type of long video generation method integrates different groups of latent space features. This method is limited by GPU memory and requires a longer cross-inference time.

[0036] The long video generation methods of related technologies have the following drawbacks.

[0037] Defect 1: The generated long videos have poor video fidelity, and when generating multiple sets of image frames, there are background jumps, resulting in poor video continuity and transitions.

[0038] Defect 2: Limited by video memory and requiring a long cross-inference time, resulting in a long video generation time.

[0039] To address the aforementioned deficiencies, no effective solution has been proposed prior to this application.

[0040] Example 1

[0041] According to an embodiment of this application, a video generation method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0042] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a video generation method according to Embodiment 1 of this application. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0043] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0044] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the video generation method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned video generation method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0045] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0046] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0047] Under the aforementioned operating environment, this application provides the following: Figure 2 The video generation method shown. Figure 2This is a flowchart of a video generation method according to Embodiment 1 of this application. Figure 2 As shown, the method may include the following steps:

[0048] Step S21: Obtain generation conditions and control conditions. The generation conditions are used to provide video materials for the target video to be generated, and the control conditions are used to guide the generation of video content that matches the generation conditions. The video length of the target video is longer than the preset length.

[0049] Step S22: Generate a target video based on the generation conditions, control conditions, and target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames contained in the generated target video. The target noise is used to control the smooth transition of the target video between different image frames.

[0050] In this embodiment, videos with a duration longer than a preset duration are defined as long videos, and correspondingly, videos with a duration less than or equal to the preset duration are defined as short videos. The preset duration can be set according to actual circumstances and is not limited here.

[0051] The target video to be generated has a duration longer than the preset duration, which can be understood as the target video to be generated being a long video. That is, the video generation method provided in this application embodiment is used to generate long videos.

[0052] The generation conditions are used to provide video materials for the target video to be generated, that is, to determine the video content to be displayed in the target video. Generation conditions can be in the form of text, images, audio, etc., and are not limited here. For example, generation conditions can be prompt text entered by the user through an input method, a specific reference image uploaded by the user, or an audio file described by the user through voice, etc., and are not limited here.

[0053] Control conditions are used to guide the generation of video content that matches the generation conditions. In other words, they guide the generation of video content that matches the video content determined by the generation conditions. This can be understood as the control conditions manipulating the video content of the target video to be generated based on the provided video materials. Control conditions can be determined by selecting an appropriate control model; no restrictions are imposed here.

[0054] Target noise is the same initialization noise added to multiple target image frames contained in the generated target video. Target noise is used to control the smooth transition between different image frames in the target video.

[0055] It is understandable that the target video to be generated includes multiple target image frames. When generating the target video, the final video is often obtained by gradually optimizing from noise. In the actual reasoning process of this application, before each group of image frames begins optimization, i.e., during the initialization phase, the same noise is used to optimize each group of image frames. That is, the same initialization noise is added to multiple target image frames contained in the target video, thereby ensuring that the starting point of each group of image frames is consistent during the optimization process. This allows for the stable generation of multiple consecutive groups of images, avoiding jumps between groups, and thus controlling the smooth transition between different image frames in the generated target video, ensuring the reasonableness of the generated target video.

[0056] In this embodiment, by obtaining generation conditions for providing video material for the target video to be generated and control conditions for guiding the generation of video content that matches the generation conditions, and generating the target video based on the generation conditions, control conditions, and the same initialization noise added to multiple target image frames of the target video to be generated, it is possible to stably generate target videos with a video duration longer than a preset duration, and control the smooth transition of the target video between different image frames, that is, to generate stable long videos.

[0057] As can be seen, this application uses the same initialization noise to optimize each group of image frames during inference, that is, it uses the same noise for initialization processing, which can avoid video jumps between different groups of image frames, ensure that the generated long video is more stable and the video content is more reasonable, and improve the video continuity and video transition of the generated long video.

[0058] The video generation method provided in this application embodiment can be applied, but is not limited to, to video generation application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, it can generate e-commerce related videos, education related videos, and legal related videos, etc., and is not limited here.

[0059] By employing the embodiments of this application, generation conditions for providing video material for the target video to be generated and control conditions for guiding the generation of video content adapted to the generation conditions are obtained. Based on the generation conditions, control conditions, and the same initialization noise added to multiple target image frames of the target video to be generated, the target video is generated. This enables the stable generation of target videos with a duration longer than a preset duration and controls the smooth transition of the target video between different image frames, thus generating stable long videos. This achieves the goal of generating stable, reasonable, continuous, and well-transitioned long videos, thereby improving the video reasonableness, video continuity, and video transition of the generated long videos. This solves the technical problem that the long videos generated by related long video generation methods have poor video reasonableness, video continuity, and video transition.

[0060] In an optional embodiment, step S22 involves generating a target video based on generation conditions, control conditions, and target noise corresponding to the target video, including the following method steps:

[0061] Step S221: The target video generation model is used to process the generation conditions, control conditions and target noise to generate a target video. The target video generation model is used to perform stable diffusion based on the generation conditions, control conditions and target noise to obtain multiple target image frames and to perform temporal linking of the multiple target image frames obtained after diffusion to obtain the target video.

[0062] It is understandable that a stable diffusion model refers to a model in which the diffusion process in time and space can reach a stable state. That is, the diffusion rate and diffusion range of a stable diffusion model remain relatively stable within a certain range and will not change drastically.

[0063] Temporal linking refers to a connection or association between two temporally adjacent events or states. This connection can be causal, sequential, or simultaneous, etc. Temporal linking allows events or states to be connected and organized in chronological order. In this embodiment, temporal linking of multiple target image frames obtained after diffusion can be understood as stitching the multiple target image frames together in chronological order to obtain the target video.

[0064] A temporal linkage model is a mathematical model used to describe and analyze the temporal relationships between events or states. It can be used to establish the chronological order of events or states and the correlations between them. Temporal linkage models can be used to predict future event trends, analyze causal relationships between events, or model and predict time series data.

[0065] In this embodiment, the target video generation model can perform stable diffusion based on generation conditions, control conditions, and target noise to obtain multiple target image frames. That is, the target video generation model in this embodiment has the function of a stable diffusion model.

[0066] It is understood that a video can be viewed as being composed of multiple sets of consecutive images stitched together. The target video generation model can stitch together the generated images and perform temporal smoothing on multiple images. That is, the target video generation model can temporally link multiple target image frames to obtain the target video. In other words, the target video generation model in this application embodiment has the function of a temporal linking model, thereby accurately capturing the temporal dependency between multiple target image frames obtained after diffusion and accurately stitching the multiple target image frames obtained after diffusion into the target video.

[0067] In this embodiment of the application, when generating a target video based on generation conditions, control conditions, and target noise corresponding to the target video, a target video generation model can be used to stably diffuse the generation conditions, control conditions, and target noise to obtain multiple target image frames. Then, the target video generation model is used to temporally link the multiple target image frames obtained after diffusion to obtain the target video.

[0068] In an optional embodiment, the target video generation model includes: a target stable diffusion model, a target control model, and a target temporal linking model. In step S221, the target video generation model is used to process the generation conditions, control conditions, and target noise to obtain the target video, including the following method steps:

[0069] Step S2211: Under the guidance of the control conditions input by the target control model, a stable diffusion model is used to perform image diffusion processing on the generation conditions and target noise to obtain multiple consecutive image frames, and a temporal linking model is used to perform temporal smoothing on the multiple consecutive image frames to obtain multiple target image frames.

[0070] Step S2212: Perform image stitching processing on multiple target image frames to obtain the target video.

[0071] In this embodiment, the target video generation model includes: a target stable diffusion model, a target control model, and a target temporal linking model. The target stable diffusion model can employ a basic stable diffusion model, which is not limited here, to ensure stable diffusion in both time and space. For example, the target stable diffusion model can use the stable diffusion structure commonly used in text-based image models, meaning that images in the training data domain can be generated through multi-step optimization.

[0072] The target control model can be a robust controller, model predictive control (MPC), or similar model that helps the system better control and regulate its behavior, thereby achieving a more stable, efficient, and reliable operating state. For example, the target control model can employ the ControlNet neural network, i.e., the ControlNet model, without limitation. Through ControlNet, the generation process of images in the target video can be better controlled, resulting in more specific and desirable images.

[0073] Temporal linking models can be models such as the animediff model, Long Short-Term Memory (LSTM) network, and Gated Recurrent Unit (GRU) used to capture temporal dependencies and sequence patterns in data. For example, the animediff model can be chosen as the temporal linking model; this is not a limitation. The animediff model is used to extend the text graph model to the animation generator without requiring adjustments for a specific model.

[0074] In this embodiment of the application, when the target video is obtained by using the target video generation model to process the generation conditions, control conditions and target noise, a stable diffusion model (i.e., the basic stable diffusion model) can be used to process the generation conditions and target noise under the guidance of the control conditions input by the target control model (e.g., the ControlNet model), thereby obtaining multiple consecutive image frames, that is, obtaining multiple consecutive image frames corresponding to each group of images.

[0075] Simultaneously, a temporal linking model (such as the Animediff model) is used to perform temporal smoothing on the obtained multiple consecutive image frames to obtain multiple target image frames. That is, the temporal linking model is used to perform temporal smoothing on the features of multiple consecutive image frames corresponding to each group of images, and a smooth transition is performed between multiple consecutive image frames to obtain multiple target image frames corresponding to each group of images, which is to obtain the generated images of each group.

[0076] Finally, multiple target image frames are stitched together to obtain the target video. This involves stitching together the generated images and then using the same initial noise to smooth the transition between different groups of images, thus obtaining the target video.

[0077] In an optional embodiment, in step S2211, a stable diffusion model is used to perform image diffusion processing on the generation conditions and target noise to obtain multiple consecutive image frames, including the following method steps:

[0078] Step S22111: In the stable diffusion model, feature encoding is performed on the generation conditions to obtain the encoding results corresponding to the generation conditions;

[0079] Step S22112: Spatial mapping is performed on the encoding results to obtain spatial mapping results corresponding to multiple initial image frames;

[0080] Step S22113: Add the target noise to the spatial mapping results corresponding to multiple initial image frames according to the preset noise optimization steps to obtain the latent space features corresponding to multiple initial image frames.

[0081] Step S22114: Perform image diffusion processing on the latent space features corresponding to multiple initial image frames to obtain image features corresponding to multiple initial image frames;

[0082] Step S22115: Decode the image features corresponding to multiple initial image frames to obtain multiple consecutive image frames;

[0083] The number of initial image frames is the pre-set number of image frames to be included in the target video.

[0084] In this embodiment, the stable diffusion model includes a Variational Autoencoder (VAE) and a denoising diffusion model. The VAE is a deep learning model used for model generation and feature learning. The VAE maps input data to a latent space through an encoder, and then generates new data from the latent space through a decoder. The denoising diffusion model is a network used to learn the process of diffusing features from noise into the image's latent space.

[0085] In this embodiment, the stable diffusion model can not only be used to extract spatial or temporal features of video frames for video analysis, identification, or prediction, but it can also process time-series data to extract key temporal information. That is, the stable diffusion model can be used to focus on and integrate encoded features temporally, enabling more effective modeling and analysis of time-series data.

[0086] The initial image frame can be understood as the initial image frame before the target image frame of the target video is obtained through video generation processing, or it can be understood as the inference frame in the actual inference process.

[0087] The number of initial image frames is the pre-set number of image frames to be included in the target video, that is, the pre-set number of video frames to be generated, which is not limited here. For example, if you want to generate a 128-frame video, then the number of initial image frames is 128, that is, the number of inference frames is 128.

[0088] The preset noise optimization steps can be understood as the number or frequency of times noise is introduced during each parameter update in the model training process. For example, the preset noise optimization steps can be adjusted according to the task and the model, and there are no restrictions here.

[0089] In this embodiment of the application, when using a stable diffusion model to perform image diffusion processing on the generation conditions and target noise to obtain image features corresponding to multiple initial image frames, a stable diffusion model can be used to encode the generation conditions to obtain the encoding results corresponding to the generation conditions.

[0090] After obtaining the encoding results corresponding to the generation conditions, a stable diffusion model is used to spatially map the encoding results to obtain spatial mapping results corresponding to multiple initial image frames. This can be understood as mapping the encoding results to the latent space through a VAE, thereby obtaining spatial mapping results corresponding to multiple initial image frames, that is, obtaining spatial mapping results corresponding to all inference image frames during the inference process.

[0091] After obtaining the spatial mapping results corresponding to multiple initial image frames, target noise is added to the spatial mapping results of the multiple initial image frames according to a preset noise optimization step number to obtain the latent spatial features corresponding to the multiple initial image frames. This can be understood as adding the same initial noise to the spatial mapping results of the multiple initial image frames according to a preset noise optimization step number to learn the noise optimization process, that is, learning how to recover the original image from the noisy image, thereby obtaining the latent spatial features corresponding to the multiple initial image frames.

[0092] After obtaining the latent space features corresponding to multiple initial image frames, image diffusion processing is performed on these latent space features to obtain the image features corresponding to the multiple initial image frames. Then, feature decoding is performed on the image features corresponding to the multiple initial image frames to obtain multiple consecutive image frames, i.e., multiple consecutive image frames corresponding to each group of images.

[0093] In one optional embodiment, the size of the noise map corresponding to the target noise is determined based on the image frame size of multiple target image frames, and the content of the noise map corresponding to the target noise is determined based on a random number within a preset range.

[0094] In this embodiment, the size of the noise map corresponding to the target noise is determined based on the image frame size of multiple target image frames, that is, by the size of the target video to be generated. For example, if each frame of the target video is 512*512 RGB (Red, Green, Blue) three-channel, the size converted to the latent space is 64*64*4, then the corresponding target noise will also be 64*64*4, and the noise size is determined by the size of the target video to be generated.

[0095] It is understandable that, regardless of whether the generation conditions are in text, image, or audio format, the size of the noise map corresponding to the target noise is determined by the size of the target video to be generated.

[0096] Understandably, target noise needs to be randomly generated before each video is generated, and the content of the noise map corresponding to the target noise is determined based on random numbers within a preset range. For example, if the target noise is 64*64*4 in size, the random numbers within the preset range are random numbers in the range of -1 to 1, that is, random numbers in the range of -1 to 1 are used to fill the 64*64*4 size, which is not limited here.

[0097] In one optional embodiment, multiple initial image frames are divided into multiple sets of image frame sequences according to a preset number and a preset ratio. The preset number determines the number of image frames contained in each set of image frame sequences, and the preset ratio determines the proportion of overlapping image frames between adjacent sets of image frame sequences. Each set of image frame sequences can use the same initialization noise to utilize target noise to control the smooth transition of the target video between different sets of image frame sequences.

[0098] In this embodiment of the application, during the actual inference process, all inference frames can be grouped according to a preset number, and adjacent groups overlap according to a preset ratio. For example, if the preset number is 24 frames and the preset ratio is 1 / 4, then when grouping all inference frames, the first group of inference frames includes images from frame 1 to frame 24, the second group of inference frames includes images from frame 17 to frame 40, the third group of inference frames includes images from frame 33 to frame 56, and so on, and subsequent groups can be inferred in the same way, which will not be elaborated here.

[0099] In an optional embodiment, in step S22112, the target noise is added to the spatial mapping results corresponding to multiple initial image frames according to the noise optimization steps to obtain the latent space features corresponding to multiple initial image frames, including the following method steps:

[0100] Step S221121: Add the target noise to the spatial mapping results of two adjacent image frame sequences in multiple image frame sequences according to the preset noise optimization steps, obtain the latent space features corresponding to the two adjacent image frame sequences, and record the number of noise optimizations for the image frames that overlap between the two adjacent image frame sequences.

[0101] Step S221122: Overlay the latent space features corresponding to two adjacent sets of image frame sequences to obtain the overlapping region;

[0102] Step S221123: Obtain the average value of features within the overlapping region;

[0103] Step S221124: Based on the feature average value and the number of noise optimization steps, obtain the latent space features corresponding to the image frames that overlap between two adjacent image frame sequences, until all noise optimization steps are completed, and obtain the latent space features corresponding to multiple initial image frames.

[0104] In this embodiment, when adding target noise to the spatial mapping results corresponding to multiple initial image frames according to the noise optimization steps to obtain latent space features corresponding to multiple initial image frames, the target noise can be added to the spatial mapping results corresponding to two adjacent image frame sequences in multiple image frame sequences according to a preset noise optimization step number, thereby obtaining latent space features corresponding to two adjacent image frame sequences. It is understood that, since this application groups multiple initial image frames, and there is overlap between adjacent groups, when the latent space features are continuously noise optimized, a certain overlapping area will appear each time noise optimization occurs. Therefore, it is also necessary to record the number of noise optimizations corresponding to the image frames that overlap between adjacent image frame sequences.

[0105] After obtaining the latent space features corresponding to two adjacent sets of image frame sequences, the latent space features of the two adjacent sets of image frame sequences are superimposed to obtain overlapping regions. Then, the feature average value is obtained within the overlapping region. Based on the feature average value and the number of noise optimization steps, the latent space features corresponding to the image frames that overlap between the two adjacent sets of image frame sequences are obtained. This process continues until all noise optimization steps are completed, resulting in latent space features corresponding to multiple initial image frames. For example, multiple initial image frames are divided into two groups: the first group of inference frames includes frames 1-24, and the second group of inference frames includes frames 17-40. Since frames 17-24 are optimized twice during the frame optimization process for both groups, the features of the two groups need to be superimposed after each frame optimization and then divided by the number of noise optimization steps. That is, the latent space features corresponding to frames 17-24 are averaged after each optimization. This ensures the stable and continuous generation of multiple sets of images, without jumps between groups, making the final generated long video more reasonable.

[0106] Figure 3 This is a schematic diagram of inference frame grouping according to Embodiment 1 of this application, as follows: Figure 3 As shown, the overlap between adjacent image frame sequences in groups M-1, M, and M+1 is illustrated. It can be seen that the M-1 image frame sequence contains two overlapping regions that coincide with both the preceding (M-1) and following (M+1) image frame sequences. Furthermore, each image frame sequence is repeatedly optimized during the continuous optimization process, meaning there are overlapping optimized regions.

[0107] In an optional embodiment, the video generation method further includes the following method steps:

[0108] Step S222: Train the initial video generation model using sample videos, sample conditions, and sample noise to generate the target video generation model.

[0109] In this embodiment of the application, the initial video generation model can be understood as the video generation model before training is completed, that is, the video generation model before the target video generation model is trained.

[0110] In this embodiment of the application, when training the target video generation model, sample videos, sample conditions, and sample noise can be used to train the initial video generation model, thereby generating the target video generation model. Here, sample videos can be understood as sample videos providing video material content during model training, and sample noise can be understood as the initial noise determined during model training.

[0111] Sample conditions can be understood as control conditions during model training, used to guide the generation of corresponding video content.

[0112] In an optional embodiment, the initial video generation model includes: an initial stable diffusion model, an initial control model, and an initial temporal linking model. In step S222, the initial video generation model is trained using sample videos, sample conditions, and sample noise to generate the target video generation model, including the following method steps:

[0113] Step S2221: Train the initial video generation model using sample videos, sample conditions, and sample noise to obtain the training results to be used.

[0114] Step S2222: Based on the training results to be used, the model parameters of the initial temporal linking model and the model parameters of the initial stable diffusion model are alternately updated to generate the target video generation model.

[0115] In this embodiment of the application, the initial video generation model includes: an initial stable diffusion model, an initial control model, and an initial temporal linking model. For example, the initial stable diffusion model can use the stable diffusion structure commonly used in text-based graph models, the initial control model can adopt the ControlNet model, and the initial temporal linking model can be animatediff model. There are no restrictions here.

[0116] In this embodiment of the application, when training the initial video generation model using sample videos, sample conditions, and sample noise to generate the target video generation model, the initial video generation model can be trained using sample videos, sample conditions, and sample noise to obtain the training results to be used. Then, based on the training results to be used, the model parameters of the initial temporal linking model and the model parameters of the initial stable diffusion model are updated alternately, that is, the initial temporal linking model and the initial stable diffusion model are trained alternately to generate the target video generation model.

[0117] In an optional embodiment, in step S2221, the initial video generation model is trained using sample videos, sample conditions, and sample noise to obtain the training results to be used, including the following method steps:

[0118] Step S2221: The sample video is split into multiple sub-videos;

[0119] Step S2222: Perform image frame sampling on multiple sub-videos according to a preset image frame sampling method to obtain sampling results. The preset image frame sampling method is used to adapt to videos with different change rates.

[0120] Step S2223: The initial video generation model is used to process the sampling results, sample conditions and sample noise to generate video, and the training results to be used are obtained.

[0121] In this embodiment, when training the initial video generation model using sample videos, sample conditions, and sample noise to obtain the training results to be used, the sample videos can be split into multiple sub-videos. This can be understood as splitting a long sample video into multiple short videos.

[0122] After obtaining multiple sub-videos, image frames are sampled from the multiple sub-videos according to a preset image frame sampling method to obtain the sampling results. In this embodiment, the preset image frame sampling method can be a random interval frame method to adapt to videos with different change rates. That is, the number of random intervals can be set based on the different change rates of the videos to be learned, and there is no limitation here. For example, the maximum number of intervals is usually set to 4 frames, and each time data is sampled, an integer value between 0 and 4 is randomly selected as the interval value, that is, the interval number is randomly selected between 0, 1, 2, 3, and 4.

[0123] After obtaining multiple short videos, image frames can be sampled from the split short videos at random intervals to obtain sampling results. Then, an initial video generation model is used to process the sampling results, sample conditions, and sample noise to generate videos, thereby obtaining the training results to be used.

[0124] For example, during the initial video generation model training process, the sample video can be split into short videos of approximately 200 frames each. Each training image frame sequence consists of approximately N frames, where N is a positive integer and can be 24 frames or higher, depending on the available video memory. These N frames are then sampled from the original short video at random intervals, allowing the initial video generation model to learn videos with different rates of change and obtain sampling results. The initial video generation model is then used to process the sampling results, sample conditions, and sample noise to generate the final training result.

[0125] Understandably, during training, noise of randomly initialized size N frames of latent space features can be used. Typically, the length and width of the latent space features will become 1 / 4 of the original image after VAE mapping. Then, the initialized noise is added to the latent space features of the training image sequence according to the number of noise optimization steps, thereby enabling the model to learn the noise optimization process.

[0126] In an optional embodiment, in step S2222, the model parameters of the initial temporal linking model and the initial stable diffusion model are alternately updated based on the training results to be used to generate the target video generation model, including the following method steps:

[0127] Step S22221: Under the premise that the model parameters of the initial stable diffusion model and the model parameters of the initial control model are fixed in advance, the model parameters of the initial temporal linking model are updated based on the training results to be used, and the first update result is obtained.

[0128] Step S22222: Under the premise that the model parameters of the initial time-series linked model are fixed in advance, the model parameters of the initial control model and the model parameters of the initial stable diffusion model are updated based on the first update result to obtain the second update result;

[0129] Step S22223: The first update result and the second update result are continuously updated alternately according to the preset number of training steps until the target video generation model is generated.

[0130] The preset number of training steps can be determined based on the actual training situation; there is no limit here.

[0131] In this embodiment, when adjusting the model parameters of the initial temporal linking model and the initial stable diffusion model alternately based on the training results to be used to generate the target video generation model, the model parameters of the initial temporal linking model can be updated based on the training results to be used, provided that the model parameters of the initial stable diffusion model and the initial control model are both fixed in advance, to obtain the first update result. This can be understood as follows: during the training of the target video generation model, the model parameters of the initial stable diffusion model and the initial control model are first fixed, only the initial temporal linking model is trained, and the model parameters of the initial temporal linking model are updated to obtain the first update result.

[0132] After training and updating the initial temporal linked model, and with the model parameters of the initial temporal linked model fixed in advance, the model parameters of the initial control model and the initial stable diffusion model are updated based on the first update result to obtain the second update result. This can be understood as follows: Given that the initial temporal linked model training is basically complete, the model parameters of the initial temporal linked model are fixed, the initial control model and the initial stable diffusion model are trained, and their model parameters are updated to obtain the second update result.

[0133] Then, following a preset number of training steps, the first and second update results are continuously and alternately updated. This involves alternately training the initial control model, the initial temporal link model, and the initial stable diffusion model until the initial control model, initial temporal link model, and initial stable diffusion model with multiple parameter updates are obtained, which in turn yield the target control model, target temporal link model, and target stable diffusion model. Finally, the target video generation model is generated based on the target temporal link model, target stable diffusion model, and target control model.

[0134] For example, during the initial training of the video generation model, the parameters of the original StableDiffusion model and ControlNet model are initially fixed, and the parameters of the Animatediff model are trained and updated separately. Then, when the Animatediff model is basically complete in training, for example, after more than 50,000 steps using 8 GPUs, the Animatediff model is fixed, and the parameters of the Stable Diffusion model and ControlNet model are trained and updated, and this process is repeated. During the loop, it can be set to alternate training and parameter updates every X training steps, for example, every 10,000 training steps, ultimately resulting in the Stable Diffusion model and the Animatediff model with updated parameters.

[0135] Figure 4This is a schematic diagram of model training according to Embodiment 1 of this application, as shown below. Figure 4 As shown, the target video generation model in this application includes a base model (i.e., a stable diffusion model), a temporal model (i.e., a temporal linking model), and a control model. During the training of the target video generation model, the model parameters of the base model and the control model are first fixed, and the temporal model is trained. After the temporal model is basically trained, its model parameters are fixed again, and the base model and the control model are trained. After the base model and the control model are basically trained, their model parameters are fixed again, and the previously trained temporal model is trained. This process is repeated, alternating between the two models.

[0136] As can be seen, this application provides a stable long video generation method based on a diffusion model. This method uses short videos for training and, by modifying the original stable diffusion model, can generate stable long videos. During training, the video is split into short videos, each containing approximately 200 frames. Then, samples are taken from the original short videos at random intervals, allowing the model to learn videos with different rates of change. In the actual long video generation process, control conditions such as skeleton points and depth maps can be used to manipulate the generation. Furthermore, noise synchronization initialization processing is required during inference to ensure the continuity and transition of the generated long videos, offering advantages such as small memory usage, short processing time, and good continuity.

[0137] It is easy to understand that the beneficial effects of the video generation method provided in this application include the following points.

[0138] Beneficial effect (1): By sampling the original short video at random intervals, it is possible to learn videos with different rates of change.

[0139] Beneficial effect (2): During the training process, the basic model and the time series model will be trained alternately, which can make the two better integrated.

[0140] Beneficial effect (3): The use of synchronous initialization noise during inference can reduce video jumps between different groups, making long videos more stable and improving video rationality, video continuity and video transition.

[0141] Beneficial effects (4): It occupies less memory and has a shorter inference time.

[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0143] Furthermore, it should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0144] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0145] Example 2

[0146] In the operating environment as described in Example 1, this application provides as follows: Figure 5 A video generation method is shown. Figure 5 This is a flowchart of a video generation method according to Embodiment 2 of this application, as follows: Figure 5 As shown, the method includes:

[0147] Step S51: Obtain a video generation request through the first application programming interface;

[0148] Step S52: Return a video generation response through the second application programming interface; wherein, the request data carried in the video generation request includes: generation conditions and control conditions, and the response data carried in the video generation response includes: target video, the generation conditions are used to provide video material for the target video to be generated, the control conditions are used to guide the generation of video content that matches the generation conditions, the video duration of the target video is greater than the preset duration, the target video is generated based on the generation conditions, the control conditions and the target noise corresponding to the target video, the target noise is the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise is used to control the smooth transition between different image frames of the target video.

[0149] The first and second application programming interfaces (APIs) mentioned above can be the same or different APIs. In one optional embodiment, the interface parameters in the first and second APIs may include, but are not limited to: a global interface identifier, an interface signing key, an interface timestamp, an interface request identifier, and a system call credential identifier. The first API can use GET or POST as the interface request method to obtain the file processing request. The second API can use JSON format to return the file processing response.

[0150] In this embodiment, a video generation request can be understood as a request to generate a video, and the request data carried in the video generation request includes: generation conditions and control conditions. A video generation response can be understood as a response to the video generation request, and the response data carried in the video generation response includes: the target video.

[0151] For a detailed description, please refer to the description in Example 1, which will not be repeated here.

[0152] In this embodiment, a video generation request carrying generation and control conditions is obtained through a first application programming interface (API), and then a video generation response carrying the target video is returned through a second API. The generation conditions are used to provide video material for the target video to be generated, and the control conditions are used to guide the generation of video content that matches the generation conditions. The video duration of the target video is longer than a preset duration. The target video is generated based on the generation conditions, control conditions, and target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames contained in the generated target video. The target noise is used to control the smooth transition between different image frames of the target video.

[0153] As can be seen, this application uses the same initialization noise to optimize each group of image frames during inference, that is, it uses the same noise for initialization processing, which can avoid video jumps between different groups of image frames, ensure that the generated long video is more stable and the video content is more reasonable, and improve the video continuity and video transition of the generated long video.

[0154] The video generation method provided in this application embodiment can be applied, but is not limited to, to video generation application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, it can generate e-commerce related videos, education related videos, and legal related videos, etc., and is not limited here.

[0155] In this embodiment, a video generation request carrying generation and control conditions is obtained through a first application programming interface (API), and then a video generation response carrying the target video is returned through a second API. The generation conditions provide video material for the target video to be generated, and the control conditions guide the generation of video content that matches the generation conditions. The target video's duration is longer than a preset duration. The target video is generated based on the generation conditions, control conditions, and target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames included in the generated target video. The target noise is used to control the smooth transition between different image frames in the target video, thereby achieving the goal of generating a stable, reasonable, continuous, and well-transitioned long video. This improves the video reasonableness, continuity, and transition of the generated long video, thus solving the technical problem of poor video reasonableness, continuity, and transition in related long video generation methods.

[0156] It should be noted that the preferred implementation of this embodiment can be found in the relevant description in Embodiment 1, and will not be repeated here.

[0157] Example 3

[0158] In the operating environment as described in Example 1, this application provides as follows: Figure 6 This illustrates a video generation method.

[0159] Figure 6 This is a flowchart of a video generation method according to Embodiment 3 of this application, as follows: Figure 6 As shown, the method includes:

[0160] Step S61: Obtain the currently input video to generate a dialogue request;

[0161] Step S62: In response to the video generation dialogue request, return a video generation dialogue reply; wherein, the request data carried in the video generation dialogue request includes: generation conditions and control conditions, and the information carried in the video generation dialogue reply includes: target video, the generation conditions are used to provide video material for the target video to be generated, the control conditions are used to guide the generation of video content that matches the generation conditions, the video duration of the target video is greater than the preset duration, the target video is generated based on the generation conditions, the control conditions and the target noise corresponding to the target video, the target noise is the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise is used to control the smooth transition between different image frames of the target video;

[0162] Step S63: Display the target video within the graphical user interface.

[0163] In this embodiment, the video generation dialogue request can be understood as a video generation dialogue request initiated by a user when conversing with a smart device, requesting the generation of a video. The request data carried in the video generation dialogue request includes: generation conditions and control conditions. The video generation dialogue response can be understood as the response made by the smart device in response to the video generation dialogue request initiated by the user. The information carried in the video generation dialogue response includes: the target video.

[0164] For a detailed description, please refer to the description in Example 1, which will not be repeated here.

[0165] In this embodiment, a video generation dialog request carrying generation and control conditions is obtained from the current input. Then, in response to the video generation dialog request, a video generation dialog response carrying the target video is returned. The generation conditions are used to provide video materials for the target video to be generated, and the control conditions are used to guide the generation of video content that matches the generation conditions. The video duration of the target video is longer than a preset duration. The target video is generated based on the generation conditions, control conditions, and target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames contained in the generated target video. The target noise is used to control the smooth transition between different image frames of the target video. After obtaining the target video, the target video is displayed in the graphical user interface to provide feedback to the user.

[0166] As can be seen, this application uses the same initialization noise to optimize each group of image frames during inference, that is, it uses the same noise for initialization processing, which can avoid video jumps between different groups of image frames, ensure that the generated long video is more stable and the video content is more reasonable, and improve the video continuity and video transition of the generated long video.

[0167] The video generation method provided in this application embodiment can be applied, but is not limited to, to video generation application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, it can generate e-commerce related videos, education related videos, and legal related videos, etc., and is not limited here.

[0168] By employing the embodiments of this application, a video generation dialog request carrying generation and control conditions is obtained from the current input. Then, in response to the video generation dialog request, a video generation dialog response carrying the target video is returned. The generation conditions provide video material for the target video to be generated, and the control conditions guide the generation of video content that matches the generation conditions. The target video's duration is longer than a preset duration. The target video is generated based on the generation conditions, control conditions, and target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames contained in the generated target video. The target noise is used to control the smooth transition between different image frames in the target video. After obtaining the target video, it is displayed in the graphical user interface to provide feedback to the user. This achieves the goal of generating a stable, reasonable, continuous, and well-transitioned long video, thereby improving the video reasonableness, video continuity, and video transition of the generated long video. This solves the technical problem of poor video reasonableness, video continuity, and video transition in related long video generation methods.

[0169] It should be noted that the preferred implementation of this embodiment can be found in the relevant description in Embodiment 1, and will not be repeated here.

[0170] Example 4

[0171] According to an embodiment of this application, an apparatus embodiment for implementing the above-described video generation method is also provided. Figure 7 This is a schematic diagram of the structure of a video generation device according to Embodiment 4 of this application, as shown below. Figure 7 As shown, the device includes:

[0172] The first acquisition module 701 is used to acquire generation conditions and control conditions. The generation conditions are used to provide video materials for the target video to be generated, and the control conditions are used to guide the generation of video content that matches the generation conditions. The video duration of the target video is longer than the preset duration.

[0173] The generation module 702 is used to generate a target video based on generation conditions, control conditions, and target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames contained in the generated target video. The target noise is used to control the smooth transition of the target video between different image frames.

[0174] Optionally, the first acquisition module 701 is further configured to: use a target video generation model to perform video generation processing on the generation conditions, control conditions and target noise to obtain a target video, wherein the target video generation model is used to perform stable diffusion based on the generation conditions, control conditions and target noise to obtain multiple target image frames and to perform temporal linking on the multiple target image frames obtained after diffusion to obtain a target video.

[0175] Optionally, the target video generation model includes: a target stable diffusion model, a target control model, and a target temporal linking model. The first acquisition module 701 is further configured to: under the guidance of control conditions input through the target control model, use the stable diffusion model to perform image diffusion processing on the generation conditions and target noise to obtain multiple consecutive image frames, and use the temporal linking model to perform temporal smoothing on the multiple consecutive image frames to obtain multiple target image frames; and perform image stitching processing on the multiple target image frames to obtain the target video.

[0176] Optionally, the first acquisition module 701 is further configured to: in a stable diffusion model, perform feature encoding on the generation conditions to obtain the encoding result corresponding to the generation conditions; perform spatial mapping on the encoding result to obtain spatial mapping results corresponding to multiple initial image frames; add target noise to the spatial mapping results corresponding to multiple initial image frames according to a preset noise optimization step number to obtain latent space features corresponding to multiple initial image frames; perform image diffusion processing on the latent space features corresponding to multiple initial image frames to obtain image features corresponding to multiple initial image frames; and perform feature decoding on the image features corresponding to multiple initial image frames to obtain multiple consecutive image frames, wherein the number of multiple initial image frames is the preset number of image frames to be included in the target video.

[0177] Optionally, the size of the noise map corresponding to the target noise is determined based on the image frame size of multiple target image frames, and the content of the noise map corresponding to the target noise is determined based on random numbers within a preset range.

[0178] Optionally, multiple initial image frames are divided into multiple sets of image frame sequences according to a preset number and a preset ratio. The preset number is used to determine the number of image frames contained in each set of image frame sequences, and the preset ratio is used to determine the proportion of image frames that overlap between two adjacent sets of image frame sequences.

[0179] Optionally, the first acquisition module 701 is further configured to: add target noise to the spatial mapping results corresponding to two adjacent image frame sequences in multiple image frame sequences according to a preset noise optimization step number, obtain the latent space features corresponding to the two adjacent image frame sequences, and record the noise optimization number corresponding to the image frames that overlap between the two adjacent image frame sequences; perform feature superposition on the current latent space features corresponding to the two adjacent image frame sequences to obtain the overlapping region; obtain the feature average value in the overlapping region; obtain the latent space features corresponding to the image frames that overlap between the two adjacent image frame sequences based on the feature average value and the noise optimization number, until all noise optimization steps are completed, and obtain the latent space features corresponding to multiple initial image frames.

[0180] Optionally, it also includes: a training module for: training the initial video generation model using sample videos, sample conditions, and sample noise to generate the target video generation model.

[0181] Optionally, the initial video generation model includes: an initial stable diffusion model, an initial control model, and an initial temporal linking model. The training module is further used to: train the initial video generation model using sample videos, sample conditions, and sample noise to obtain the training results to be used; and alternately update the model parameters of the initial temporal linking model and the model parameters of the initial stable diffusion model based on the training results to be used to generate the target video generation model.

[0182] Optionally, the training module is further configured to: update the model parameters of the initial temporal linking model based on the training results to be used, given that the model parameters of the initial stable diffusion model and the initial control model are both fixed in advance, to obtain a first update result; update the model parameters of the initial control model and the initial stable diffusion model based on the first update result, given that the model parameters of the initial temporal linking model are fixed in advance, to obtain a second update result; and continuously update the first update result and the second update result alternately according to a preset number of training steps until the target video generation model is generated.

[0183] Optionally, the training module is further configured to: split the sample video into multiple sub-videos; sample the multiple sub-videos using a preset image frame sampling method to obtain sampling results, wherein the preset image frame sampling method is used to adapt to videos with different rates of change; and use an initial video generation model to process the sampling results, sample conditions, and sample noise to obtain training results to be used.

[0184] By employing the embodiments of this application, generation conditions for providing video material for the target video to be generated and control conditions for guiding the generation of video content adapted to the generation conditions are obtained. Based on the generation conditions, control conditions, and the same initialization noise added to multiple target image frames of the target video to be generated, the target video is generated. This enables the stable generation of target videos with a duration longer than a preset duration and controls the smooth transition of the target video between different image frames, thus generating stable long videos. This achieves the goal of generating stable, reasonable, continuous, and well-transitioned long videos, thereby improving the video reasonableness, video continuity, and video transition of the generated long videos. This solves the technical problem that the long videos generated by related long video generation methods have poor video reasonableness, video continuity, and video transition.

[0185] It should be noted that the first acquisition module 701 and the generation module 702 mentioned above correspond to steps S21 and S22 in Embodiment 1. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware components or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of the device and run in the computer terminal 10 provided in Embodiment 1.

[0186] According to an embodiment of this application, another apparatus embodiment for implementing the above-described video generation method is also provided. Figure 8 This is a schematic diagram of the structure of another video generation method apparatus according to Embodiment 4 of this application, as shown below. Figure 8 As shown, the device includes:

[0187] The second acquisition module 801 is used to acquire a video generation request through the first application programming interface;

[0188] The first return module 802 is used to return a video generation response through the second application programming interface. The request data carried in the video generation request includes generation conditions and control conditions. The response data carried in the video generation response includes a target video. The generation conditions are used to provide video material for the target video to be generated. The control conditions are used to guide the generation of video content that matches the generation conditions. The video duration of the target video is longer than a preset duration. The target video is generated based on the generation conditions, control conditions, and target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames contained in the generated target video. The target noise is used to control the smooth transition between different image frames of the target video.

[0189] In this embodiment, a video generation request carrying generation and control conditions is obtained through a first application programming interface (API), and then a video generation response carrying the target video is returned through a second API. The generation conditions provide video material for the target video to be generated, and the control conditions guide the generation of video content that matches the generation conditions. The target video's duration is longer than a preset duration. The target video is generated based on the generation conditions, control conditions, and target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames included in the generated target video. The target noise is used to control the smooth transition between different image frames in the target video, thereby achieving the goal of generating a stable, reasonable, continuous, and well-transitioned long video. This improves the video reasonableness, continuity, and transition of the generated long video, thus solving the technical problem of poor video reasonableness, continuity, and transition in related long video generation methods.

[0190] It should be noted that the second acquisition module 801 and the first return module 802 mentioned above correspond to steps S51 and S52 in Embodiment 2. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware components or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of the device and run in the computer terminal 10 provided in Embodiment 1.

[0191] According to an embodiment of this application, another embodiment of an apparatus for implementing the above-described video generation method is also provided. Figure 9 This is a schematic diagram of another video generation apparatus according to Embodiment 4 of this application, as shown below. Figure 9 As shown, the device includes:

[0192] The third acquisition module 901 is used to acquire the currently input video generation dialogue request;

[0193] The second return module 902 is used to respond to the video generation dialogue request and return a video generation dialogue response. The request data carried in the video generation dialogue request includes: generation conditions and control conditions. The information carried in the video generation dialogue response includes: target video, generation conditions are used to provide video material for the target video to be generated, control conditions are used to guide the generation of video content that matches the generation conditions, the video duration of the target video is longer than the preset duration, the target video is generated based on the generation conditions, control conditions and the target noise corresponding to the target video, the target noise is the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise is used to control the smooth transition between different image frames of the target video.

[0194] Display module 903 is used to display the target video within a graphical user interface.

[0195] By employing the embodiments of this application, a video generation dialog request carrying generation and control conditions is obtained from the current input. Then, in response to the video generation dialog request, a video generation dialog response carrying the target video is returned. The generation conditions provide video material for the target video to be generated, and the control conditions guide the generation of video content that matches the generation conditions. The target video's duration is longer than a preset duration. The target video is generated based on the generation conditions, control conditions, and target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames contained in the generated target video. The target noise is used to control the smooth transition between different image frames in the target video. After obtaining the target video, it is displayed in the graphical user interface to provide feedback to the user. This achieves the goal of generating a stable, reasonable, continuous, and well-transitioned long video, thereby improving the video reasonableness, video continuity, and video transition of the generated long video. This solves the technical problem of poor video reasonableness, video continuity, and video transition in related long video generation methods.

[0196] It should be noted that the third acquisition module 901, the second return module 902, and the display module 903 mentioned above correspond to steps S61 to S63 in Embodiment 3. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0197] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0198] Example 5

[0199] Embodiments of this application may provide a computer terminal (or electronic device), which may be any one of a group of computer terminal devices. Optionally, in this embodiment, the computer terminal may also be replaced by a mobile terminal or other terminal device.

[0200] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0201] In this embodiment, the computer terminal described above can execute the program code for the following steps in the video generation method: obtaining generation conditions and control conditions, wherein the generation conditions are used to provide video material for the target video to be generated, and the control conditions are used to guide the generation of video content that matches the generation conditions, and the video duration of the target video is greater than a preset duration; generating the target video based on the generation conditions, control conditions, and target noise corresponding to the target video, wherein the target noise is the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise is used to control the smooth transition between different image frames of the target video.

[0202] Optionally, Figure 10 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 10 As shown, the electronic device is exemplified by computer terminal A. Computer terminal A may include: one or more (only one is shown in the figure) processors 1002, memory 1004, memory controller, and peripheral interfaces, wherein the peripheral interfaces are connected to a radio frequency module, an audio module, and a display.

[0203] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the video generation method and apparatus in this application embodiment. The processor executes the stored software programs and modules to perform various functional applications and data processing, thereby realizing the aforementioned video generation method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to computer terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0204] The processor can invoke information and application programs stored in the memory through the transmission device to perform the following steps: obtaining generation conditions and control conditions, wherein the generation conditions are used to provide video material for the target video to be generated, and the control conditions are used to guide the generation of video content that is compatible with the generation conditions, and the video duration of the target video is longer than a preset duration; generating the target video based on the generation conditions, control conditions, and target noise corresponding to the target video, wherein the target noise is the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise is used to control the smooth transition between different image frames of the target video.

[0205] Optionally, the processor may also execute program code for the following steps: using a target video generation model to perform video generation processing on generation conditions, control conditions, and target noise to obtain a target video, wherein the target video generation model is used to perform stable diffusion based on generation conditions, control conditions, and target noise to obtain multiple target image frames and to perform temporal linking of the multiple target image frames obtained after diffusion to obtain a target video.

[0206] Optionally, the target video generation model includes: a target stable diffusion model, a target control model, and a target temporal linking model. The processor can also execute program code with the following steps: under the guidance of control conditions input through the target control model, the stable diffusion model is used to perform image diffusion processing on the generation conditions and target noise to obtain multiple consecutive image frames, and the temporal linking model is used to perform temporal smoothing on the multiple consecutive image frames to obtain multiple target image frames; the multiple target image frames are then image stitched together to obtain the target video.

[0207] Optionally, the processor may also execute program code for the following steps: in a stable diffusion model, feature encoding is performed on the generation conditions to obtain the encoding results corresponding to the generation conditions; spatial mapping is performed on the encoding results to obtain spatial mapping results corresponding to multiple initial image frames; target noise is added to the spatial mapping results corresponding to multiple initial image frames according to a preset noise optimization step number to obtain latent space features corresponding to multiple initial image frames; image diffusion processing is performed on the latent space features corresponding to multiple initial image frames to obtain image features corresponding to multiple initial image frames; feature decoding is performed on the image features corresponding to multiple initial image frames to obtain multiple consecutive image frames; wherein, the number of multiple initial image frames is the preset number of image frames to be included in the target video.

[0208] Optionally, the size of the noise map corresponding to the target noise is determined based on the image frame size of multiple target image frames, and the content of the noise map corresponding to the target noise is determined based on random numbers within a preset range.

[0209] Optionally, multiple initial image frames are divided into multiple sets of image frame sequences according to a preset number and a preset ratio. The preset number is used to determine the number of image frames contained in each set of image frame sequences, and the preset ratio is used to determine the proportion of image frames that overlap between two adjacent sets of image frame sequences.

[0210] Optionally, the processor may also execute program code for the following steps: adding target noise to the spatial mapping results of adjacent two sets of image frame sequences according to a preset number of noise optimization steps, obtaining latent space features corresponding to adjacent two sets of image frame sequences, and recording the number of noise optimization steps for image frames that overlap between adjacent two sets of image frame sequences; performing feature superposition on the current latent space features corresponding to adjacent two sets of image frame sequences to obtain overlapping regions; obtaining the feature average value within the overlapping regions; obtaining the latent space features corresponding to image frames that overlap between adjacent two sets of image frame sequences based on the feature average value and the number of noise optimization steps, until all noise optimization steps have been executed, and obtaining latent space features corresponding to multiple initial image frames.

[0211] Optionally, the processor may also execute program code that performs the following steps: training the initial video generation model using sample videos, sample conditions, and sample noise to generate the target video generation model.

[0212] Optionally, the initial video generation model includes: an initial stable diffusion model, an initial control model, and an initial temporal linking model. The processor can also execute program code for the following steps: training the initial video generation model using sample videos, sample conditions, and sample noise to obtain the training results to be used; and alternately updating the model parameters of the initial temporal linking model and the model parameters of the initial stable diffusion model based on the training results to be used, thereby generating the target video generation model.

[0213] Optionally, the processor may also execute program code with the following steps: Given that the model parameters of the initial stable diffusion model and the initial control model are both fixed in advance, update the model parameters of the initial temporal linking model based on the training results to be used, to obtain a first update result; given that the model parameters of the initial temporal linking model are fixed in advance, update the model parameters of the initial control model and the initial stable diffusion model based on the first update result, to obtain a second update result; and continuously update the first update result and the second update result alternately according to a preset number of training steps until the target video generation model is generated.

[0214] Optionally, the processor may also execute program code for the following steps: splitting the sample video into multiple sub-videos; sampling the multiple sub-videos according to a preset image frame sampling method to obtain sampling results, wherein the preset image frame sampling method is used to adapt to videos with different change rates; and using an initial video generation model to perform video generation processing on the sampling results, sample conditions, and sample noise to obtain training results to be used.

[0215] By employing the embodiments of this application, generation conditions for providing video material for the target video to be generated and control conditions for guiding the generation of video content adapted to the generation conditions are obtained. Based on the generation conditions, control conditions, and the same initialization noise added to multiple target image frames of the target video to be generated, the target video is generated. This enables the stable generation of target videos with a duration longer than a preset duration and controls the smooth transition of the target video between different image frames, thus generating stable long videos. This achieves the goal of generating stable, reasonable, continuous, and well-transitioned long videos, thereby improving the video reasonableness, video continuity, and video transition of the generated long videos. This solves the technical problem that the long videos generated by related long video generation methods have poor video reasonableness, video continuity, and video transition.

[0216] Those skilled in the art will understand that Figure 10 The structure shown is for illustrative purposes only. Computer terminal A can also be a smartphone (such as an Android phone, iOS phone, etc.), tablet computer, PDA, mobile Internet Device (MID), PAD, and other terminal devices. Figure 10 This does not limit the structure of the aforementioned electronic device. For example, computer terminal A may also include components that are more... Figure 10 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 10 The different configurations shown.

[0217] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0218] Example 6

[0219] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the aforementioned computer-readable storage medium can be used to store the program code executed by the video generation method provided in Embodiment 1.

[0220] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0221] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining generation conditions and control conditions, wherein the generation conditions are used to provide video material for the target video to be generated, and the control conditions are used to guide the generation of video content that is compatible with the generation conditions, and the video duration of the target video is greater than a preset duration; generating the target video based on the generation conditions, control conditions, and target noise corresponding to the target video, wherein the target noise is the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise is used to control the smooth transition of the target video between different image frames.

[0222] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements any of the above-described video generation methods.

[0223] Optionally, in this embodiment, when the computer program product is executed by the processor, the program code is used to perform the following steps: obtaining generation conditions and control conditions, wherein the generation conditions are used to provide video material for the target video to be generated, the control conditions are used to guide the generation of video content that is compatible with the generation conditions, and the video duration of the target video is greater than a preset duration; generating the target video based on the generation conditions, control conditions, and target noise corresponding to the target video, wherein the target noise is the same initialization noise added to multiple target image frames contained in the generated target video, and the target noise is used to control the smooth transition between different image frames of the target video.

[0224] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0225] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0226] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0227] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0228] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0229] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0230] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A video generation method, characterized in that, include: The generation conditions and control conditions are obtained, wherein the generation conditions are used to provide video materials for the target video to be generated, and the control conditions are used to guide the generation of video content that is compatible with the generation conditions, and the video duration of the target video is greater than a preset duration; The target video is generated based on the generation conditions, the control conditions, and the target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames included in the generation of the target video. The target noise is used to control the smooth transition of the target video between different image frames.

2. The video generation method according to claim 1, characterized in that, Generating the target video based on the generation conditions, the control conditions, and the target noise corresponding to the target video includes: A target video generation model is used to perform video generation processing on the generation conditions, the control conditions, and the target noise to obtain the target video. The target video generation model is used to perform stable diffusion based on the generation conditions, the control conditions, and the target noise to obtain the plurality of target image frames, and to perform temporal linking on the plurality of target image frames obtained after diffusion to obtain the target video.

3. The video generation method according to claim 2, characterized in that, The target video generation model includes: a target stable diffusion model, a target control model, and a target temporal linking model. The target video is generated by applying the generation conditions, the control conditions, and the target noise using the target video generation model, resulting in the following target video: Guided by the control conditions input through the target control model, the stable diffusion model is used to perform image diffusion processing on the generation conditions and the target noise to obtain multiple consecutive image frames, and the temporal linking model is used to perform temporal smoothing on the multiple consecutive image frames to obtain the multiple target image frames. The target video is obtained by stitching together the multiple target image frames.

4. The video generation method according to claim 3, characterized in that, The image diffusion process, which uses the stable diffusion model to perform image diffusion processing on the generation conditions and the target noise, yields the multiple consecutive image frames, including: In the stable diffusion model, the generation conditions are feature-encoded to obtain the encoding results corresponding to the generation conditions; Spatial mapping is performed on the encoding results to obtain spatial mapping results corresponding to multiple initial image frames; The target noise is added to the spatial mapping result corresponding to the multiple initial image frames according to the preset noise optimization steps to obtain the latent space features corresponding to the multiple initial image frames; Image diffusion processing is performed on the latent space features corresponding to the plurality of initial image frames to obtain the image features corresponding to the plurality of initial image frames; The image features corresponding to the plurality of initial image frames are decoded to obtain the plurality of consecutive image frames; The number of the plurality of initial image frames is the number of image frames to be included in the target video as preset.

5. The video generation method according to claim 4, characterized in that, The size of the noise map corresponding to the target noise is determined based on the image frame size of the plurality of target image frames, and the content of the noise map corresponding to the target noise is determined based on random numbers within a preset range.

6. The video generation method according to claim 4, characterized in that, The initial image frames are divided into multiple image frame sequences according to a preset number and a preset ratio. The preset number is used to determine the number of image frames contained in each image frame sequence, and the preset ratio is used to determine the proportion of image frames that overlap between two adjacent image frame sequences.

7. The video generation method according to claim 6, characterized in that, The target noise is added to the spatial mapping results corresponding to the plurality of initial image frames according to the noise optimization steps, and the latent spatial features corresponding to the plurality of initial image frames are obtained as follows: According to the preset noise optimization steps, the target noise is added to the spatial mapping results corresponding to two adjacent image frame sequences in the multiple image frame sequences to obtain the latent spatial features corresponding to the two adjacent image frame sequences, and the number of noise optimizations corresponding to the image frames that overlap between the two adjacent image frame sequences is recorded. The latent space features corresponding to the two adjacent sets of image frame sequences are superimposed to obtain the overlapping region; The average value of the features is obtained within the overlapping region; Based on the average value of the features and the number of noise optimization steps, the latent space features corresponding to the image frames that overlap between the two adjacent image frame sequences are obtained, until all the noise optimization steps are completed, and the latent space features corresponding to the multiple initial image frames are obtained.

8. The video generation method according to claim 2, characterized in that, The video generation method further includes: The initial video generation model is trained using sample videos, sample conditions, and sample noise to generate the target video generation model.

9. The video generation method according to claim 8, characterized in that, The initial video generation model includes: an initial stable diffusion model, an initial control model, and an initial temporal linking model. The initial video generation model is trained using the sample video, the sample conditions, and the sample noise. Generating the target video generation model includes: The initial video generation model is trained using the sample video, the sample conditions, and the sample noise to obtain the training results to be used. Based on the training results to be used, the model parameters of the initial temporal linking model and the model parameters of the initial stable diffusion model are alternately updated to generate the target video generation model.

10. The video generation method according to claim 9, characterized in that, Based on the training results to be used, the model parameters of the initial temporal linking model and the initial stable diffusion model are alternately updated to generate the target video generation model, including: With the model parameters of the initial stable diffusion model and the initial control model both fixed in advance, the model parameters of the initial temporal linking model are updated based on the training results to be used, to obtain the first update result; With the model parameters of the initial time-series linkage model fixed in advance, the model parameters of the initial control model and the model parameters of the initial stable diffusion model are updated based on the first update result to obtain the second update result; The first update result and the second update result are continuously updated alternately according to a preset number of training steps until the target video generation model is generated.

11. The video generation method according to claim 9, characterized in that, The initial video generation model is trained using the sample video, the sample conditions, and the sample noise to obtain the training results to be used, including: The sample video is split into multiple sub-videos; The multiple sub-videos are sampled according to a preset image frame sampling method to obtain sampling results. The preset image frame sampling method is used to adapt to videos with different change rates. The initial video generation model is used to process the sampling results, the sample conditions, and the sample noise to generate a video, thereby obtaining the training results to be used.

12. A video generation method, characterized in that, include: The video generation request is obtained through the first application programming interface; The video generation response is returned via the second application programming interface; The video generation request carries request data including generation conditions and control conditions. The video generation response carries response data including a target video. The generation conditions are used to provide video material for the target video to be generated. The control conditions are used to guide the generation of video content that matches the generation conditions. The video duration of the target video is greater than a preset duration. The target video is generated based on the generation conditions, the control conditions, and the target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames included in the generation of the target video. The target noise is used to control the smooth transition of the target video between different image frames.

13. A video generation method, characterized in that, include: Get the currently input video and generate a dialogue request; In response to the video generation dialogue request, a video generation dialogue reply is returned; The video generation dialogue request carries request data including generation conditions and control conditions. The video generation dialogue response carries information including a target video. The generation conditions are used to provide video material for the target video to be generated. The control conditions are used to guide the generation of video content that matches the generation conditions. The video duration of the target video is longer than a preset duration. The target video is generated based on the generation conditions, the control conditions, and the target noise corresponding to the target video. The target noise is the same initialization noise added to multiple target image frames contained in the target video. The target noise is used to control the smooth transition of the target video between different image frames. The target video is displayed within a graphical user interface.

14. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the video generation method according to any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the video generation method according to any one of claims 1 to 13.

16. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the video generation method according to any one of claims 1 to 13.

Images