Video generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a video generation method and device, electronic equipment and a storage medium, and the method comprises the steps: generating a first video based on a target image, the first video comprises the target image and multiple frames of all-black images, and the first frame image of the first video is the target image; and inputting the preset noise information and conditional guidance information into a pre-trained video generation model to obtain a second video, the conditional guidance information comprising the first video, the first frame of the second video being the target image and the duration of the second video being equal to the duration of the first video. Therefore, when the video generation model generates the second video based on the preset noise information and the first video, the target image can be comprehensively considered to ensure that the generated second video comprises the target image, and meanwhile, the target image is taken as the first frame image of the first video, so that the first frame image of the second video is ensured to be the target image; the video is generated through the first frame image.

Description

Technical Field

[0001] The present disclosure relates to the technical field of video generation, and in particular to a video generation method, device, electronic device, and storage medium. Background Art

[0002] In recent years, with the rapid development of Vincent video models, Vincent video models have made great achievements in the field of AIGC (Artificial Intelligence Generated Content). In related technologies, in the process of generating videos, it is often necessary to include specific images in the generated videos, but Vincent video models cannot generate videos that include specific images, which reduces the matching degree between the generated videos and the videos required by users, affecting the user experience. Summary of the invention

[0003] In order to solve the above technical problems, the embodiments of the present disclosure provide a video generation method, device, electronic device, storage medium and program product.

[0004] One aspect of an embodiment of the present disclosure provides a video generation method, comprising: generating a first video based on a target image, the first video including the target image and multiple frames of all-black images, and the first frame image of the first video is the target image; inputting preset noise information and conditional guidance information into a pre-trained video generation model to obtain a second video, the conditional guidance information including the first video, the first frame of the second video is the target image, and the duration of the second video is the same as the duration of the first video.

[0005] Another aspect of the embodiments of the present disclosure provides a video generation method, including: generating a first video based on a target image, the first video including the target image and multiple frames of all-black images, and the first frame image of the first video is the target image; inputting preset noise information and conditional guidance information into a pre-trained video generation model to obtain a second video, the conditional guidance information including the first video, the first frame of the second video is the target image, and the duration of the second video is the same as the duration of the first video.

[0006] Another aspect of the embodiments of the present disclosure provides an electronic device, including: a memory for storing a computer program; a processor for executing the computer program stored in the memory, and when the computer program is executed, the above-mentioned video generation method is implemented.

[0007] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned video generation method is implemented.

[0008] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, including computer program instructions, which implement the above-mentioned video generation method when executed by a processor.

[0009] In an embodiment of the present disclosure, by generating a first video including a target image, the video generation model can comprehensively consider the target image when generating a second video based on preset noise information and the first video, so as to ensure that the generated second video includes the target image; at the same time, in an embodiment of the present disclosure, the target image is also used as the first frame image of the first video, thereby ensuring that the first frame image of the second video is the target image, thereby realizing the generation of video through the first frame image.

[0010] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0012] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which: Figure 1 is a flowchart of a video generation method provided by an exemplary embodiment of the present disclosure; Figure 2 is a flowchart of step S110 provided by an exemplary embodiment of the present disclosure; Figure 3 is a schematic diagram of an input layer in an adapter network module and a video generation network module provided by an exemplary embodiment of the present disclosure; Figure 4 is a flowchart of a video generation method provided by another exemplary embodiment of the present disclosure; Figure 5 It is a structural schematic diagram of an embodiment of the video generating device disclosed in the present invention; Figure 6 It is a structural schematic diagram of another embodiment of the video generating device disclosed in the present invention; Figure 7 The figure is a schematic diagram of the structure of an application embodiment of the electronic device disclosed in the present invention. DETAILED DESCRIPTION

[0013] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0014] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0015] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0016] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0017] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0018] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0019] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0020] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0021] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0022] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0023] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0024] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0025] In the process of implementing the present disclosure, the inventors found that in practical applications, when generating a video, it is usually necessary to include the specified image in the generated video. However, the existing text-generated video model cannot generate a video including the specified image, which results in a low match between the video generated by the text-generated video model and the video required by the user, affecting the user experience.

[0026] Figure 1 FIG. 1 is a flow chart of a video generation method provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the following steps are included: Step S100: generating a first video based on a target image.

[0027] The first video includes a target image and multiple frames of all-black images, and the first frame image of the first video is the target image.

[0028] In one embodiment, multiple frames of all-black images and the target image may be combined according to preset timing information to obtain the first video. The preset timing information may include, for example, that the target image is located in the first frame of the first video.

[0029] Step S110, inputting preset noise information and conditional guidance information into a pre-trained video generation model to obtain a second video.

[0030] The conditional guidance information includes the first video, the first frame of the second video is the target image, and the duration of the second video is the same as the duration of the first video. The video generation model can be a deep learning model that extends the video according to the video data.

[0031] In an embodiment of the present disclosure, by generating a first video including a target image, the video generation model can comprehensively consider the target image when generating a second video based on preset noise information and the first video, so as to ensure that the generated second video includes the target image; at the same time, in an embodiment of the present disclosure, the target image is also used as the first frame image of the first video, thereby ensuring that the first frame image of the second video is the target image, thereby realizing the generation of video through the first frame image.

[0032] In some optional implementations, in the embodiments of the present disclosure, the conditional guidance information also includes video description information.

[0033] The video description information may be used to describe the video, and the video description information may include, for example, text information describing the video content.

[0034] Correspondingly, in the embodiment of the present disclosure, step S110 may include: the adapter network module in the video generation model processes the first video to obtain the first vector output by each first network layer in the adapter network module; the video generation network module in the video generation model processes multiple first vectors, preset noise information and video description information to obtain the second video.

[0035] Wherein, the video generation model may include: an adapter network module and a video generation network module. The adapter network module and the video generation network module may be pre-trained deep learning models. The pre-trained video generation model is obtained by joint training of the video generation network module and the adapter network module to be trained. Optionally, in the process of joint training based on the video generation network module and the adapter network module to be trained, the video generation network module and the adapter network module to be trained are first obtained, wherein the network layer structure of the adapter network module to be trained is the same as the network structure of the input layer of the video generation network module, and each network layer of the adapter network module to be trained corresponds to the network layer of the input layer of the video generation network module, that is, the number of layers of the network layer of the adapter network module to be trained is the same as the number of layers of the network layer of the input layer of the video generation network module, and each network layer of the adapter network module to be trained has the same network parameters as the network layer of the corresponding input layer of the video generation network module. Secondly, the network parameters of each network layer of the adapter network module to be trained are adjusted through the loss function of the video generation network module, so as to obtain the adapter network module in the video generation model.

[0036] In this embodiment, the network layer in the adapter network module can be referred to as a first network layer, and the adapter network module can include multiple first network layers. The input layer of the video generation network module includes multiple second network layers, and the multiple first network layers of the adapter network module correspond to the multiple second network layers of the video generation network module in a one-to-one mapping manner, wherein the first vector outputted by each first network layer is used to be superimposed with the second vector outputted by the corresponding second network layer to serve as the input vector of the next second network layer.

[0037] Furthermore, in some embodiments, the structure of each first network layer is the same as that of the corresponding second network layer, and thus the dimension of the output vector of each first network layer is the same as that of the output vector of the corresponding second network layer, thereby facilitating the same-dimensional superposition of the output vector of the first network layer and the output vector of the corresponding second network layer, so as to achieve comprehensive fine-tuning of the video generation network module through the adapter network module.

[0038] Exemplarily, the adapter network module can adopt an adapter model; the video generation network module is a diffusion model structure. Among them, the diffusion model is a generation model based on a deep learning algorithm, which generates high-quality content through a gradual denoising and denoising process, performs well in generating images, videos, audio and other high-dimensional data, and has strong performance in generation tasks. In the embodiment of the present disclosure, the video generation network module can be obtained by training the diffusion model. The video generation network module has universality in the field of video generation, but the matching effect between the content of the generated video and the input is not necessarily very good. In the embodiment of the present disclosure, by combining the adapter network module with the video generation network module, it can be based on the universality of the video generation network module. When the input includes a target image, it can efficiently generate a high-quality video (that is, the content of the generated video matches the input content well).

[0039] The preset noise information may include Gaussian noise. Exemplarily, a Gaussian noise matrix may be randomly generated in advance, and the Gaussian noise matrix may be determined as the preset noise information.

[0040] In one embodiment, the video generation network module is a diffusion model structure, and the diffusion model mainly includes two processes: noise addition and denoising: in the noise addition process, the original data is converted into an expression form of latent space through a variational autoencoder, which can make the model calculation more efficient, and obtain noisy content after T (T≥1) steps of noise addition; in the denoising process, noise expectation is performed for the given noisy content, and conditional guidance information is received during the process, which includes the first video and video description information, and finally restored from the latent space to the pixel space through the variational autoencoder to obtain a video that meets the expectations. In this embodiment, the video description information and the first video can be better referenced in the video generation process, so that the generated second video and the input content match better.

[0041] In one embodiment, when preset noise information, a first video, and video description information are input into a video generation model, the adapter network module processes the first video and records the first vector output by each first network layer; then the video generation network module processes the first vector output by each first network layer, the video description information, and the preset noise information to obtain a second video including a target image, and outputs the second video.

[0042] In the disclosed embodiment, the video generation model is designed to include an adapter network module and a video generation network module. The adapter network module is used to process a first video including a target image to obtain multiple first vectors. Then, the video generation network module is used to process the multiple first vectors, video description information and preset noise information, so that the video generation network module takes into account the first video and the video description information when generating a second video, so that the generated second video can not only include the target image, but also the video content conforms to the video description information. This not only improves the matching degree between the second video generated by the video generation model and the video required by the user, improves the user experience, but also expands the application scope of the video generation model.

[0043] Figure 2 is a flow chart of step S100 provided by an exemplary embodiment of the present disclosure. In some optional implementations, such as Figure 2 As shown, step S100 may include the following steps: Step S101, obtaining a preset video.

[0044] The preset video may be a content-free video with a certain duration, and the preset video may include multiple frames of completely black images.

[0045] In one implementation, a video including a preset number of frames of all-black images may be created as the preset video. For example, a video including 160 frames of all-black images may be created as the preset video.

[0046] Step S102, replacing the first frame image of the preset video with the target image to obtain a first video.

[0047] In one embodiment, the first frame image in the preset video can be replaced with the target image to obtain a first video. Exemplarily, assuming that the frame rate of the preset video is 160 FPS and the duration is 10 seconds, the preset video includes 160 frames of all-black images. Video editing software such as Hand Brake, Freemake Video Converter or Quick Clip can be used to replace the first frame image (the first frame image) in the preset video with the target image to obtain a first video, in which the first frame image is the target image and the remaining 159 frames of images are all-black images.

[0048] Exemplarily, the acquired preset video is represented by 000000000000000000000, wherein each 0 represents a completely black video frame, and the target image is represented by 1. The first frame image of the preset video is replaced by the target image to obtain the first video: 100000000000000000000. The preset noise information and the conditional guidance information including the conditional first video are input into the pre-trained video generation model to obtain the final second video: 122222222222222222222, wherein 2 represents a video frame of completely new visual content generated by the pre-trained video generation model, the first frame of the second video is the target image 1, and the duration of the second video is the same as that of the first video. The temporal position of the target image in the first video is consistent with the temporal position of the target image in the second video. Therefore, through the embodiments of the present invention, the generation of video through the first frame can be realized quickly and efficiently, which meets the current user's demand for generating a second video based on a target image with the target image as the first frame of the video, and improves the interest of video generation in the field of AIGC technology.

[0049] In the embodiment of the present disclosure, the preset video is designed to be composed of multiple frames of all-black images, and the first frame image in the preset video is replaced by the target image to obtain the first video. In this way, not only can the first video including the target image be generated quickly, but also because the first video only includes the target image and the all-black image, interference of other content images to the video generation model is avoided.

[0050] In some optional embodiments, in the presently disclosed embodiment, the adapter network module includes n first network layers, the input layer of the video generation network module includes n second network layers, and the n first network layers of the adapter network module correspond one-to-one to the n second network layers of the video generation network module, respectively.

[0051] The video generation network module includes an input layer and an output layer. At each time step, the video generation network module adds or removes noise for one time step through noise prediction of the input layer and the output layer, and then obtains the latent space expression of the final generated video after performing noise prediction for T (T≥1) time steps. In this embodiment, the network layer in the input layer of the video generation network module can be called a second network layer, and any second network layer corresponds to a first network layer.

[0052] Among them, the input layer in the video generation network module includes n second network layers, where n is an integer greater than or equal to 1. The network structure of the input layer is the same as the network structure of the adapter network module, that is, the adapter network module also includes n first network layers, and the first network layers with the same sequence number correspond to the second network layers. For example, the first first network layer corresponds to the first second network layer, and so on, the nth first network layer corresponds to the nth second network layer. The shape (dimension) of the data output by the first network layer is the same as the shape of the data output by the second network layer.

[0053] Exemplarily, both the first network layer and the second network layer can adopt the UNet network. UNet can denoise random noise to achieve the generation from noise to video. In this embodiment, the input layer of the video generation network module is used to gradually downsample the input noise, extract features and reduce the spatial resolution of the image. The output layer of the video generation network module gradually restores the spatial resolution of the video through deconvolution (or transposed convolution) and upsampling operations; through the jump connection structure of the UNet network, the output layer can combine the low-level features in the input layer (such as edge, texture and other detail information) with high-level features (such as semantic information of video description information), and the fusion of features helps to reconstruct the details of the video; finally, the second video with the same resolution as the input first video is output. In the present application, based on the multi-layer second network layer structure in the input layer of the video generation network module, an independent adapter network module is designed to fine-tune the vector generated by the input layer downsampling process. That is to say, the adapter network module adapts to new tasks by inserting additional parameters into the pre-trained video generation network module, rather than directly adjusting the parameters of the entire video generation network module (the parameters of the video generation network module are very large, such as billions or tens of billions of parameters). This can not only reduce the number of parameters required for fine-tuning, but also reduce the demand for computing resources during training and speed up training. At the same time, when the video generation network module performs a specific task, the trained adapter network module is used to fine-tune the output vectors of each layer of the output layer downsampled, so that the video generation network module does not need to be trained on a large scale as a whole, and the specific task can be completed efficiently and accurately. It can be seen that through the combination and joint training of the adapter network module and the video generation network module, it is possible to quickly generate high-quality videos based on the target image based on AIGC technology, and it will not affect the versatility of the video generation network module.

[0054] Correspondingly, in the embodiment of the present disclosure, the video generation network module can process multiple first vectors, preset noise information and video description information in the following manner: in the denoising process of each time step of the video generation network module, based on the preset noise information and video description information, the n first vectors output by the n first network layers are respectively calculated with the corresponding n second vectors output by the n second network layers to obtain a second video.

[0055] Among them, it can be understood that the input layer and output layer of the video generation network module perform one round of calculation as a noise prediction of one time step, that is, a denoising process of one time step, and the video generation network module performs T rounds of iterative denoising to obtain the final generated second video. After the output layer outputs the output vector of the first round, it is input into the input layer again as the input vector of the second round. After T rounds of loop execution according to the above process, the output layer outputs the output vector of the Tth round to input the decoder to obtain the final generated video. The decoder can reconstruct the video from the latent space expression, map the generation result of the video generation network module from the latent space to the pixel space, and generate the final second video.

[0056] In the disclosed embodiment, the adapter network module is set to include n first network layers, and the video network module is set to include n second network layers, so that each second network layer can process the first vector output by the corresponding first network layer, thereby enabling the video network module to comprehensively consider the first video when generating the second video, thereby ensuring that the target image is included in the second video.

[0057] In some optional implementations, in the embodiment of the present disclosure, using n second network layers to process the multiple first vectors, the preset noise information, and the video description information may include: For the denoising process at each time step, the first second network layer processes the preset noise information based on the video description information to obtain the second vector output by the first second network layer; for the second second network layer to the nth second network layer, the second network layer processes the third vector based on the video description information to obtain the second vector output by the second network layer, and the second video is determined based on the second vector output by the nth second network layer in the denoising process at the last time step.

[0058] The third vector is obtained by adding a second vector output by a second network layer previous to the second network layer to a first vector output by a first network layer corresponding to the previous second network layer. In one embodiment, the second video is determined based on the second vector output by the nth second network layer and the first vector output by the nth first network layer.

[0059] For example, Figure 3 FIG. 1 is a schematic diagram of an input layer in an adapter network module and a video generation network module provided by an exemplary embodiment of the present disclosure. Figure 3 As shown, a variational auto-encoder (VAE) can be used to encode the first video, and then the encoded first video is input into the adapter network module, and the first vectors output by each first network layer in the adapter network module are recorded; The VAE can be used to encode the video description information and the preset noise information. For the denoising process at each time step, the encoded video description information and the preset noise information are input into the first second network layer to obtain a second vector output by the first second network layer. The second vector is added to the first vector output by the first first network layer to obtain a third vector. The third vector can be used as input data of the second second network layer. A third vector obtained by adding the second vector output by the i-1th second network layer to the first vector output by the i-1th first network layer and the encoded video description information are input into the i-th second network layer to obtain the second vector output by the i-th second network layer, and the second vector is added to the first vector output by the i-th first network layer to obtain a third vector, which can be used as input data of the i+1th second network layer, wherein 2≤i≤n; The result of adding the second vector output by the nth second network layer in the denoising process of the last time step to the first vector output by the nth first network layer is input into the VAE for decoding processing to obtain the second video.

[0060] In the embodiment of the present disclosure, the third vector obtained by adding the second vector output by each second network layer to the first vector output by the corresponding first network layer is used as the input data of the next second network layer, so that the video generation network module can take the first video into consideration when generating the second video, thereby ensuring that the generated second video includes the target image and the video content corresponds to the video description information.

[0061] Figure 4 FIG. 1 is a flow chart of a video generation method provided by another exemplary embodiment of the present disclosure. In some optional implementations, such as Figure 4 As shown, the video generation model can be obtained as follows: Step S200, obtaining the model to be trained and sample data.

[0062] The model to be trained includes: a video generation network module and an adapter network module to be trained.

[0063] The sample data includes multiple sample videos, and label videos and label video description information corresponding to each sample video. For multiple sample videos, the sample video includes a frame of image with content and multiple frames of all-black images, and the image with content is the first frame image.

[0064] In some optional implementations, in the embodiments of the present disclosure, a sample video can be obtained in the following manner: obtain multiple labeled videos; for the multiple labeled videos, replace the remaining frame images in the labeled video except the first frame image with all-black images to obtain a sample video corresponding to the labeled video.

[0065] Among them, multiple videos with the same duration and frame rate can be obtained as label videos. Exemplarily, a complete video with a longer duration can be cut into multiple videos of the same duration as label videos. For each label video, the remaining frame images in the sample video except the first frame image can be replaced with a completely black image to obtain a sample video corresponding to the label video. The above operation of replacing the completely black image is performed for each label video to obtain multiple sample videos.

[0066] For example, assuming that the duration of each tagged video is 10s and the frame rate is 16FPS, each tagged video includes 160 frames of images. For each tagged video, the first frame of the tagged video is not processed, and the second to 160 frames of the initial sample image are all replaced with black images to obtain a sample video corresponding to the tagged video. In the sample video, the first frame is an image with content, and the remaining 159 frames are all black images without content.

[0067] In an optional embodiment, the labeled video description information can be obtained in the following manner: for each labeled video, the labeled video is input into a visual language model (VLMs), the visual language model outputs the video description information corresponding to the labeled video, and the video description information is used as the labeled video description information corresponding to the sample video obtained based on the labeled video.

[0068] Step S210, for multiple sample videos, the sample video, the label video corresponding to the sample video and the label video description information are input into the model to be trained, the sample video is processed by the adapter network module to be trained, and the fourth vectors output by each first network layer in the adapter network module to be trained are obtained, and the video generation network module processes the multiple fourth vectors and the label video description information to obtain the predicted video.

[0069] The adapter network module to be trained includes n first network layers. Each sample video, as well as the label video and label video description information corresponding to each sample video can be sequentially input into the model to be trained, and the model to be trained sequentially outputs the predicted video corresponding to each sample video.

[0070] Step S220, adjusting the parameters of the adapter network module to be trained according to the predicted video and the label video corresponding to each sample video until the preset training end condition is met, and obtaining the video generation model from the model to be trained.

[0071] In the model training stage, the parameters of the video generation network module can be frozen, and the parameters of the input layer of the video generation network module can be used as the initialization parameters of the adapter network module to be trained.

[0072] Specifically, the training process is to iteratively adjust the network parameters of the adapter network module, and the network parameters of each layer of the video generation network module remain unchanged. Based on this, according to the loss function of the video generation network module, the network parameters of each first network layer of the adapter network module are adjusted by back propagation. In this way, according to S210 to S220, the loop iteration is repeated until the loss function converges, and a trained adapter network module can be obtained, and each first network layer has a trained network parameter.

[0073] The training method disclosed in the present invention can form sample data including a sample video of a frame of image with content and label video description information, and externally connect an adapter network module during the downsampling process of a pre-trained video generation network module to realize the training of the adapter network module, thereby efficiently and low-costly obtaining a video generation model that can restore or reconstruct video content.

[0074] In one embodiment, the loss function value can be determined based on the difference between the label video and the predicted video corresponding to each sample video using a preset loss function. The preset loss function may include, but is not limited to, a cross entropy error function or a mean square error function. The operation of determining the loss function value may be iteratively performed, and the parameters of the adapter network module to be trained may be iteratively adjusted to continuously reduce the loss function value until the loss function value converges, and it is determined that the preset training end condition is met, and the training of the model to be trained is completed, and the trained model to be trained is used as the video generation model.

[0075] Parameter optimizers such as Stochastic Gradient Descent (SGD), Adaptive Gradient Algorithm (Adagrad), Adaptive Moment Estimation (Adam), Root Mean Square Prop (RMSprop) can be used to adjust the parameters of the adapter network module to be trained. For example, the parameter optimizer can be used to calculate the gradient of the parameters of the adapter network module to be trained, and the parameters can be adjusted along the direction of the gradient, where the gradient represents the direction in which the loss function value is reduced the most, and the operation of iteratively determining the loss function value is performed until the loss function value no longer decreases, and the training of the model to be trained is completed, and a video generation model is obtained.

[0076] In the disclosed embodiment, a model to be trained, including a video generation network module and an adapter network module to be trained, is trained through multiple sample videos, as well as label videos and label video description information corresponding to each sample video. During the model training process, only the parameters of the adapter network module to be trained need to be adjusted, which not only reduces the difficulty of model training and improves the efficiency of model training, but also enables the trained video generation model to generate a video including the image based on the image.

[0077] In some optional implementations, in the embodiment of the present disclosure, the video generation network module may process the plurality of fourth vectors and the label video description information in the following manner: For the denoising process at each time step, the first second network layer in the video generation network module processes the label video description information to obtain the fifth vector output by the first second network layer; for the second second network layer to the nth second network layer in the video generation network module, the second network layer processes the label video description information and the sixth vector to obtain the fifth vector output by the second network layer, and based on the fifth vector output by the nth second network layer in the denoising process at the last time step, the predicted video corresponding to the sample video is determined.

[0078] The sixth vector is obtained by adding a fifth vector output by a second network layer preceding the second network layer and a fourth vector output by a first network layer corresponding to the second network layer preceding the second network layer.

[0079] In one embodiment, the network structure of the adapter network module to be trained is the same as the network structure of the input layer of the video generation network module, that is, the adapter network module to be trained also includes n first network layers. In the model to be trained, the first network layer with the same sequence number corresponds to the second network layer, and the shape of the data output by the first network layer is the same as the shape of the data output by the second network layer. For example, the first network layer and the second network layer can adopt the UNet network.

[0080] In the forward propagation process of the video generation network module, for the denoising process of each time step, the label video description information is input into the first second network layer, the first second network layer outputs a fifth vector, the fifth vector is added to the fourth vector output by the first first network layer, and a sixth vector is obtained, and the sixth vector is used as input data of the second second network layer; A sixth vector obtained by adding the fifth vector output by the i-1th second network layer and the fourth vector output by the i-1th first network layer and the label video description information are input into the i-th second network layer, and a fifth vector is output by the i-th second network layer, and the fifth vector is added to the fourth vector output by the i-th first network layer to obtain a sixth vector, and the sixth vector is used as input data of the i+1th second network layer; The predicted video is determined based on the result of adding the fifth vector output by the nth second network layer and the fourth vector output by the nth first network layer in the denoising process of the last time step.

[0081] In the disclosed embodiment, the sixth vector obtained by adding the fifth vector output by each second network layer and the fourth vector output by the corresponding first network layer is used as the input data of the next second network layer, so that the video generation network module can take into account the data output by the adapter network module to be trained when generating a predicted video, thereby facilitating better learning of the adapter network module to be trained.

[0082] Figure 5 FIG. 1 is a schematic diagram of the structure of an embodiment of the video generation device disclosed in the present invention. Figure 5 As shown, the video generating device of this embodiment may include: a first video generating module 310 and a second video generating module 320 .

[0083] A first video generating module 310 is used to generate a first video based on a target image, wherein the first video includes the target image and a plurality of frames of all-black images, and the first frame image of the first video is the target image; The second video generation module 320 is used to input preset noise information and conditional guidance information into a pre-trained video generation model to obtain a second video, wherein the conditional guidance information includes the first video, the first frame of the second video is the target image, and the duration of the second video is the same as that of the first video.

[0084] In some possible implementations of the present disclosure, the conditional guidance information in the embodiments of the present disclosure further includes video description information; The preset noise information and conditional guidance information are input into a pre-trained video generation model to obtain a second video, which is further used for: The first video is processed by an adapter network module in the video generation model to obtain a first vector output by each first network layer in the adapter network module; The video generation network module in the video generation model processes the multiple first vectors, the preset noise information and the video description information to obtain the second video.

[0085] Figure 6 FIG. 1 is a schematic diagram of the structure of another embodiment of the video generation device of the present disclosure. In some possible implementations of the present disclosure, such as Figure 6 As shown, the first video generation module 310 includes: The video acquisition submodule 311 is used to acquire a preset video, wherein the preset video includes a plurality of frames of completely black images; The video generation submodule 312 is used to replace the first frame image of the preset video with the target image to obtain the first video.

[0086] In some possible implementations of the present disclosure, the adapter network module in the embodiment of the present disclosure includes n first network layers, the input layer of the video generation network module includes n second network layers, and the n first network layers of the adapter network module correspond one-to-one to the n second network layers of the video generation network module respectively; The video generation network module in the video generation model processes the multiple first vectors, the preset noise information and the video description information to obtain the second video, further used for: In the denoising process of each time step of the video generation network module, based on the preset noise information and the video description information, the n first vectors output by the n first network layers are respectively calculated with the corresponding n second vectors output by the n second network layers to obtain the second video.

[0087] In some possible implementations of the present disclosure, in the denoising process of each time step of the video generation network module in the embodiment of the present disclosure, based on the preset noise information and the video description information, the n first vectors output by the n first network layers are respectively calculated with the corresponding n second vectors output by the n second network layers to obtain the second video, which is further used to: For the denoising process at each time step, the first second network layer processes the preset noise information based on the video description information to obtain a second vector output by the first second network layer; For the second second network layer to the nth second network layer, the second network layer processes the third vector based on the video description information to obtain a second vector output by the second network layer, wherein the third vector is obtained by adding a second vector output by a second network layer immediately preceding the second network layer to a first vector output by a first network layer corresponding to the immediately preceding second network layer; The second video is determined based on the second vector output by the nth second network layer in the denoising process of the last time step.

[0088] In some possible implementations of the present disclosure, the video generating device in the embodiment of the present disclosure further includes: The training data acquisition module 330 is used to acquire a model to be trained and sample data, wherein the model to be trained includes a video generation network module and an adapter network module to be trained, and the sample data includes multiple sample videos, and label videos and label video description information corresponding to each sample video, wherein the sample videos include one frame of image with content and multiple frames of completely black images; The model training module 340 is used for inputting the sample videos, the label videos corresponding to the sample videos, and the label video description information into the model to be trained, and the adapter network module to be trained processes the sample videos to obtain the fourth vectors output by each first network layer in the adapter network module to be trained, and the video generation network module processes the plurality of fourth vectors and the label video description information to obtain the predicted video; The parameter adjustment module 350 is used to adjust the parameters of the adapter network module to be trained according to the predicted video and the label video corresponding to each sample video until the preset training end condition is met, and the video generation model is obtained from the model to be trained.

[0089] In some possible implementations of the present disclosure, the video generation network module in the embodiment of the present disclosure processes the plurality of fourth vectors and the label video description information, and is further used to: For the denoising process at each time step, the first second network layer in the video generation network module processes the label video description information to obtain a fifth vector output by the first second network layer; For the second second network layer to the nth second network layer in the video generation network module, the second network layer processes the label video description information and the sixth vector to obtain a fifth vector output by the second network layer, and the sixth vector is obtained by adding the fifth vector output by the second network layer above the second network layer and the fourth vector output by the first network layer corresponding to the above second network layer; The predicted video is determined based on the fifth vector output by the nth second network layer in the denoising process of the last time step.

[0090] The video generating device of the embodiment of the present disclosure corresponds to the embodiment of the video generating method of the present disclosure, and the relevant contents can be referenced to each other and will not be repeated here.

[0091] The beneficial technical effects corresponding to the exemplary embodiments of the video generating device of the disclosed embodiments can be found in the corresponding beneficial technical effects of the above-mentioned corresponding exemplary method part, which will not be repeated here.

[0092] In addition, an embodiment of the present disclosure further provides an electronic device, including: Memory for storing computer programs; The processor is used to execute the computer program stored in the memory, and when the computer program is executed, the video generation method described in any of the above embodiments of the present disclosure is implemented.

[0093] Figure 7 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed in the present invention. Figure 7 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.

[0094] like Figure 7 As shown, the electronic device includes one or more processors and memory.

[0095] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0096] The memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the video generation method of each embodiment of the present disclosure described above and / or other desired functions.

[0097] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0098] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0099] The output device can output various information to the outside, including the determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0100] Of course, to simplify, Figure 7 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.

[0101] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the video generation method according to various embodiments of the present disclosure described in the above part of this specification.

[0102] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0103] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the video generation method according to various embodiments of the present disclosure described in the above part of this specification.

[0104] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0105] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0106] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0107] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0108] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0109] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0110] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0111] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0112] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A video generation method, characterized in that: include: Based on the target image, a first video is generated, wherein the first video includes the target image and a plurality of frames of all-black images, and the first frame image of the first video is the target image; The preset noise information and conditional guidance information are input into a pre-trained video generation model to obtain a second video, wherein the conditional guidance information includes the first video, the first frame of the second video is the target image, and the duration of the second video is the same as that of the first video.

2. The method according to claim 1, characterized in that The conditional guidance information also includes video description information; The step of inputting the preset noise information and the conditional guidance information into a pre-trained video generation model to obtain a second video includes: The first video is processed by an adapter network module in the video generation model to obtain a first vector output by each first network layer in the adapter network module; The video generation network module in the video generation model processes the multiple first vectors, the preset noise information and the video description information to obtain the second video.

3. The method according to claim 1, characterized in that The step of generating a first video based on the target image includes: Acquire a preset video, wherein the preset video includes a plurality of frames of completely black images; The first frame image of the preset video is replaced with the target image to obtain the first video.

4. The method according to claim 2, characterized in that: The adapter network module includes n first network layers, the input layer of the video generation network module includes n second network layers, and the n first network layers of the adapter network module correspond one-to-one to the n second network layers of the video generation network module respectively; The step of processing the plurality of first vectors, the preset noise information and the video description information by the video generation network module in the video generation model to obtain the second video includes: In the denoising process of each time step of the video generation network module, based on the preset noise information and the video description information, the n first vectors output by the n first network layers are respectively calculated with the corresponding n second vectors output by the n second network layers to obtain the second video.

5. The method according to claim 4, characterized in that In the denoising process of each time step of the video generation network module, based on the preset noise information and the video description information, the n first vectors output by the n first network layers are respectively calculated with the corresponding n second vectors output by the n second network layers to obtain the second video, including: For the denoising process at each time step, the first second network layer processes the preset noise information based on the video description information to obtain a second vector output by the first second network layer; For the second second network layer to the nth second network layer, the second network layer processes the third vector based on the video description information to obtain a second vector output by the second network layer, wherein the third vector is obtained by adding a second vector output by a second network layer immediately preceding the second network layer to a first vector output by a first network layer corresponding to the immediately preceding second network layer; The second video is determined based on the second vector output by the nth second network layer in the denoising process of the last time step.

6. The method according to claim 1, characterized in that The video generation model is obtained in the following way: Acquire a model to be trained and sample data, wherein the model to be trained includes a video generation network module and an adapter network module to be trained, and the sample data includes multiple sample videos, and label videos and label video description information corresponding to each sample video, and for the multiple sample videos, the sample videos include one frame of image with content and multiple frames of completely black images; For the multiple sample videos, the sample videos, the label videos corresponding to the sample videos, and the label video description information are input into the model to be trained, the adapter network module to be trained processes the sample videos to obtain fourth vectors output by each first network layer in the adapter network module to be trained, and the video generation network module processes the multiple fourth vectors and the label video description information to obtain a predicted video; The parameters of the adapter network module to be trained are adjusted according to the predicted video and the label video corresponding to each sample video until the preset training end condition is met, and the video generation model is obtained from the model to be trained.

7. The method according to claim 6, characterized in that The processing of the plurality of fourth vectors and the label video description information by the video generation network module includes: For the denoising process at each time step, the first second network layer in the video generation network module processes the label video description information to obtain a fifth vector output by the first second network layer; For the second second network layer to the nth second network layer in the video generation network module, the second network layer processes the label video description information and the sixth vector to obtain a fifth vector output by the second network layer, and the sixth vector is obtained by adding the fifth vector output by the second network layer above the second network layer and the fourth vector output by the first network layer corresponding to the above second network layer; The predicted video is determined based on the fifth vector output by the nth second network layer in the denoising process of the last time step.

8. A video generating device, characterized in that: include: A first video generating module, configured to generate a first video based on a target image, wherein the first video includes the target image and a plurality of frames of all-black images, and the first frame image of the first video is the target image; The second video generation module is used to input preset noise information and conditional guidance information into a pre-trained video generation model to obtain a second video, wherein the conditional guidance information includes the first video, the first frame of the second video is the target image, and the duration of the second video is the same as that of the first video.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program stored in the memory, and when the computer program is executed, the video generation method described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video generation method described in any one of claims 1 to 7 is implemented.

Images