Method and device for generating video by using image, equipment and medium

Abstract

The embodiment of the invention discloses a method and device for generating a video by using an image, equipment and a medium, and the method comprises the steps: obtaining conditional guidance information which comprises an input image; the preset noise information and the condition guidance information are input into a pre-trained video generation model to obtain a target video, the target video comprises an input image, an adapter network module in the video generation model processes the input image to obtain a plurality of first vectors, and the plurality of first vectors are sent to the video generation model; and processing the plurality of first vectors and the preset noise information by a video generation network module in the video generation model to obtain a target video. Therefore, the video generation network module can consider the input image when generating the target video, so that the generated target video comprises the input image, and the user experience is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of video generation technology, and in particular to a method, device, equipment, and medium for generating a video using an image. Background Art

[0002] In recent years, with the rapid development of Vincent video models, Vincent video models have made great progress in the field of AIGC (Artificial Intelligence Generated Content). In related technologies, Vincent video models usually generate corresponding videos based on text descriptions. In the process of video generation, it is sometimes necessary to generate videos based on specified images. However, existing Vincent video models usually generate corresponding videos based on text descriptions, and cannot generate corresponding videos based on images, thus failing to meet user needs and affecting user experience. Summary of the invention

[0003] In order to solve the above technical problems, the embodiments of the present disclosure provide a method, an apparatus, a device, and a medium for generating a video using an image.

[0004] One aspect of an embodiment of the present disclosure provides a method for generating a video using an image, comprising: obtaining conditional guidance information, the conditional guidance information including an input image; inputting preset noise information and the conditional guidance information into a pre-trained video generation model to obtain a target video, the target video including the input image; wherein an adapter network module in the video generation model processes the input image to obtain a first vector output by each first network layer in the adapter network module, and a video generation network module in the video generation model processes multiple first vectors and the preset noise information to obtain the target video.

[0005] Another aspect of the embodiments of the present disclosure provides a device for generating videos using images, including: an information acquisition module, used to obtain conditional guidance information, the conditional guidance information including an input image; a video generation module, used to input preset noise information and the conditional guidance information into a pre-trained video generation model to obtain a target video, the target video including the input image; wherein the input image is processed by an adapter network module in the video generation model to obtain a first vector output by each first network layer in the adapter network module, and the video generation network module in the video generation model processes multiple first vectors and the preset noise information to obtain the target video.

[0006] Another aspect of the embodiments of the present disclosure provides an electronic device, comprising: a memory for storing a computer program; a processor for executing the computer program stored in the memory, and when the computer program is executed, the above-mentioned method of generating a video using an image is implemented.

[0007] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method of generating a video using an image is implemented.

[0008] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, including computer program instructions, which, when executed by a processor, implement the above-mentioned method of generating a video using an image.

[0009] In the disclosed embodiment, the video generation model is designed to include an adapter network module and a video generation network module. The input image is processed by the adapter network module to obtain multiple first vectors. The multiple first vectors and preset noise information are then processed by the video generation network module, so that the video generation network module can take the input image into consideration when generating a target video, so that the generated target video includes the input image, thereby improving the user experience.

[0010] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0012] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which: Figure 1 is a flowchart of a method for generating a video using an image provided by an exemplary embodiment of the present disclosure; Figure 2 is a flowchart of step S100 provided by an exemplary embodiment of the present disclosure; Figure 3 is a schematic diagram of an input layer in an adapter network module and a video generation network module provided by an exemplary embodiment of the present disclosure; Figure 4 is a schematic diagram of a process of generating a video using an image provided by another exemplary embodiment of the present disclosure; Figure 5 A schematic diagram of the structure of an embodiment of the apparatus for generating a video using an image disclosed herein; Figure 6 A schematic diagram of the structure of another embodiment of the apparatus for generating a video using an image disclosed herein; Figure 7 The figure is a schematic diagram of the structure of an application embodiment of the electronic device disclosed in the present invention. DETAILED DESCRIPTION

[0013] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0014] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0015] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0016] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0017] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0018] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0019] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0020] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0021] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0022] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0023] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0024] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0025] In the process of implementing the present disclosure, the inventors found that in practical applications, in the process of video generation, it is sometimes necessary to generate a video related to a specific image based on a specific image. For example, when creating an animation video, an animation video related to an image can be created based on an image. However, the existing text-generated video model generates a video based on a text description, and cannot generate a video related to an image based on an image. This makes the text-generated image model unable to meet the needs of users and affects the user experience.

[0026] Figure 1 FIG. 1 is a flow chart of a method for generating a video using an image provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the following steps are included: Step S100, obtaining conditional guidance information.

[0027] The conditional guidance information includes an input image, which can be used to generate a video. The input image can be, for example, a grayscale image or a true color image.

[0028] Step S110, inputting preset noise information and conditional guidance information into a pre-trained video generation model to obtain a target video.

[0029] Among them, the target video includes an input image. The conditional guidance information includes multiple separate input images, which are input into a pre-trained video generation model according to the preset order of the multiple input images to obtain a target video including multiple input images, and the timing order of the multiple input images in the target video is the same as the preset order. Exemplarily, the conditional guidance information includes input image A, input image B, and input image C, wherein the preset order of the three input images is: input image A, input image C, input image B, then the conditional guidance information is input into the pre-trained video generation model to obtain the target video, and the order in which the three input images appear in the target video is also input image A, input image C, input image B. Through this embodiment, the content of the target video finally generated can be controlled based on business needs or the preset order of input images set by the user, which not only makes the use of video generation technology more interesting, but also meets more user needs.

[0030] In step S110, the adapter network module in the video generation model processes the input image to obtain the first vectors output by each first network layer in the adapter network module, and the video generation network module in the video generation model processes multiple first vectors and preset noise information to obtain the target video.

[0031] The video generation model may include: an adapter network module and a video generation network module. The adapter network module and the video generation network module may be pre-trained deep learning models. In this embodiment, the pre-trained video generation model is obtained by joint training of the video generation network module and the adapter network module to be trained. Optionally, in the process of joint training based on the video generation network module and the adapter network module to be trained, the video generation network module and the adapter network module to be trained are first obtained, wherein the network layer structure of the adapter network module to be trained is the same as the network structure of the input layer of the video generation network module, and each network layer of the adapter network module to be trained corresponds to the network layer of the input layer of the video generation network module, that is, the number of layers of the network layer of the adapter network module to be trained is the same as the number of layers of the network layer of the input layer of the video generation network module, and each network layer of the adapter network module to be trained has the same network parameters as the network layer of the corresponding input layer of the video generation network module. Secondly, the network parameters of each network layer of the adapter network module to be trained are adjusted through the loss function of the video generation network module, so as to obtain the adapter network module in the video generation model.

[0032] In this embodiment, the network layer in the adapter network module can be referred to as a first network layer, and the adapter network module can include multiple first network layers. The input layer of the video generation network module includes multiple second network layers, and the multiple first network layers of the adapter network module correspond to the multiple second network layers of the video generation network module in a one-to-one mapping manner, wherein the output first vector of each first network layer is used to be superimposed with the output second vector of the corresponding second network layer to serve as the input vector of the next second network layer.

[0033] Furthermore, in some embodiments, the structure of each first network layer is the same as that of the corresponding second network layer, and thus the dimension of the output vector of each first network layer is the same as that of the output vector of the corresponding second network layer, thereby facilitating the same-dimensional superposition of the output vector of the first network layer and the output vector of the corresponding second network layer, so as to achieve comprehensive fine-tuning of the video generation network module through the adapter network module.

[0034] Exemplarily, the adapter network module may adopt an adapter model; the video generation network module may be a diffusion model structure. The diffusion model is a generation model based on a deep learning algorithm, which generates high-quality content through a gradual denoising and denoising process, performs well in generating images, videos, audios and other high-dimensional data, and has strong performance in generation tasks. In the disclosed embodiment, a video generation network module can be obtained by training the diffusion model. The video generation network module is universal in the field of video generation, but the matching effect between the content of the generated video and the input is not necessarily very good. In the disclosed embodiment, by combining the adapter network module with the video generation network module, it is possible to efficiently generate a higher quality video when the input includes an image without changing the universality of the video generation network module, that is, the matching effect between the content of the generated video and the input content is good.

[0035] The preset noise information may include Gaussian noise. Exemplarily, a Gaussian noise matrix may be randomly generated in advance, and the Gaussian noise matrix may be determined as the preset noise information.

[0036] In one embodiment, the video generation network module is a diffusion model structure, and the diffusion model mainly includes two processes: noise addition and denoising: in the noise addition process, the original data is converted into an expression form of latent space through a variational autoencoder, which can make the model calculation more efficient, and the noisy content is obtained after T (T≥1) steps of noise addition; in the denoising process, noise is expected for the given noisy content, and conditional guidance information is received during the process, which includes the input image, and finally restored from the latent space to the pixel space through the variational autoencoder to obtain a video that meets the expectations. In this embodiment, the input image can be better referenced in the video generation process, so that the generated video and the input content match better.

[0037] In one embodiment, when preset noise information and an input image are input into a video generation model, the adapter network module processes the input image and records the first vector output by each first network layer; then the video generation network module processes the first vector output by each first network layer and the preset noise information, and outputs a target video including the input image.

[0038] In the disclosed embodiment, the video generation model is designed to include an adapter network module and a video generation network module. The input image is processed by the adapter network module to obtain multiple first vectors. The multiple first vectors and preset noise information are then processed by the video generation network module, so that the video generation network module can take the input image into consideration when generating a target video, so that the generated target video includes the input image, thereby improving the user experience.

[0039] In some optional implementations, in the embodiments of the present disclosure, the conditional guidance information also includes video description information.

[0040] The video description information may be used to describe the video. The video description information may include, for example, text information describing the video content.

[0041] Correspondingly, step S110 in the embodiment of the present disclosure may include: the adapter network module in the video generation model processes the input image to obtain the first vector output by each first network layer in the adapter network module; the video generation network module in the video generation model processes multiple first vectors, preset noise information and video description information to obtain the target video.

[0042] In the disclosed embodiment, the video description information and the input image are used together as conditional guiding information, so that the video generation network module takes both the input image and the video description information into consideration when generating the target video. Not only does the generated target video include the input image, but the video content also conforms to the video description information, thereby improving the user experience.

[0043] Figure 2 is a flow chart of step S100 provided by an exemplary embodiment of the present disclosure. In some optional implementations, such as Figure 2 Said step S100 may include the following steps: Step S101, acquiring an image sequence.

[0044] The image sequence includes multiple frames of images.

[0045] Step S102: extracting an input image from a preset position in an image sequence based on the preset position information.

[0046] The preset position information may include the extraction position of the image sequence. For example, the preset position information may include the p-th frame image, etc., where p is an integer greater than or equal to 1, indicating that the p-th frame image in the image sequence is extracted as the input image.

[0047] In one implementation, the input image may also be subjected to denoising, sharpening, smoothing and other processing to improve the image quality of the input image.

[0048] Step S103, determining conditional guidance information based on the input image.

[0049] Among them, the input image can be used as conditional guidance information.

[0050] In the embodiment of the present disclosure, the input image is extracted from multiple frames of images in the image sequence according to preset position information, thereby enabling rapid acquisition of the input image.

[0051] In some optional embodiments, in the presently disclosed embodiment, the adapter network module includes n first network layers, the input layer of the video generation network module includes n second network layers, and the n first network layers of the adapter network module correspond one-to-one to the n second network layers of the video generation network module, respectively.

[0052] In this embodiment, the network layer in the video generation network module may be referred to as a second network layer, and any second network layer corresponds to a first network layer.

[0053] The video generation network module includes an input layer and an output layer. At each time step, the video generation network module adds or removes noise for one time step through noise prediction of the input layer and the output layer, and then obtains the latent space expression of the final generated video after performing noise prediction for T (T≥1) time steps. In this embodiment, the network layer in the input layer of the video generation network module can be called a second network layer, and any second network layer corresponds to a first network layer.

[0054] Among them, the input layer of the video generation network module includes n second network layers, where n is an integer greater than or equal to 1. The network structure of the input layer is the same as the network structure of the adapter network module, that is, the adapter network module also includes n first network layers, and the first network layers with the same sequence number correspond to the second network layers. For example, the first first network layer corresponds to the first second network layer, and so on, the nth first network layer corresponds to the nth second network layer. The shape (dimension) of the data output by the first network layer is the same as the shape of the data output by the second network layer.

[0055] Exemplarily, both the first network layer and the second network layer can adopt the UNet network. UNet can denoise random noise to achieve the generation from noise to video. In this embodiment, the input layer of the video generation network module is used to gradually downsample the input noise, extract features and reduce the spatial resolution of the image. The output layer of the video generation network module gradually restores the spatial resolution of the video through deconvolution (or transposed convolution) and upsampling operations; through the jump connection structure of the UNet network, the output layer can combine the low-level features in the input layer (such as edge, texture and other detail information) with high-level features (such as semantic information of video description information), and the fusion of features helps to reconstruct the details of the video; finally, the target video with the same resolution as the input video is output. In the present disclosure, based on the multi-layer second network layer structure in the input layer of the video generation network module, an independent adapter network module is designed to fine-tune the vector generated by the input layer downsampling process. That is to say, the adapter network module adapts to new tasks by inserting additional parameters into the pre-trained video generation network module, rather than directly adjusting the parameters of the entire video generation network module (the parameters of the video generation network module are very large, such as billions or tens of billions of parameters). This can not only reduce the number of parameters required for fine-tuning, but also reduce the demand for computing resources during training and speed up training. At the same time, when the video generation network module performs a specific task, the trained adapter network module is used to fine-tune the output vectors of each layer of the output layer downsampled, so that the video generation network module does not need to be trained on a large scale as a whole, and the specific task can be completed efficiently and accurately. It can be seen that through the combination and joint training of the adapter network module and the video generation network module, it is possible to quickly generate high-quality videos based on the target image based on AIGC technology, and it will not affect the versatility of the video generation network module.

[0056] Correspondingly, in the embodiment of the present disclosure, the video generation network module can process multiple first vectors, preset noise information and video description information in the following manner: in the denoising process of each time step of the video generation network module, based on the preset noise information and the video description information, the n first vectors output by the n first network layers are respectively calculated with the n second vectors output by the corresponding n second network layers to obtain the target video.

[0057] In the embodiment of the present disclosure, the adapter network module is set to include n first network layers, and the video network module is set to include n second network layers, so that each second network layer can process the first vector output by the corresponding first network layer, thereby enabling the video network module to comprehensively consider the input image when generating a target video, so that the generated target video includes the input image.

[0058] In some optional implementations, in the embodiment of the present disclosure, using n second network layers to process the multiple first vectors, the preset noise information, and the video description information may include: For the denoising process at each time step, the first second network layer processes the preset noise information based on the video description information to obtain the second vector output by the first second network layer; for the second second network layer to the nth second network layer, the second network layer processes the third vector based on the video description information to obtain the second vector output by the second network layer, and the target video is determined based on the second vector output by the nth second network layer in the denoising process at the last time step.

[0059] The third vector is obtained by adding a second vector output by a second network layer previous to the second network layer to a first vector output by a first network layer corresponding to the previous second network layer. In one embodiment, the target video is determined based on the second vector output by the nth second network layer and the first vector output by the nth first network layer.

[0060] For example, Figure 3 FIG. 1 is a schematic diagram of an input layer in an adapter network module and a video generation network module provided by an exemplary embodiment of the present disclosure. Figure 3 As shown, a variational auto-encoder (VAE) can be used to encode the input image, and then the encoded input image is input into the adapter network module, and the first vector output by each first network layer in the adapter network module is recorded; The VAE can be used to encode the video description information and the preset noise information. For the denoising process at each time step, the encoded video description information and the preset noise information are input into the first second network layer to obtain a second vector output by the first second network layer. The second vector is added to the first vector output by the first first network layer to obtain a third vector. The third vector can be used as input data of the second second network layer. A third vector obtained by adding the second vector output by the i-1th second network layer to the first vector output by the i-1th first network layer and the encoded video description information are input into the i-th second network layer to obtain the second vector output by the i-th second network layer, and the second vector is added to the first vector output by the i-th first network layer to obtain a third vector, which can be used as input data of the i+1th second network layer, wherein 2≤i≤n; The result of adding the second vector output by the nth second network layer in the denoising process of the last time step to the first vector output by the nth first network layer is input into the VAE for decoding processing to obtain the target video.

[0061] In the embodiment of the present disclosure, the third vector obtained by adding the second vector output by each second network layer to the first vector output by the corresponding first network layer is used as the input data of the next second network layer, so that the video generation network module can take the input image into consideration when generating the target video, thereby ensuring that the generated target video can include the input image and the video content corresponds to the video description information.

[0062] Figure 4 FIG. 1 is a flow chart of generating a video using an image provided by another exemplary embodiment of the present disclosure. In some optional implementations, such as Figure 4 As shown, the video generation model can be obtained as follows: Step S200, obtaining the model to be trained and sample data.

[0063] The model to be trained includes: a video generation network module and an adapter network module to be trained. The sample data includes: a plurality of sample images, and a label video and label video description information corresponding to each sample image.

[0064] In some optional implementations, the sample image may be obtained in the following manner: a plurality of tagged videos are acquired, and for the plurality of tagged videos, preset frame images in the tagged videos are extracted as sample images corresponding to the tagged videos.

[0065] The preset frame image may be, for example, the first frame image of the labeled video. For example, a complete video with a long duration may be cut into multiple videos of the same duration as labeled videos. For each labeled video, the first frame image (first frame image) in the labeled video is extracted as a sample image.

[0066] In an optional embodiment, the labeled video description information can be obtained in the following manner: for each labeled video, the labeled video is input into a visual language model (VLMs), the visual language model outputs the video description information corresponding to the labeled video, and the video description information is used as the labeled video description information corresponding to the sample image obtained based on the labeled video.

[0067] Step S210, for multiple sample images, the sample image, the label video corresponding to the sample image and the label video description information are input into the model to be trained, the sample image is processed by the adapter network module to be trained to obtain the fourth vector output by each first network layer in the adapter network module to be trained, and the video generation network module processes the multiple fourth vectors and the label video description information to obtain the predicted video.

[0068] The adapter network module to be trained includes a plurality of first network layers. Each sample image, and the label video and label video description information corresponding to each sample image can be sequentially input into the model to be trained, and the model to be trained sequentially outputs the predicted video corresponding to each sample image.

[0069] Step S220, adjusting the parameters of the adapter network module to be trained according to the predicted video and the label video corresponding to each sample image until the preset training end condition is met, and obtaining the video generation model from the model to be trained.

[0070] In the model training stage, the parameters of the video generation network module can be frozen, and the parameters of the input layer of the video generation network module can be used as the initialization parameters of the adapter network module to be trained.

[0071] Specifically, the training process is to iteratively adjust the network parameters of the adapter network module, and the network parameters of each layer of the video generation network module remain unchanged. Based on this, according to the loss function of the video generation network module, the network parameters of each first network layer of the adapter network module are adjusted by back propagation. In this way, according to S210 to S220, the loop iteration is repeated until the loss function converges, and a trained adapter network module can be obtained, and each first network layer has a trained network parameter.

[0072] The training method disclosed in the present invention can form sample data from sample images and label text information, and due to the sparse content characteristics of the sample images, an external adapter network module is used during the downsampling process of the pre-trained video generation network module to implement training of the adapter network module, thereby efficiently and at low cost obtaining a video generation model that can restore or reconstruct video content.

[0073] In one embodiment, the loss function value can be determined based on the difference between the label video and the predicted video corresponding to each sample image using a preset loss function. The preset loss function may include, but is not limited to, a cross entropy error function or a mean square error function. The operation of determining the loss function value may be iteratively performed, and the parameters of the adapter network module to be trained may be iteratively adjusted to continuously reduce the loss function value until the loss function value converges, and it is determined that the preset training end condition is met, and the training of the model to be trained is completed, and the trained model to be trained is used as the video generation model.

[0074] Parameter optimizers such as Stochastic Gradient Descent (SGD), Adaptive Gradient Algorithm (Adagrad), Adaptive Moment Estimation (Adam), Root Mean Square Prop (RMSprop) can be used to adjust the parameters of the adapter network module to be trained. For example, the parameter optimizer can be used to calculate the gradient of the parameters of the adapter network module to be trained, and the parameters can be adjusted along the direction of the gradient, where the gradient represents the direction in which the loss function value is reduced the most, and the operation of iteratively determining the loss function value is performed until the loss function value no longer decreases, and the training of the model to be trained is completed, and a video generation model is obtained.

[0075] In the disclosed embodiment, a model to be trained, including a video generation network module and an adapter network module to be trained, is trained using multiple sample images, as well as label videos and label video description information corresponding to each sample image. During the model training process, only the parameters of the adapter network module to be trained need to be adjusted, which not only reduces the difficulty of model training and improves the efficiency of model training, but also enables the trained video generation model to generate a video including the image based on the image.

[0076] In some optional embodiments, in the embodiments of the present disclosure, the video generation network module can process multiple fourth vectors and label video description information in the following manner: the first second network layer in the video generation network module processes the label video description information to obtain the fifth vector output by the first second network layer; for the second second network layer to the nth second network layer in the video generation network module, the second network layer processes the label video description information and the sixth vector to obtain the fifth vector output by the second network layer, and determines the predicted video based on the fifth vector output by the nth second network layer.

[0077] The sixth vector is obtained by adding a fifth vector output by a second network layer preceding the second network layer and a fourth vector output by a first network layer corresponding to the second network layer preceding the second network layer.

[0078] In one embodiment, the network structure of the adapter network module to be trained is the same as the network structure of the input layer of the video generation network module, that is, the adapter network module to be trained also includes n first network layers. In the model to be trained, the first network layer with the same sequence number corresponds to the second network layer, and the shape of the data output by the first network layer is the same as the shape of the data output by the second network layer. For example, the first network layer and the second network layer can adopt the UNet network.

[0079] In the forward propagation process of the video generation network module, the label video description information is input into the first second network layer, the first second network layer outputs a fifth vector, the fifth vector is added to the fourth vector output by the first first network layer to obtain a sixth vector, and the sixth vector is used as input data of the second second network layer; A sixth vector obtained by adding the fifth vector output by the i-1th second network layer and the fourth vector output by the i-1th first network layer and the label video description information are input into the i-th second network layer, and a fifth vector is output by the i-th second network layer, and the fifth vector is added to the fourth vector output by the i-th first network layer to obtain a sixth vector, and the sixth vector is used as input data of the i+1th second network layer; The predicted video is determined according to the result of adding the fifth vector output by the nth second network layer and the fourth vector output by the nth first network layer.

[0080] In the disclosed embodiment, the sixth vector obtained by adding the fifth vector output by each second network layer and the fourth vector output by the corresponding first network layer is used as the input data of the next second network layer, so that the video generation network module can take into account the data output by the adapter network module to be trained when generating a predicted video, thereby facilitating better learning of the adapter network module to be trained.

[0081] Figure 5 FIG. 1 is a schematic diagram of a structure of an embodiment of a device for generating a video using an image according to the present disclosure. Figure 5 As shown, the apparatus for generating a video using an image in this embodiment may include: an information acquisition module 300 and a video generation module 310 .

[0082] An information acquisition module 300 is used to acquire conditional guidance information, wherein the conditional guidance information includes an input image; The video generation module 310 is used to input the preset noise information and the conditional guidance information into a pre-trained video generation model to obtain a target video, wherein the target video includes the input image; wherein the input image is processed by an adapter network module in the video generation model to obtain a first vector output by each first network layer in the adapter network module, and the video generation network module in the video generation model processes multiple first vectors and the preset noise information to obtain the target video.

[0083] In some possible implementations of the present disclosure, the conditional guidance information in the embodiments of the present disclosure further includes video description information; The preset noise information and the conditional guidance information are input into a pre-trained video generation model to obtain a target video, which is further used for: The adapter network module in the video generation model processes the input image to obtain the first vector output by each first network layer in the adapter network module, and the video generation network module in the video generation model processes multiple first vectors, the preset noise information and the video description information to obtain the target video.

[0084] Figure 6 FIG. 1 is a schematic diagram of another embodiment of the apparatus for generating a video using an image in the present disclosure. In some possible implementations of the present disclosure, for example, Figure 6 As shown, the information acquisition module 300 includes: The image sequence acquisition submodule 301 is used to acquire an image sequence; An image extraction submodule 302 is used to extract the input image from the image sequence based on preset position information; The generating submodule 303 is used to determine the conditional guidance information based on the input image.

[0085] In some possible implementations of the present disclosure, the adapter network module in the embodiment of the present disclosure includes n first network layers, the input layer of the video generation network module includes n second network layers, and the n first network layers of the adapter network module correspond one-to-one to the n second network layers of the video generation network module respectively; The processing of the plurality of first vectors, the preset noise information and the video description information by the video generation network module in the video generation model is further used to: In the denoising process of each time step of the video generation network module, based on the preset noise information and the video description information, the n first vectors output by the n first network layers are respectively calculated with the n second vectors output by the corresponding n second network layers to obtain the target video. In some possible implementations of the present disclosure, in the denoising process of each time step of the video generation network module in the embodiment of the present disclosure, based on the preset noise information and the video description information, the n first vectors output by the n first network layers are respectively calculated with the n second vectors output by the corresponding n second network layers to obtain the target video, further used for: For the denoising process at each time step, the first second network layer processes the preset noise information based on the video description information to obtain a second vector output by the first second network layer; For the second second network layer to the nth second network layer, the second network layer processes the third vector based on the video description information to obtain a second vector output by the second network layer, wherein the third vector is obtained by adding a second vector output by a second network layer immediately preceding the second network layer to a first vector output by a first network layer corresponding to the immediately preceding second network layer; The target video is determined based on the second vector output by the nth second network layer in the denoising process of the last time step.

[0086] In some possible implementations of the present disclosure, the apparatus for generating a video using an image in an embodiment of the present disclosure further includes: The training data acquisition module 320 is used to acquire a model to be trained and sample data, wherein the model to be trained includes a video generation network module and an adapter network module to be trained, and the sample data includes a plurality of sample images, and a label video and label video description information corresponding to each sample image; The model training module 330 is used for inputting the sample images, the label videos corresponding to the sample images, and the label video description information into the model to be trained, the adapter network module to be trained processes the sample images to obtain the fourth vectors output by each first network layer in the adapter network module to be trained, and the video generation network module processes the plurality of fourth vectors and the label video description information to obtain the predicted video; The parameter adjustment module 340 is used to adjust the parameters of the adapter network module to be trained according to the predicted video and the label video corresponding to each sample image until the preset training end condition is met, and the video generation model is obtained from the model to be trained.

[0087] In some possible implementations of the present disclosure, the video generation network module in the embodiment of the present disclosure processes the plurality of fourth vectors and the label video description information, and is further used to: For the denoising process at each time step, the first second network layer in the video generation network module processes the label video description information to obtain a fifth vector output by the first second network layer; For the second second network layer to the nth second network layer in the video generation network module, the second network layer processes the label video description information and the sixth vector to obtain a fifth vector output by the second network layer, and the sixth vector is obtained by adding the fifth vector output by the second network layer above the second network layer and the fourth vector output by the first network layer corresponding to the above second network layer; The predicted video is determined based on the fifth vector output by the nth second network layer in the denoising process of the last time step.

[0088] The device for generating a video using an image in an embodiment of the present disclosure corresponds to the embodiment of the method for generating a video using an image in the present disclosure, and the relevant contents can be referenced to each other and will not be repeated here.

[0089] The beneficial technical effects corresponding to the exemplary embodiment of the device for generating videos using images in the embodiments of the present disclosure can be found in the corresponding beneficial technical effects of the above-mentioned corresponding exemplary method part, which will not be repeated here.

[0090] In addition, an embodiment of the present disclosure further provides an electronic device, including: Memory for storing computer programs; The processor is used to execute the computer program stored in the memory, and when the computer program is executed, the method for generating a video using an image as described in any of the above embodiments of the present disclosure is implemented.

[0091] Figure 7 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed in the present invention. Figure 7 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.

[0092] like Figure 7 As shown, the electronic device includes one or more processors and memory.

[0093] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0094] The memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the method for generating a video using an image and / or other desired functions of the various embodiments of the present disclosure described above.

[0095] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0096] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0097] The output device can output various information to the outside, including the determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0098] Of course, to simplify, Figure 7 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.

[0099] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the method of generating videos using images according to various embodiments of the present disclosure described in the above part of this specification.

[0100] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0101] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the method for generating videos using images according to various embodiments of the present disclosure described in the above part of this specification.

[0102] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0103] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0104] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0105] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0106] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0107] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0108] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0109] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0110] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for generating a video using an image, characterized in that: include: Acquiring conditional guidance information, wherein the conditional guidance information includes an input image; The preset noise information and the conditional guidance information are input into a pre-trained video generation model to obtain a target video, wherein the target video includes the input image; wherein the input image is processed by an adapter network module in the video generation model to obtain a first vector output by each first network layer in the adapter network module, and the video generation network module in the video generation model processes multiple first vectors and the preset noise information to obtain the target video.

2. The method according to claim 1, characterized in that: The conditional guidance information also includes video description information; The step of inputting the preset noise information and the conditional guidance information into a pre-trained video generation model to obtain a target video includes: The adapter network module in the video generation model processes the input image to obtain the first vector output by each first network layer in the adapter network module, and the video generation network module in the video generation model processes multiple first vectors, the preset noise information and the video description information to obtain the target video.

3. The method according to claim 1 or 2, characterized in that: The conditional guidance information for obtaining video generation includes: Get image sequence; Extracting the input image from the image sequence based on preset position information; Based on the input image, the conditional guidance information is determined.

4. The method according to claim 2, characterized in that: The adapter network module includes n first network layers, the input layer of the video generation network module includes n second network layers, and the n first network layers of the adapter network module correspond one-to-one to the n second network layers of the video generation network module respectively; The processing of the plurality of first vectors, the preset noise information and the video description information by the video generation network module in the video generation model includes: In the denoising process of each time step of the video generation network module, based on the preset noise information and the video description information, the n first vectors output by the n first network layers are respectively calculated with the corresponding n second vectors output by the n second network layers to obtain the target video.

5. The method according to claim 4, characterized in that In the denoising process at each time step of the video generation network module, based on the preset noise information and the video description information, the n first vectors output by the n first network layers are respectively calculated with the corresponding n second vectors output by the n second network layers to obtain the target video, including: For the denoising process at each time step, the first second network layer processes the preset noise information based on the video description information to obtain a second vector output by the first second network layer; For the second second network layer to the nth second network layer, the second network layer processes the third vector based on the video description information to obtain a second vector output by the second network layer, wherein the third vector is obtained by adding a second vector output by a second network layer immediately preceding the second network layer to a first vector output by a first network layer corresponding to the immediately preceding second network layer; The target video is determined based on the second vector output by the nth second network layer in the denoising process of the last time step.

6. The method according to claim 1, characterized in that The video generation model is obtained in the following way: Acquire a model to be trained and sample data, wherein the model to be trained includes a video generation network module and an adapter network module to be trained, and the sample data includes a plurality of sample images, and a label video and label video description information corresponding to each sample image; For the multiple sample images, the sample images, the label videos and the label video description information corresponding to the sample images are input into the model to be trained, the adapter network module to be trained processes the sample images to obtain fourth vectors output by each first network layer in the adapter network module to be trained, and the video generation network module processes the multiple fourth vectors and the label video description information to obtain a predicted video; The parameters of the adapter network module to be trained are adjusted according to the predicted video and the label video corresponding to each sample image until the preset training end condition is met, and the video generation model is obtained from the model to be trained.

7. The method according to claim 6, characterized in that The processing of the plurality of fourth vectors and the label video description information by the video generation network module includes: The first second network layer in the video generation network module processes the label video description information to obtain a fifth vector output by the first second network layer; For the denoising process at each time step, for the second second network layer to the nth second network layer in the video generation network module, the second network layer processes the label video description information and the sixth vector to obtain a fifth vector output by the second network layer, and the sixth vector is obtained by adding the fifth vector output by the second network layer immediately preceding the second network layer to the fourth vector output by the first network layer corresponding to the immediately preceding second network layer; The predicted video is determined based on the fifth vector output by the nth second network layer in the denoising process of the last time step.

8. A device for generating a video using an image, characterized in that: include: An information acquisition module, used to acquire conditional guidance information, wherein the conditional guidance information includes an input image; A video generation module is used to input preset noise information and the conditional guidance information into a pre-trained video generation model to obtain a target video, wherein the target video includes the input image; wherein the input image is processed by an adapter network module in the video generation model to obtain a first vector output by each first network layer in the adapter network module, and the video generation network module in the video generation model processes multiple first vectors and the preset noise information to obtain the target video.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program stored in the memory, and when the computer program is executed, the method for generating a video using an image as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating a video using an image as described in any one of claims 1 to 7 is implemented.

Images