Video generation method and device, electronic equipment, storage medium and product

Abstract

The embodiment of the invention discloses a video generation method and device, electronic equipment, a storage medium and a product, and the method comprises the steps: determining the image features of a guide image and the text features of a prompt text, and carrying out the denoising processing of a preset number of time steps through a pre-trained video generation model based on the image features, the text features and preset noise, a target video is determined, and a cross attention module in the video generation model comprises a query feature mapping layer, a text key feature mapping layer, a text value feature mapping layer, an image key feature mapping layer and an image value feature mapping layer; and in the denoising processing of each time step, a cross attention module performs cross attention calculation on the image features, the text features and the preset noise to obtain cross attention output so as to determine the target video based on the denoising processing of each time step and the cross attention output of each time step. Therefore, the matching degree between the generated target video and the user requirement is improved, and the user experience is improved.

Description

Technical Field

[0001] The present disclosure relates to video generation technology, and in particular to a method, device, electronic device, storage medium, and product for video generation. Background Art

[0002] In recent years, the rapid development of video models has led to a surge in the field of AIGC (Artificial Intelligence Generated Content). In AIGC, it's sometimes necessary to create a video based on a specific image, with the first frame being a specific image and matching the content of that image. However, in practical applications, video generation models often generate the corresponding video from text and are unable to generate videos with a specific first frame. This reduces the match between the generated video and the user's desired video, impacting the user experience. Summary of the Invention

[0003] In order to solve the above technical problems, the embodiments of the present disclosure provide a method, device, electronic device, storage medium and product for video generation.

[0004] One aspect of an embodiment of the present disclosure provides a method for video generation, comprising: obtaining a guide image and prompt text corresponding to a target video, wherein the guide image is the first frame image of the target video; determining image features of the guide image and text features of the prompt text; performing denoising processing for a preset number of time steps using a pre-trained video generation model based on the image features, the text features and preset noise to determine the target video, wherein the video generation model includes a cross-attention module, the cross-attention module includes a query feature mapping layer, a text key feature mapping layer, a text value feature mapping layer, an image key feature mapping layer and an image value feature mapping layer, and in the denoising processing of each time step: the cross-attention module performs cross-attention calculation on the image features, the text features and the preset noise to obtain a cross-attention output, so as to determine the target video based on the denoising processing of each time step and the cross-attention output of each time step.

[0005] Another aspect of the embodiments of the present disclosure provides a device for generating a video from an image, comprising: a conditional acquisition module for acquiring a guide image and prompt text corresponding to a target video, wherein the guide image is the first frame image of the target video; a first feature extraction module for determining the image features of the guide image and the text features of the prompt text; a video generation module for performing denoising processing for a preset number of time steps using a pre-trained video generation model based on the image features, the text features and preset noise to determine the target video, wherein the video generation model includes a cross-attention module, the cross-attention module includes a query feature mapping layer, a text key feature mapping layer, a text value feature mapping layer, an image key feature mapping layer and an image value feature mapping layer, and in the denoising processing of each time step: the cross-attention module performs cross-attention calculation on the image features, the text features and the preset noise to obtain a cross-attention output, so as to determine the target video based on the denoising processing of each time step and the cross-attention output of each time step. Another aspect of the embodiments of the present disclosure provides an electronic device, including: a memory for storing a computer program; a processor for executing the computer program stored in the memory, and implementing the above method when the computer program is executed.

[0006] Another aspect of the embodiments of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above method when executed by a processor.

[0007] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, comprising computer program instructions, wherein the computer program instructions implement the above method when executed by a processor.

[0008] In the disclosed embodiment, by introducing the image features of the guide image during the target video generation process and configuring the cross-attention module in the video generation model to include a query feature mapping layer, a text key feature mapping layer, a text value feature mapping layer, an image key feature mapping layer, and an image value feature mapping layer, the guide image can be better focused on during the video generation process, and the guide image and the prompt text can be better integrated, thereby generating a target video that conforms to the prompt text, has the guide image as the first frame image, and has continuous video content. This improves the quality of the generated target video, ensures the match between the target video and user needs, and enhances the user experience.

[0009] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0011] The present disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which: Figure 1 is a flowchart of a method for generating a video provided by an exemplary embodiment of the present disclosure; Figure 2 This is a process diagram of a method for generating a video provided by an application example of the present disclosure; Figure 3 is a flowchart of a method for generating a video provided by another exemplary embodiment of the present disclosure; Figure 4 A schematic structural diagram of an embodiment of a device for generating video according to the present disclosure; Figure 5 The figure is a schematic structural diagram of an application embodiment of the electronic device disclosed herein. DETAILED DESCRIPTION

[0012] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure.

[0013] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, and do not represent any specific technical meanings, nor do they indicate a necessary logical order between them.

[0014] It should also be understood that in the embodiments of the present disclosure, “a plurality of” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0015] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0016] In addition, the term "and / or" in this disclosure is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the related objects are in an "or" relationship.

[0017] It should also be understood that the description of the various embodiments in this disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced with each other. For the sake of brevity, they will not be described one by one.

[0018] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0019] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0020] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0021] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0022] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate in conjunction with numerous other general-purpose or specialized computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with terminal devices, computer systems, servers, and other electronic devices include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above systems, among others.

[0023] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system-executable instructions (such as program modules) executed by the computer system. Generally, program modules can include routines, programs, object programs, components, logic, data structures, and the like that perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked via a communications network. In distributed cloud computing environments, program modules can be located on local or remote computer system storage media, including storage devices.

[0024] In the process of implementing the present disclosure, the inventors discovered that in practical applications, some video generation scenarios require the generation of videos whose first frame image is a specific image. For example, a video creator wants to animate a photo, or an animation creator needs to generate a complete video based on a drawn image. However, existing video models are unable to generate videos whose first frame image is a specific image. This results in a low degree of match between the videos generated by the text generation video model and the videos required by users, affecting the user experience.

[0025] Figure 1 FIG. 1 is a flow chart of a method for generating a video provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the video generation method may include the following steps: Step S100: Obtain a guide image and prompt text corresponding to a target video.

[0026] The guide image is the first frame of the target video. In one embodiment, the guide image can be used to generate the target video. The guide image can be, for example, a photograph or a work of art. The prompt text includes content information indicating the target video, such as element information, shot information, composition information, style information, or technical parameters such as frame rate and resolution. Element information refers to information about objects that are desired to appear or not appear in the target video. The prompt text can include content description information for the portion connecting to the guide image, content description information for the guide image, or content description information for a portion of the target video's video frames. The portion of the target video's video frames can be the last frame of the target video or a video frame corresponding to a specified position and duration in the target video. For example, if the guide image is "a girl in a skirt standing on a wide meadow," the text information may include: a girl in a skirt running on a wide meadow, slowly turning back while running, with a happy smile on her face. This prompt text can better guide the video generation model to generate the video content that the user expects.

[0027] Step S110 : determining the image features of the guide image and the text features of the prompt text.

[0028] Image features are vector representations extracted from the guide image that characterize its visual content, spatial structure, and semantic information. Text features are vector representations extracted from the prompt text that indicate the content of the target video. Optionally, feature extraction of the guide image is performed based on a pre-trained deep learning model to obtain image features. Deep learning models can employ convolutional neural networks (CNNs) or recurrent neural networks (RNNs). A text encoder is used to extract features from the prompt text to obtain its text features.

[0029] Step S120 , based on the image features, text features and preset noise, a pre-trained video generation model is used to perform denoising processing for a preset number of time steps to determine the target video.

[0030] Among them, the video generation model is a diffusion model based on the Transformer structure (DiffusionTransformer, DiT. Compared with the traditional deep neural network model structure, the video generation model in this embodiment has stronger cross-modal information alignment capabilities and complex semantic control capabilities in video generation tasks. Specifically, the video generation model includes a cross-attention module, which includes a query feature mapping layer, a text key feature mapping layer, a text value feature mapping layer, an image key feature mapping layer, and an image value feature mapping layer. In the denoising process of each time step: the cross-attention module performs cross-attention calculations on image features, text features, and preset noise to obtain a cross-attention output, so as to determine the target video based on the denoising process of each time step and the cross-attention output of each time step. Among them, the cross-attention module is used to perform cross-attention calculations on data. The query feature mapping layer, the text key feature mapping layer, the text value feature mapping layer, the image key feature mapping layer, and the image value feature mapping layer are used to perform feature mapping processing. Through the video generation model with the above model structure, cross-model fusion between image features and text features can be achieved.

[0031] Cross-Attention is a deep learning technique that allows dynamic associations between different input feature sequences. It enables cross-modal or cross-sequence information fusion by interacting query features with key-value features. In this embodiment, query features are derived based on the query feature mapping layer and preset noise in the video generation model. Text features and image features are mapped to text key-value features and image key-value features, respectively, based on the text key feature mapping layer, text value feature mapping layer, image key feature mapping layer, and image value feature mapping layer, thereby establishing cross-modal semantic dependencies.

[0032] In one embodiment, the preset noise may include Gaussian noise. Exemplarily, a Gaussian noise matrix may be randomly generated in advance, and the Gaussian noise matrix may be determined as the preset noise. The preset noise is input into a pre-trained video generation model, and image features and text features are used as conditions of the pre-trained video generation model. By performing denoising processing for a preset number of time steps, a target video that conforms to the semantics of the guide image and prompt text may be obtained. In the denoising processing of each time step, a cross-attention calculation is performed through a cross-attention module in the video generation model, and the output of the video generation model at each time step T may be used as the input of the next time step T+1. Based on the processing of the last time step, the video generation model may output the generated target video.

[0033] In one embodiment, in the denoising process of each time step, the cross-attention module in the Lth network layer performs cross-attention calculations on image features, text features, and preset noise to obtain a cross-attention output, which is then used as the input of the L+1th network layer. The cross-attention module in the L+1th network layer then repeats the operation of the cross-attention module in the Lth network layer, and the cross-attention output output by the cross-attention module of the last network layer is determined as the output of this time step.

[0034] In the disclosed embodiment, by introducing the image features of the guide image during the target video generation process and configuring the cross-attention module in the video generation model to include a query feature mapping layer, a text key feature mapping layer, a text value feature mapping layer, an image key feature mapping layer, and an image value feature mapping layer, the guide image can be better focused on during the video generation process, and the guide image and the prompt text can be better integrated, thereby generating a target video that conforms to the prompt text, has the guide image as the first frame image, and has continuous video content. This improves the quality of the generated target video, ensures the match between the target video and user needs, and enhances the user experience.

[0035] In some optional implementations, the cross-attention module in the disclosed embodiment performs cross-attention calculations on image features, text features, and preset noise, including: Based on the preset noise, the query feature mapping layer is used to perform feature mapping to obtain the query feature (Quary). Based on the text feature, the text key feature mapping layer and the text value feature mapping layer are used to perform feature mapping respectively to obtain the first key feature (Key1) and the first value feature (Value1). Based on the image feature, the image key feature mapping layer and the image value feature mapping layer are used to perform feature mapping respectively to obtain the second key feature (Key2) and the second value feature (Value2). The cross-attention mechanism is calculated for the query feature, the first key feature, the first value feature, the second key feature and the second value feature to obtain the cross-attention output.

[0036] In one embodiment, the query feature may represent the state of the current video generation process. The first key feature and the first value feature may represent key information of the prompt text. The second key feature and the second value feature may represent key information of the guide image.

[0037] Accordingly, in the embodiment of the present disclosure, performing cross-attention mechanism calculation on the query feature, the first key feature, the first value feature, the second key feature and the second value feature may include: performing cross-attention calculation on the query feature, the first key feature and the first value feature to obtain a first sub-fusion feature, performing cross-attention calculation on the query feature, the second key feature and the second value feature to obtain a second sub-fusion feature, and obtaining a cross-attention output based on the first sub-fusion feature and the second sub-fusion feature.

[0038] In one embodiment, a matrix multiplication operation is performed on the query feature and the first key feature to obtain a first attention feature map (Attention Map), which is then multiplied by the first value feature to obtain a first sub-fusion feature. A matrix multiplication operation is performed on the query feature and the second key feature to obtain a second attention feature map (CilpAttention Map), which is then multiplied by the second attention feature map and the second value feature sequence to obtain a second sub-fusion feature. The cross-attention output can be obtained by multiplying the first sub-fusion feature and the second sub-fusion feature or by weighted multiplication.

[0039] Accordingly, in one embodiment, obtaining the cross-attention output based on the first sub-fusion feature and the second sub-fusion feature may include: performing linear calculation on the first sub-fusion feature and the second sub-fusion feature to obtain the cross-attention output.

[0040] Exemplarily, the first sub-fusion feature and the second sub-fusion feature can be added to obtain a cross-attention output.

[0041] In one embodiment, the preset noise is initialized and the initialized preset noise x is input into the to_q layer (query feature mapping layer), and the to_q layer outputs the query matrix Q (query feature); the text features are respectively input into the to_k layer (text key feature mapping layer) and the to_v layer (text value feature mapping layer), the to_k layer outputs K (first key feature), and the to_v layer outputs V (first value feature); the image features are respectively input into the to_k_clip layer (image key feature mapping layer) and the to_v_clip layer (image value feature mapping layer), the to_k_clip layer outputs K_clip (second key feature), and the to_v_clip layer outputs V_clip (second value feature). First, based on Q, K and V, use formula (1) to perform cross-attention calculation to obtain the first sub-fusion feature, that is, the Q and K matrices are multiplied to obtain the Attention Map (first attention feature map), and then multiplied by V; then, based on Q, K_clip and V_clip, use formula (2) to perform cross-attention calculation to obtain the second sub-fusion feature, that is, the Q and K_clip matrices are multiplied to obtain the Cilp Attention Map (second attention feature map), and then multiplied by V_clip. The first sub-fusion feature and the second sub-fusion feature are added to obtain the cross-attention output.

[0042] Formula (1) Formula (2) In formulas (1) and (2), Attention(Q,K,V) represents the first sub-fusion feature, represents the vector dimension (depth) of K, T represents the transpose calculation, represents the second sub-fusion feature, express In the embodiment of the present disclosure, query features corresponding to preset noise, first key features and first value features corresponding to text features, and second key features and second value features corresponding to image features are generated respectively through the query feature mapping layer, the text key feature mapping layer, the text value feature mapping layer, the image key feature mapping layer, and the image value feature mapping layer. By performing cross-attention calculation processing on the query features, the first key features, the first value features, the second key features, and the second value features, the video generation model can effectively focus on and align the prompt texts and guide images from different sources, thereby helping the video generation model to better capture the correlation between the prompt texts and the guide images, so that the generated target video can simultaneously meet the requirements that the first frame is the guide image, matches the prompt text, and the video content is continuous.

[0043] For example, Figure 2This is a process diagram of a method for generating a video provided by an application example of the present disclosure. Figure 2 As shown, the video generation model includes n network layers, at least one of which includes a cross-attention module. In this example, the network layer uses a Transformer network. The cross-attention module includes: a query feature map layer (not shown in the figure), a text key feature map layer (not shown in the figure), a text value feature map layer (not shown in the figure), an image key feature map layer (not shown in the figure), and an image value feature map layer (not shown in the figure).

[0044] The image features of the guide image, the text features of the prompt text, and the preset noise are obtained, and the image features and the preset noise are input into the encoder of the Variational Auto-Encoder (VAE) for encoding. Then, the image features, text features, and the preset noise are input into the video generation model for denoising processing for m time steps (preset time steps) to determine the target video.

[0045] In the denoising process of the i-th (0<i<m, i is an integer) time step: in the cross attention module in the f-th (0<f<n, f is an integer) network layer, feature mapping is performed using the query feature mapping layer based on the preset noise to obtain a query feature, feature mapping is performed using the text key feature mapping layer and the text value feature mapping layer based on the text feature to obtain a first key feature and a first value feature, feature mapping is performed using the image key feature mapping layer and the image value feature mapping layer based on the image feature to obtain a second key feature and a second value feature, matrix multiplication operation is performed on the query feature and the first key feature to obtain a first attention feature map, and then the first attention feature map and the first value feature are multiplied to obtain a first sub-fusion feature; Perform a matrix multiplication operation on the query feature and the second key feature to obtain a second attention feature map, then perform a multiplication operation on the second attention feature map and the second value feature sequence to obtain a second sub-fusion feature, add the first sub-fusion feature and the second sub-fusion feature to obtain a cross-attention output, use the cross-attention output as the input of the cross-attention module in the f+1th network layer, repeat the above operation, determine the cross-attention output output of the cross-attention module of the nth network layer as the cross-attention output of the i-th time step, use the cross-attention output of the i-th time step as the preset noise in the denoising process of the i+1th time step, and determine the target video based on the cross-attention output of the denoising process of the m-th time step. Exemplarily, the cross-attention output of the denoising process of the m-th time step can be input into the decoder of the VAE for decoding to obtain the target video.

[0046] In some optional implementations, step S110 in the embodiment of the present disclosure may include: using a pre-trained multimodal model to extract image features from the guide image to obtain image features, and encoding the prompt text to obtain text features.

[0047] In one embodiment, the multimodal model may employ a contrastive language-image pretraining (CLIP) model, for example. A text encoder, such as the T5 (Text-To-Text Transfer Transformer) model, may be employed to encode the prompt text.

[0048] In the disclosed embodiment, image features of the guide image are extracted through a multimodal model, and prompt text is encoded, thereby achieving efficient acquisition of image features and text features.

[0049] Figure 3 FIG. 1 is a flow chart of a method for generating a video according to another exemplary embodiment of the present disclosure. Figure 3 As shown in Figure 2, the video generation model is obtained as follows: Step S200: Obtain the model to be trained and sample data.

[0050] The sample data includes: multiple sample videos, and the label text, label image, and label noise corresponding to any sample video. The label image corresponding to any sample video is the first frame image of any sample video. The label noise corresponding to any sample video is generated by adding noise to any sample video. The model to be trained includes: a cross-attention module, which includes: a query feature mapping layer, a text key feature mapping layer, a text value feature mapping layer, an image key feature mapping layer, and an image value feature mapping layer.

[0051] In one embodiment, multiple video clips can be obtained and cropped to a fixed resolution and duration to obtain multiple sample videos. The first frame image from the sample video can be selected as the label image corresponding to the sample video. The sample video can be input into a vision-language model (VLM), which outputs the label text corresponding to the sample video. Random Gaussian noise can be added to the sample video to obtain the label noise corresponding to the sample video.

[0052] Step S210 : For a plurality of sample videos, feature extraction is performed on the label images and label texts corresponding to the sample videos to obtain label image features and label text features corresponding to the sample videos.

[0053] Among them, the method of obtaining the label image features and the label text features can refer to the same method of determining the image features and the text features, and will not be repeated here.

[0054] In step S220, the sample video, and the label noise, label image features, and label text features corresponding to the sample video are input into the model to be trained, and the model to be trained performs noise addition processing for a preset number of time steps and outputs predicted noise.

[0055] Among them, in the noise addition processing at each time step: the cross-attention module performs cross-attention calculation on the label image features, label text features, and sample video to obtain the label cross-attention output, so as to determine the predicted noise based on the noise addition processing at each time step and the label cross-attention output at each time step.

[0056] In one embodiment, the cross-attention module performs cross-attention calculation on the label image feature, the label text feature, and the sample video, including: performing feature mapping based on the sample video using the query feature mapping layer to obtain the label query feature; performing feature mapping based on the label text feature using the text key feature mapping layer and the text value feature mapping layer to obtain the label first key feature and the label first value feature; performing feature mapping based on the label image feature using the image key feature mapping layer and the image value feature mapping layer to obtain the label second key feature and the label second value feature; performing cross-attention mechanism calculation on the label query feature, the label first key feature, the label first value feature, the label second key feature, and the label second value feature to obtain the label cross-attention output; specifically, performing cross-attention mechanism calculation on the label query feature, the label first key feature, the label first value feature, the label second key feature, and the label second value feature may include: performing cross-attention calculation on the label query feature, the label first key feature, and the label first value feature to obtain the label first sub-fusion feature; performing cross-attention calculation on the label query feature, the label second key feature, and the label second value feature to obtain the label second sub-fusion feature; performing linear calculation on the label first sub-fusion feature and the label second sub-fusion feature to obtain the label cross-attention output.

[0057] For example, Figure 2The corresponding video generation model is used as an example to illustrate. In the noise addition processing of the i-th time step: in the cross attention module in the f-th network layer, based on the sample video, the query feature mapping layer is used to perform feature mapping to obtain the label query feature, based on the label text feature, the text key feature mapping layer and the text value feature mapping layer are used to perform feature mapping respectively to obtain the label first key feature and the label first value feature, based on the label image feature, the image key feature mapping layer and the image value feature mapping layer are used to perform feature mapping respectively to obtain the label second key feature and the label second value feature, the label query feature and the label first key feature are matrix multiplied to obtain the label first attention feature map, and then the label first attention feature map and the label first value feature are multiplied to obtain the label first sub-fusion feature; the label query feature and the label The second key feature of the label is matrix multiplied to obtain the second attention feature map of the label, and then the second attention feature map of the label is multiplied with the second value feature sequence of the label to obtain the second sub-fusion feature of the label, and the first sub-fusion feature of the label and the second sub-fusion feature of the label are added to obtain the label cross attention output, and the label cross attention output is used as the input of the cross attention module in the f+1th network layer. The above operation is repeated, and the label cross attention output output of the cross attention module of the nth network layer is determined as the label cross attention output of the i-th time step, and the label cross attention output of the i-th time step is used as the sample video in the noise addition processing of the i+1th time step. The predicted noise is determined based on the label cross attention output of the m-th time step. Exemplarily, the label cross attention output of the m-th time step can be determined as the predicted noise.

[0058] In step S230 , the parameters of the model to be trained are adjusted according to the predicted noises and the corresponding label noises until a preset training end condition is met, and a video generation model is obtained from the model to be trained.

[0059] A loss function value can be determined based on the difference between the label noise and the predicted noise corresponding to each sample video using a preset loss function. Examples of the preset loss function include, but are not limited to, a cross-entropy error function or a mean square error function. The loss function value determination operation can be iteratively performed, and the parameters of the model to be trained can be iteratively adjusted to continuously reduce the loss function value until the loss function value converges. Preset training termination conditions are then determined to be met, and the training of the model to be trained is completed. The trained model to be trained is then used as the video generation model. Parameter optimizers such as stochastic gradient descent (SGD), adaptive gradient algorithm (Adagrad), adaptive moment estimation (Adam), and root mean square prop (RMSprop) can be used to adjust the parameters of the model to be trained. For example, the parameter optimizer can be used to calculate the gradient of each parameter of the model to be trained, and the parameters can be adjusted along the gradient direction, where the gradient represents the direction of the greatest loss function value reduction. The loss function value determination operation is iteratively performed until the loss function value no longer decreases, thus completing the training of the model to be trained and obtaining the video generation model.

[0060] In the embodiment of the present disclosure, the first frame image in the sample video is used as the label image, and the label image features, label text features and sample video are used to train the training model. This not only achieves efficient training of the training model, but also enables the trained video generation model to generate a video with the first frame image as the specified image.

[0061] Figure 4 This is a schematic diagram of the structure of an embodiment of the apparatus for generating video according to the present disclosure. Figure 4 As shown, the device of this embodiment may include: The condition acquisition module 300 is used to acquire a guide image and prompt text corresponding to a target video, wherein the guide image is the first frame image of the target video; A first feature extraction module 310 is used to determine the image features of the guide image and the text features of the prompt text; The video generation module 320 is used to perform denoising processing for a preset number of time steps based on the image features, the text features and the preset noise using a pre-trained video generation model to determine the target video, wherein the video generation model includes a cross-attention module, and the cross-attention module includes a query feature mapping layer, a text key feature mapping layer, a text value feature mapping layer, an image key feature mapping layer and an image value feature mapping layer. In the denoising processing of each time step: the cross-attention module performs cross-attention calculation on the image features, the text features and the preset noise to obtain a cross-attention output, so as to determine the target video based on the denoising processing of each time step and the cross-attention output of each time step.

[0062] In some possible implementations of the present disclosure, the cross-attention module in the embodiment of the present disclosure performs cross-attention calculation on the image features, the text features, and the preset noise, further generally speaking: Based on the preset noise, feature mapping is performed using the query feature mapping layer to obtain query features; based on the text features, feature mapping is performed using the text key feature mapping layer and the text value feature mapping layer respectively to obtain first key features and first value features; based on the image features, feature mapping is performed using the image key feature mapping layer and the image value feature mapping layer respectively to obtain second key features and second value features; a cross-attention mechanism is calculated for the query features, the first key features, the first value features, the second key features and the second value features to obtain the cross-attention output.

[0063] In some possible implementations of the present disclosure, a cross-attention mechanism calculation is performed on the query feature, the first key feature, the first value feature, the second key feature, and the second value feature in the embodiment of the present disclosure, and is further used to: A cross-attention calculation is performed on the query feature, the first key feature, and the first value feature to obtain a first sub-fusion feature; a cross-attention calculation is performed on the query feature, the second key feature, and the second value feature to obtain a second sub-fusion feature; and the cross-attention output is obtained based on the first sub-fusion feature and the second sub-fusion feature.

[0064] In some possible implementations of the present disclosure, the cross-attention output is obtained based on the first sub-fusion feature and the second sub-fusion feature in the embodiment of the present disclosure, and is further used to: perform linear calculation on the first sub-fusion feature and the second sub-fusion feature to obtain the cross-attention output.

[0065] In some possible implementations of the present disclosure, the first feature extraction module 310 in the embodiment of the present disclosure is specifically used to use a pre-trained multimodal model to extract image features of the guide image to obtain the image features; and to encode the prompt text to obtain the text features.

[0066] In some possible implementations of the present disclosure, the apparatus for generating a video in an embodiment of the present disclosure further includes: A sample acquisition module is used to acquire a model to be trained and sample data, wherein the sample data includes multiple sample videos, and label text, label image and label noise corresponding to any sample video, the label image corresponding to any sample video is the first frame image of any sample video, and the label noise is generated by adding noise to any sample video, and the model to be trained includes a cross-attention module, and the cross-attention module includes a query feature mapping layer, a text key feature mapping layer, a text value feature mapping layer, an image key feature mapping layer and an image value feature mapping layer; A second feature extraction module is configured to extract features of the label images and label texts corresponding to the plurality of sample videos to obtain label image features and label text features corresponding to the sample videos; A first training module is configured to input the sample video, and the label noise, label image features, and label text features corresponding to the sample video, into the to-be-trained model, and the to-be-trained model performs noise addition processing for a preset number of time steps and outputs predicted noise, wherein, in the noise addition processing at each time step: the cross-attention module performs cross-attention calculation on the label image features, the label text features, and the sample video to obtain a label cross-attention output, so as to determine the predicted noise based on the noise addition processing at each time step and the label cross-attention output at each time step; The second training module is used to adjust the parameters of the model to be trained according to each predicted noise and the corresponding label noise until a preset training end condition is met, and the video generation model is obtained from the model to be trained.

[0067] In some possible implementations of the present disclosure, the cross-attention module in the embodiment of the present disclosure performs cross-attention calculations on the label image features, the label text features, and the sample video, and is further used to perform feature mapping based on the sample video using the query feature mapping layer to obtain label query features; based on the label text features, perform feature mapping using the text key feature mapping layer and the text value feature mapping layer respectively to obtain label first key features and label first value features; based on the label image features, perform feature mapping using the image key feature mapping layer and the image value feature mapping layer respectively to obtain label second key features and label second value features; perform cross-attention mechanism calculations on the label query features, the label first key features, the label first value features, the label second key features, and the label second value features to obtain the label cross-attention output.

[0068] The video generation device of the embodiment of the present disclosure corresponds to the embodiment of the video generation method of the present disclosure, and the relevant contents can be referenced to each other and will not be repeated here.

[0069] The beneficial technical effects corresponding to the exemplary embodiments of the video generation device of the embodiments of the present disclosure can be found in the corresponding beneficial technical effects of the above-mentioned corresponding exemplary method part, and will not be repeated here.

[0070] In addition, an embodiment of the present disclosure further provides an electronic device, including: memory for storing computer programs; The processor is configured to execute the computer program stored in the memory, and when the computer program is executed, the method for generating a video as described in any one of the above embodiments of the present disclosure is implemented.

[0071] Figure 5 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed in the present invention. Figure 5 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be either or both of the first device and the second device, or a standalone device independent of them, and the standalone device may communicate with the first device and the second device to receive collected input signals from them.

[0072] like Figure 5 As shown, the electronic device includes one or more processors and memory.

[0073] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0074] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the video generation methods of the various embodiments of the present disclosure described above and / or other desired functions.

[0075] In one example, the electronic device may further include an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0076] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0077] The output device can output various information to the outside, including the determined target video, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0078] Of course, to simplify, Figure 5 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application scenarios.

[0079] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to perform the steps in the method of video generation according to various embodiments of the present disclosure described in the above part of this specification.

[0080] The computer program product may be written in any combination of one or more programming languages to implement the operations of the disclosed embodiments, including object-oriented programming languages such as Java, C++, and conventional procedural programming languages such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0081] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps in the method of video generation according to various embodiments of the present disclosure described in the above part of this specification.

[0082] The computer-readable storage medium may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0083] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk, etc. Various media that can store program codes.

[0084] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0085] Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For system embodiments, since they are essentially identical to the method embodiments, their description is relatively simple. For relevant parts, refer to the descriptions of the method embodiments.

[0086] The block diagrams of the devices, devices, equipment, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0087] The methods and apparatus of the present disclosure may be implemented in many ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.

[0088] It should also be noted that in the apparatus, device, and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0089] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0090] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for generating a video, characterized in that: include: Obtaining a guide image and prompt text corresponding to a target video, wherein the guide image is the first frame image of the target video; Determining image features of the guide image and text features of the prompt text; Based on the image features, the text features and the preset noise, a pre-trained video generation model is used to perform denoising processing for a preset time step to determine the target video, wherein the video generation model includes a cross-attention module, and the cross-attention module includes a query feature mapping layer, a text key feature mapping layer, a text value feature mapping layer, an image key feature mapping layer and an image value feature mapping layer. In the denoising processing of each time step: the cross-attention module performs cross-attention calculation on the image features, the text features and the preset noise to obtain a cross-attention output, so as to determine the target video based on the denoising processing of each time step and the cross-attention output of each time step.

2. The method according to claim 1, characterized in that The cross attention module performs cross attention calculation on the image feature, the text feature, and the preset noise, including: Based on the preset noise, feature mapping is performed using the query feature mapping layer to obtain query features; Based on the text feature, feature mapping is performed using the text key feature mapping layer and the text value feature mapping layer to obtain a first key feature and a first value feature; Based on the image features, feature mapping is performed using the image key feature mapping layer and the image value feature mapping layer to obtain a second key feature and a second value feature; A cross-attention mechanism is performed on the query feature, the first key feature, the first value feature, the second key feature, and the second value feature to obtain the cross-attention output.

3. The method according to claim 2, characterized in that The performing a cross attention mechanism calculation on the query feature, the first key feature, the first value feature, the second key feature, and the second value feature includes: Performing a cross attention calculation on the query feature, the first key feature, and the first value feature to obtain a first sub-fusion feature; Performing a cross attention calculation on the query feature, the second key feature, and the second value feature to obtain a second sub-fusion feature; The cross attention output is obtained based on the first sub-fusion feature and the second sub-fusion feature.

4. The method according to claim 3, characterized in that The obtaining the cross attention output based on the first sub-fusion feature and the second sub-fusion feature includes: Performing linear calculation on the first sub-fusion feature and the second sub-fusion feature to obtain the cross attention output.

5. The method according to any one of claims 1 to 4, characterized in that: The determining of the image features of the guide image and the text features of the prompt text includes: Extracting image features from the guide image using a pre-trained multimodal model to obtain the image features; The prompt text is encoded to obtain the text features.

6. The method according to claim 1, characterized in that The video generation model is obtained in the following way: Obtain a model to be trained and sample data, wherein the sample data includes multiple sample videos, and label text, label image, and label noise corresponding to any sample video, the label image corresponding to any sample video is the first frame image of any sample video, and the label noise is generated by adding noise to any sample video, and the model to be trained includes a cross-attention module, and the cross-attention module includes a query feature mapping layer, a text key feature mapping layer, a text value feature mapping layer, an image key feature mapping layer, and an image value feature mapping layer; For the multiple sample videos, feature extraction is performed on the label images and label texts corresponding to the sample videos to obtain label image features and label text features corresponding to the sample videos; The sample video, as well as the label noise, label image features, and label text features corresponding to the sample video, are input into the to-be-trained model, and the to-be-trained model performs noise addition processing for a preset number of time steps and outputs predicted noise, wherein, in the noise addition processing at each time step: the cross-attention module performs cross-attention calculation on the label image features, the label text features, and the sample video to obtain a label cross-attention output, so as to determine the predicted noise based on the noise addition processing at each time step and the label cross-attention output at each time step; The parameters of the model to be trained are adjusted according to each predicted noise and the corresponding label noise until a preset training end condition is met, and the video generation model is obtained from the model to be trained.

7. The method according to claim 6, characterized in that The cross attention module performs cross attention calculation on the label image features, the label text features, and the sample video, including: Based on the sample video, feature mapping is performed using the query feature mapping layer to obtain a label query feature; Based on the tag text feature, feature mapping is performed using the text key feature mapping layer and the text value feature mapping layer to obtain a tag first key feature and a tag first value feature; Based on the label image feature, feature mapping is performed using the image key feature mapping layer and the image value feature mapping layer to obtain a label second key feature and a label second value feature; A cross-attention mechanism is calculated for the tag query feature, the tag first key feature, the tag first value feature, the tag second key feature and the tag second value feature to obtain the tag cross-attention output.

8. A video generation device, characterized in that: include: A condition acquisition module, configured to acquire a guide image and prompt text corresponding to a target video, wherein the guide image is the first frame image of the target video; A first feature extraction module, configured to determine image features of the guide image and text features of the prompt text; A video generation module is used to perform denoising processing for a preset number of time steps based on the image features, the text features and the preset noise using a pre-trained video generation model to determine the target video, wherein the video generation model includes a cross-attention module, and the cross-attention module includes a query feature mapping layer, a text key feature mapping layer, a text value feature mapping layer, an image key feature mapping layer and an image value feature mapping layer. In the denoising processing of each time step: the cross-attention module performs cross-attention calculation on the image features, the text features and the preset noise to obtain a cross-attention output, so as to determine the target video based on the denoising processing of each time step and the cross-attention output of each time step.

9. An electronic device, characterized in that: include: Memory for storing computer programs; The processor is configured to execute the computer program stored in the memory, and when the computer program is executed, the method for generating a video according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising computer program instructions, characterized in that When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images