Video generation method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a video generation method and device, electronic equipment and a computer readable storage medium, and belongs to the technical field of video generation. According to the embodiment of the invention, the reference image and the prompt text are acquired, and the reference image and the prompt text are input into the video generation model, so that the target video output by the video generation model can be obtained; wherein the video generation model comprises an image prompt module and a target generation module. According to the embodiment of the application, the reference image and the prompt text are decoupled by using the image prompt module, so that the image features can be reserved more explicitly, semantic confusion caused by direct coupling is effectively avoided, efficient alignment and fine control of the image-text features are realized, and the user experience is improved. Therefore, the target generation module can generate a high-quality video which is more accurate and accords with user expectations based on the target image features and the target text features output by the image prompt module, and the user experience is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video generation technology, and specifically to a video generation method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the continuous development of AI (Artificial Intelligence) technology, it is now possible to use large AI models to automatically generate videos based on prompt text.

[0003] However, current models that rely on text prompts are prone to semantic ambiguity, resulting in generated videos that do not conform to expectations, which in turn leads to a poor user experience. Summary of the Invention

[0004] The embodiments of the present application provide a video generation method, device, electronic device, and computer-readable storage medium, which can generate high-quality videos that meet user expectations.

[0005] In a first aspect, an embodiment of the present application provides a video generation method, comprising:

[0006] Get reference images and prompt text;

[0007] Inputting the reference image and the prompt text into a video generation model to obtain a target video output by the video generation model; wherein the video generation model includes an image prompt module and a target generation module;

[0008] The image prompt module is configured to perform decoupling processing on the reference image and the prompt text to obtain target image features and target text features;

[0009] The target generation module is configured to generate the target video according to the target image features and the target text features.

[0010] In one embodiment, the image prompt module includes an image preprocessing module, a text preprocessing module and a decoupled cross attention module, the decoupled cross attention module includes a first cross attention layer and a second cross attention layer; the input end of the image preprocessing module is configured to receive the reference image, the input end of the text preprocessing module is configured to receive the prompt text, the output end of the image preprocessing module is connected to the input end of the first cross attention layer, and the output end of the text preprocessing module is connected to the second cross attention layer; the output end of the first cross attention layer and the output end of the second cross attention layer are respectively connected to the input end of the target generation module;

[0011] The image preprocessing module is configured to perform feature extraction on the reference image to obtain initial image features corresponding to the reference image;

[0012] The text preprocessing module is configured to perform feature extraction on the prompt text to obtain initial text features corresponding to the prompt text;

[0013] The first cross-attention layer is configured to perform cross-attention calculation on the initial image features to obtain target image features;

[0014] The second cross-attention layer is configured to perform cross-attention calculation on the initial text features to obtain target text features.

[0015] In one embodiment, the video generation model further includes a diffusion enhancement module, wherein an input end of the diffusion enhancement module is configured to receive the reference image, and an output end of the diffusion enhancement module is connected to an input end of the target generation module;

[0016] The diffusion enhancement module is configured to generate at least one corresponding conditional control signal based on the reference image;

[0017] Generating the target video according to the target image feature and the target text feature includes:

[0018] The target video is generated according to the target image feature, the target text feature and at least one of the conditional control signals.

[0019] In one embodiment, the diffusion enhancement module includes at least one control signal generating unit, and different control signal generating units are used to generate different conditional control signals; at least one of the conditional control signals includes one or more of an edge detection map, a posture skeleton map, a depth map, and a semantic segmentation map.

[0020] In one embodiment, the video generation model further includes a mask processing module, wherein an input end of the mask processing module is configured to receive the reference image, and an output end of the mask processing module is connected to an input end of the target generation module;

[0021] The mask processing module is configured to generate a mask corresponding to the target area in the reference image; the mask is used to indicate that the target area is a motion area of ​​the reference image;

[0022] Generating the target video according to the target image feature, the target text feature, and at least one of the conditional control signals includes:

[0023] The target video is generated according to the target image feature, the target text feature, the mask and at least one of the conditional control signals.

[0024] In one embodiment, the method further comprises:

[0025] In response to a user's region selection operation on the reference image, determining a target region of the reference image; or,

[0026] The reference image and the prompt text are input into a target detection model to obtain the target area output by the target detection model.

[0027] In one embodiment, the video generation model further includes a post-processing module, and the output end of the target generation module is connected to the input end of the post-processing module;

[0028] The post-processing module is configured to perform post-processing on the target video to obtain an optimized video; the post-processing includes super-resolution processing and / or frame insertion processing.

[0029] In a second aspect, an embodiment of the present application provides a video generation device, the device comprising:

[0030] A data acquisition module, used to obtain reference images and prompt texts;

[0031] A video generation module, configured to input the reference image and the prompt text into a video generation model to obtain a target video output by the video generation model; wherein the video generation model includes an image prompt module and a target generation module;

[0032] The image prompt module is configured to perform decoupling processing on the reference image and the prompt text to obtain target image features and target text features;

[0033] The target generation module is configured to generate the target video according to the target image features and the target text features.

[0034] In one embodiment, the image prompt module includes an image preprocessing module, a text preprocessing module and a decoupled cross attention module, the decoupled cross attention module includes a first cross attention layer and a second cross attention layer; the input end of the image preprocessing module is configured to receive the reference image, the input end of the text preprocessing module is configured to receive the prompt text, the output end of the image preprocessing module is connected to the input end of the first cross attention layer, and the output end of the text preprocessing module is connected to the second cross attention layer; the output end of the first cross attention layer and the output end of the second cross attention layer are respectively connected to the input end of the target generation module;

[0035] The image preprocessing module is configured to perform feature extraction on the reference image to obtain initial image features corresponding to the reference image;

[0036] The text preprocessing module is configured to perform feature extraction on the prompt text to obtain initial text features corresponding to the prompt text;

[0037] The first cross-attention layer is configured to perform cross-attention calculation on the initial image features to obtain target image features;

[0038] The second cross-attention layer is configured to perform cross-attention calculation on the initial text features to obtain target text features.

[0039] In one embodiment, the video generation model further includes a diffusion enhancement module, wherein an input end of the diffusion enhancement module is configured to receive the reference image, and an output end of the diffusion enhancement module is connected to an input end of the target generation module;

[0040] The diffusion enhancement module is configured to generate at least one corresponding conditional control signal based on the reference image;

[0041] Generating the target video according to the target image feature and the target text feature includes:

[0042] The target video is generated according to the target image feature, the target text feature and at least one of the conditional control signals.

[0043] In one embodiment, the diffusion enhancement module includes at least one control signal generating unit, and different control signal generating units are used to generate different conditional control signals; at least one of the conditional control signals includes one or more of an edge detection map, a posture skeleton map, a depth map, and a semantic segmentation map.

[0044] In one embodiment, the video generation model further includes a mask processing module, wherein an input end of the mask processing module is configured to receive the reference image, and an output end of the mask processing module is connected to an input end of the target generation module;

[0045] The mask processing module is configured to generate a mask corresponding to the target area in the reference image; the mask is used to indicate that the target area is a motion area of ​​the reference image;

[0046] Generating the target video according to the target image feature, the target text feature, and at least one of the conditional control signals includes:

[0047] The target video is generated according to the target image feature, the target text feature, the mask and at least one of the conditional control signals.

[0048] In one embodiment, the video generating device further includes:

[0049] A first determining module is configured to determine a target area of ​​the reference image in response to a user's area selection operation on the reference image; or

[0050] The second determination module is used to input the reference image and the prompt text into a target detection model to obtain the target area output by the target detection model.

[0051] In one embodiment, the video generation model further includes a post-processing module, and the output end of the target generation module is connected to the input end of the post-processing module;

[0052] The post-processing module is configured to perform post-processing on the target video to obtain an optimized video; the post-processing includes super-resolution processing and / or frame insertion processing.

[0053] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps in the above-mentioned video generation method are implemented.

[0054] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned video generation method are implemented.

[0055] In a fifth aspect, an embodiment of the present application further provides a computer program product or a computer program, which includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the various optional implementations described in the embodiments of the present application.

[0056] In summary, in the embodiment of the present application, a reference image and a prompt text can be obtained, and the reference image and the prompt text can be input into a video generation model to obtain a target video output by the video generation model; wherein the video generation model includes an image prompt module and a target generation module; the image prompt module is configured to decouple the reference image and the prompt text to obtain target image features and target text features, and the target generation module is configured to generate a target video based on the target image features and target text features. The embodiment of the present application utilizes the image prompt module to decouple the reference image and the prompt text, which can retain the image features more explicitly, thereby effectively avoiding semantic confusion caused by direct coupling, and realizing efficient alignment and refined control of image and text features, so that the target generation module can generate a more accurate, high-quality video that meets user expectations based on the target image features and target text features, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in this application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 This is a schematic diagram of the steps of a video generation method provided in one embodiment of the present application;

[0059] Figure 2 This is a structural diagram of an image prompt module provided in one embodiment of the present application;

[0060] Figure 3 This is a structural diagram of a video generation model provided by an embodiment of the present application;

[0061] Figure 4 is a structural diagram of a video generating device provided in one embodiment of the present application;

[0062] Figure 5 It is a structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0064] It should be noted that with the development of AIGC (Artificial Intelligence Generated Content) technology, the technology of using AI large models to generate images has become more mature, and there are already many mature commercial applications in the field of cultural images.

[0065] However, AI-generated video technology is far from mature. Specifically, current models that rely on text prompts are prone to semantic ambiguity, resulting in the generated video not matching expectations. This means the generated video will contain content unrelated to the prompt, leading to a poor user experience.

[0066] In response to the current problem of satisfactorily generating high-quality videos that meet user expectations, this application aims to provide a video generation method. By configuring an image prompt module and a target generation module in a video generation model, and using the image prompt module to decouple the reference image and prompt text, the image features can be more explicitly retained, thereby effectively avoiding semantic confusion caused by direct coupling, and achieving efficient alignment and refined control of image and text features, so that the target generation module can generate more accurate and high-quality videos that meet user expectations based on the target image features and target text features, thereby improving the user experience.

[0067] It should be noted that the order of description of the following embodiments does not limit the priority order of the embodiments.

[0068] Figure 1 The following schematically illustrates a flow chart of a video generation method according to one embodiment of the present application. The execution subject of the video generation method can be any computing service device with data processing, network communication, and program execution functions, such as cloud servers and cloud computing platforms, or electronic devices with the above functions, such as computers, mobile phones, televisions, and smart watches.

[0069] Reference Figure 1 , shows a video generation method of the present application, which may specifically include the following steps:

[0070] S101: Acquire a reference image and prompt text.

[0071] In this embodiment, the user can input a reference image and prompt text for generating a video according to his or her personalized needs.

[0072] In this embodiment, an image selection button and a text input button may be pre-configured so that the user can quickly select a corresponding reference image through the image selection button and quickly input a prompt text through the text input button.

[0073] In a specific implementation, in response to an image selection instruction triggered by a user for an image selection button, the user's photo album can be called through a preset interface, and in response to the user's image selection operation for a reference image in the user's photo album, a reference image can be obtained; in response to a text input instruction triggered by a user for a text input button, a text input box can be displayed, and the prompt text entered by the user through the text input box can be obtained.

[0074] S102: Input the reference image and prompt text into the video generation model to obtain the target video output by the video generation model.

[0075] In this embodiment, the video generation model includes an image prompt module and a target generation module; the image prompt module is configured to decouple the reference image and the prompt text to obtain target image features and target text features; the target generation module is configured to generate a target video based on the target image features and target text features.

[0076] It should be noted that decoupling processing means performing feature processing on the reference image and prompt text separately to avoid semantic confusion caused by direct coupling of image and text features, and to improve the consistency between the generated content and the text.

[0077] In this embodiment, the target generation module comprehensively considers the target image features and the target text features during the video generation process, so that the target video can retain the composition of the reference image and accurately match the text description of the prompt text.

[0078] In a specific implementation, in order to further improve the video quality of the target video, the target generation module can adopt the AnimateDiff model.

[0079] It's important to note that the AnimateDiff model is a text-to-video model built on the SD (Stable Diffusion) model. The AnimateDiff model incorporates a motion module to learn inter-frame motion information. Therefore, the AnimateDiff model can generate high-quality dynamic images or videos. Diffusion is an open-source diffusion model that is often used as the foundational model for most text-to-video production.

[0080] In this embodiment, by injecting the target image features and target text features generated by the image prompt module into the AnimateDiff model, the AnimateDiff model can generate more accurate image frames based on the target image features and target text features, and use the motion modeling module to generate an image frame sequence based on the motion information of the image frames, thereby obtaining the target video.

[0081] In this embodiment, by configuring an image prompt module and a target generation module in the video generation model, and using the image prompt module to decouple the reference image and prompt text, the image features can be more explicitly retained, thereby effectively avoiding semantic confusion caused by direct coupling, and achieving efficient alignment and refined control of image and text features, so that the target generation module can generate more accurate and user-expected high-quality videos based on the target image features and target text features, thereby improving the user experience.

[0082] In one possible embodiment, referring to Figure 2 , showing a schematic diagram of the structure of the image prompt module. The image prompt module may specifically include an image preprocessing module, a text preprocessing module and a decoupled cross-attention module, wherein the decoupled cross-attention module includes a first cross-attention layer and a second cross-attention layer; the input end of the image preprocessing module is configured to receive a reference image, the input end of the text preprocessing module is configured to receive a prompt text, the output end of the image preprocessing module is connected to the input end of the first cross-attention layer, and the output end of the text preprocessing module is connected to the second cross-attention layer; the output end of the first cross-attention layer and the output end of the second cross-attention layer are respectively connected to the input end of the target generation module.

[0083] In this embodiment, the image preprocessing module is configured to perform feature extraction on the reference image to obtain initial image features corresponding to the reference image.

[0084] In this embodiment, the image prompt module can specifically adopt the IPAdapter model. The IPAdapter model is a generation technology based on image prompts, which can decouple images and text prompts, thereby more explicitly retaining image features and generating more accurate and diverse images in combination with text prompts.

[0085] Specifically, the image preprocessing module can include an image encoder, a linear layer, and layer normalization (LN). The image encoder is used to convert the reference image into a high-dimensional feature representation, the linear layer is used to transform or reduce the dimension of the features, and the layer normalization is used to standardize the feature distribution and improve training stability.

[0086] In this embodiment, the reference image is preprocessed by the image preprocessing module to obtain initial image features (Imaga Features) that are convenient for the first cross attention layer to recognize.

[0087] In this embodiment, the text preprocessing module is configured to perform feature extraction on the prompt text to obtain initial text features corresponding to the prompt text.

[0088] In this embodiment, the text preprocessing module may specifically use a text encoder to encode the prompt text (e.g., "girl wearing sunglasses") into initial text features. The text encoder may use a text encoder from CLIP or a pre-trained model such as BERT.

[0089] In this embodiment, the prompt text is preprocessed by a text preprocessing module to obtain initial text features (Imaga Features) that are convenient for the second cross attention layer to recognize.

[0090] In this embodiment, the first cross-attention layer is configured to perform cross-attention calculations on the initial image features to obtain target image features.

[0091] In this implementation, the first cross-attention layer is responsible for image-to-text interaction.

[0092] Specifically, the first cross-attention layer uses image features as queries and text features as keys and values ​​to extract regions in the reference image that are semantically relevant to the text, thereby obtaining target image features. For example, if the text prompt is "girl wearing sunglasses," the first cross-attention layer will focus on visual regions in the reference image that are relevant to "sunglasses" and "girl."

[0093] In this embodiment, the second cross-attention layer is configured to perform cross-attention calculation on the initial text features to obtain target text features.

[0094] In this embodiment, the second cross-attention layer is responsible for the text-to-image interaction.

[0095] Specifically, the second cross-attention layer can use text features as queries and image features as keys and values ​​to inject text semantics into the image generation process. For example, based on the text "sunglasses", the details of the glasses in the generated image (such as color and shape) can be adjusted.

[0096] In this implementation, two independent Cross Attention layers are responsible for image-to-text and text-to-image interactions, respectively. This enables feature decoupling of image and text, avoids semantic confusion caused by direct coupling, and thus improves the accuracy of multimodal alignment.

[0097] In this embodiment, the target generation module can adopt a denoising U-Net and utilize the encoder-decoder structure of the denoising U-Net to combine the target image features and target text features input by the first cross attention layer and the second cross attention layer respectively to generate the noise-containing latent representation X. t Perform gradual denoising to gradually restore image details and obtain a clearer potential representation X t-1 , and combined with the motion information of the image frame input by the motion modeling module, it is finally decoded into the target video.

[0098] It should be noted that, continue to refer to Figure 2 , where the linear layer (Linear), layer normalization (LN) and the first cross-attention layer are all trainable modules (Trainable Modules). Specifically, during the training process, the difference between the generated image and the real image can be minimized (such as L1 / L2 loss, adversarial loss), and the parameters of the trainable modules can be optimized through backpropagation. By training the trainable modules, the image prompt module can continuously adapt to new data and new tasks to achieve better performance. It should be further explained that, in addition to the trainable modules, other modules can adopt frozen modules whose parameters remain unchanged during the model training process. Frozen modules can reduce the number of model parameters, speed up training, and help improve the model's generalization ability.

[0099] In a feasible embodiment, the video generation model further includes a diffusion enhancement module, the input end of the diffusion enhancement module is configured to receive a reference image, and the output end of the diffusion enhancement module is connected to the input end of the target generation module.

[0100] In this embodiment, the diffusion enhancement module is configured to generate at least one corresponding conditional control signal based on the reference image.

[0101] It should be noted that the conditional control signal is used to guide the target generation module to generate an image that meets the conditions.

[0102] In a specific implementation, the diffusion enhancement module includes at least one control signal generating unit, and different control signal generating units are used to generate different conditional control signals; at least one conditional control signal includes one or more of an edge detection map, a posture skeleton map, a depth map, and a semantic segmentation map.

[0103] In this embodiment, referring to Figure 3, which shows a schematic diagram of the structure of a video generation model provided by one embodiment of the present application. Specifically, the control signal generation unit can use ControlNet, a deep learning technology designed to enhance the capabilities of existing diffusion models (such as Stable Diffusion) by introducing additional structural or semantic information to finely control the generated content.

[0104] In this embodiment, one or more ControlNets can be selected to inject one or more conditional control signals into the target generation module based on the required accuracy. When there are multiple ControlNets, a ControlNet stack can be generated to generate multiple conditional control signals. This, in turn, further improves the control accuracy of the target video by introducing multiple conditional control signals.

[0105] In a video generation scenario, an image of a girl wearing sunglasses is generated, but the text prompt may not accurately describe the shape or position of the glasses. Then, ControlNet is used to obtain an edge detection map (CannyEdge) based on the reference image as a conditional control signal to mark the facial contour and glasses position of the character. In this way, the sunglasses in the image generated by the target generation module strictly fit the facial contour, and the shape and edge of the frame are consistent. Figure 1 At the same time, the character posture (such as head angle) is matched with the edge detection map to avoid misalignment or deformation.

[0106] In this embodiment, generating a target video according to target image features and target text features includes: generating a target video according to target image features, target text features and at least one conditional control signal.

[0107] In this embodiment, by further considering the conditional control signal input by the diffusion enhancement module on the basis of the target image features and the target text features, a target video that better meets expectations can be generated.

[0108] In one possible embodiment, continue to refer to Figure 3 The video generation model further includes a mask processing module, an input end of the mask processing module is configured to receive a reference image, and an output end of the mask processing module is connected to an input end of the target generation module.

[0109] In this embodiment, the mask processing module is configured to generate a mask corresponding to the target area in the reference image; the mask is used to indicate that the target area is a motion area of ​​the reference image.

[0110] In this embodiment, the prompt text may be imprecise or ambiguous, making it difficult to generate a desired video. Therefore, a mask processing module can be used to generate a mask corresponding to the target area in the reference image. This allows the motion portion of the reference image to be specified, improving the quality of the generated result.

[0111] In a specific implementation, in order to achieve accurate identification of the target area, the video generation method may also include the following steps: determining the target area of ​​the reference image in response to the user's area selection operation on the reference image; or, inputting the reference image and prompt text into the target detection model to obtain the target area output by the target detection model.

[0112] In this embodiment, the user can manually select the target area of ​​the reference image where the object to be moved is located, thereby improving the accuracy of the target area; or, to simplify user operations, a target detection model can be used, such as the GroundingDino SAM model (a segmentation model combined with GroundingDino), which automatically identifies the target area of ​​the target detection model based on the reference image and prompt text.

[0113] In a video generation scenario, the prompt text is: cloud drift, plant drift. The mask processing module identifies the cloud and plant areas in the reference image based on the reference image, generates masks corresponding to the cloud and plant areas, and injects the masks into the target generation module. The target generation module then generates and edits only the clouds and plants in the target area based on the masks, obtaining a generation result that meets the dynamic mask constraints, that is, only the clouds and plants drift in the target video.

[0114] In this embodiment, generating a target video according to target image features, target text features and at least one conditional control signal includes generating a target video according to target image features, target text features, a mask and at least one conditional control signal.

[0115] In this embodiment, by further considering the mask input by the mask processing module on the basis of the target image features, target text features and at least one conditional control signal, the target generation module can generate a target video with higher quality.

[0116] In one possible embodiment, continue to refer to Figure 3 ,The video generation model also includes a post-processing module (PostProcess), and the output end of the ,target generation module is connected to the input end of the post-processing ,module.

[0117] In this embodiment, the post-processing module is configured to perform post-processing on the target video to obtain an optimized video; the post-processing includes super-resolution processing, frame insertion processing and / or HDR (High Dynamic Range).

[0118] It should be noted that super-resolution processing is used to improve the resolution of the video; interpolation processing refers to adding frames to the video to ensure the clarity of the video, and interpolation processing is usually performed after the frame rate of the video changes; HDR is used to render the image frames of the target video to improve image quality.

[0119] In this embodiment, by performing super-resolution processing and frame insertion processing on the target video, the video resolution can be improved while the video smoothness can be improved.

[0120] In this implementation, considering that video generation services provided by commercial platforms both domestically and internationally typically charge by volume, resulting in high fees and a negative impact on user experience, a video generation model is constructed based on the fully open-source AnimateDiff model, IPAdapter model, ControlNet, and Post Process modules. This effectively reduces video generation costs while improving video quality.

[0121] To facilitate better implementation of the video generation method of the present application, the present application also provides a video generation device based on the above video generation method. The meanings of the terms are the same as those in the above video generation method, and the specific implementation details can be referred to the description in the method embodiment.

[0122] Based on the same inventive concept, Figure 4 , an embodiment of the present application provides a video generating device 400, the video generating device 400 comprising:

[0123] Data acquisition module 401, used to acquire reference images and prompt text;

[0124] The video generation module 402 is used to input the reference image and prompt text into the video generation model to obtain the target video output by the video generation model; wherein the video generation model includes an image prompt module and a target generation module;

[0125] The image prompt module is configured to decouple the reference image and the prompt text to obtain target image features and target text features;

[0126] The target generation module is configured to generate a target video according to target image features and target text features.

[0127] In one embodiment, the image prompt module includes an image preprocessing module, a text preprocessing module and a decoupled cross attention module, the decoupled cross attention module includes a first cross attention layer and a second cross attention layer; the input end of the image preprocessing module is configured to receive a reference image, the input end of the text preprocessing module is configured to receive a prompt text, the output end of the image preprocessing module is connected to the input end of the first cross attention layer, and the output end of the text preprocessing module is connected to the second cross attention layer; the output end of the first cross attention layer and the output end of the second cross attention layer are respectively connected to the input end of the target generation module;

[0128] The image preprocessing module is configured to perform feature extraction on the reference image to obtain initial image features corresponding to the reference image;

[0129] The text preprocessing module is configured to perform feature extraction on the prompt text to obtain initial text features corresponding to the prompt text;

[0130] The first cross-attention layer is configured to perform cross-attention calculations on the initial image features to obtain target image features;

[0131] The second cross-attention layer is configured to perform cross-attention calculation on the initial text features to obtain target text features.

[0132] In one embodiment, the video generation model further includes a diffusion enhancement module, wherein an input end of the diffusion enhancement module is configured to receive a reference image, and an output end of the diffusion enhancement module is connected to an input end of the target generation module;

[0133] The diffusion enhancement module is configured to generate at least one corresponding conditional control signal based on the reference image;

[0134] Generate target video based on target image features and target text features, including:

[0135] A target video is generated according to target image features, target text features and at least one conditional control signal.

[0136] In one embodiment, the diffusion enhancement module includes at least one control signal generating unit, and different control signal generating units are used to generate different conditional control signals; at least one conditional control signal includes one or more of an edge detection map, a posture skeleton map, a depth map, and a semantic segmentation map.

[0137] In one embodiment, the video generation model further includes a mask processing module, wherein an input end of the mask processing module is configured to receive a reference image, and an output end of the mask processing module is connected to an input end of the target generation module;

[0138] The mask processing module is configured to generate a mask corresponding to the target area in the reference image; the mask is used to indicate that the target area is a motion area of ​​the reference image;

[0139] Generating a target video according to target image features, target text features, and at least one conditional control signal, including:

[0140] A target video is generated according to the target image feature, the target text feature, the mask, and at least one conditional control signal.

[0141] In one embodiment, the video generating device further includes:

[0142] A first determining module is configured to determine a target area of ​​the reference image in response to a user's area selection operation on the reference image; or

[0143] The second determination module is used to input the reference image and the prompt text into the target detection model to obtain the target area output by the target detection model.

[0144] In one embodiment, the video generation model further includes a post-processing module, and the output end of the target generation module is connected to the input end of the post-processing module;

[0145] The post-processing module is configured to perform post-processing on the target video to obtain an optimized video; the post-processing includes super-resolution processing and / or frame insertion processing.

[0146] By adopting the technical solution of the embodiment of the present application, by utilizing the image prompt module to decouple the reference image and the prompt text, the image features can be retained more explicitly, thereby effectively avoiding semantic confusion caused by direct coupling, and achieving efficient alignment and refined control of image and text features, so that the target generation module can generate more accurate and user-expected high-quality videos based on the target image features and target text features, thereby improving the user experience.

[0147] For the specific definition of the video generation device, please refer to the definition of the video generation method above and will not be repeated here. Each module in the above-mentioned video generation device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0148] In addition, the present application also provides an electronic device, such as Figure 5 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0149] The electronic device may include one or more processors 501 of processing cores and one or more computer-readable storage media memories 502 and other components. Figure 3 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0150] The processor 501 is the control center of the electronic device. It connects the various parts of the entire electronic device using various interfaces and lines. By running or executing software programs and / or modules stored in the memory 502 and accessing data stored in the memory 502, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 501 may include one or more processing cores; preferably, the processor 501 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, user interface, and application programs, and the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into the processor 501.

[0151] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0152] In one feasible embodiment, the electronic device further includes a power supply 503 for supplying power to various components. Preferably, the power supply 503 can be logically connected to the processor 501 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 503 can also include any of one or more DC or AC power supplies, a recharging system, a power supply device debugging circuit, a power converter or inverter, a power status indicator, and other components.

[0153] In a feasible embodiment, the electronic device may further include an input unit 504, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0154] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically, in this embodiment, the processor 501 in the electronic device loads the executable files corresponding to one or more application processes into the memory 502 according to the following instructions, and the processor 501 runs the application stored in the memory 502, thereby implementing the steps of any of the video generation methods provided in the embodiments of the present application.

[0155] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0156] In a feasible embodiment, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method described in any embodiment of the present application is implemented.

[0157] In one feasible embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in any embodiment of the present application is implemented.

[0158] In a feasible implementation, a computer program product is also proposed, including a computer program or instructions, which implement the method described in any embodiment of the present application when executed by a processor.

[0159] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0160] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0161] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any video generation method provided in the present application.

[0162] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0163] The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0164] Since the instructions stored in the computer-readable storage medium can execute the steps in any video generation method provided in the present application, the beneficial effects that can be achieved by any video generation method provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0165] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "includes," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements that are inherent to such process, method, article, or terminal device. In the absence of further restrictions, an element defined by the phrase "comprises a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0166] The above is a detailed introduction to a video generation method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A video generation method, characterized in that: include: Get reference images and prompt text; Inputting the reference image and the prompt text into a video generation model to obtain a target video output by the video generation model; wherein the video generation model includes an image prompt module and a target generation module; The image prompt module is configured to perform decoupling processing on the reference image and the prompt text to obtain target image features and target text features; The target generation module is configured to generate the target video according to the target image features and the target text features.

2. The video generation method according to claim 1, wherein: The image prompt module includes an image preprocessing module, a text preprocessing module and a decoupled cross attention module, wherein the decoupled cross attention module includes a first cross attention layer and a second cross attention layer; an input end of the image preprocessing module is configured to receive the reference image, an input end of the text preprocessing module is configured to receive the prompt text, an output end of the image preprocessing module is connected to an input end of the first cross attention layer, and an output end of the text preprocessing module is connected to the second cross attention layer; An output end of the first cross attention layer and an output end of the second cross attention layer are respectively connected to an input end of the target generation module; The image preprocessing module is configured to perform feature extraction on the reference image to obtain initial image features corresponding to the reference image; The text preprocessing module is configured to perform feature extraction on the prompt text to obtain initial text features corresponding to the prompt text; The first cross-attention layer is configured to perform cross-attention calculation on the initial image features to obtain target image features; The second cross-attention layer is configured to perform cross-attention calculation on the initial text features to obtain target text features.

3. The video generation method according to claim 1, wherein: The video generation model further includes a diffusion enhancement module, wherein an input end of the diffusion enhancement module is configured to receive the reference image, and an output end of the diffusion enhancement module is connected to an input end of the target generation module; The diffusion enhancement module is configured to generate at least one corresponding conditional control signal based on the reference image; Generating the target video according to the target image feature and the target text feature includes: The target video is generated according to the target image feature, the target text feature and at least one of the conditional control signals.

4. The video generation method according to claim 3, wherein: The diffusion enhancement module includes at least one control signal generating unit, and different control signal generating units are used to generate different conditional control signals; at least one of the conditional control signals includes one or more of an edge detection map, a posture skeleton map, a depth map, and a semantic segmentation map.

5. The video generation method according to claim 3, characterized in that: The video generation model further includes a mask processing module, wherein an input end of the mask processing module is configured to receive the reference image, and an output end of the mask processing module is connected to an input end of the target generation module; The mask processing module is configured to generate a mask corresponding to the target area in the reference image; the mask is used to indicate that the target area is a motion area of ​​the reference image; Generating the target video according to the target image feature, the target text feature, and at least one of the conditional control signals includes: The target video is generated according to the target image feature, the target text feature, the mask and at least one of the conditional control signals.

6. The video generation method according to claim 5, characterized in that The method further comprises: In response to a user's region selection operation on the reference image, determining a target region of the reference image; or, The reference image and the prompt text are input into a target detection model to obtain the target area output by the target detection model.

7. The video generation method according to claim 1, characterized in that: The video generation model further includes a post-processing module, and the output end of the target generation module is connected to the input end of the post-processing module; The post-processing module is configured to perform post-processing on the target video to obtain an optimized video; the post-processing includes super-resolution processing and / or frame insertion processing.

8. A video generating device, characterized in that: The device comprises: A data acquisition module, used to obtain reference images and prompt texts; A video generation module, configured to input the reference image and the prompt text into a video generation model to obtain a target video output by the video generation model; wherein the video generation model includes an image prompt module and a target generation module; The image prompt module is configured to perform decoupling processing on the reference image and the prompt text to obtain target image features and target text features; The target generation module is configured to generate the target video according to the target image features and the target text features.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the video generation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the video generation method according to any one of claims 1 to 7 are implemented.

Images