Text-based motion video generation method and device, storage medium and equipment

Abstract

The invention discloses a text-based motion video generation method and device, a storage medium and equipment, and belongs to the technical field of computers. Obtaining a positive prompt word and a negative prompt word; performing text coding on the positive prompt words and the negative prompt words to obtain text features; the Animatediff model and the LCM Lora model are inserted into a Stable-Diffusion model, and a video generation model is obtained; processing the text features by using a video generation model to obtain a motion video; and performing video enhancement processing on the motion video by using the video enhancement processing model to obtain a high-definition motion video. According to the method, the LCM Lora model can be used for reasoning acceleration, the ControlNet model is used for enabling the video action to be consistent with the action in the action reference video, and the video enhancement processing model is used for generating the high-definition video.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a text-based motion video generation method, device, storage medium and equipment. Background Art

[0002] Animatediff is an effective framework to extend the text-to-image model to the animation generator without tuning for a specific model. As long as the motion prior knowledge is learned in a large video dataset, AnimateDiff can be inserted into a personalized text-to-image model, which is compatible with existing text-to-image models and with large models that are fine-tuned by yourself. However, the AnimateDiff model has problems such as slow video generation, deformed and discontinuous motion, and low clarity of generated videos. Summary of the invention

[0003] The present application provides a text-based motion video generation method, device, storage medium and equipment, which are used to solve the problems of slow video generation, deformed and discontinuous motion, and low definition of the generated video in the AnimateDiff model. The technical solution is as follows:

[0004] According to a first aspect of the present application, a text-based motion video generation method is provided, the method comprising:

[0005] Obtaining positive prompt words and negative prompt words for the motion video to be generated;

[0006] Performing text encoding on the positive prompt words and the negative prompt words to obtain text features;

[0007] Insert the Animatediff model and the LCM Lora model into the Stable-Diffusion model to obtain a video generation model, wherein the Animatediff model is used to impart motion capability, and the LCM Lora model is used to accelerate reasoning;

[0008] Processing the text features using the video generation model to obtain a motion video;

[0009] The motion video is subjected to video enhancement processing by using a video enhancement processing model to obtain a high-definition motion video, wherein the video enhancement processing model is used for performing video enhancement.

[0010] In one possible implementation,

[0011] The method further comprises: obtaining a reference video of the action of the character;

[0012] The inserting of the Animatediff model and the LCM Lora model into the Stable-Diffusion model to obtain the video generation model includes: inserting the Animatediff model, the LCM Lora model and the ControlNet model into the Stable-Diffusion model to obtain the video generation model, wherein the ControlNet module is used to provide reference actions;

[0013] The method of using the video generation model to process the text features to obtain a motion video includes: using the video generation model to process the text features and the action reference video to obtain a motion video.

[0014] In a possible implementation, inserting the Animatediff model, the LCM Lora model, and the ControlNet model into the Stable-Diffusion model to obtain the video generation model includes:

[0015] Connecting each module in the LCM Lora model in parallel with a corresponding module in the Stable-Diffusion model to obtain a plurality of parallel modules;

[0016] Adding some modules in the ControlNet model to corresponding parallel modules to obtain parallel modules;

[0017] After inserting each module in the Animatediff model into each parallel module, a video generation model is obtained.

[0018] In a possible implementation, the using the video generation model to process the text features and the action reference video to obtain the motion video includes:

[0019] The text features are processed respectively by the LCM Lora module and the Stable-Diffusion module in the first parallel module to obtain two first processing results; the two first processing results are fused into a second processing result; the action reference video is processed by the jth ControlNet module, and the third processing result obtained is added to the second processing result to obtain a fourth processing result; the fourth processing result is processed by the first Animatediff module to obtain a first feature map, j≥1;

[0020] The first feature map output by the i-1th Animatediff module is processed by the LCM Lora module and the Stable-Diffusion module in the i-th parallel module to obtain two first processing results; the two first processing results are merged into a second processing result; the third processing result output by the j+i-2th ControlNet module is processed by the j+i-1th ControlNet module, and the third processing result is added to the second processing result to obtain a fourth processing result; the fourth processing result is processed by the i-th Animatediff module to obtain a first feature map, i≥2;

[0021] Combine the first feature maps output by the last Animatediff module multiple times into a motion video.

[0022] In a possible implementation, the step of performing video enhancement processing on the motion video using a video enhancement processing model to obtain a high-definition motion video includes:

[0023] When the video enhancement processing model includes multiple video enhancement modules and dual regulators, each video enhancement module is used to process the motion video to obtain each second feature map; the dual regulator is used to adjust each second feature map and the motion video to obtain a high-definition motion video.

[0024] In a possible implementation, when the first video enhancement module includes a maximum value processing module, an edge prior processing module, an AIE-BAN module and a three-dimensional convolution layer, and other video enhancement modules include an AIE-BAN module and a three-dimensional convolution layer, the motion video is processed by each video enhancement module to obtain each second feature map, including:

[0025] Using the maximum value processing module to perform maximum value processing on the motion video to obtain a first fifth processing result; using the edge prior processing module to perform edge prior processing on the motion video to obtain a first sixth processing result; using the AIE-BAN module in the first video enhancement module to process the first fifth processing result, the first sixth processing result and the motion video to obtain a first seventh processing result; using the three-dimensional convolution layer in the first video enhancement module to perform three-dimensional convolution on the seventh processing result to obtain a first second feature map;

[0026] Perform tensor shape matching on the first fifth processing result to obtain the kth fifth processing result; perform tensor shape matching on the first sixth processing result to obtain the kth sixth processing result; use the AIE-BAN module in the kth video enhancement module to process the kth fifth processing result, the kth sixth processing result and the k-1th second feature map output by the k-1th video enhancement module to obtain the kth seventh processing result; use the three-dimensional convolution layer in the kth video enhancement module to perform three-dimensional convolution on the kth seventh processing result to obtain the kth second feature map, k≥2.

[0027] In a possible implementation, when the AIE-BAN module includes a convolution activation unit, a dual regulator, and a sampling unit,

[0028] The using the AIE-BAN module in the first video enhancement module to process the first fifth processing result, the first sixth processing result and the motion video to obtain the first seventh processing result includes: using the convolution activation unit in the first AIE-BAN module to process the motion video and the first fifth processing result respectively and then add them to obtain the first eighth processing result; using the dual regulator in the first AIE-BAN module to process the motion video and the first eighth processing result respectively to obtain the first ninth processing result; using the convolution activation unit to process the first sixth processing result and the first ninth processing result respectively to obtain the first tenth processing result; using the sampling unit in the first AIE-BAN module to process the first tenth processing result to obtain the first seventh processing result;

[0029] The method of using the AIE-BAN module in the kth video enhancement module to process the kth fifth processing result, the kth sixth processing result and the k-1th second feature map output by the k-1th video enhancement module to obtain the kth seventh processing result includes: using the convolution activation unit in the kth AIE-BAN module to process the k-1th second feature map output by the k-1th video enhancement module and the kth fifth processing result respectively and then add them to obtain the kth eighth processing result; using the dual regulator in the kth AIE-BAN module to process the k-1th second feature map output by the k-1th video enhancement module and the kth eighth processing result respectively to obtain the kth ninth processing result; using the convolution activation unit to process the kth sixth processing result and the kth ninth processing result respectively to obtain the kth tenth processing result; using the sampling unit in the kth AIE-BAN module to process the kth tenth processing result to obtain the kth seventh processing result.

[0030] According to a second aspect of the present application, a text-based motion video generation device is provided, the device comprising:

[0031] An acquisition module, used to acquire positive prompt words and negative prompt words of the motion video to be generated;

[0032] An encoding module, used for performing text encoding on the positive prompt words and the negative prompt words to obtain text features;

[0033] An insertion module is used to insert the Animatediff model and the LCM Lora model into the Stable-Diffusion model to obtain a video generation model, wherein the Animatediff model is used to impart motion capability, and the LCM Lora model is used to accelerate reasoning;

[0034] A generation module, used for processing the text features using the video generation model to obtain a motion video;

[0035] The generation module is also used to perform video enhancement processing on the motion video using a video enhancement processing model to obtain a high-definition motion video, and the video enhancement processing model is used to perform video enhancement.

[0036] According to a third aspect of the present application, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the text-based motion video generation method as described above.

[0037] According to a fourth aspect of the present application, a computer device is provided, wherein the computer device includes the above-mentioned text-based motion video generation apparatus.

[0038] The beneficial effects of the technical solution provided by this application include at least:

[0039] The LCM-LoRA model is a lightweight module. By introducing the LCM LoRA module to accelerate the inference of the Stable Diffusion model, the inference process of the model can be optimized, the inference speed can be significantly improved, the video generation time can be shortened, the video generation efficiency can be greatly improved, and the consumption of computing resources can be reduced.

[0040] The ControlNet model can capture and encode the motion information in the motion reference video and pass the motion information to the Stable Diffusion model, so that the generated video motion is consistent with the motion in the motion reference video, achieving precise control, thereby ensuring that the generated video motion is highly consistent and controllable.

[0041] By introducing a video enhancement processing model, the generated motion video is restored to high definition frame by frame, making the generated video clearer and having higher visual quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 It is a schematic diagram of the network structure of the video generation model;

[0044] Figure 2 It is a schematic diagram of the network structure of the video enhancement processing model;

[0045] Figure 3 It is a schematic diagram of the network structure of the AIE-BAN model;

[0046] Figure 4 is a flow chart of a text-based motion video generation method;

[0047] Figure 5 is a schematic diagram of a text-based motion video generation method;

[0048] Figure 6 is a flow chart of a text-based motion video generation method;

[0049] Figure 7 It is a structural block diagram of a text-based motion video generation device. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0051] The AnimateDiff model adds animation capabilities to the text-to-image model. The core of the AnimateDiff model is to insert a newly initialized motion modeling module into the text-to-image model and extract reasonable motion prior knowledge by training it on a video dataset. Once trained, the text-to-image model can be transformed into a text-driven video model by simply injecting the motion model module, which can generate diverse and personalized dynamic images, covering the fields of anime pictures and realistic photos, while maintaining the domain characteristics and diversity of its output. However, the Animatediff model has problems such as slow video generation, deformed and discontinuous motion, and low clarity of generated videos.

[0052] In order to solve the problem of slow video generation in the Animatediff model, the improvement method is to introduce the Latent Consistency Model (LCM) Low-Rank Adaptation (Lora) model to accelerate reasoning. The LCM Lora model is a general training-free acceleration module that can be directly inserted into various Stable Diffusion models, supporting resource savings brought by fast reasoning technology with minimal steps. Generally speaking, the training of the latent consistency model is carried out in a single-stage guided distillation method. This method uses the pre-trained autoencoder latent space to distill the guided diffusion model into LCM, which ensures that the generated samples follow the trajectory of generating high-quality images.

[0053] In order to solve the problem of motion deformation and discontinuity in the Animatediff model, the improvement method is to introduce the ControlNet model to control motion generation. The ControlNet module can be used to selectively use the character motion in the motion reference video as a condition to control the batch generation of images with continuous specified motions. The ControlNet model can capture and encode the motion information in the motion reference video, pass the motion information to the Stable Diffusion model, make the generated video motion consistent with the motion in the motion reference video, achieve precise control, and ensure that the generated video motion is highly consistent and controllable.

[0054] Aiming at the problem of low video clarity generated by Animatediff model, the improvement method is to introduce a video enhancement processing model and use the video enhancement processing model to perform video enhancement processing on motion video to obtain high-definition motion video, thereby improving the video clarity.

[0055] If you need to use action reference videos to guide actions, you need to insert the Animatediff model, LCM Lora model, and ControlNet model into the Stable-Diffusion model to obtain the first video generation model; if you do not need to use action reference videos to guide actions, you need to insert the Animatediff model and LCM Lora model into the Stable-Diffusion model to obtain the second video generation model. The following describes the model structures of these two video generation models.

[0056] (1) The first video generation model:

[0057] Inserting the Animatediff model, the LCM Lora model and the ControlNet model into the Stable-Diffusion model to obtain a video generation model can include: connecting each module in the LCM Lora model in parallel with the corresponding module in the Stable-Diffusion model to obtain multiple parallel modules; adding some modules in the ControlNet model to the corresponding parallel modules to obtain parallel modules; and inserting each module in the Animatediff model into each parallel module to obtain the video generation model.

[0058] like Figure 1 As shown, the green hexagon represents the Stable-Diffusion model, the gray square represents the Stable-Diffusion module, the large blue square above represents the Animatediff model, the small blue square in the green hexagon represents the Animatediff module, the large orange square above represents the LCM Lora model, the small orange square in the green hexagon represents the LCM Lora module, the pink hexagon represents the ControlNet model, and the pink square represents the ControlNet module. Figure 1 In the figure, the LCM Lora module and the Stable-Diffusion module form a parallel module, the ControlNet module is added to the parallel module to form a parallel module, and the Animatediff module is located after each parallel module to combine into a video generation model.

[0059] (2) The second video generation model:

[0060] Inserting the Animatediff model and the LCM Lora model into the Stable-Diffusion model to obtain a video generation model can include: connecting each LCM Lora module in the LCM Lora model in parallel with a corresponding Stable-Diffusion module in the Stable-Diffusion model to obtain a plurality of parallel modules; and inserting each Animatediff module in the Animatediff model after each parallel module to obtain the video generation model.

[0061] like Figure 1 As shown in Figure 1, after removing the ControlNet model in the dotted box, all the modules in the green hexagon form a video generation model.

[0062] The model structure of the video enhancement processing model is explained below.

[0063] like Figure 2As shown, the video enhancement processing model includes multiple video enhancement modules and dual regulators ( Figure 2 (not shown in the figure), the first video enhancement module includes a maximum processing module, an edge prior processing module, an AIE-BAN module and a three-dimensional convolution layer, and other video enhancement modules include an AIE-BAN module and a three-dimensional convolution layer, and the three-dimensional convolution layer includes a three-dimensional convolution module and a nonlinear activation function. Figure 2 The green hexagon in the figure represents the video enhancement processing model. The first green square represents the combination of the maximum processing module, the edge prior processing module and the AIE-BAN module. The other green squares represent the AIE-BAN modules. The pink squares represent the three-dimensional convolutional layers.

[0064] After obtaining the weight parameters of the AIE-BAN module, the video can be used to train the video enhancement processing model. During the training process, the weight parameters of the AIE-BAN module need to be fixed and the weight parameters of the three-dimensional convolution layer need to be trained. Specifically, the video enhancement processing model can be used to process the input video to obtain the output video, and the mean square error (MSE) and structural similarity (SSIM) are used to calculate the loss values ​​of the input video and the output video, and the weight parameters of the three-dimensional convolution layer are updated according to the loss values.

[0065] like Figure 3 As shown, Figure 3 The orange block IL in the figure represents the feature map output by the maximum (max) processing module, the orange block CI represents the feature map output by the edge prior model (EPM) module, and the light pink block and the dark pink block both represent the AIE-BAN module.

[0066] The AIE-BAN module includes a convolutional activation unit, a dual regulator, and a sampling unit. The convolutional activation unit includes a convolution operation + activation function, which is represented by a gray square; the dual regulator includes a brightener and a dimmer, both represented by a yellow square; the sampling unit includes downsampling or upsampling, both represented by a blue square. When the sampling unit in the AIE-BAN module is a downsampling unit, the light pink square represents the AIE-BAN module + downsampling; when the sampling unit in the AIE-BAN module is an upsampling unit, the dark pink square represents the AIE-BAN module + upsampling.

[0067] like Figure 2 As shown, the original video can be subjected to any one or more of blur processing, noise processing, and brightness processing to obtain multiple degraded videos. The original video and the corresponding degraded videos are combined into a group of training samples, and the video enhancement processing model is trained using multiple groups of training samples.

[0068] like Figure 4 As shown, it shows a method flow chart of a text-based motion video generation method provided by an embodiment of the present application, and the text-based motion video generation method can be applied to a computer device. The text-based motion video generation method may include:

[0069] Step 401, obtaining positive prompt words and negative prompt words of the motion video to be generated.

[0070] Positive prompt words are used to describe elements that you want to appear in the video, such as a red-haired girl dancing, etc.

[0071] Negative cue words are used to describe elements that are not desired to appear in the video, such as poor quality, blurry, etc.

[0072] Step 402: Perform text encoding on the positive prompt words and the negative prompt words to obtain text features.

[0073] In this embodiment, a text encoder may be used to encode the positive prompt words and the negative prompt words respectively to obtain two text features.

[0074] Step 403: insert the Animatediff model and the LCM Lora model into the Stable-Diffusion model to obtain a video generation model. The Animatediff model is used to impart motion capability, and the LCM Lora model is used to accelerate reasoning.

[0075] After inserting the Animatediff model and the LCM Lora model into the Stable-Diffusion model, a video generation model can be obtained. The structure of the video generation model is as follows: Figure 1 shown.

[0076] Step 404: Use the video generation model to process the text features to obtain a motion video.

[0077] Specifically, using the video generation model to process text features to obtain motion video may include:

[0078] (1) Using the LCM Lora module and the Stable-Diffusion module in the first parallel module to process the text features respectively, and obtain two first processing results; fusing the two first processing results into a second processing result; using the first Animatediff module to process the second processing result, and obtain a third feature map.

[0079] (2) Use the LCM Lora module and the Stable-Diffusion module in the i-th parallel module to process the third feature map output by the i-1th Animatediff module respectively to obtain two first processing results; fuse the two first processing results into a second processing result; use the i-th Animatediff module to process the second processing result to obtain a third feature map, i≥2.

[0080] (3) Combine the third feature maps output multiple times by the last Animatediff module into a motion video.

[0081] Simply put, the input of the first parallel module is the text feature, the input of the i-th (i≥2) parallel module is the third feature map output by the i-1-th parallel module, and the last parallel module outputs the third feature map multiple times, and all the third feature maps are combined into a motion video.

[0082] Step 405, using the video enhancement processing model to perform video enhancement processing on the motion video to obtain a high-definition motion video, the video enhancement processing model is used to perform video enhancement.

[0083] The video enhancement processing model can perform any one or more of blur processing, noise processing, and brightness processing on the motion video to obtain a high-definition motion video.

[0084] like Figure 5 As shown in the figure, the Animatediff model and the LCM Lora model are inserted into the Stable-Diffusion model to obtain a video generation model; the positive prompt words and the negative prompt words are encoded and input into the video generation model; the video generation model is used to process the text features to obtain a motion video; the video enhancement processing model is used to process the motion video to obtain a high-definition motion video.

[0085] To sum up, the text-based motion video generation method provided in the embodiment of the present application, the LCM-LoRA model is a lightweight module. By introducing the LCM LoRA module to accelerate the reasoning of the Stable Diffusion model, the reasoning process of the model can be optimized, the reasoning speed can be significantly improved, the video generation time can be shortened, the video generation efficiency can be greatly improved, and the consumption of computing resources can be reduced.

[0086] By introducing a video enhancement processing model, the generated motion video is restored to high definition frame by frame, making the generated video clearer and having higher visual quality.

[0087] like Figure 6As shown, it shows a flowchart of a text-based motion video generation method provided by an embodiment of the present application, and the text-based motion video generation method can be applied to a computer device. The text-based motion video generation method may include:

[0088] Step 601, obtain positive prompt words and negative prompt words of the motion video to be generated, and obtain a reference video of the character's action.

[0089] Positive prompt words are used to describe elements that you want to appear in the video, such as a red-haired girl dancing, etc.

[0090] Negative cue words are used to describe elements that are not desired to appear in the video, such as poor quality, blurry, etc.

[0091] The action reference video may be a video containing reference actions, such as a video of a person dancing.

[0092] Step 602: Perform text encoding on the positive prompt words and the negative prompt words to obtain text features.

[0093] In this embodiment, a text encoder may be used to encode the positive prompt words and the negative prompt words respectively to obtain two text features.

[0094] Step 603: insert the Animatediff model, the LCM Lora model and the ControlNet model into the Stable-Diffusion model to obtain a video generation model. The ControlNet module is used to provide reference actions.

[0095] After inserting the Animatediff model, LCM Lora model and ControlNet model into the Stable-Diffusion model, a video generation model can be obtained. The structure of the video generation model is as follows: Figure 1 shown.

[0096] Step 604: Use the video generation model to process the text features and the action reference video to obtain a motion video.

[0097] Specifically, using the video generation model to process the text features and the action reference video to obtain the motion video may include:

[0098] (1) Use the LCM Lora module and the Stable-Diffusion module in the first parallel module to process the text features respectively to obtain two first processing results; fuse the two first processing results into a second processing result; use the jth ControlNet module to process the action reference video, add the third processing result obtained to the second processing result to obtain a fourth processing result; use the first Animatediff module to process the fourth processing result to obtain a first feature map, j≥1.

[0099] The value of j can be set according to actual needs. Figure 1 For example, starting from the most central ControlNet module in the ControlNet model, assuming that the ControlNet model includes 5 ControlNet modules, the value of j is 3; assuming that the ControlNet model includes 7 ControlNet modules, the value of j is 4, and so on.

[0100] (2) Use the LCM Lora module and the Stable-Diffusion module in the i-th parallel module to process the first feature map output by the i-1th Animatediff module respectively to obtain two first processing results; merge the two first processing results into a second processing result; use the j+i-1th ControlNet module to process the third processing result output by the j+i-2th ControlNet module, add the third processing result to the second processing result to obtain a fourth processing result; use the i-th Animatediff module to process the fourth processing result to obtain a first feature map, i≥2.

[0101] (3) Combine the first feature maps output multiple times by the last Animatediff module into a motion video.

[0102] Simply put, the input of the first parallel module is the text feature, the input of the i-th (i≥2) parallel module is the first feature map output by the i-1-th parallel module, and the last parallel module outputs the first feature map multiple times, and all the first feature maps are combined into a motion video.

[0103] Step 605, when the video enhancement processing model includes multiple video enhancement modules and dual regulators, each video enhancement module is used to process the motion video to obtain each second feature map; each second feature map and motion video is adjusted by the dual regulator to obtain a high-definition motion video.

[0104] The dual regulator includes a brightener and a dimmer. The calculation formula of the brightener is: The calculation formula for the dimming function is in, represents the weight graph, I(x) represents the input, and γ represents a constant.

[0105] When the first video enhancement module includes a maximum value processing module, an edge prior processing module, an AIE-BAN module and a three-dimensional convolution layer, and other video enhancement modules include an AIE-BAN module and a three-dimensional convolution layer, using each video enhancement module to process the motion video to obtain each second feature map may include:

[0106] (1) Performing maximum processing on the motion video using the maximum processing module to obtain a first fifth processing result; performing edge prior processing on the motion video using the edge prior processing module to obtain a first sixth processing result; using the AIE-BAN module in the first video enhancement module to process the first fifth processing result, the first sixth processing result and the motion video to obtain a first seventh processing result; performing three-dimensional convolution on the first seventh processing result using the three-dimensional convolution layer in the first video enhancement module to obtain a first second feature map.

[0107] (2) performing tensor shape matching on the first fifth processing result to obtain the kth fifth processing result; performing tensor shape matching on the first sixth processing result to obtain the kth sixth processing result; using the AIE-BAN module in the kth video enhancement module to process the kth fifth processing result, the kth sixth processing result and the k-1th second feature map output by the k-1th video enhancement module to obtain the kth seventh processing result; using the three-dimensional convolution layer in the kth video enhancement module to perform three-dimensional convolution on the kth seventh processing result to obtain the kth second feature map, k≥2.

[0108] Tensor matching refers to adjusting the first fifth processing result and the first sixth processing result to the same size as the k-1th second feature map, which may be unchanged or downsampled by 2 times, 4 times or more.

[0109] In simple terms, the input of the first video enhancement module is the motion video, and the input of the kth (k ≥ 2) video enhancement module is the k-1th second feature map output by the k-1th video enhancement module.

[0110] Among them, when the AIE-BAN module includes a convolutional activation unit, a dual regulator, and a sampling unit,

[0111] (1) Using the AIE-BAN module in the first video enhancement module to process the first fifth processing result, the first sixth processing result and the motion video to obtain a first seventh processing result, including: using the convolution activation unit in the first AIE-BAN module to process the motion video and the first fifth processing result respectively and then add them to obtain a first eighth processing result; using the dual regulator in the first AIE-BAN module to process the motion video and the first eighth processing result respectively to obtain a first ninth processing result; using the convolution activation unit to process the first sixth processing result and the first ninth processing result respectively to obtain a first tenth processing result; using the sampling unit in the first AIE-BAN module to process the first tenth processing result to obtain a first seventh processing result.

[0112] The AIE-BAN module can include multiple convolutional activation units. Figure 3 As shown, in one embodiment, the motion video is processed by 3 convolution activation units, the fifth processing result is processed by 3 convolution activation units, and the sixth processing result and the ninth processing result are processed by 3 convolution activation units.

[0113] (2) using the AIE-BAN module in the kth video enhancement module to process the kth fifth processing result, the kth sixth processing result and the k-1th second feature map output by the k-1th video enhancement module to obtain the kth seventh processing result, including: using the convolution activation unit in the kth AIE-BAN module to process the k-1th second feature map output by the k-1th video enhancement module and the kth fifth processing result respectively and then add them to obtain the kth eighth processing result; using the dual regulator in the kth AIE-BAN module to process the k-1th second feature map output by the k-1th video enhancement module and the kth eighth processing result respectively to obtain the kth ninth processing result; using the convolution activation unit to process the kth sixth processing result and the kth ninth processing result respectively to obtain the kth tenth processing result; using the sampling unit in the kth AIE-BAN module to process the kth tenth processing result to obtain the kth seventh processing result.

[0114] Simply put, the input of the convolutional activation unit in the first AIE-BAN module is the motion video, and the input of the convolutional activation unit in the kth (k ≥ 2)th AIE-BAN module is the k-1th second feature map output by the k-1th video enhancement module.

[0115] like Figure 5As shown in the figure, the Animatediff model, LCM Lora model and ControlNet model are inserted into the Stable-Diffusion model to obtain a video generation model; the positive prompt words and negative prompt words are encoded and input into the video generation model; the video generation model is used to process the text features to obtain a motion video; the video enhancement processing model is used to process the motion video to obtain a high-definition motion video.

[0116] To sum up, the text-based motion video generation method provided in the embodiment of the present application, the LCM-LoRA model is a lightweight module. By introducing the LCM LoRA module to accelerate the reasoning of the Stable Diffusion model, the reasoning process of the model can be optimized, the reasoning speed can be significantly improved, the video generation time can be shortened, the video generation efficiency can be greatly improved, and the consumption of computing resources can be reduced.

[0117] The ControlNet model can capture and encode the motion information in the motion reference video and pass the motion information to the Stable Diffusion model, so that the generated video motion is consistent with the motion in the motion reference video, achieving precise control, thereby ensuring that the generated video motion is highly consistent and controllable.

[0118] By introducing a video enhancement processing model, the generated motion video is restored to high definition frame by frame, making the generated video clearer and having higher visual quality.

[0119] like Figure 7 As shown, it shows a structural block diagram of a text-based motion video generation device provided by an embodiment of the present application, and the text-based motion video generation device can be applied to a computer device. The text-based motion video generation device may include:

[0120] An acquisition module 710 is used to acquire positive prompt words and negative prompt words of the sports video to be generated;

[0121] The encoding module 720 is used to perform text encoding on the positive prompt words and the negative prompt words to obtain text features;

[0122] Insertion module 730, used to insert the Animatediff model and the LCM Lora model into the Stable-Diffusion model to obtain a video generation model, wherein the Animatediff model is used to impart motion capability, and the LCM Lora model is used to accelerate reasoning;

[0123] A generation module 740 is used to process text features using a video generation model to obtain a motion video;

[0124] The generation module 740 is also used to perform video enhancement processing on the motion video using the video enhancement processing model to obtain a high-definition motion video. The video enhancement processing model is used to perform video enhancement.

[0125] In an optional embodiment, the acquisition module 710 is further used to acquire a reference video of a character's action;

[0126] The insertion module 730 is also used to insert the Animatediff model, the LCM Lora model and the ControlNet model into the Stable-Diffusion model to obtain a video generation model, and the ControlNet module is used to provide a reference action;

[0127] The generation module 740 is further used to process the text features and the action reference video using the video generation model to obtain the motion video.

[0128] In an optional embodiment, the insertion module 730 is further configured to include:

[0129] Each module in the LCM Lora model is connected in parallel with the corresponding module in the Stable-Diffusion model to obtain multiple parallel modules;

[0130] Add some modules in the ControlNet model to the corresponding parallel modules to obtain parallel modules;

[0131] After inserting each module in the Animatediff model into each parallel module, a video generation model is obtained.

[0132] In an optional embodiment, the generating module 740 is further configured to:

[0133] The text features are processed respectively by the LCM Lora module and the Stable-Diffusion module in the first parallel module to obtain two first processing results; the two first processing results are fused into a second processing result; the action reference video is processed by the jth ControlNet module, and the third processing result is added to the second processing result to obtain a fourth processing result; the fourth processing result is processed by the first Animatediff module to obtain a first feature map, j≥1;

[0134] The first feature map output by the i-1th Animatediff module is processed by the LCM Lora module and the Stable-Diffusion module in the i-th parallel module to obtain two first processing results; the two first processing results are merged into a second processing result; the third processing result output by the j+i-2th ControlNet module is processed by the j+i-1th ControlNet module, and the third processing result is added to the second processing result to obtain a fourth processing result; the fourth processing result is processed by the i-th Animatediff module to obtain a first feature map, i≥2;

[0135] Combine the first feature maps output by the last Animatediff module multiple times into a motion video.

[0136] In an optional embodiment, the generating module 740 is further configured to:

[0137] When the video enhancement processing model includes multiple video enhancement modules and dual regulators, each video enhancement module is used to process the motion video to obtain each second feature map; the dual regulator is used to adjust each second feature map and the motion video to obtain a high-definition motion video.

[0138] In an optional embodiment, when the first video enhancement module includes a maximum value processing module, an edge prior processing module, an AIE-BAN module and a three-dimensional convolution layer, and other video enhancement modules include an AIE-BAN module and a three-dimensional convolution layer, the generation module 740 is further used to:

[0139] Using the maximum value processing module to perform maximum value processing on the motion video, a first fifth processing result is obtained; using the edge prior processing module to perform edge prior processing on the motion video, a first sixth processing result is obtained; using the AIE-BAN module in the first video enhancement module to process the first fifth processing result, the first sixth processing result and the motion video, a first seventh processing result is obtained; using the three-dimensional convolution layer in the first video enhancement module to perform three-dimensional convolution on the first seventh processing result, a first second feature map is obtained;

[0140] Perform tensor shape matching on the first fifth processing result to obtain the kth fifth processing result; perform tensor shape matching on the first sixth processing result to obtain the kth sixth processing result; use the AIE-BAN module in the kth video enhancement module to process the kth fifth processing result, the kth sixth processing result and the k-1th second feature map output by the k-1th video enhancement module to obtain the kth seventh processing result; use the three-dimensional convolution layer in the kth video enhancement module to perform three-dimensional convolution on the kth seventh processing result to obtain the kth second feature map, k≥2.

[0141] In an optional embodiment, when the AIE-BAN module includes a convolution activation unit, a dual regulator, and a sampling unit, the generating module 740 is further configured to:

[0142] The motion video and the first fifth processing result are processed by the convolution activation unit in the first AIE-BAN module respectively and then added to obtain a first eighth processing result; the motion video and the first eighth processing result are processed by the dual regulator in the first AIE-BAN module respectively to obtain a first ninth processing result; the first sixth processing result and the first ninth processing result are processed by the convolution activation unit respectively to obtain a first tenth processing result; the first tenth processing result is processed by the sampling unit in the first AIE-BAN module to obtain a first seventh processing result;

[0143] The k-1th second feature map and the k-th fifth processing result output by the k-1th video enhancement module are processed and added by the convolution activation unit in the k-th AIE-BAN module to obtain the k-th eighth processing result; the k-1th second feature map and the k-th eighth processing result output by the k-1th video enhancement module are processed by the dual regulator in the k-th AIE-BAN module to obtain the k-th ninth processing result; the k-th sixth processing result and the k-th ninth processing result are processed by the convolution activation unit to obtain the k-th tenth processing result; the k-th tenth processing result is processed by the sampling unit in the k-th AIE-BAN module to obtain the k-th seventh processing result.

[0144] To sum up, the text-based motion video generation device provided in the embodiment of the present application, the LCM-LoRA model is a lightweight module. By introducing the LCM LoRA module to accelerate the reasoning of the Stable Diffusion model, the reasoning process of the model can be optimized, the reasoning speed can be significantly improved, the video generation time can be shortened, the video generation efficiency can be greatly improved, and the consumption of computing resources can be reduced.

[0145] The ControlNet model can capture and encode the motion information in the motion reference video and pass the motion information to the Stable Diffusion model, so that the generated video motion is consistent with the motion in the motion reference video, achieving precise control, thereby ensuring that the generated video motion is highly consistent and controllable.

[0146] By introducing a video enhancement processing model, the generated motion video is restored to high definition frame by frame, making the generated video clearer and having higher visual quality.

[0147] An embodiment of the present application provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the text-based motion video generation method as described above.

[0148] An embodiment of the present application provides a computer device, wherein the computer device includes any of the above-mentioned text-based motion video generation devices.

[0149] It should be noted that: the text-based motion video generation device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate when performing portrait control generation of a specific person. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the text-based motion video generation device is divided into different functional modules to complete all or part of the functions described above. In addition, the text-based motion video generation device provided in the above embodiment and the text-based motion video generation method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0150] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0151] The above description is not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. A method for generating a motion video based on text, characterized in that: The method comprises: Obtaining positive prompt words and negative prompt words for the motion video to be generated; Performing text encoding on the positive prompt words and the negative prompt words to obtain text features; Insert the Animatediff model and the LCM Lora model into the Stable-Diffusion model to obtain a video generation model, wherein the Animatediff model is used to impart motion capability, and the LCM Lora model is used to accelerate reasoning; Processing the text features using the video generation model to obtain a motion video; The motion video is subjected to video enhancement processing by using a video enhancement processing model to obtain a high-definition motion video, wherein the video enhancement processing model is used for performing video enhancement.

2. The method for generating a motion video based on text according to claim 1, characterized in that: The method further comprises: obtaining a reference video of the action of the character; The inserting of the Animatediff model and the LCM Lora model into the Stable-Diffusion model to obtain the video generation model includes: inserting the Animatediff model, the LCM Lora model and the ControlNet model into the Stable-Diffusion model to obtain the video generation model, wherein the ControlNet module is used to provide reference actions; The method of using the video generation model to process the text features to obtain a motion video includes: using the video generation model to process the text features and the action reference video to obtain a motion video.

3. The method for generating a text-based motion video according to claim 2, characterized in that: The method of inserting the Animatediff model, the LCM Lora model and the ControlNet model into the Stable-Diffusion model to obtain the video generation model includes: Connecting each module in the LCM Lora model in parallel with a corresponding module in the Stable-Diffusion model to obtain a plurality of parallel modules; Adding some modules in the ControlNet model to corresponding parallel modules to obtain parallel modules; After inserting each module in the Animatediff model into each parallel module, a video generation model is obtained.

4. The method for generating a motion video based on text according to claim 3, characterized in that: The step of using the video generation model to process the text features and the action reference video to obtain a motion video includes: The text features are processed respectively by the LCM Lora module and the Stable-Diffusion module in the first parallel module to obtain two first processing results; the two first processing results are fused into a second processing result; the action reference video is processed by the jth ControlNet module, and the third processing result obtained is added to the second processing result to obtain a fourth processing result; the fourth processing result is processed by the first Animatediff module to obtain a first feature map, j≥1; The first feature map output by the i-1th Animatediff module is processed by the LCM Lora module and the Stable-Diffusion module in the i-th parallel module to obtain two first processing results; the two first processing results are merged into a second processing result; the third processing result output by the j+i-2th ControlNet module is processed by the j+i-1th ControlNet module, and the third processing result is added to the second processing result to obtain a fourth processing result; the fourth processing result is processed by the i-th Animatediff module to obtain a first feature map, i≥2; Combine the first feature maps output by the last Animatediff module multiple times into a motion video.

5. The method for generating a motion video based on text according to claim 1, characterized in that: The step of performing video enhancement processing on the motion video using the video enhancement processing model to obtain a high-definition motion video includes: When the video enhancement processing model includes multiple video enhancement modules and dual regulators, each video enhancement module is used to process the motion video to obtain each second feature map; the dual regulator is used to adjust each second feature map and the motion video to obtain a high-definition motion video.

6. The method for generating a text-based motion video according to claim 5, characterized in that: When the first video enhancement module includes a maximum value processing module, an edge prior processing module, an AIE-BAN module and a three-dimensional convolution layer, and the other video enhancement modules include an AIE-BAN module and a three-dimensional convolution layer, the motion video is processed by each video enhancement module to obtain each second feature map, including: Using the maximum value processing module to perform maximum value processing on the motion video to obtain a first fifth processing result; using the edge prior processing module to perform edge prior processing on the motion video to obtain a first sixth processing result; using the AIE-BAN module in the first video enhancement module to process the first fifth processing result, the first sixth processing result and the motion video to obtain a first seventh processing result; using the three-dimensional convolution layer in the first video enhancement module to perform three-dimensional convolution on the first seventh processing result to obtain a first second feature map; Perform tensor shape matching on the first fifth processing result to obtain the kth fifth processing result; perform tensor shape matching on the first sixth processing result to obtain the kth sixth processing result; use the AIE-BAN module in the kth video enhancement module to process the kth fifth processing result, the kth sixth processing result and the k-1th second feature map output by the k-1th video enhancement module to obtain the kth seventh processing result; use the three-dimensional convolution layer in the kth video enhancement module to perform three-dimensional convolution on the kth seventh processing result to obtain the kth second feature map, k≥2.

7. The method for generating a motion video based on text according to claim 6, characterized in that: When the AIE-BAN module includes a convolutional activation unit, a dual regulator, and a sampling unit, The using the AIE-BAN module in the first video enhancement module to process the first fifth processing result, the first sixth processing result and the motion video to obtain the first seventh processing result includes: using the convolution activation unit in the first AIE-BAN module to process the motion video and the first fifth processing result respectively and then add them to obtain the first eighth processing result; using the dual regulator in the first AIE-BAN module to process the motion video and the first eighth processing result respectively to obtain the first ninth processing result; using the convolution activation unit to process the first sixth processing result and the first ninth processing result respectively to obtain the first tenth processing result; using the sampling unit in the first AIE-BAN module to process the first tenth processing result to obtain the first seventh processing result; The method of using the AIE-BAN module in the kth video enhancement module to process the kth fifth processing result, the kth sixth processing result and the k-1th second feature map output by the k-1th video enhancement module to obtain the kth seventh processing result includes: using the convolution activation unit in the kth AIE-BAN module to process the k-1th second feature map output by the k-1th video enhancement module and the kth fifth processing result respectively and then add them to obtain the kth eighth processing result; using the dual regulator in the kth AIE-BAN module to process the k-1th second feature map output by the k-1th video enhancement module and the kth eighth processing result respectively to obtain the kth ninth processing result; using the convolution activation unit to process the kth sixth processing result and the kth ninth processing result respectively to obtain the kth tenth processing result; using the sampling unit in the kth AIE-BAN module to process the kth tenth processing result to obtain the kth seventh processing result.

8. A text-based motion video generation device, characterized in that: The device comprises: An acquisition module, used to acquire positive prompt words and negative prompt words of the motion video to be generated; An encoding module, used for performing text encoding on the positive prompt words and the negative prompt words to obtain text features; An insertion module is used to insert the Animatediff model and the LCM Lora model into the Stable-Diffusion model to obtain a video generation model, wherein the Animatediff model is used to impart motion capability, and the LCM Lora model is used to accelerate reasoning; A generation module, used for processing the text features using the video generation model to obtain a motion video; The generation module is also used to perform video enhancement processing on the motion video using a video enhancement processing model to obtain a high-definition motion video, and the video enhancement processing model is used to perform video enhancement.

9. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the text-based motion video generation method according to any one of claims 1 to 7.

10. A computer device, characterized in that: The computer device comprises: the text-based motion video generating device as described in claim 8.

Images