Motion video generation method, device, electronic device and storage medium

Abstract

The present application relates to the field of artificial intelligence technology and provides a motion video generation method, device, electronic device and storage medium. The method obtains a reference video and a base model for motion video generation, injects LoRA components into the spatial module and time module of the base model respectively, trains the LoRA component using an asynchronous training strategy, then receives an instruction to generate a target motion video, determines the target weight of the LoRA component, fuses the LoRA component with the base model using the target weight, and generates the target motion video based on the motion video instruction using the fused base model. This method generates a motion video using the motion information provided by the reference video without the need for trajectory guidance, and sequentially trains and fine-tunes the parameters of the spatial module LoRA component and the time module LoRA component through an asynchronous training strategy, thereby improving the controllability and robustness of the target motion video.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a motion video generation method, device, electronic device, and storage medium. Background Art

[0002] In recent years, significant progress has been made in VDM (Video Diffusion Model) technology, driving the rapid development of a wide range of cutting-edge applications, including advertising generation and virtual fitting. However, most current methods are limited in motion control. This is particularly true for subject editing tasks in creative marketing scenarios, where maintaining consistent motion before and after editing is a significant challenge.

[0003] Most current motion control methods guide the production process based on user-specified trajectories. However, these methods still have the following problems: trajectory information makes it difficult to model small movements, such as shaking the head or raising the hand; and trajectory information cannot describe camera movements, such as zooming in, zooming out, panning, and rotating the lens. Summary of the Invention

[0004] In view of this, embodiments of the present application provide a motion video generation method, device, electronic device, and storage medium to solve the problem of poor automatic motion video generation in the prior art.

[0005] A first aspect of the embodiments of the present application provides a method for generating a motion video, comprising:

[0006] Acquire a reference video, where the reference video at least includes motion information of a first moving object;

[0007] Obtain a base model generated by a motion video, inject low-rank adaptive LoRA components into the spatial module and temporal module of the base model respectively, and train the spatial module LoRA component and the temporal module LoRA component using an asynchronous training strategy to obtain the trained spatial module LoRA component and the trained temporal module LoRA component;

[0008] receiving a target motion video generation instruction, wherein the target motion video generation instruction includes at least motion information of a second motion object in the target motion video, wherein the motion information of the second motion object is consistent with the motion information of the first motion object in the reference video;

[0009] Determine the target weight of the space module LoRA component and the target weight of the time module LoRA component;

[0010] Use the target weights of the spatial module LoRA component and the target weights of the temporal module LoRA component to fuse the trained spatial module LoRA component and the trained temporal module LoRA component with the base model to obtain a fused video generation model;

[0011] Based on the instructions for generating target motion video, the target motion video is generated using the fused video generation model.

[0012] A second aspect of the embodiments of the present application provides a motion video generation device, including:

[0013] an acquisition module configured to acquire a reference video, where the reference video at least includes motion information of a first moving object;

[0014] A training module is configured to obtain a base model generated by a motion video, inject a low-rank adaptive LoRA component into a spatial module and a temporal module of the base model respectively, and train the spatial module LoRA component and the temporal module LoRA component using an asynchronous training strategy to obtain a trained spatial module LoRA component and a trained temporal module LoRA component;

[0015] A receiving module is configured to receive an instruction for generating a target motion video, wherein the instruction for generating a target motion video includes at least motion information of a second motion object in the target motion video, wherein the motion information of the second motion object is consistent with the motion information of the first motion object in the reference video;

[0016] A determination module configured to determine a target weight of a spatial module LoRA component and a target weight of a temporal module LoRA component;

[0017] A fusion module is configured to fuse the trained spatial module LoRA component and the trained temporal module LoRA component with the base model using the target weight of the spatial module LoRA component and the target weight of the temporal module LoRA component to obtain a fused video generation model;

[0018] The generation module is configured to generate the target motion video based on the target motion video generation instruction and using the fused video generation model.

[0019] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0020] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above method are implemented.

[0021] The beneficial effects of the embodiments of the present application compared with the prior art are as follows: the embodiments of the present application obtain a reference video and a base model for generating a motion video, inject LoRA components into the spatial module and the temporal module of the base model respectively, train the LoRA component using an asynchronous training strategy, then receive an instruction to generate a target motion video, determine the target weight of the LoRA component, use the target weight to fuse the LoRA component with the base model, and use the fused base model based on the motion video instruction to generate the target motion video, thereby achieving the generation of motion video using the motion information provided by the reference video without the need for trajectory guidance, and sequentially training and fine-tuning the parameters of the spatial module LoRA component and the temporal module LoRA component through the asynchronous training strategy, thereby achieving the decoupling of the temporal and spatial modules, avoiding the spatial module LoRA component from learning the timing parameters of the reference video, and enabling the temporal module LoRA component to learn pure motion information to the greatest extent, thereby improving the controllability and robustness of the target motion video. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] Figure 1 This is a flow chart of a motion video generation method provided in an embodiment of the present application.

[0024] Figure 2 This is a flowchart of a method for training the spatial module LoRA component and the temporal module LoRA component using an asynchronous training strategy provided in an embodiment of the present application.

[0025] Figure 3 This is a flowchart of a method for training the LoRA component of a spatial module using a reference video provided in an embodiment of the present application.

[0026] Figure 4 This is a flowchart of a method for training a time module LoRA component using a reference video provided in an embodiment of the present application.

[0027] Figure 5 It is a flowchart of a method for determining the target weight of the LoRA component of the space module provided in an embodiment of the present application.

[0028] Figure 6 1 is a flow chart of a method for determining the target weight of the LoRA component of the time module provided in an embodiment of the present application.

[0029] Figure 7 This is a flowchart of another motion video generation method provided in an embodiment of the present application.

[0030] Figure 8 2 is a schematic diagram of a motion video generating device provided in an embodiment of the present application.

[0031] Figure 9 Schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0033] A motion video generation method and apparatus according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0034] As mentioned above, most current motion control methods guide the production process based on user-specified trajectories. However, these methods still have the following problems: trajectory information makes it difficult to model small movements, such as shaking the head or raising the hand; and trajectory information cannot describe camera movements, such as zooming in, zooming out, panning, and rotating.

[0035] In view of this, an embodiment of the present application provides a motion video generation method, which obtains a reference video and a base model for motion video generation, injects LoRA components into the spatial module and time module of the base model respectively, trains the LoRA component using an asynchronous training strategy, and then receives an instruction to generate a target motion video, determines the target weight of the LoRA component, uses the target weight to fuse the LoRA component with the base model, and uses the fused base model based on the motion video instruction to generate the target motion video, thereby achieving the generation of motion video without trajectory guidance using the motion information provided by the reference video, and sequentially training and fine-tuning the parameters of the spatial module LoRA component and the time module LoRA component through the asynchronous training strategy, which can achieve decoupling of the time and space modules, avoid the spatial module LoRA component from learning the timing parameters of the reference video, and enable the time module LoRA component to learn pure motion information to the greatest extent, thereby improving the controllability and robustness of the target motion video.

[0036] Figure 1 This is a flow chart of a motion video generation method provided by an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0037] In step S101 , a reference video is obtained.

[0038] The reference video at least includes motion information of the first moving object.

[0039] In step S102, a base model generated by a motion video is obtained, and low-rank adaptive LoRA components are injected into the spatial module and the temporal module of the base model respectively. The spatial module LoRA component and the temporal module LoRA component are trained using an asynchronous training strategy to obtain a trained spatial module LoRA component and a trained temporal module LoRA component.

[0040] In step S103, an instruction to generate a target motion video is received.

[0041] The instruction for generating the target motion video at least includes motion information of a second motion object in the target motion video, and the motion information of the second motion object is consistent with the motion information of the first motion object in the reference video.

[0042] In step S104, the target weight of the spatial module LoRA component and the target weight of the temporal module LoRA component are determined.

[0043] In step S105, the trained spatial module LoRA component and the trained temporal module LoRA component are fused with the base model using the target weight of the spatial module LoRA component and the target weight of the temporal module LoRA component to obtain a fused video generation model.

[0044] In step S106 , the target motion video is generated based on the instruction to generate the target motion video and using the fused video generation model.

[0045] In some embodiments of the present application, the method can be executed by a terminal device or a server. In some implementations, the method can be used to generate motion videos in application scenarios such as creative migration advertising design, video virtual fitting, animation production, and video face swapping.

[0046] In some embodiments of the present application, a reference video may be first obtained, where the reference video includes at least motion information of a first moving object. For example, the reference video may be a virtual fitting video, which includes motion information of a first object trying on one or more items of clothing. In another example, the reference video may be an advertising video, which includes motion information of a first displayed product.

[0047] In certain embodiments of the present application, a base model for motion video generation can also be obtained, and LoRA components can be injected into the spatial module and temporal module of the base model respectively. Among them, the base model can be a conventional motion video generation model, and its model parameters are usually fixed. In order to improve the accuracy and flexibility of the base model in generating motion video, a LoRA component can be injected into it, and the parameters of the LoRA component can be trained by bypassing the work mode. Then, the trained LoRA component is fused with the base model, and finally the fused model is used to generate motion video.

[0048] In some embodiments of the present application, in order to improve the model training efficiency of the LoRA component, so that the trained LoRA component can better generate motion videos with higher consistency and robustness after being fused with the base model, an asynchronous training strategy can be used to train the spatial module LoRA component and the temporal module LoRA component to achieve decoupling of the time and space modules, avoid the spatial module LoRA component from learning the timing parameters of the reference video, and enable the temporal module LoRA component to learn pure motion information to the greatest extent.

[0049] In some embodiments, a reference video can be used to train the spatial module LoRA component and the temporal module LoRA component. During training, a reference video and a text description of the reference video can be obtained, and the reference video and the text description of the reference video can be used together to train the LoRA component. The text description of the reference video can be input by the user or generated based on the reference video, which is not limited here.

[0050] In some embodiments of the present application, a target motion video generation instruction input by a user can also be received, and the target motion video generation instruction includes at least motion information of a second motion object in the target motion video, and the motion information of the second motion object is consistent with the motion information of the first motion object in the reference video.

[0051] For example, if the reference video is the aforementioned virtual fitting video, the target motion video the user wishes to generate could be a video of a second person trying on the same clothing as in the reference video. For another example, if the reference video is the aforementioned advertising video, the target motion video the user wishes to generate could be a video of the second product moving in the same manner as in the reference video.

[0052] In some embodiments of the present application, the target weight of the spatial module LoRA component and the target weight of the temporal module LoRA component can be determined. Among them, the target weight of the spatial module LoRA component can be determined according to the generate target motion video instruction, and the target weight of the temporal module LoRA component can be dynamically adjusted during the motion video generation process. The target weight of the spatial module LoRA component and the target weight of the temporal module LoRA component are determined in detail below and will not be repeated here.

[0053] After determining the target weights of the spatial module LoRA component and the temporal module LoRA component, the trained spatial module LoRA component and the trained temporal module LoRA component can be fused with the base model using the target weights of the spatial module LoRA component and the temporal module LoRA component to obtain a fused video generation model. Finally, based on the generate target motion video instruction, the target motion video can be generated using the fused video generation model.

[0054] According to the technical solution provided in the embodiment of the present application, by obtaining a reference video and a base model for generating a motion video, LoRA components are injected into the spatial module and the temporal module of the base model respectively, the LoRA component is trained using an asynchronous training strategy, and then an instruction to generate a target motion video is received, the target weight of the LoRA component is determined, the LoRA component is fused with the base model using the target weight, and the target motion video is generated using the fused base model based on the motion video instruction. This realizes the generation of motion video using the motion information provided by the reference video without the need for trajectory guidance, and sequentially trains and fine-tunes the parameters of the spatial module LoRA component and the temporal module LoRA component through the asynchronous training strategy, which can realize the decoupling of the temporal and spatial modules, avoid the spatial module LoRA component from learning the timing parameters of the reference video, and enable the temporal module LoRA component to learn pure motion information to the greatest extent, thereby improving the controllability and robustness of the target motion video.

[0055] In some embodiments of the present application, the base model may be a diffusion model. In this case, the spatial module of the base model may be the spatial self-attention and text-space cross-attention modules in the diffusion model, wherein the text is the text corresponding to the target motion video instruction, which may be input by the user, that is, the target motion video instruction is directly input by the user in the form of a text instruction. On the other hand, the text may also be generated by parsing other instructions input by the user, such as voice instructions, which are not limited here.

[0056] Meanwhile, the temporal module of the base model can be the temporal self-attention module in the diffusion model.

[0057] Figure 21 is a flow chart of a method for training the spatial module LoRA component and the time module LoRA component using an asynchronous training strategy provided by an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0058] In step S201, the spatial module LoRA component is trained using a reference video to obtain a trained spatial module LoRA component.

[0059] In step S202, in response to determining that the training of the spatial module LoRA component is completed, the temporal module LoRA component is trained using a reference video to obtain a trained temporal module LoRA component.

[0060] In some embodiments of the present application, when the spatial module LoRA component and the temporal module LoRA component are trained using an asynchronous training strategy, the spatial module LoRA component can be first trained using a reference video to obtain a trained spatial module LoRA component. Then, when it is determined that the training of the spatial module LoRA component is completed, the temporal module LoRA component is trained using the reference video to obtain a trained temporal module LoRA component.

[0061] That is, when using an asynchronous training strategy for the spatial module LoRA component and the temporal module LoRA component, in each round of training, the spatial module LoRA component can be trained first to learn the appearance information of the reference video, and then the temporal module LoRA component. This asynchronous training strategy allows the temporal module LoRA component to further learn the motion pattern of the reference video based on the content generated by the spatial module LoRA component without additional learning of appearance information.

[0062] Figure 3 1 is a flow chart of a method for training a spatial module LoRA component using a reference video provided in an embodiment of the present application. Figure 3 As shown, the method includes the following steps:

[0063] In step S301 , a random frame of image in a reference video is obtained as a target training image.

[0064] In step S302, the target training image is used to train the spatial module LoRA component to obtain a trained spatial module LoRA component.

[0065] In some embodiments of the present application, when training the spatial module LoRA component, a random frame image in the reference video can be obtained as a target training image, and then the target training image is used to train the spatial module LoRA component, thereby obtaining a trained spatial module LoRA component.

[0066] The above training process can be repeated N times, each time randomly selecting a random frame image from the reference video as the target training image to train the spatial module LoRA component. N is a positive integer.

[0067] This approach uses a diffusion model that includes the spatial module LoRA component to denoise a random frame from a reference video to train the spatial module LoRA component. This allows the spatial module LoRA component to learn only the fixed appearance content of the reference video, preventing it from learning the temporal information in the reference video. Furthermore, using a random frame for denoising prevents the spatial module LoRA component from learning only the content of a single frame, which could hinder the learning of subsequent temporal information.

[0068] Figure 4 1 is a flow chart of a method for training a time module LoRA component using a reference video provided in an embodiment of the present application. Figure 4 As shown, the method includes the following steps:

[0069] In step S401, any two frames of images in a reference video are obtained.

[0070] In step S402 , the difference between any two frames of images is predicted to obtain the cross-frame difference between any two frames of images.

[0071] In step S403, the time module LoRA component is trained using the difference between each frame image and the cross-frame difference between each frame image and any other frame image as training data to obtain a trained time module LoRA component.

[0072] In some embodiments of the present application, when training the time module LoRA component, one can first obtain any two frames of images in a reference video and predict the difference between the two frames to obtain the cross-frame difference between the two frames. Then, the difference between each frame image and the cross-frame difference between it and any other frame image is used as training data to train the time module LoRA component, thereby obtaining a trained time module LoRA component.

[0073] In other words, when training the temporal module LoRA component, all frames in the reference video can be denoised for training, allowing the temporal module LoRA component to fully learn the motion information in the reference video. In addition, the appearance information in the reference video can be eliminated by predicting cross-frame differences, allowing the temporal module LoRA component to learn the pure motion information to the greatest extent possible.

[0074] For example, if the reference video includes n frames, the cross-frame difference between the i-th frame and the j-th frame can be predicted respectively. The i-th frame minus the cross-frame difference is then used as the j-th value of the i-th frame, and the j-th frame minus the cross-frame difference is used as the i-th value of the j-th frame. Finally, the temporal module LoRA component is trained based on all the values ​​of the n-frame image. Where n is a positive integer greater than 1, i and j are both positive integers greater than or equal to 1 and less than or equal to n, and i and j are different.

[0075] Figure 5 : is a flow chart of a method for determining the target weight of a space module LoRA component provided in an embodiment of the present application. Figure 5 As shown, the method includes the following steps:

[0076] In step S501, the type of instruction for generating a target motion video is obtained and received.

[0077] In step S502 , the editing type is determined based on the instruction type.

[0078] In step S503 , the spatial module LoRA component weight corresponding to the editing type is determined as the target weight of the spatial module LoRA component.

[0079] In some embodiments of the present application, when determining the target weight of the spatial module LoRA component, the target motion video instruction type can be first obtained and received, and the editing type can be determined based on the instruction type. Then, the spatial module LoRA component weight corresponding to the editing type is determined as the target weight of the spatial module LoRA component.

[0080] That is, when generating motion videos, different types of target motion video generation instructions may result in different sizes of the edited video area. For example, some types of instructions only require changing the main body of the video, in which case the video generation model needs to retain most of the area outside the main body; for another example, some types of instructions also require changing the style of the video, in which case the video generation model needs to change the entire area of ​​the video. Therefore, different target weights can be set for the spatial module LoRA component based on the instruction type.

[0081] In one example, the spatial module LoRA component weights corresponding to different editing types can be predefined. Then, the text corresponding to the target motion video generation instruction can be classified, for example using a pre-trained text classification model, to obtain the instruction type for the target motion video generation instruction and determine the editing type corresponding to the instruction type. Then, the spatial module LoRA component weight corresponding to the editing type can be found to determine the target weight of the corresponding spatial module LoRA component.

[0082] Figure 6: is a flow chart of a method for determining the target weight of the time module LoRA component provided in an embodiment of the present application. Figure 6 As shown, the method includes the following steps:

[0083] In step S601, the initial weight of the LoRA component of the time module is determined.

[0084] In step S602, a first motion video is generated based on the target motion video generation instruction and using the base model without the fusion of the temporal module LoRA component weights.

[0085] In step S603, based on the target motion video generation instruction, a second motion video is generated using a base model that incorporates the initial weights of the time module LoRA component.

[0086] In step S604, in response to determining that the content consistency between the first motion video and the second motion video is less than or equal to the first preset threshold, the initial weight is increased to obtain an updated time module LoRA component weight.

[0087] In step S605, in response to determining that the content consistency between the first motion video and the second motion video is greater than or equal to a second preset threshold, the initial weight is reduced to obtain an updated time module LoRA component weight; the second preset threshold is greater than the first preset threshold.

[0088] In step S606, the operation of generating an updated second motion video based on the target motion video instruction and using the base model that incorporates the updated initial weights of the time module LoRA component is iteratively executed until the iteration termination condition is met.

[0089] The iteration termination condition includes that the content consistency between the first motion video and the updated second motion video is greater than a first preset threshold and less than a second preset threshold.

[0090] In step S607 , the updated time module LoRA component weight when the iteration termination condition is satisfied is determined as the target weight of the time module LoRA component.

[0091] In some embodiments of the present application, when determining the target weight of the time module LoRA component, the initial weight of the time module LoRA component can be determined first. The initial weight can be an empirical value or a fixed value, which is not limited here.

[0092] In some embodiments, a first motion video can be generated based on a target motion video instruction and a base model that does not integrate the weights of the time module LoRA component, and a second motion video can be generated based on a target motion video instruction and a base model that integrates the initial weights of the time module LoRA component.

[0093] The first motion video and the second motion video can be compared for content consistency. If the content consistency between the first motion video and the second motion video is less than or equal to a first preset threshold, it can be considered that the motion information of the reference video has not been migrated to the newly generated motion video. In this case, the initial weight can be increased to obtain the updated time module LoRA component weight.

[0094] On the other hand, if it is determined that the content consistency between the first motion video and the second motion video is greater than or equal to the second preset threshold, it can be considered that the subject of the newly generated motion video is not different enough from the subject in the reference video. In this case, the initial weight is reduced to obtain the updated time module LoRA component weight.

[0095] After updating the weight of the time module LoRA component, an operation of generating an updated second motion video based on the instruction to generate a target motion video and using the base model that incorporates the updated initial weight of the time module LoRA component can be iteratively executed until the content consistency between the first motion video and the updated second motion video is greater than a first preset threshold and less than a second preset threshold. Finally, the updated weight of the time module LoRA component at the end of the iteration can be used as the target weight of the time module LoRA component.

[0096] That is to say, the target weight of the time module LoRA component can be adaptively updated based on the content consistency of the generated video under different circumstances during the video generation process until the optimal value is reached. Because the weight of the time module LoRA component is too small to be able to migrate the learned motion information to the newly generated video, it is too large to limit the flexibility of the model generation subject, and the constraint subject is highly consistent with the reference video. Therefore, the content consistency of the first motion video generated by the base model and the second motion video generated by the base model with the time module LoRA component added can be compared during the motion video generation process. When the consistency is low, it means that the motion has not yet been injected into the newly generated video. At this time, the time module LoRA component can be given a larger weight. If the consistency is too high, the weight of the time module LoRA component can be reduced accordingly to support the model to flexibly generate new subjects.

[0097] The technical solution provided by the embodiments of this application achieves efficient fine-tuning by adding trainable LoRA modules to the temporal and spatial modules of the base model. Using reference videos as training data, the spatial module LoRA component is trained to memorize video appearance, and the temporal module LoRA component is trained to memorize motion patterns, thereby obtaining a motion-controllable video generation model that does not require a trajectory. During the fusion process of the base model and LoRA component parameters, an adaptive calculation method for LoRA component weights is also used, enabling the model to combine motion controllability with subject generation flexibility.

[0098] At the same time, the technical solution provided by the embodiments of this application uses dual-granularity activated motion customization technology to efficiently fine-tune a small number of parameters in the LoRA component to "memorize" the motion patterns in the reference video and adaptively inject them into the generation process of the new video, thereby improving the controllability and robustness of the motion. The technical solution provided by the embodiments can effectively train the motion customization model and batch generate creative videos with controllable motion, controllable scenes, and diverse content. The model only needs to fine-tune a small number of parameters, ensuring the efficiency of the technology.

[0099] Figure 7 FIG. 1 is a flow chart of another motion video generation method provided by an embodiment of the present application. Figure 7 As shown, users can first enter a reference video and a text description of the reference video, where the text description of the reference video can also be automatically parsed from the reference video. In one example, users can directly upload a customized reference video and customize the corresponding text description of the reference video. Alternatively, a large multimodal understanding model can be used to automatically generate a text description of the reference video.

[0100] Then, the video base model is obtained and the temporal module LoRa component and the spatial module LoRa component are injected into it. The LoRa components of the base model can be trained using the reference video to achieve motion customization capabilities based on the reference video.

[0101] On the other hand, the user can also enter editing instructions, which can be the editing instructions that the user wants to generate for the target motion video. The multimodal understanding model can be called to optimize the editing instructions entered by the user and, after feedback to the user for confirmation, serve as input to the base model.

[0102] The classification model can be called according to the editing type of the editing instruction to obtain the target weight of the spatial module LoRA component. The time module LoRA component and the spatial module LoRA component are trained using the user input video and text description. The training process can also generate target motion videos according to the editing instructions, and adaptively adjust the target weight of the time module LoRA component based on the content consistency of the target motion videos generated under different conditions, until the trained spatial module LoRA component and time module LoRA component are obtained. In other words,

[0103] Finally, the trained spatial module LoRA component and temporal module LoRA component are fused with the base model, and the fused base model is used to generate the target motion video, thereby realizing the automatic generation of motion-controllable video.

[0104] The technical solutions provided by the embodiments of this application can effectively solve the problem of trajectory guidance being insensitive to small movements and camera motion in motion video generation, providing efficient training of motion controllable models. Furthermore, by adjusting the target weights of the temporal module LoRA component and the spatial module LoRA component, the model achieves both robust motion controllability and flexible subject generation capabilities.

[0105] All of the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0106] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0107] Figure 8 Schematic diagram of a motion video generation device provided by an embodiment of the present application. Figure 8 As shown, the device includes:

[0108] The acquisition module 801 is configured to acquire a reference video, where the reference video at least includes motion information of a first moving object.

[0109] The training module 802 is configured to obtain a base model generated by a motion video, inject low-rank adaptive LoRA components into the spatial module and the temporal module of the base model respectively, and train the spatial module LoRA component and the temporal module LoRA component using an asynchronous training strategy to obtain a trained spatial module LoRA component and a trained temporal module LoRA component.

[0110] The receiving module 803 is configured to receive an instruction for generating a target motion video, where the instruction for generating a target motion video at least includes motion information of a second motion object in the target motion video, where the motion information of the second motion object is consistent with the motion information of the first motion object in the reference video.

[0111] The determination module 804 is configured to determine the target weight of the spatial module LoRA component and the target weight of the temporal module LoRA component.

[0112] The fusion module 805 is configured to use the target weight of the spatial module LoRA component and the target weight of the temporal module LoRA component to fuse the trained spatial module LoRA component and the trained temporal module LoRA component with the base model to obtain a fused video generation model.

[0113] The generation module 806 is configured to generate the target motion video based on the instruction to generate the target motion video and using the fused video generation model.

[0114] According to the technical solution provided in the embodiment of the present application, by obtaining a reference video and a base model for generating a motion video, LoRA components are injected into the spatial module and the temporal module of the base model respectively, the LoRA component is trained using an asynchronous training strategy, and then an instruction to generate a target motion video is received, the target weight of the LoRA component is determined, the LoRA component is fused with the base model using the target weight, and the target motion video is generated using the fused base model based on the motion video instruction. This realizes the generation of motion video using the motion information provided by the reference video without the need for trajectory guidance, and sequentially trains and fine-tunes the parameters of the spatial module LoRA component and the temporal module LoRA component through the asynchronous training strategy, which can realize the decoupling of the temporal and spatial modules, avoid the spatial module LoRA component from learning the timing parameters of the reference video, and enable the temporal module LoRA component to learn pure motion information to the greatest extent, thereby improving the controllability and robustness of the target motion video.

[0115] In some embodiments, the base model is a diffusion model; the spatial module of the base model is the spatial self-attention and text-spatial cross-attention modules in the diffusion model, and the text is the text corresponding to the target motion video instructions; the temporal module of the base model is the temporal self-attention module in the diffusion model.

[0116] In some embodiments, the spatial module LoRA component and the temporal module LoRA component are trained using an asynchronous training strategy, including: using a reference video to train the spatial module LoRA component to obtain a trained spatial module LoRA component; in response to determining that the training of the spatial module LoRA component is completed, using a reference video to train the temporal module LoRA component to obtain a trained temporal module LoRA component.

[0117] In some embodiments, the spatial module LoRA component is trained using a reference video, including: obtaining a random frame image in the reference video as a target training image; using the target training image to train the spatial module LoRA component to obtain a trained spatial module LoRA component.

[0118] In some embodiments, a reference video is used to train the time module LoRA component, including: obtaining any two frames of images in the reference video; predicting the difference between any two frames of images to obtain the cross-frame difference between any two frames of images; using the difference between each frame image and the cross-frame difference between it and any other frame image as training data to train the time module LoRA component to obtain a trained time module LoRA component.

[0119] In some embodiments, determining the target weight of the spatial module LoRA component includes: obtaining a command type for receiving and generating a target motion video; determining an editing type based on the command type; and determining the weight of the spatial module LoRA component corresponding to the editing type as the target weight of the spatial module LoRA component.

[0120] In some embodiments, determining the target weight of the time module LoRA component includes: determining the initial weight of the time module LoRA component; generating a first motion video based on a target motion video instruction and using a base model that does not integrate the weight of the time module LoRA component; generating a second motion video based on a target motion video instruction and using a base model that integrates the initial weight of the time module LoRA component; in response to determining that the content consistency between the first motion video and the second motion video is less than or equal to a first preset threshold, increasing the initial weight to obtain an updated time module LoRA component weight; or in response to determining that the content consistency between the first motion video and the second motion video is greater than or equal to a second preset threshold, reducing the initial weight to obtain an updated time module LoRA component weight; the second preset threshold is greater than the first preset threshold; iteratively executing the operation of generating an updated second motion video based on the target motion video instruction and using the base model that integrates the initial weight of the updated time module LoRA component until an iteration termination condition is met; the iteration termination condition includes that the content consistency between the first motion video and the updated second motion video is greater than the first preset threshold and less than the second preset threshold; determining that the updated time module LoRA component weight when the iteration termination condition is met is the target weight of the time module LoRA component.

[0121] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0122] Figure 9 Schematic diagram of an electronic device provided in an embodiment of the present application. Figure 9 As shown, the electronic device 9 of this embodiment includes: a processor 901, a memory 902, and a computer program 903 stored in the memory 902 and executable by the processor 901. When the processor 901 executes the computer program 903, the steps of the above-described method embodiments are implemented. Alternatively, when the processor 901 executes the computer program 903, the functions of the modules / units in the above-described device embodiments are implemented.

[0123] The electronic device 9 may be a desktop computer, a notebook, a PDA, a cloud server or other electronic device. The electronic device 9 may include but is not limited to a processor 901 and a memory 902. Those skilled in the art will understand that Figure 9This is merely an example of the electronic device 9 and does not limit the electronic device 9 . The electronic device 9 may include more or fewer components than shown in the figure, or different components.

[0124] The processor 901 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0125] The memory 902 can be an internal storage unit of the electronic device 9, such as the hard disk or memory of the electronic device 9. The memory 902 can also be an external storage device of the electronic device 9, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. The memory 902 can also include both the internal storage unit of the electronic device 9 and an external storage device. The memory 902 is used to store computer programs and other programs and data required by the electronic device.

[0126] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0127] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the processes in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program may include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium.

[0128] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A motion video generation method for generating motion videos in creative transfer advertising design, video virtual fitting, animation production or video face swap application scenarios, characterized in that: include: Acquire a reference video, where the reference video at least includes motion information of a first moving object; Obtain a base model generated by a motion video, inject a low-rank adaptive LoRA component into the spatial module and the temporal module of the base model respectively, and train the spatial module LoRA component and the temporal module LoRA component using an asynchronous training strategy to obtain a trained spatial module LoRA component and a trained temporal module LoRA component; receiving an instruction for generating a target motion video, wherein the instruction for generating a target motion video includes at least motion information of a second motion object in the target motion video, wherein the motion information of the second motion object is consistent with the motion information of the first motion object in the reference video; Determine the target weight of the space module LoRA component and the target weight of the time module LoRA component; The trained spatial module LoRA component and the trained temporal module LoRA component are fused with the base model using the target weight of the spatial module LoRA component and the target weight of the temporal module LoRA component to obtain a fused video generation model; Based on the instruction to generate the target motion video, the target motion video is generated using the fused video generation model; Determine the target weights for the LoRA components of the space module, including: Get the instruction type for receiving and generating target motion video; determining an edit type based on the instruction type; Determine the spatial module LoRA component weight corresponding to the editing type as the target weight of the spatial module LoRA component; Determine the target weights for the LoRA components of the time module, including: Determine the initial weight of the time module LoRA component; Generate a first motion video based on the target motion video generation instruction and using a base model that does not incorporate the time module LoRA component weights; Generate a second motion video based on the target motion video generation instruction and using a base model that incorporates the initial weights of the time module LoRA component; In response to determining that the content consistency between the first motion video and the second motion video is less than or equal to a first preset threshold, increasing the initial weight to obtain an updated time module LoRA component weight; or In response to determining that the content consistency between the first motion video and the second motion video is greater than or equal to a second preset threshold, reducing the initial weight to obtain an updated time module LoRA component weight; the second preset threshold is greater than the first preset threshold; Iteratively executing an operation of generating an updated second motion video based on the instruction to generate a target motion video and using a base model that incorporates the initial weights of the updated time module LoRA component until an iteration termination condition is satisfied; the iteration termination condition includes that the content consistency between the first motion video and the updated second motion video is greater than a first preset threshold and less than a second preset threshold; The updated time module LoRA component weight when the iteration termination condition is satisfied is determined as the target weight of the time module LoRA component.

2. The method according to claim 1, characterized in that The base model is a diffusion model; The spatial modules of the base model are the spatial self-attention and text-space cross-attention modules in the diffusion model, and the text is the text corresponding to the target motion video instruction; The time module of the base model is the temporal self-attention module in the diffusion model.

3. The method according to claim 2, characterized in that The spatial module LoRA component and the temporal module LoRA component are trained using an asynchronous training strategy, including: Using the reference video to train the spatial module LoRA component to obtain a trained spatial module LoRA component; In response to determining that the training of the spatial module LoRA component is completed, the time module LoRA component is trained using the reference video to obtain a trained time module LoRA component.

4. The method according to claim 3, characterized in that Training the space module LoRA component using the reference video includes: Obtaining a random frame image from the reference video as a target training image; The target training image is used to train the spatial module LoRA component to obtain a trained spatial module LoRA component.

5. The method according to claim 3, characterized in that Training the time module LoRA component using the reference video includes: Obtain any two frames of images in the reference video; Predicting the difference between the two arbitrary frames of image to obtain the cross-frame difference between the two arbitrary frames of image; The time module LoRA component is trained using the difference between each frame image and the cross-frame difference between the frame image and any other frame image as training data to obtain a trained time module LoRA component.

6. A motion video generation device for generating motion videos in creative migration advertising design, video virtual fitting, animation production or video face-changing application scenarios, characterized in that: include: an acquisition module, configured to acquire a reference video, wherein the reference video at least includes motion information of a first moving object; A training module is configured to obtain a base model generated by a motion video, inject a low-rank adaptive LoRA component into a spatial module and a temporal module of the base model respectively, and train the spatial module LoRA component and the temporal module LoRA component using an asynchronous training strategy to obtain a trained spatial module LoRA component and a trained temporal module LoRA component; a receiving module configured to receive an instruction for generating a target motion video, wherein the instruction for generating a target motion video includes at least motion information of a second motion object in the target motion video, wherein the motion information of the second motion object is consistent with the motion information of the first motion object in the reference video; A determination module configured to determine a target weight of a spatial module LoRA component and a target weight of a temporal module LoRA component; A fusion module is configured to fuse the trained spatial module LoRA component and the trained temporal module LoRA component with the base model using the target weight of the spatial module LoRA component and the target weight of the temporal module LoRA component to obtain a fused video generation model; A generation module is configured to generate the target motion video based on the instruction to generate the target motion video and using the fused video generation model; The determination module is specifically configured to: obtain and receive a target motion video generation instruction type; determine an editing type based on the instruction type; determine a spatial module LoRA component weight corresponding to the editing type as a target weight of the spatial module LoRA component; The determination module is further configured to: determine the initial weight of the LoRA component of the time module; Based on the instruction to generate the target motion video, a first motion video is generated using a base model that does not fuse the weights of the time module LoRA component; based on the instruction to generate the target motion video, a second motion video is generated using a base model that fuses the initial weights of the time module LoRA component; In response to determining that the content consistency between the first motion video and the second motion video is less than or equal to a first preset threshold, increasing the initial weight to obtain an updated time module LoRA component weight; Or in response to determining that the content consistency between the first motion video and the second motion video is greater than or equal to a second preset threshold, reducing the initial weight to obtain an updated time module LoRA component weight; The second preset threshold is greater than the first preset threshold; the operation of generating an updated second motion video based on the target motion video generation instruction and using the base model that integrates the initial weight of the updated time module LoRA component is iteratively executed until the iteration termination condition is met; the iteration termination condition includes that the content consistency between the first motion video and the updated second motion video is greater than the first preset threshold and less than the second preset threshold; it is determined that the weight of the updated time module LoRA component when the iteration termination condition is met is the target weight of the time module LoRA component.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images