Video generation method and device, computer equipment and storage medium
By combining the multi-layer MLPs model, the Pose Transformer model and the SVD model embedded in the Gated Cross-Attention model, the problems of poor video generation and unclear pose expression are solved, and high-quality and coherent human action video generation are achieved.
Patent Information
- Application Number
- CN202411858695.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-17
- Publication Date
- 2025-04-15
AI Technical Summary
In the video generation, the prior art has problems such as poor video generation effect, unclear pose expression, difficulty in handling multi-person interactive scenes, and insufficient model performance in capturing details.
The multi-layer MLPs model, Pose Transformer model and SVD model are adopted. The SVD model is embedded with the Gated Cross-Attention model. By obtaining the key point position data and reference pictures of the video frame, high-quality and highly coherent human action videos are generated.
It realizes a delicate posture expression effect, ensures high consistency between the output video and the input conditions, and improves the clarity, consistency and smoothness of video generation.
Smart Images
Figure CN119835497A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of deep learning, computer vision, and generative artificial intelligence technology, and in particular to a video generation method, device, computer equipment, and storage medium. Background Art
[0002] Diffusion model-based video generation technology is a new method to gradually generate image sequences to form videos. It originates from the diffusion model, which converts random noise into clear images through a series of step-by-step denoising processes. When applied to video generation, the diffusion model not only needs to generate images for each frame, but also must ensure the temporal consistency between adjacent frames to achieve a natural video effect. Such techniques usually adopt conditional generation and introduce specific control information (such as text description or posture) to guide the diffusion process to generate video sequences that meet the expected content. In recent years, many studies have improved the cross-frame consistency mechanism and temporal dependency of video diffusion models, gradually improving the coherence and physical rationality of generated videos.
[0003] Posture-guided human motion video generation technology refers to the use of posture information as a control condition to generate human motion videos with specific movements or behaviors. The key to this technology is to use the human posture sequence as input to drive the generation model to synthesize motion videos corresponding to the posture frame by frame. This method has important application value in animation production, virtual human driving and immersive entertainment.
[0004] When using explicit motion signals to control human motion, image skeleton graphs are often used to express postures, but this expression method has limitations. First, the skeleton graph cannot distinguish spatial directions such as left and right, front and back, resulting in poor posture expression. In addition, the skeleton graph is difficult to effectively express complex scenes of multi-person interaction, which easily causes overlapping or confusion of movements, affecting the realism and consistency of the generated video.
[0005] Training on a weaker basic model for video generation will affect the generation effect. The reason is that the basic model lacks the ability to model high-quality videos, resulting in poor clarity, coherence, and smoothness of the generated videos. In addition, the weaker model is not good at capturing posture and movement details, resulting in distorted or unnatural human movements, which is difficult to meet the needs of practical applications. Summary of the invention
[0006] Based on this, it is necessary to address the technical problem that the video generation effect of the prior art is poor, and propose a video generation method, device, computer equipment and storage medium.
[0007] In a first aspect, a video generation method is provided, the method comprising:
[0008] Obtain key point position data and a reference picture corresponding to each video frame, wherein the reference picture is extracted based on the target video, and the key point position data corresponding to the video frame is obtained by generating key points based on the target video;
[0009] Video generation is performed based on the key point position data, the reference image, and a video generation model, wherein the video generation model includes a multi-layer MLPs model, a Pose Transformer model, and an SVD model, and the SVD model is embedded with a Gated Cross-Attention model.
[0010] In a second aspect, a video generation device is provided, the device comprising:
[0011] An acquisition module, used to acquire key point position data and a reference picture corresponding to each video frame, wherein the reference picture is extracted based on the target video, and the key point position data corresponding to the video frame is obtained by generating key points based on the target video;
[0012] A generation module is used to generate a video based on the key point position data, the reference image and a video generation model, wherein the video generation model includes a multi-layer MLPs model, a Pose Transformer model, and an SVD model, and the SVD model is embedded with a Gated Cross-Attention model.
[0013] According to a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned video generation method when executing the computer program.
[0014] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned video generation method are implemented.
[0015] The video generation method proposed in the present invention obtains key point position data and reference images corresponding to each video frame, wherein the reference image is extracted based on the target video, and the key point position data corresponding to the video frame is obtained by generating key points based on the target video, and then generates video based on the key point position data, the reference image and the video generation model, wherein the video generation model includes a multi-layer MLPs model, a PoseTransformer model, and an SVD model, wherein the SVD model is embedded with a Gated Cross-Attention model, and the SVD model is a large model for video generation. The present invention generates accurate action videos according to input postures through an SVD model embedded with a Gated Cross-Attention model, thereby achieving a delicate posture expression effect and ensuring a high degree of consistency between the output video content and the input conditions. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0017] in:
[0018] Figure 1 A diagram showing an application environment of a video generation method in an embodiment;
[0019] Figure 2 is a flowchart of a video generation method in one embodiment;
[0020] Figure 3 is a structural block diagram of a video generating device in one embodiment;
[0021] Figure 4 is a structural block diagram of a computer device in one embodiment;
[0022] Figure 5 It is a structural block diagram of a computer device in another embodiment. DETAILED DESCRIPTION
[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0024] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0026] The video generation method provided by the embodiment of the present invention can be applied in Figure 1In the application environment, the client 110 communicates with the server 120 through the network. The server 120 can receive the key point position data and reference pictures corresponding to each video frame through the client 110, wherein the reference picture is extracted based on the target video, and the key point position data corresponding to the video frame is obtained by generating key points based on the target video, and then the server 120 generates the video based on the key point position data, the reference picture and the video generation model, wherein the video generation model includes a multi-layer MLPs model, a Pose Transformer model, and an SVD model, wherein the SVD model is embedded with a Gated Cross-Attention model, and the SVD model is a large model for video generation. The present invention generates an accurate action video according to the input posture through the SVD model embedded with the Gated Cross-Attention model, realizes a delicate posture expression effect, and ensures that the output video content is highly consistent with the input conditions. Among them, the client 110 can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server 120 can be implemented by an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.
[0027] See also Figure 2 As shown, Figure 2 A schematic flow chart of a video generation method provided by an embodiment of the present invention includes the following steps:
[0028] Step S101: obtaining key point position data and a reference picture corresponding to each video frame, wherein the reference picture is extracted based on a target video, and the key point position data corresponding to the video frame is obtained by generating key points based on the target video;
[0029] Among them, the video frames can be multiple video frames randomly selected from the target video, for example, 14 video frames are selected, the target video can contain a video of a human body, the reference picture can be a video frame containing a human body image randomly selected from the target video, and the key point position data can be in the form of coordinates.
[0030] Step S102: Generate a video based on the key point position data, the reference image, and a video generation model, wherein the video generation model includes a multi-layer MLPs model, a Pose Transformer model, and an SVD model, and the SVD model is embedded with a Gated Cross-Attention model.
[0031] Among them, SVD (Stable Video Diffusion) is a large model for video generation; Pose Transformer is mainly composed of spatial modules (Spatial Module) and temporal modules (Temporal Module). Each module realizes self-attention calculation within and between frames, and gradually captures the spatiotemporal information of key points of the human body, thereby improving the modeling ability of complex movements and subtle posture changes; MLPs (Multi-layer Perceptron) is a common artificial neural network model, which is widely used in various tasks, including classification, regression and feature learning; Gated Cross-Attention model uses gated cross-attention technology.
[0032] As an example, all key point position data of 14 frames are first encoded through three layers of MLPs and then input into the Pose Transformer architecture for calculation. The Pose Transformer model uses the transformer's self-attention mechanism to mine the expression of human posture information from spatial and temporal dimensions.
[0033] As an example, the video generation model is used to generate 14 frames of video in which a person in a reference image performs an action according to the key point position data corresponding to the 14 frames of video frames and a reference image.
[0034] In one embodiment, the step of generating a video based on the key point position data, the reference picture and the video generation model includes:
[0035] Step S1021: inputting the key point position data into the multi-layer MLPs model for encoding to obtain the first feature;
[0036] Step S1022: inputting the encoded feature into the Pose Transformer model to obtain a second feature;
[0037] Step S1023: Based on the second feature and the reference image, the video is input into the SVD model for video generation, wherein the SVD model includes a Self Attention spatial layer and a Cross-Attention layer, an input port of the Gated Cross-Attention model is connected to the output port of the Self Attention spatial layer, another input port of the Cross-Attention model is connected to the output port of the Pose Transformer model, and the output port of the Gated Cross-Attention model is connected to the input port of the Cross-Attention layer. The Gated Cross-Attention model is used to process the features from the Self Attention spatial layer and the second feature output by the Pose Transformer model to obtain a third feature, and output the third feature to the Cross-Attention layer.
[0038] In this embodiment, after obtaining the second feature of the Pose Transformer output, the key point information is injected into the Stable Video Diffusion model through a Gated Cross-Attention module, wherein the Gated Cross-Attention is specifically embedded in the spatial layer Self Attention and Cross-Attention layers of the SVD, because the Pose information focuses more on spatial signals. The parameters of the gating mechanism used by Gated Cross-Attention are set to 0 before the start of training, which means that the output of SVD can be guaranteed to be normal when the training starts, which can greatly promote the effect of training. At the same time, in order to ensure the stability of SVD during training, only the Gated Cross-Attention module and the timing module of SVD are trained, which can ensure that the basic model SVD maintains the original video generation capability to the greatest extent.
[0039] It should be noted that injecting key point position data into the SVD model in the form of coordinates is an implicit control signal, so the difficulty of training is much greater than the currently popular explicit control signal injection. In order to improve the training speed of the video generation model, a two-stage training method is proposed: in the first stage, a lower resolution, that is, 360x240 resolution video is used for multiple rounds of iterative training to converge quickly; in the second stage, a higher resolution, such as 896x640 resolution, is used to fine-tune the model to improve the quality of the picture.
[0040] The video generation method proposed in the present invention obtains key point position data and reference images corresponding to each video frame, wherein the reference image is extracted based on the target video, and the key point position data corresponding to the video frame is obtained by generating key points based on the target video, and then generates video based on the key point position data, the reference image and the video generation model, wherein the video generation model includes a multi-layer MLPs model, a PoseTransformer model, and an SVD model, wherein the SVD model is embedded with a Gated Cross-Attention model, and the SVD model is a large model for video generation. The present invention generates accurate action videos according to input postures through an SVD model embedded with a Gated Cross-Attention model, thereby achieving a delicate posture expression effect and ensuring a high degree of consistency between the output video content and the input conditions.
[0041] See also Figure 3 As shown, in one embodiment, a video generating device is provided, the device comprising:
[0042] An acquisition module 10 is used to acquire key point position data and a reference picture corresponding to each video frame, wherein the reference picture is extracted based on the target video, and the key point position data corresponding to the video frame is obtained by generating key points based on the target video;
[0043] The generation module 20 is used to generate a video based on the key point position data, the reference image and the video generation model, wherein the video generation model includes a multi-layer MLPs model, a Pose Transformer model, and an SVD model, and the SVD model is embedded with a Gated Cross-Attention model. The SVD model is a large model for video generation.
[0044] In one embodiment, the generating module 20 is further used for:
[0045] The key point position data is input into the multi-layer MLPs model for encoding to obtain the first feature;
[0046] Inputting the encoded feature into the Pose Transformer model to obtain a second feature;
[0047] Based on the second feature and the reference image, the video is generated by inputting into the SVD model, wherein the SVD model includes a Self Attention space layer and a Cross-Attention layer, an input port of the Gated Cross-Attention model is connected to the output port of the Self Attention space layer, another input port of the Cross-Attention model is connected to the output port of the Pose Transformer model, and the output port of the Gated Cross-Attention model is connected to the input port of the Cross-Attention layer.
[0048] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the functions or steps on the server side of a video generation method are implemented.
[0049] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, the functions or steps on the client side of a video generation method are implemented.
[0050] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:
[0051] Obtain key point position data and a reference picture corresponding to each video frame, wherein the reference picture is extracted based on the target video, and the key point position data corresponding to the video frame is obtained by generating key points based on the target video;
[0052] Video generation is performed based on the key point position data, the reference image and a video generation model, wherein the video generation model includes a multi-layer MLPs model, a Pose Transformer model and an SVD model, the SVD model is embedded with a Gated Cross-Attention model, and the SVD model is a large video generation model.
[0053] The present invention generates accurate action videos according to input postures through an SVD model embedded with a Gated Cross-Attention model, achieves a delicate posture expression effect, and ensures that the output video content is highly consistent with the input conditions.
[0054] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:
[0055] Obtain key point position data and a reference picture corresponding to each video frame, wherein the reference picture is extracted based on the target video, and the key point position data corresponding to the video frame is obtained by generating key points based on the target video;
[0056] Video generation is performed based on the key point position data, the reference image and a video generation model, wherein the video generation model includes a multi-layer MLPs model, a Pose Transformer model and an SVD model, the SVD model is embedded with a Gated Cross-Attention model, and the SVD model is a large video generation model.
[0057] The present invention generates accurate action videos according to input postures through an SVD model embedded with a Gated Cross-Attention model, achieves a delicate posture expression effect, and ensures that the output video content is highly consistent with the input conditions.
[0058] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.
[0059] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0060] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0061] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.
Claims
1. A video generation method, characterized in that: The video generation method comprises: Obtain key point position data and a reference picture corresponding to each video frame, wherein the reference picture is extracted based on the target video, and the key point position data corresponding to the video frame is obtained by generating key points based on the target video; Video generation is performed based on the key point position data, the reference image and a video generation model, wherein the video generation model includes a multi-layer MLPs model, a Pose Transformer model and an SVD model, the SVD model is embedded with a Gated Cross-Attention model, and the SVD model is a large video generation model.
2. The video generation method according to claim 1, characterized in that: The step of generating a video based on the key point position data, the reference picture and the video generation model comprises: The key point position data is input into the multi-layer MLPs model for encoding to obtain the first feature; Inputting the encoded feature into the Pose Transformer model to obtain a second feature; Based on the second feature and the reference image, the video is generated by inputting into the SVD model, wherein the SVD model includes a Self Attention space layer and a Cross-Attention layer, an input port of the Gated Cross-Attention model is connected to the output port of the Self Attention space layer, another input port of the Cross-Attention model is connected to the output port of the Pose Transformer model, and the output port of the Gated Cross-Attention model is connected to the input port of the Cross-Attention layer.
3. The video generation method according to claim 2, characterized in that: The Gated Cross-Attention model is used to process the features from the Self Attention spatial layer and the second features output by the Pose Transformer model to obtain a third feature, and output the third feature to the Cross-Attention layer.
4. A video generating device, characterized in that: The video generating device comprises: An acquisition module, used to acquire key point position data and a reference picture corresponding to each video frame, wherein the reference picture is extracted based on the target video, and the key point position data corresponding to the video frame is obtained by generating key points based on the target video; A generation module is used to generate a video based on the key point position data, the reference image and a video generation model, wherein the video generation model includes a multi-layer MLPs model, a Pose Transformer model, and an SVD model, wherein the SVD model is embedded with a Gated Cross-Attention model, and the SVD model is a large video generation model.
5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the video generation method according to any one of claims 1 to 3 are implemented.
6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the video generation method according to any one of claims 1 to 3 are implemented.