Method and system for generating double-person armed video
In the video generation method in multi-subject scenes, the reference ID and text description are characterized by using VAE and CLIP models, and combined with the personalized attention layer and background generation module, high-quality two-player martial arts videos are generated, which solves the problem of difficulty in coordination of subject relationships in multi-subject scenes.
Patent Information
- Application Number
- CN202411887459.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-20
- Publication Date
- 2025-04-15
AI Technical Summary
The prior art image and video generation methods in multi-subject scenes are not ideal, and it is difficult to effectively balance and coordinate the interactions and relationships between multiple subjects.
A two-player martial arts video generation method and system is proposed. By obtaining target information, including reference ID, text description, initial background image and action posture sequence, processing and feature extraction are performed, reference ID and text description are characterized by using VAE and CLIP models, and combining personalized attention layer and background generation module, a video that integrates characters and backgrounds is generated.
It realizes the generation of characters that match the text description, ensures the natural fusion of characters and background, and outputs high-quality two-player martial arts videos, solving the problem of difficulty in coordination of subject relationships in multi-subject scenes.
Smart Images
Figure CN119835498A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of video generation, and in particular to a method and system for generating a two-person martial arts video. Background Art
[0002] In the early days, the research on video generation mainly focused on the generation of images and videos of single-subject scenes. The difficulty of implementing single-subject scenes and multi-subject scenes in terms of the mutual logic of the subjects, the interaction between the subject and the background, and the fidelity of the subject's details has been greatly reduced. As the generation methods of single-subject scenes continue to increase and the effects continue to improve, especially with the accumulation of data and computing power, people are slowly turning their attention to multi-subject scenes.
[0003] The current methods for generating images and videos in multi-agent scenarios are still not very ideal. The interactions and relationships between multiple subjects are complex. In multi-agent scenarios, there are various interactions and relationships between subjects, and the subjects in an image or video may have different attributes and characteristics. There is still no good general method to balance and coordinate these relationships.
[0004] In order to overcome these defects, the present application proposes a method and system for generating a two-person martial arts video. Summary of the invention
[0005] The purpose of this application is to provide a method and system for generating a two-person martial arts video, aiming to solve the above-mentioned problems.
[0006] To achieve the above objectives, this application provides the following technical solutions:
[0007] In a first aspect, the present application provides a method for generating a two-person martial arts video, comprising:
[0008] Acquire target information, the target information including: reference ID, text description, initial background image, action posture sequence;
[0009] Processing the target information to obtain condition information;
[0010] The conditional information is input into a network model for synthesis, and a synthesized video is output.
[0011] In a second aspect, the present application provides a two-person martial arts video generation system, comprising:
[0012] Acquiring a model: acquiring target information, the target information including: a reference ID, a text description, an initial background image, and an action posture sequence;
[0013] Processing module: processes the target information to obtain condition information;
[0014] Video synthesis module: inputs the conditional information into the network model for synthesis, and outputs the synthesized video.
[0015] In a third aspect, the present application provides a device, comprising a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing a method for generating a two-player martial arts video; and the processor is used to execute the program instructions stored in the memory to implement a two-player martial arts video generation.
[0016] In a fourth aspect, the present application provides a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute a method for generating a two-person martial arts video.
[0017] The present application provides a method and system for generating a two-person martial arts video, which has the following beneficial effects:
[0018] (1) Encode the original image provided by the reference ID through VAE to obtain a representation in the latent space; Use the CLIP model to align the reference ID with the text description to ensure that the features of the character in the generated video are consistent with the text semantics, so that the network model can accurately understand and generate a character that matches the text description;
[0019] (2) The feature-extracted and aligned representations and other conditional information are input into the network model, and random noise is gradually denoised in multiple rounds of iterations to generate an intermediate image sequence in which the character and the background are fused together. In this process, the personalized attention layer uses the attention mechanism corresponding to the reference ID to enable the generator to correctly inject the appearance features of the corresponding character in each frame, thereby ensuring that the generated character is consistent with the text description.
[0020] (3) Based on the initially generated character action sequence, the background generation module is used to generate and enhance the background, making the scene in which the character is located more realistic and detailed; all time series frames are decoded back to the pixel space through VAE and synthesized into the final output video. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 A schematic diagram of a process for generating a two-person martial arts video according to Embodiment 1 of the present application;
[0022] Figure 2 This is a schematic diagram of the traditional two-person video generation effect of Example 1 of the present application;
[0023] Figure 3 A schematic diagram of a framework of a method for generating a two-person martial arts video according to Embodiment 1 of the present application;
[0024] Figure 4 This is a structural schematic diagram of a two-person martial arts video generation system according to Embodiment 2 of the present application;
[0025] Figure 5 This is a schematic diagram of the device structure of Example 3 of the present application;
[0026] Figure 6 This is a schematic diagram of the storage medium structure of Example 4 of the present application. DETAILED DESCRIPTION
[0027] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0028] The following is an analysis of the solutions in the prior art in combination with related technologies.
[0029] In the video generation scenario under multi-scene scenarios, there are currently many problems, such as Figure 2 As shown, we can see that under the traditional generation method, there are several problems:
[0030] Whether the number of individuals in the video corresponds accurately: As the number of subjects increases, it is difficult to accurately capture and reflect the number of individuals in the video. The currently generated videos have incorrect numbers of subjects and do not match the expected number.
[0031] Whether the individual correspondence in the video is accurate: There is often a mismatch between the reference image and the generated video. The generated portrait is not the same as the original image, or there is an incorrect correspondence relationship, where the portrait of subject A corresponds to another individual (such as subject B).
[0032] Whether the individual information of the video remains consistent: the generated single-person dancing video often cannot maintain the same facial information as the reference image; the two-person fighting video often has subject confusion and other problems.
[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0034] Example 1
[0035] See also Figure 1 , is a flow chart of a method for generating a two-person martial arts video according to Example 1 of the present application; the steps include:
[0036] S1: Obtain target information, which includes: reference ID, text description, initial background image, and action posture sequence.
[0037] In this embodiment, the reference ID corresponds to an image of a person with specific appearance or character features; the text description includes semantic information such as character features and action scenes, for example, "a fat man with green skin and a young man in yellow tights are performing a martial arts fight"; the initial background image can be a simple background, such as a building scene, or a pure white background, or a random noise background image; the action posture sequence is a character posture that changes over time, which is used to guide the changes in the character's actions and time steps in the video; the action posture sequence is usually generated by a pre-trained posture estimation model DWPose, which specifically parses the video frames or images, extracts the joint point information of the human body, and outputs the action posture sequence.
[0038] S2: Process the target information to obtain condition information.
[0039] In this embodiment, the reference ID, text description, initial background image, and action posture sequence in the target information are processed, specifically including:
[0040] The original image provided by the reference ID is encoded through VAE (Variational AutoEncoder), and the representation in the latent space is recorded as z_id; z_id contains the visual feature information of the reference character.
[0041] The CLIP model is used to align the reference ID with the text description, and the matching representation of the text and the reference character in the common embedding space is recorded as z_clip_id, which ensures that the features of the character in the generated video are consistent with the text semantics.
[0042] The acquired text description, such as “a fat man with green skin and a young man in yellow tights engaging in martial arts fighting”, is mapped to a text embedding z_text through the CLIP text encoder; this text embedding will provide semantic guidance for the subsequent image generation process.
[0043] If the user provides a simple background image, such as a simple scene photo, a pure white background or a random noise image, the initial background image is input into the network model as conditional information. If it is a complex background, such as "two men fighting in an abandoned factory", the background generation module in the network model gradually "expands" or "enhances" the simple background into a complex and reasonable scene background according to the text description, and obtains BG_enhanced.
[0044] The pose sequence is input into the pose guidance module in the network model, and the corresponding position information and joint distribution pose_info are obtained by parsing the skeleton pose of each frame. Pose_info is used to guide the position and action of the character in each time frame, so that the final generated video character action accurately follows the given pose changes.
[0045] S3: Input the conditional information into the network model for synthesis, and output the synthesized video.
[0046] In this embodiment, the condition information includes: z_id, z_clip_id, z_text, BG_enhanced, pose_info; the condition information is input into the network model for synthesis, specifically including:
[0047] The personalized attention layer uses the attention mechanism corresponding to the reference ID to enable the generator to inject the appearance features of the corresponding character in each frame and place the character's appearance information in the corresponding spatiotemporal position.
[0048] The text criss-cross attention layer semantically constrains the scene and character appearance based on the text features in the conditional information.
[0049] The temporal attention layer utilizes the time series information to make the generation results between adjacent frames coherent.
[0050] After multiple rounds of iterations, the network model generates an intermediate image sequence frames_intermediate in which the character and the background are fused together.
[0051] Based on the initially generated character action sequence, if you choose to generate a more complex background through text description, the background generator uses the text description to generate and enhance the background, converting the initial background image into BG_complex, making the scene where the character is more realistic and detailed. The background of the entire frame is synthesized and enriched to obtain the final time frame sequence frames_final with complex details.
[0052] All time series frames frames_final are decoded back to pixel space through VAE, and a synthetic video is output. The video contains a realistic scene of two reference ID characters performing martial arts fighting in the acquired action posture sequence and text description.
[0053] Furthermore, the network model includes: a core module, U-Net, a personalized attention layer, a text cross attention layer, a temporal attention layer, a posture guidance module, and a background generation module.
[0054] The core module is used to fuse the input condition information;
[0055] The U-Net is a denoising network model used to generate specific features of a frame; wherein the U-Net consists of an encoder, a decoder, and a skip connection.
[0056] The encoder is responsible for extracting the features of the input image layer by layer, gradually reducing the spatial resolution of the feature map through multi-layer convolution operations and downsampling, while capturing high-level global features. The decoder restores the features extracted by the encoder to the original resolution through layer-by-layer upsampling and convolution operations, which not only restores the detailed features of the image, but also gradually generates high-quality output results. The jump connection is a jump connection between the corresponding layers of the encoder and the decoder to directly transfer high-resolution local features from the encoder to the decoder. While maintaining the detail information, the learning ability and reconstruction effect of the network are enhanced. Through the above structure, U-Net can effectively combine global semantic information and local detail features, and is suitable for tasks such as image generation, segmentation, and denoising. Its symmetrical design and feature fusion mechanism have significant advantages in improving generation quality and detail retention.
[0057] Furthermore, the output composite video can be generated at any length according to user needs, and is used to demonstrate character interaction and scene generation under complex conditions, which is suitable for subsequent applications such as film and television creation, game character animation design, etc. If the user needs further processing, the composite video can be input into the post-processing module, such as image quality enhancement, stylization processing or video post-processing special effects, to meet higher quality and more scene requirements.
[0058] In summary, Example 1 of the present application extracts and aligns the reference ID and text description through VAE and CLIP models based on the acquisition of target information, ensuring that the generated video character is consistent with the text description; based on the initially generated character action sequence, a complex and reasonable background is generated through the background generation module, and merged with the character to ensure that the action and background can be naturally integrated, and an action sequence containing all key action details is output; all time series frames are decoded back to the pixel space through VAE and synthesized into the final high-quality video. This application solves the problem of clothing and body misattribution in the generation of two-person fighting videos by introducing innovative points such as personalized attention networks and body shape adaptation strategies, providing strong technical support for film and television creation, game character animation design, etc.
[0059] Example 2
[0060] See also Figure 4 , is a structural diagram of a two-person martial arts video generation system in Example 2 of the present application; the specific contents include:
[0061] Acquiring a model: acquiring target information, the target information including: a reference ID, a text description, an initial background image, and an action posture sequence;
[0062] Processing module: processes the target information to obtain condition information;
[0063] Video synthesis module: inputs the conditional information into the network model for synthesis, and outputs the synthesized video.
[0064] Example 3
[0065] See also Figure 5 , is a schematic diagram of the device structure of Embodiment 3 of the present application. The device 50 includes a processor 51 and a memory 52 coupled to the processor 51 .
[0066] The memory 52 stores program instructions for implementing the above-mentioned two-person martial arts video generation method.
[0067] The processor 51 is used to execute the program instructions stored in the memory 52 to realize the generation of a two-person martial arts video.
[0068] The processor 51 may also be referred to as a CPU (Central Processing Unit).
[0069] The processor 51 may be an integrated circuit chip with signal processing capabilities. The processor 51 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0070] Example 4
[0071] See also Figure 6 , which is a schematic diagram of the structure of the storage medium of Example 4 of the present application. The storage medium of the embodiment of the present application stores a program file 61 that can implement all the above methods, wherein the program file 61 can be stored in the above storage medium in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or a computer, a server, a mobile phone, a tablet, and other devices.
[0072] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.
[0073] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.
[0074] Although the embodiments of the present application have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the appended claims and their equivalents.
[0075] Of course, the present invention may have many other implementations. Based on this implementation, other implementations obtained by ordinary technicians in this field without any creative work are all within the scope of protection of the present invention.
Claims
1. A method for generating a two-person martial arts video, characterized in that: include: Acquire target information, the target information including: reference ID, text description, initial background image, action posture sequence; Processing the target information to obtain condition information; The conditional information is input into a network model for synthesis, and a synthesized video is output.
2. A two-person martial arts video generation method according to claim 1, characterized in that: The step of processing the target information to obtain the condition information specifically includes the following steps: The original image provided by the reference ID is encoded through VAE, and the representation in the latent space is recorded as z_id, where z_id contains the visual feature information of the reference character; The reference ID and the text description are feature aligned by the CLIP model, and a matching representation of the text and the reference role in a common embedding space is obtained and recorded as z_clip_id; Mapping the text description to a text embedding z_text via a CLIP text encoder; The initial background image is processed by a background generation module in a network model to obtain BG_enhanced; The action posture sequence is input into the posture guidance module in the network model, and the corresponding position information and joint point distribution pose_info are obtained by parsing the skeleton posture of each frame; wherein pose_info is used to guide the position and action of the character in each time frame.
3. The method for generating a two-person martial arts video according to claim 1, characterized in that: The step of inputting the condition information into the network model for synthesis and outputting the synthesized video specifically includes the following steps: The condition information includes: z_id, z_clip_id, z_text, BG_enhanced, pose_info; The condition information is input into the network model for synthesis, specifically including: The personalized attention layer uses the attention mechanism corresponding to the reference ID to enable the generator to inject the appearance features of the corresponding character in each frame and place the character's appearance information in the corresponding spatiotemporal position; The text cross-attention layer semantically constrains the scene and character appearance based on the text features in the conditional information; The temporal attention layer uses time series information to make the generation results between adjacent frames coherent; After multiple rounds of iterations, the network model generates an intermediate image sequence frames_intermediate in which the character and the background are fused together; The intermediate image sequence frames_intermediate is post-processed to obtain a time series of frames frames_final.
4. The method for generating a two-person martial arts video according to claim 1, characterized in that: The time series frames frames_final are decoded into pixel space through VAE and output as a synthetic video; the synthetic video includes a realistic scene of two reference ID characters performing martial arts fighting in an action posture sequence and text description.
5. The method for generating a two-person martial arts video according to claim 1, characterized in that: The network model includes: a core module, U-Net, a personalized attention layer, a text cross-attention layer, a temporal attention layer, a posture guidance module, and a background generation module. The core module is used to fuse the input condition information; The U-Net is a denoising network model used to generate specific features of a frame; wherein the U-Net consists of an encoder, a decoder, and a skip connection.
6. A system according to claim 1, characterized in that: include: Acquire model: acquire target information, the target information includes: reference ID, text description, initial background image, action posture sequence; Processing module: processes the target information to obtain condition information; Video synthesis module: inputs the conditional information into the network model for synthesis, and outputs the synthesized video.
7. A device, characterized in that: The device includes a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing a method for generating a two-player martial arts video as described in any one of claims 1-5; and the processor is used to execute the program instructions stored in the memory to realize the generation of a two-player martial arts video.
8. A storage medium, characterized in that: The device stores program instructions executable by a processor, wherein the program instructions are used to execute a method for generating a two-person martial arts video as described in any one of claims 1-5.