Cloth bag play video generation method and device, equipment and medium

Abstract

The invention relates to a cloth bag play video generation method and device, equipment and a medium, and the method comprises the steps: obtaining a target script and a picture with a background; inputting the target script and the picture into a cloth bag play video generation model to generate a cloth bag play video; wherein the cloth bag play video generation model comprises a text analysis and coding module which is used for performing text analysis on the target script to obtain a plurality of pieces of text information and coding the plurality of pieces of text information to obtain a text information vector; the video generation module is used for generating a plurality of continuous canvas frames and foreground objects based on the picture under the condition of the text information vector, and the canvas frames have a complete background and a transparent foreground; and the synthesis module is used for superposing the foreground object onto the transparent foreground on the canvas frame. According to the invention, the characteristics of the cloth bag play can be embodied under the condition of not increasing the computing power.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video generation, in particular to a video generation method, device, equipment and medium. BACKGROUND

[0002] In recent years, generative artificial intelligence (Generative AI) has made remarkable rapid development, especially in the field of image and video generation, a series of breakthrough achievements have emerged. From lifelike image synthesis to coherent and smooth video creation, the creative ability of AI is expanding the boundaries of human imagination at an unprecedented speed. In this wave of technological innovation, diffusion models (Diffusion Models) and Transformer architecture undoubtedly play the roles of two major driving forces. Diffusion models, with their strong generation quality and stability, have become the mainstream in image generation tasks; while Transformer, with its excellent sequence processing and long-distance dependency capturing ability, has achieved revolutionary success in natural language processing (NLP) and has begun to penetrate into the field of computer vision (CV). In this context, Diffusion Transformer (DiT) emerged and quickly became a representative work in the field of video generation. DiT creatively uses the Transformer architecture as the backbone network of the diffusion model, challenging the traditional paradigm dominated by convolutional networks such as U-Net.

[0003] Puppet play, also known as puppet play or palm play, is a puppet play composed of wooden puppet heads and cloth trunks, performed by finger and palm manipulation. When generating puppet play videos using DiT models, existing DiT models can only generate one canvas containing all objects in the video. At this time, the generated objects have reduced clarity and are small, which cannot reflect the characteristics of puppet play. If you want to improve the clarity of the objects generated by the DiT model, the clarity of the background also needs to be improved, which requires a huge amount of training or inference. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a puppet play video generation method, device, equipment and medium, which can reflect the characteristics of puppet play without increasing the amount of calculation.

[0005] The technical solution adopted by the present application to solve its technical problem is to provide a puppet play video generation method, comprising the following steps: obtaining a target script and a picture with a background; inputting the target script and the picture into a puppet play video generation model to generate a puppet play video; wherein the puppet play video generation model comprises: The text analysis and coding module is configured to perform text analysis on the target script to obtain a plurality of text information, and to code the plurality of text information to obtain a text information vector; The video generation module is configured to generate a plurality of continuous canvas frames and foreground objects based on the picture conditioned on the text information vector, wherein the canvas frames have complete backgrounds and transparent foregrounds; The synthesis module is configured to superimpose the foreground objects onto the transparent foregrounds on the canvas frames.

[0006] The video generation module is configured to set a maximum of three foreground objects at the beginning of training, to manually label the foreground objects in the process of preparing the training set, and to generate a dynamic video of a plurality of foreground objects and a background of a transparent foreground when outputting.

[0007] The process of preparing the training set further includes automatically identifying and filling the remaining backgrounds to complete the cut-out area.

[0008] The video generation module is configured based on a DiT model, and the penultimate layer of the DiT model architecture is connected to a fully connected layer to output the corresponding superimposed positions of the transparent foregrounds.

[0009] The technical solution adopted by the present application to solve the technical problems is to provide a cloth bag play video generation device, comprising: The acquisition module is configured to acquire a target script and a picture with a background. The generation module is configured to input the target script and the picture into a cloth bag play video generation model to generate a cloth bag play video. The cloth bag play video generation model comprises: The text analysis and coding module is configured to perform text analysis on the target script to obtain a plurality of text information, and to code the plurality of text information to obtain a text information vector; The video generation module is configured to generate a plurality of continuous canvas frames and foreground objects based on the picture conditioned on the text information vector, wherein the canvas frames have complete backgrounds and transparent foregrounds; The synthesis module is configured to superimpose the foreground objects onto the transparent foregrounds on the canvas frames.

[0010] The video generation module is configured to set a maximum of three foreground objects at the beginning of training, to manually label the foreground objects in the process of preparing the training set, and to generate a dynamic video of a plurality of foreground objects and a background of a transparent foreground when outputting.

[0011] The process of preparing the training set further includes automatically identifying and filling the remaining backgrounds to complete the cut-out area.

[0012] The video generation module is configured based on a DiT model, and an output of a second-to-last layer of the DiT model architecture accesses a fully connected layer to output a superimposed position of the corresponding transparent foreground.

[0013] The technical solution adopted by the present application to solve its technical problems is to provide an electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned baguio video generation method when executing the computer program.

[0014] The technical solution adopted by the present application to solve its technical problems is to provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-mentioned baguio video generation method.

[0015] Advantages Due to the adoption of the above technical solution, the present application has the following advantages and positive effects compared with the prior art: when generating a baguio video, the present application first generates a plurality of frames of continuous canvas frames with transparent foregrounds and foreground objects, and then superimposes the foreground objects on the transparent foregrounds on the canvas frames. In this way, different objects are decoupled to a certain extent, so that a more efficient video with clearer subjects is achieved, the characteristics of baguio are reflected, and the computing power does not need to be increased. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a flowchart of the baguio video generation method of the first embodiment of the present application. DETAILED DESCRIPTION

[0017] The present application will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present application and not to limit the scope of the present application. In addition, it should be understood that after reading the content taught by the present application, those skilled in the art can make various modifications or modifications to the present application, and these equivalent forms also fall within the scope defined by the claims attached hereto.

[0018] The first embodiment of the present application relates to a baguio video generation method, as shown in Figure 1 The method comprises the following steps: Step 1, obtaining a target script and a picture with a background; Step 2, inputting the target script and the picture into a baguio video generation model to generate a baguio video.

[0019] In this embodiment, the baguio video generation model comprises: a text analysis and encoding module for text analysis of the target script to obtain a plurality of text information, and encoding the plurality of text information to obtain a text information vector; a video generation module configured to generate, based on the picture, a plurality of continuous canvas frames and foreground objects on the canvas frames with complete backgrounds and transparent foregrounds, conditioned on the text information vector; a synthesis module configured to superimpose the foreground objects on the transparent foregrounds on the canvas frames.

[0020] The baguio video generation model in the embodiment can adopt a model of a TI2V architecture. The text analysis and encoding module of the TI2V architecture first analyzes the input target script to obtain a plurality of text information, encodes the plurality of text information to obtain a text information vector, and then embeds the text information vector into the video generation module as a condition.

[0021] The video generation module in the embodiment is constituted based on a DiT model. The DiT model divides an input picture into patches, converts the patches into token sequences, and then extracts features and generates through a Transformer module. The embodiment improves the DiT model. Specifically, a fully connected layer is connected to the second-to-last layer of the original DiT model to correspond to the superimposition position of the transparent foreground. To decouple the foreground objects and the background in the generated video, the embodiment initially sets up to have at most three foreground objects when training the video generation module. The foreground objects are manually labeled and cut out during the preparation of the training set, and the cut-out areas are filled in by automatic recognition and filling. At the output, at most three foreground objects and a background canvas frame with a transparent foreground are generated, and the resolution of the foreground objects is the same as that of the background canvas frame. The embodiment converts the single-canvas generation of the DiT model into multi-canvas (i.e., a plurality of foreground objects and a background canvas frame) generation, leaves a transparent foreground on the background canvas frame, and finally superimposes the foreground objects on the transparent foreground on the background canvas frame.

[0022] As can be seen, the embodiment can decouple different objects to a certain extent and greatly improve the clarity of the characters, so that the main body in the generated baguio video is large and clear enough, thereby achieving a video with a clearer main body with higher efficiency. In this way, the characteristics of baguio can be embodied without increasing the computing power.

[0023] A second embodiment of the present application relates to a baguio video generation device, comprising: an acquisition module configured to acquire a target script and a picture with a background; a generation module configured to input the target script and the picture into a baguio video generation model to generate a baguio video; The baguio video generation model comprises: The text analysis and coding module is configured to analyze the target script text, obtain a plurality of text information, and encode the plurality of text information to obtain a text information vector. The video generation module is configured to generate a plurality of continuous canvas frames and foreground objects based on the picture conditioned on the text information vector, wherein the canvas frames have complete backgrounds and transparent foregrounds. The synthesis module is configured to superimpose the foreground objects on the transparent foregrounds of the canvas frames.

[0024] The video generation module is configured to set a maximum of three foreground objects at the beginning of training, manually label the foreground objects in the process of preparing the training set, and generate a dynamic video of three foreground objects and a background of a transparent foreground when outputting.

[0025] In the process of preparing the training set, the remaining background is also supplemented by automatic recognition and filling.

[0026] The video generation module is based on a DiT model, and the penultimate layer of the DiT model architecture is connected to a fully connected layer to output the superimposed position of the corresponding transparent foreground.

[0027] The third embodiment of the present application relates to an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the bag play video generation method.

[0028] The fourth embodiment of the present application relates to a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of the bag play video generation method.

[0029] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer usable program code.

[0030] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart Figure 1 one or more flows and / or blocks. Figure 1 one or more flows and / or blocks.

[0031] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart Figure 1 one or more flows and / or blocks. Figure 1 one or more flows and / or blocks.

[0032] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart Figure 1 one or more flows and / or blocks. Figure 1 one or more flows and / or blocks.

[0033] The above generally describes only some embodiments of the present application, but the protection scope of the present application is not limited to this, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for generating puppet show videos, characterized in that, Includes the following steps: Obtain the target script and an image with a background; The target script and images are input into the puppet show video generation model to generate a puppet show video; The puppet show video generation model includes: The text parsing and encoding module is used to parse the target script to obtain multiple text information, and to encode the multiple text information to obtain a text information vector; The video generation module is used to generate multiple consecutive canvas frames and foreground objects based on the image, using the text information vector as a condition. The canvas frames have a complete background and a transparent foreground. A compositing module is used to overlay the foreground object onto a transparent foreground on the canvas frame.

2. The method for generating puppet show videos according to claim 1, characterized in that, The video generation module is set to a maximum of three foreground objects in the early stage of training. During the preparation of the training set, the foreground objects are extracted using a manual annotation method. When outputting, a dynamic video with several foreground objects and a transparent foreground background is generated.

3. The method for generating puppet show videos according to claim 2, characterized in that, The process of preparing the training set also includes automatically identifying and filling in the remaining background to complete the cut-out area.

4. The method for generating puppet show videos according to claim 1, characterized in that, The video generation module is based on the DiT model. The output of the penultimate layer of the DiT model architecture is connected to a fully connected layer to output the stacking position of the corresponding transparent foreground.

5. A device for generating videos of puppet shows, characterized in that, include: The acquisition module is used to acquire the target script and an image with a background. The generation module is used to input the target script and images into the puppet show video generation model to generate puppet show videos; The puppet show video generation model includes: The text parsing and encoding module is used to parse the target script to obtain multiple text information, and to encode the multiple text information to obtain a text information vector; The video generation module is used to generate multiple consecutive canvas frames and foreground objects based on the image, using the text information vector as a condition. The canvas frames have a complete background and a transparent foreground. A compositing module is used to overlay the foreground object onto a transparent foreground on the canvas frame.

6. The puppet show video generation device according to claim 5, characterized in that, The video generation module is set to a maximum of three foreground objects in the early stage of training. During the preparation of the training set, the foreground objects are extracted using a manual annotation method. When outputting, a dynamic video with several foreground objects and a transparent foreground background is generated.

7. The puppet show video generation device according to claim 6, characterized in that, The process of preparing the training set also includes automatically identifying and filling in the remaining background to complete the cut-out area.

8. The puppet show video generation device according to claim 5, characterized in that, The video generation module is based on the DiT model. The output of the penultimate layer of the DiT model architecture is connected to a fully connected layer to output the stacking position of the corresponding transparent foreground.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the puppet show video generation method as described in any one of claims 1-4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the puppet show video generation method as described in any one of claims 1-4.

Images