Transparent video generation method and device, electronic equipment, storage medium and computer program product

Abstract

The invention relates to a transparent video generation method and apparatus, an electronic device, a storage medium and a computer program product. The method comprises the steps of obtaining an RGB video sequence and a mask picture sequence; carrying out noise addition on other pictures except the target RGB picture contained in the RGB video sequence; inputting the RGB video sequence after noise addition and the mask picture sequence into a transparent encoder; inputting the RGB video sequence after noise addition into a VAE encoder to obtain an RGB latent feature matrix; calculating a summation matrix of the first transparent latent characteristic matrix and the RGB latent characteristic matrix, and inputting the summation matrix into a diffusion model; a transparent video is generated based on the intermediate summed element (Xm) and the other summed elements (Xi). Therefore, prior information can be provided for the generation process of the transparent video by using an intermediate frame attention mechanism, so that the content information, the transparent information and the action information in the video frame can be decoupled, and the transparent video with an accurate transparent region and without artifacts can be ensured to be generated.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to a method, device, electronic device, storage medium and computer program product for generating a transparent video. Background Art

[0002] Green screen technology is widely used in film, television and video production. The current mainstream method of obtaining green screen videos is to shoot them in a professional studio, which often requires a lot of manpower, material resources and time, resulting in high shooting costs. In contrast, transparent videos retain the transparency information in the frame, which allows videos to be directly superimposed, thereby achieving the same effect as green screen technology.

[0003] In the related art, the transparent image generation model is mainly expanded into a transparent video generation model by directly adding a time series model to the transparent image generation model to generate a transparent video. However, directly adding a time series model will bring about the problem of inconsistent data distribution, which will cause the generated transparent video to have defects such as inaccurate transparent areas and artifacts. Summary of the invention

[0004] The present disclosure provides a method, device, electronic device, storage medium and computer program product for generating a transparent video, so as to at least solve the problem that the generated transparent video has inaccurate transparent areas and artifacts in the above-mentioned related technologies.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for generating a transparent video is provided, comprising: obtaining an RGB video sequence and a mask picture sequence, wherein the RGB video sequence uses a target RGB picture as an intermediate frame, and the mask picture sequence uses a target mask picture as an intermediate frame, and obtaining the target RGB picture and the target mask picture by decomposing the target transparent picture; performing noise addition on other pictures except the target RGB picture contained in the RGB video sequence to obtain a noisy RGB video sequence; inputting the noisy RGB video sequence and the mask picture sequence into a transparent encoder to obtain a first transparent latent feature matrix; inputting the noisy RGB video sequence into a VAE encoder to obtain an RGB latent feature matrix; calculating a sum matrix of the first transparent latent feature matrix and the RGB latent feature matrix, and inputting the sum matrix into a diffusion model, wherein the sum matrix includes intermediate sum elements (X m ) and other sum elements corresponding to other images that have been noised (X i ); Based on the intermediate sum element (X m ) and the other summation elements (X i ), generate transparent video.

[0006] Optionally, obtaining the RGB video sequence and the mask picture sequence includes: inputting the target text into a transparent image generation model to obtain the target transparent picture, wherein the picture content of the target transparent picture is the content indicated by the target text; decomposing the target transparent picture to obtain the target RGB picture and the target mask picture; obtaining the RGB video sequence with the target RGB picture as the intermediate frame, and the mask picture sequence with the target mask picture as the intermediate frame.

[0007] Optionally, the obtaining of the RGB video sequence with the target RGB picture as the intermediate frame, and the obtaining of the mask picture sequence with the target mask picture as the intermediate frame, includes: obtaining a preset number of pure noise pictures arranged in sequence, and replacing the pure noise picture located in the middle position of the preset number of pure noise pictures arranged in sequence with the target RGB picture, to obtain the RGB video sequence; and, obtaining a preset number of all-0 mask pictures arranged in sequence, and replacing the all-0 mask picture located in the middle position of the preset number of all-0 mask pictures arranged in sequence with the target mask picture, to obtain the mask picture sequence.

[0008] Optionally, the intermediate sum element (X m ) and the other summation elements (X i ), generating a transparent video, including: using the diffusion model based on the intermediate summation element (X m ) to obtain prior information (K m , V m ), and based on the prior information and the other summation elements (X i ) obtain an intermediate attention matrix; obtain a second transparent latent feature matrix based on the intermediate attention matrix through the diffusion model; and generate the transparent video based on the second transparent latent feature matrix.

[0009] Optionally, the diffusion model is based on the intermediate summation element (X m ) to obtain prior information (K m , V m ), including: obtaining the prior information through the diffusion model based on the following formula:

[0010] K m =X m W K , V m =X m W V

[0011] Among them, X m is the intermediate summation element, WK is the bond weight matrix of the diffusion model, W V is the value weight matrix of the diffusion model, K m and V m is the prior information.

[0012] Optionally, the prior information and the other summation elements (X i ) Obtaining an intermediate attention matrix, comprising: calculating the intermediate attention matrix by the following formula:

[0013]

[0014] (Q i )′=X i W′ Q

[0015] Among them, Out′ is the intermediate attention matrix, (Q i ) is the query feature matrix corresponding to the i-th other picture, X i is the other summed element corresponding to the i-th other picture, W′ O is the output weight matrix of the diffusion model, W′ Q is the query weight matrix of the diffusion model, d is the feature dimension, and T represents the transposition operation.

[0016] Optionally, generating the transparent video based on the second transparent latent feature matrix includes: inputting the second transparent latent feature matrix into a VAE decoder to obtain a decoded video sequence; and inputting the decoded video sequence and the second transparent latent feature matrix into a transparent decoder to obtain the decoded transparent video.

[0017] According to a second aspect of an embodiment of the present disclosure, a transparent video generation device is provided, comprising: a video sequence acquisition module, configured to acquire an RGB video sequence and a mask picture sequence, wherein the RGB video sequence uses a target RGB picture as an intermediate frame, and the mask picture sequence uses a target mask picture as an intermediate frame, and the target RGB picture and the target mask picture are obtained by decomposing the target transparent picture; a noise addition module, configured to perform noise addition on other pictures contained in the RGB video sequence except the target RGB picture, to obtain a noisy RGB video sequence; a transparent latent feature matrix acquisition module, configured to input the noisy RGB video sequence and the mask picture sequence into a transparent encoder to obtain a first transparent latent feature matrix; an RGB latent feature matrix acquisition module, configured to input the noisy RGB video sequence into a VAE encoder to obtain an RGB latent feature matrix; a sum matrix calculation module, configured to calculate a sum matrix of the first transparent latent feature matrix and the RGB latent feature matrix, and input the sum matrix into a diffusion model, wherein the sum matrix contains intermediate sum elements (X m ) and other sum elements corresponding to other images that have been noised (X i ); A transparent video generation module configured to generate a transparent video based on the intermediate sum element (X m ) and the other summation elements (X i ), generate transparent video.

[0018] Optionally, the video sequence acquisition module is configured to: input the target text into a transparent image generation model to obtain the target transparent picture, wherein the picture content of the target transparent picture is the content indicated by the target text; decompose the target transparent picture to obtain the target RGB picture and the target mask picture; obtain the RGB video sequence with the target RGB picture as the intermediate frame, and the mask picture sequence with the target mask picture as the intermediate frame.

[0019] Optionally, the video sequence acquisition module is configured to: obtain a preset number of pure noise pictures arranged in sequence, and replace the pure noise picture located in the middle position of the preset number of pure noise pictures arranged in sequence with the target RGB picture, so as to obtain the RGB video sequence; and obtain a preset number of all-0 mask pictures arranged in sequence, and replace the all-0 mask picture located in the middle position of the preset number of all-0 mask pictures arranged in sequence with the target mask picture, so as to obtain the mask picture sequence.

[0020] Optionally, the transparent video generation module is configured to: generate a transparent video based on the intermediate sum element (X) by using the diffusion model m) to obtain prior information (K m , V m ), and based on the prior information and the other summation elements (X i ) obtain an intermediate attention matrix; obtain a second transparent latent feature matrix based on the intermediate attention matrix through the diffusion model; and generate the transparent video based on the second transparent latent feature matrix.

[0021] Optionally, the transparent video generation module is configured to: obtain the prior information through the diffusion model based on the following formula:

[0022] K m =X m W K , V m =X m W V

[0023] Among them, X m is the intermediate summation element, W K is the bond weight matrix of the diffusion model, W V is the value weight matrix of the diffusion model, K m and V m is the prior information.

[0024] Optionally, the transparent video generation module is configured to calculate the intermediate attention matrix by the following formula:

[0025]

[0026] (Q i )′=X i W′ Q

[0027] Among them, Out′ is the intermediate attention matrix, (Q i ) is the query feature matrix corresponding to the i-th other picture, X i is the other summed element corresponding to the i-th other picture, W′ O is the output weight matrix of the diffusion model, W′ Q is the query weight matrix of the diffusion model, d is the feature dimension, and T represents the transposition operation.

[0028] Optionally, the transparent video generation module is configured to: input the second transparent latent feature matrix into a VAE decoder to obtain a decoded video sequence; input the decoded video sequence and the second transparent latent feature matrix into a transparent decoder to obtain the decoded transparent video.

[0029] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a method for generating a transparent video according to the present disclosure.

[0030] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method for generating a transparent video according to the present disclosure.

[0031] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method for generating a transparent video according to the present disclosure is implemented.

[0032] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0033] In the present disclosure, the intermediate sum element (X) corresponding to the RGB intermediate frame without noise can be obtained by using a diffusion model. m ) and other sum elements corresponding to other images that have been noised (X i ) to generate a transparent video. That is, the present disclosure can use the intermediate frame attention mechanism to provide prior information for the generation process of the transparent video, and then decouple the content information, transparency information and action information in the video frame, which can ensure the generation of a transparent video with accurate transparent areas and no artifacts.

[0034] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0036] Figure 1 is a flowchart illustrating a method for generating a transparent video according to an exemplary embodiment of the present disclosure;

[0037] Figure 2 is a schematic diagram showing 16 video frames included in an RGB video sequence according to an exemplary embodiment of the present disclosure;

[0038] Figure 3 is a structural diagram showing a system for generating a transparent video according to an exemplary embodiment of the present disclosure;

[0039] Figure 4is a block diagram illustrating a generating apparatus of a transparent video according to an exemplary embodiment of the present disclosure;

[0040] Figure 5 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0042] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0043] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0044] Green screen technology is widely used in film, television and video production. The current mainstream method of obtaining green screen videos is to shoot them in a professional studio, which often requires a lot of manpower, material resources and time, resulting in high shooting costs. In contrast, transparent videos retain the transparency information in the frame, which allows videos to be directly superimposed, thereby achieving the same effect as green screen technology.

[0045] However, current models can usually only generate standard videos without transparent information, and research on transparent video technology is still limited. In related technologies, transparent image generation technology can inject transparent information into the potential feature distribution of the diffusion model. On this basis, the simplest way to extend transparent image generation to transparent video generation is to integrate the pre-trained motion module into the layer diffusion model. However, since the pre-trained transparent image diffusion model is trained on transparent images, and the motion module is trained on standard video data, this will lead to a data distribution mismatch between the two. This mismatch will cause the following problems: inaccurate transparent areas and severe distortion. In addition, since the size of the model is very large, training with a small amount of data will prevent the model from learning the true distribution of transparent data, resulting in poor transparent video generation. However, the cost of training with a large data set is also very high. Therefore, how to achieve transparent video generation while maintaining high video frame quality is a problem worthy of further study.

[0046] In order to solve the above problems existing in the related art, the present disclosure provides a method, device, electronic device, storage medium and computer program product for generating transparent video, which can generate a transparent video by using a diffusion model based on the intermediate sum element (X) corresponding to the RGB intermediate frame without noise. m ) and other sum elements corresponding to other images that have been noised (X i ) to generate a transparent video. That is, the present disclosure can use the intermediate frame attention mechanism to provide prior information for the generation process of the transparent video, and then decouple the content information, transparency information and action information in the video frame, which can ensure the generation of a transparent video with accurate transparent areas and no artifacts.

[0047] Figure 1 is a flowchart illustrating a method of generating a transparent video according to an exemplary embodiment of the present disclosure.

[0048] Reference Figure 1 In step 101, an RGB video sequence and a mask picture sequence may be obtained, wherein the RGB video sequence may use a target RGB picture as an intermediate frame, and the mask picture sequence may use a target mask picture as an intermediate frame. In addition, the target RGB picture and the target mask picture may be obtained by decomposing the target transparent picture.

[0049] According to an exemplary embodiment of the present disclosure, the target text can be input into the transparent image generation model to obtain a target transparent picture, wherein the picture content of the target transparent picture can be the content indicated by the target text. That is, a text prompt (Prompt) can be input into the transparent image generation model to obtain a transparent picture containing an alpha channel. Exemplarily, assuming that the text content of the target text is: "A little boy in a blue T-shirt is playing a violin", the transparent image generation model can generate a transparent picture with a picture content of "a little boy in a blue T-shirt is playing a violin" based on the target text.

[0050] It should be noted that the transparent image may refer to an image containing RGBA four-channel information, and the last alpha channel may represent transparency information, and the transparent image may be used as subsequent prior information. In addition, the transparent image containing the alpha channel may be a grayscale image, and the element value range of the pixels contained in the grayscale image may be 0 to 255, wherein the element value "0" may represent complete transparency, and the corresponding pixel is completely invisible; the element value "255" may represent complete opacity.

[0051] Next, the target transparent image can be decomposed to obtain the target RGB image and the target mask image. That is, the transparent image can be decomposed into an RGB image with a gray background and a mask image with only transparency information. "Mask image" M∈R(H×W), where H can represent the height of the mask image, W can represent the width of the mask image, and its element value range can be: 0~255, and the element value "0" can represent complete transparency, and its corresponding pixel is completely invisible; the element value "255" can represent complete opacity, and R can represent a two-dimensional real space. "RGB image with a gray background" I∈R(H×W×C), where H can represent the height of the RGB image, W can represent the width of the RGB image, C can represent the number of channels of the RGB image, and R can represent a three-dimensional real space.

[0052] Then, an RGB video sequence with the target RGB image as the intermediate frame and a mask image sequence with the target mask image as the intermediate frame can be obtained. That is, the gray background RGB image and the target mask image can be used as intermediate frames to expand into a time-series video.

[0053] According to an exemplary embodiment of the present disclosure, a preset number of pure noise pictures arranged in sequence can be obtained, and the pure noise picture located in the middle position of the preset number of pure noise pictures arranged in sequence can be replaced with a target RGB picture, thereby obtaining an RGB video sequence. Exemplarily, 16 pure noise pictures arranged in sequence can be obtained, and the pure noise picture located in the middle position, that is, the 7th position of the 16 pure noise pictures arranged in sequence can be replaced with a target RGB picture, thereby obtaining an RGB video sequence.

[0054] Figure 2 is a schematic diagram showing 16 video frames included in an RGB video sequence according to an exemplary embodiment of the present disclosure. Figure 2 , the video frame at the middle position, that is, the 7th position, is the target RGB image, the video frames at the 0th position to the 6th position are pure noise images, and the video frames at the 8th position to the 15th position are pure noise images.

[0055] In addition, a preset number of all-0 mask pictures arranged in sequence can be obtained, and the all-0 mask picture located in the middle position of the preset number of all-0 mask pictures arranged in sequence can be replaced with the target mask picture, thereby obtaining a mask picture sequence. Exemplarily, 16 all-0 mask pictures arranged in sequence can be obtained, and the all-0 mask picture located in the middle position, that is, the 7th position, of the 16 all-0 mask pictures arranged in sequence can be replaced with the target mask picture, thereby obtaining a mask picture sequence.

[0056] In step 102, noise may be added to other pictures included in the RGB video sequence except the target RGB picture to obtain a noisy RGB video sequence.

[0057] In step 103, the noisy RGB video sequence and the mask image sequence can be input into the transparent encoder to obtain the first transparent latent feature matrix. It should be noted that the transparent encoder is pre-trained with a large number of transparent images, so it can ensure that the original distribution will not be changed while adding transparent information to the feature, thereby reducing the artifact phenomenon caused by inconsistent data distribution. In this way, the transparent encoder can take the mask image sequence and the RGB video sequence as input, which can ensure that the encoded features contain transparent information without changing the normal feature distribution.

[0058] In step 104, the noisy RGB video sequence can be input into the VAE encoder to obtain an RGB latent feature matrix, that is, the noisy RGB video sequence can be encoded using the VAE encoder, and then the normally distributed RGB latent features can be obtained. It should be noted that a variational auto-encoder (VAE) is usually composed of an encoder and a decoder, and the above-mentioned "VAE encoder" may refer to an encoder included in the variational auto-encoder.

[0059] It should be noted that the RGB latent feature x∈R (B×F×H×W×C) , where B can represent a batch of transparent pictures. For example, the batch can be, but not limited to, 2, 3, or 5. In this case, a transparent video can be generated for each transparent picture in the batch of transparent pictures, and the processes of generating transparent videos for different transparent pictures can not interfere with each other. F can represent the number of video frames; H can represent the height of the RGB latent feature matrix, W can represent the width of the RGB latent feature matrix, and C can represent the number of channels of the target RGB picture. R can represent a 5-dimensional real number space.

[0060] In step 105, a sum matrix of the first transparent latent feature matrix and the RGB latent feature matrix may be calculated, and the sum matrix may be input into the diffusion model, wherein the sum matrix may include intermediate sum elements (X m ) and other sum elements corresponding to other images that have been noised (X i ).

[0061] Exemplarily, as mentioned above, the RGB video sequence and the mask image sequence may both include 16 video frames. In this case, the sum matrix may include an unnoised RGB intermediate frame, that is, the intermediate sum element (X) corresponding to the 7th RGB video frame. m ), and the other sum elements (X) corresponding to each of the other pictures that have been noisy in the 0th other picture to the 6th other picture, the 8th other picture to the 15th other picture. i ).

[0062] It should be noted that the "diffusion model" in the present disclosure can be composed of multiple spatial modules and temporal modules, wherein the "spatial module" can be composed of a convolutional layer and an attention layer composed of multiple attention components, and the "temporal module" can be composed of an inter-frame attention layer.

[0063] Specifically, the attention layer composed of multiple attentions in the spatial module can include "Self-Attention Layer", "Cross-Attention Layer" and "Inter-Frame Attention Layer". The "Inter-Frame Attention Layer" can obtain information from the unnoised intermediate frames by using the unnoised intermediate frames as keys and values, and the remaining frames as query frames, that is, it can provide prior information for the frames that need to be predicted. In addition, when the spatial module processes the features, it needs to merge the dimension of video length into the dimension of batch for calculation.

[0064] The attention layer in the timing module is mainly a self-attention layer. When processing features, the timing module can perform attention calculations on the dimension of video frames to obtain the relationship between different frames.

[0065] In step 106, the intermediate sum element (X m ) and other summation elements (X i ), generate transparent video.

[0066] According to an exemplary embodiment of the present disclosure, the diffusion model can be used based on the intermediate summation element (X m ) to obtain prior information (K m , V m ), and can be based on prior information and other summation elements (X i ) obtains the intermediate attention matrix Out′. Then, the second transparent latent feature matrix can be obtained based on the intermediate attention matrix through the diffusion model. Next, a transparent video can be generated based on the second transparent latent feature matrix.

[0067] According to an exemplary embodiment of the present disclosure, prior information may be acquired through a diffusion model based on the following formula:

[0068] K m =X m W K , V m =X m W V

[0069] Among them, X m is the intermediate sum element, W K is the bond weight matrix of the diffusion model, W V is the value weight matrix of the diffusion model, K m and V m is prior information.

[0070] That is, in the present disclosure, the intermediate frame attention layer in the spatial module included in the diffusion model can be used to take the unnoised intermediate frame as the key and value, and the remaining frames can be used as query frames, so as to obtain prior information from the unnoised intermediate frame.

[0071] According to an exemplary embodiment of the present disclosure, the intermediate attention matrix can be calculated by the following formula, that is, the intermediate frame attention mechanism can be formulated as follows:

[0072]

[0073] (Q i )′=X i W′ Q

[0074] Among them, Out′ is the intermediate attention matrix, (Q i )′ is the query feature matrix corresponding to the i-th other picture, X i is the other summed elements corresponding to the i-th other picture, W′ O is the output weight matrix of the diffusion model, W′ Q is the query weight matrix of the diffusion model, d is the feature dimension, and T represents the transposition operation.

[0075] It should be noted that in the intermediate attention matrix Out′, only W′ Q and W′ O is trainable, and the remaining parameters can be set as non-trainable during the training phase. Since the intermediate frames contain high-quality detail information and transparency information, in the present disclosure, the intermediate frames without noise can be set as keys and values ​​through the intermediate frame attention mechanism, and the remaining frames can be set as query frames, so that the content information and transparency information of the intermediate frames can be obtained as prior information, so that the model can decouple the content and transparency information from the action, that is, the model can focus more on the learning of the action.

[0076] In addition, in the present disclosure, the intermediate frame attention mechanism and the self-attention mechanism can be executed in parallel, and the outputs of the two can be added, and then the addition result can be input into the cross-attention layer to learn inter-frame information.

[0077] According to an exemplary embodiment of the present disclosure, the second transparent latent feature matrix can be input into a VAE decoder to obtain a decoded video sequence. Then, the decoded video sequence and the second transparent latent feature matrix can be input into a transparent decoder to obtain a decoded transparent video. It should be noted that, as mentioned above, a variational auto-encoder (VAE) is generally composed of an encoder and a decoder, and the above-mentioned "VAE decoder" may refer to a decoder contained in the variational auto-encoder.

[0078] Figure 3 is a diagram showing a system structure for generating a transparent video according to an exemplary embodiment of the present disclosure. Figure 3 When generating transparent videos, it is mainly necessary to use a transparent image generation model (Layer Diffuse), a contrastive language-image pre-training model (Contrastive Language-Image Pre-training, CLIP), CLIP is mainly used to extract text features of the target text. For example, the target text can be: "A little boy in a blue T-shirt is playing the violin", a transparent encoder (TansparencyEncoder), a transparent decoder (TansparencyDecoder), a VAE encoder (VAE Encoder), a VAE decoder (VAEDecoder), an adder (Add) and a diffusion model. The diffusion model can mainly include a spatial block and a temporal module (Motion Module), and the spatial module can mainly include a self-attention layer, an intermediate frame attention layer and a cross attention layer.

[0079] In the present disclosure, the intermediate sum element (X) corresponding to the RGB intermediate frame without noise can be obtained by using a diffusion model. m ) and other sum elements corresponding to other images that have been noised (X i ) to generate a transparent video. That is, the present disclosure can use the intermediate frame attention mechanism to provide prior information for the transparent video generation process, and then decouple the content information, transparency information and action information in the video frame, so that the model can pay more attention to the learning of the action, and then ensure the generation of transparent areas accurately and without artifacts.

[0080] Figure 4 is a block diagram illustrating a transparent video generating apparatus 400 according to an exemplary embodiment of the present disclosure.

[0081] Reference Figure 4 The transparent video generation device 400 may include a video sequence acquisition module 401, a noise addition module 402, a transparent latent feature matrix acquisition module 403, an RGB latent feature matrix acquisition module 404, a sum matrix calculation module 405 and a transparent video generation module 406.

[0082] The video sequence acquisition module 401 can acquire an RGB video sequence and a mask picture sequence, wherein the RGB video sequence can use the target RGB picture as an intermediate frame, and the mask picture sequence can use the target mask picture as an intermediate frame. In addition, the target RGB picture and the target mask picture can be obtained by decomposing the target transparent picture.

[0083] According to an exemplary embodiment of the present disclosure, the video sequence acquisition module 401 can input the target text into the transparent image generation model to obtain a target transparent picture, wherein the picture content of the target transparent picture can be the content indicated by the target text. That is, Prompt can be input into the transparent image generation model to obtain a transparent picture containing an alpha channel. Exemplarily, assuming that the text content of the target text is: "A little boy in a blue T-shirt is playing the violin", the transparent image generation model can generate a transparent picture with the picture content of "A little boy in a blue T-shirt is playing the violin" based on the target text.

[0084] It should be noted that the transparent image may refer to an image containing RGBA four-channel information, and the last alpha channel may represent transparency information, and the transparent image may be used as subsequent prior information. In addition, the transparent image containing the alpha channel may be a grayscale image, and the element value range of the pixels contained in the grayscale image may be 0 to 255, wherein the element value "0" may represent complete transparency, and the corresponding pixel is completely invisible; the element value "255" may represent complete opacity.

[0085] Next, the video sequence acquisition module 401 can decompose the target transparent image to obtain the target RGB image and the target mask image. That is, the transparent image can be decomposed into an RGB image with a gray background and a mask image with only transparency information. "Mask image" M∈R(H×W), where H can represent the height of the mask image, W can represent the width of the mask image, and its element value range can be: 0~255, and the element value "0" can represent complete transparency, and its corresponding pixel is completely invisible; the element value "255" can represent complete opacity, and R can represent a two-dimensional real space. "RGB image with a gray background" I∈R(H×W×C), where H can represent the height of the RGB image, W can represent the width of the RGB image, C can represent the number of channels of the RGB image, and R can represent a three-dimensional real space.

[0086] Then, the video sequence acquisition module 401 can acquire an RGB video sequence with the target RGB picture as the intermediate frame, and a mask picture sequence with the target mask picture as the intermediate frame. That is, the gray background RGB picture and the target mask picture can be used as intermediate frames to expand into a time-series video.

[0087] According to an exemplary embodiment of the present disclosure, the video sequence acquisition module 401 can acquire a preset number of pure noise pictures arranged in sequence, and can replace the pure noise picture located in the middle position of the preset number of pure noise pictures arranged in sequence with the target RGB picture, thereby obtaining an RGB video sequence. Exemplarily, 16 pure noise pictures arranged in sequence can be acquired, and the pure noise picture located in the middle position, i.e., the 7th position, of the 16 pure noise pictures arranged in sequence can be replaced with the target RGB picture, thereby obtaining an RGB video sequence.

[0088] In addition, the video sequence acquisition module 401 can acquire a preset number of all-0 mask pictures arranged in sequence, and can replace the all-0 mask picture located in the middle position of the preset number of all-0 mask pictures arranged in sequence with the target mask picture, thereby obtaining a mask picture sequence. Exemplarily, 16 all-0 mask pictures arranged in sequence can be acquired, and the all-0 mask picture located in the middle position, i.e., the 7th position, of the 16 all-0 mask pictures arranged in sequence can be replaced with the target mask picture, thereby obtaining a mask picture sequence.

[0089] The noise adding module 402 may add noise to other pictures included in the RGB video sequence except the target RGB picture to obtain a noisy RGB video sequence.

[0090] The transparent latent feature matrix acquisition module 403 can input the noisy RGB video sequence and the mask image sequence into the transparent encoder to obtain the first transparent latent feature matrix. It should be noted that the transparent encoder is pre-trained with a large number of transparent images, so it can ensure that the original distribution will not be changed while adding transparent information to the feature, thereby reducing the artifact phenomenon caused by inconsistent data distribution. In this way, the transparent encoder can take the mask image sequence and the RGB video sequence as input, which can ensure that the encoded features contain transparent information without changing the normal feature distribution.

[0091] The RGB latent feature matrix acquisition module 404 can input the noisy RGB video sequence into the VAE encoder to obtain the RGB latent feature matrix, that is, the noisy RGB video sequence can be encoded using the VAE encoder to obtain normally distributed RGB latent features.

[0092] It should be noted that the RGB latent feature x∈R (B×F×H×W×C), where B can represent a batch of transparent pictures. For example, the batch can be, but not limited to, 2, 3, or 5. In this case, a transparent video can be generated for each transparent picture in the batch of transparent pictures, and the processes of generating transparent videos for different transparent pictures can not interfere with each other. F can represent the number of video frames; H can represent the height of the RGB latent feature matrix, W can represent the width of the RGB latent feature matrix, and C can represent the number of channels of the target RGB picture. R can represent a 5-dimensional real number space.

[0093] The sum matrix calculation module 405 can calculate the sum matrix of the first transparent latent feature matrix and the RGB latent feature matrix, and can input the sum matrix into the diffusion model, wherein the sum matrix can include the intermediate sum elements (X m ) and other sum elements corresponding to other images that have been noised (X i ).

[0094] Exemplarily, as mentioned above, the RGB video sequence and the mask image sequence may both include 16 video frames. In this case, the sum matrix may include an unnoised RGB intermediate frame, that is, the intermediate sum element (X) corresponding to the 7th RGB video frame. m ), and the other sum elements (X) corresponding to each of the other pictures that have been noisy in the 0th other picture to the 6th other picture, the 8th other picture to the 15th other picture. i ).

[0095] It should be noted that the "diffusion model" in the present disclosure can be composed of multiple spatial modules and temporal modules, wherein the "spatial module" can be composed of a convolutional layer and an attention layer composed of multiple attention components, and the "temporal module" can be composed of an inter-frame attention layer.

[0096] Specifically, the attention layer composed of multiple attentions in the spatial module can include "Self-Attention Layer", "Cross-Attention Layer" and "Inter-Frame Attention Layer". The "Inter-Frame Attention Layer" can obtain information from the unnoised intermediate frames by using the unnoised intermediate frames as keys and values, and the remaining frames as query frames, that is, it can provide prior information for the frames that need to be predicted. In addition, when the spatial module processes the features, it needs to merge the dimension of video length into the dimension of batch for calculation.

[0097] The attention layer in the timing module is mainly a self-attention layer. When processing features, the timing module can perform attention calculations on the dimension of video frames to obtain the relationship between different frames.

[0098] The transparent video generation module 406 can be based on the intermediate sum element (X m ) and other summation elements (X i ), generate transparent video.

[0099] According to an exemplary embodiment of the present disclosure, the transparent video generation module 406 can generate a transparent video by using a diffusion model based on the intermediate sum element (X m ) to obtain prior information (K m , V m ), and can be based on prior information and other summation elements (X i ) obtains the intermediate attention matrix Out′. Then, the transparent video generation module 406 can obtain the second transparent latent feature matrix based on the intermediate attention matrix through the diffusion model. Next, the transparent video generation module 406 can generate a transparent video based on the second transparent latent feature matrix.

[0100] According to an exemplary embodiment of the present disclosure, the transparent video generation module 406 may obtain prior information through a diffusion model based on the following formula:

[0101] K m =X m W K , V m =X m W V

[0102] Among them, X m is the intermediate sum element, W K is the bond weight matrix of the diffusion model, W V is the value weight matrix of the diffusion model, K m and V m is prior information.

[0103] That is, in the present disclosure, the intermediate frame attention layer in the spatial module included in the diffusion model can be used to take the unnoised intermediate frame as the key and value, and the remaining frames can be used as query frames, so as to obtain prior information from the unnoised intermediate frame.

[0104] According to an exemplary embodiment of the present disclosure, the transparent video generation module 406 can calculate the intermediate attention matrix by the following formula, that is, the intermediate frame attention mechanism can be formulated as follows:

[0105]

[0106] (Q i )′=X i W′ Q

[0107] Among them, Out′ is the intermediate attention matrix, (Q i )′ is the query feature matrix corresponding to the i-th other picture, X i is the other summed elements corresponding to the i-th other picture, W′ O is the output weight matrix of the diffusion model, W′ Q is the query weight matrix of the diffusion model, d is the feature dimension, and T represents the transposition operation.

[0108] It should be noted that in the intermediate attention matrix Out′, only W′ Q and W′ O is trainable, and the remaining parameters can be set as non-trainable during the training phase. Since the intermediate frames contain high-quality detail information and transparency information, in the present disclosure, the intermediate frames without noise can be set as keys and values ​​through the intermediate frame attention mechanism, and the remaining frames can be set as query frames, so that the content information and transparency information of the intermediate frames can be obtained as prior information, so that the model can decouple the content and transparency information from the action, that is, the model can focus more on the learning of the action.

[0109] In addition, in the present disclosure, the intermediate frame attention mechanism and the self-attention mechanism can be executed in parallel, and the outputs of the two can be added, and then the addition result can be input into the cross-attention layer to learn inter-frame information.

[0110] According to an exemplary embodiment of the present disclosure, the transparent video generation module 406 may input the second transparent latent feature matrix into the VAE decoder to obtain a decoded video sequence. Then, the transparent video generation module 406 may input the decoded video sequence and the second transparent latent feature matrix into the transparent decoder to obtain a decoded transparent video.

[0111] Figure 5 is a block diagram illustrating an electronic device 500 according to an exemplary embodiment of the present disclosure.

[0112] Reference Figure 5 The electronic device 500 includes at least one memory 501 and at least one processor 502, wherein the at least one memory 501 stores instructions, and when the instructions are executed by the at least one processor 502, a method for generating a transparent video according to an exemplary embodiment of the present disclosure is executed.

[0113] As an example, the electronic device 500 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instructions. Here, the electronic device 500 is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above instructions (or instruction sets) individually or in combination. The electronic device 500 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0114] In the electronic device 500, the processor 502 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0115] The processor 502 may execute instructions or codes stored in the memory 501, wherein the memory 501 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0116] The memory 501 may be integrated with the processor 502, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 501 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 501 and the processor 502 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 502 can read files stored in the memory.

[0117] In addition, the electronic device 500 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 500 may be connected to each other via a bus and / or a network.

[0118] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium may also be provided, and when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above-mentioned method for generating transparent video. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0119] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, including a computer program, and when the computer program is executed by a processor, the method for generating a transparent video according to the present disclosure is implemented.

[0120] According to the transparent video generation method, device, electronic device, storage medium and computer program product disclosed in the present invention, the intermediate sum element (X) corresponding to the RGB intermediate frame without noise can be obtained by using a diffusion model. m ) and other sum elements corresponding to other images that have been noised (X i ) to generate a transparent video. That is, the present disclosure can use the intermediate frame attention mechanism to provide prior information for the generation process of the transparent video, and then decouple the content information, transparency information and action information in the video frame, which can ensure the generation of a transparent video with accurate transparent areas and no artifacts.

[0121] According to an exemplary embodiment of the present disclosure, a transparent encoder can take a mask image sequence and an RGB video sequence as input, which can ensure that the encoded features contain transparent information without changing the normal feature distribution.

[0122] According to an exemplary embodiment of the present disclosure, since the intermediate frames contain high-quality detail information and transparency information, in the present disclosure, the intermediate frames without noise can be set as keys and values ​​through the intermediate frame attention mechanism, and the remaining frames can be set as query frames, so that the content information and transparency information of the intermediate frames can be obtained as prior information, so that the model can decouple the content and transparency information from the action, that is, the model can focus more on the learning of the action.

[0123] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0124] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for generating a transparent video, characterized in that: include: Acquire an RGB video sequence and a mask picture sequence, wherein the RGB video sequence uses a target RGB picture as an intermediate frame, and the mask picture sequence uses a target mask picture as an intermediate frame, and obtain the target RGB picture and the target mask picture by decomposing a target transparent picture; Noising other pictures contained in the RGB video sequence except the target RGB picture to obtain a noisy RGB video sequence; Inputting the noisy RGB video sequence and the mask image sequence into a transparent encoder to obtain a first transparent latent feature matrix; Input the noisy RGB video sequence into a VAE encoder to obtain an RGB latent feature matrix; Calculate the sum matrix of the first transparent latent feature matrix and the RGB latent feature matrix, and input the sum matrix into the diffusion model, wherein the sum matrix includes the intermediate sum elements (X m ) and other sum elements corresponding to other images that have been noised (X i ); Based on the intermediate summation element (X m ) and the other summation elements (X i ), generate transparent video.

2. The generation method according to claim 1, characterized in that: The obtaining of the RGB video sequence and the mask image sequence comprises: Inputting the target text into a transparent image generation model to obtain the target transparent image, wherein the image content of the target transparent image is the content indicated by the target text; Decomposing the target transparent image to obtain the target RGB image and the target mask image; The RGB video sequence with the target RGB picture as an intermediate frame and the mask picture sequence with the target mask picture as an intermediate frame are obtained.

3. The generation method according to claim 2, characterized in that: The acquiring the RGB video sequence with the target RGB picture as an intermediate frame and the mask picture sequence with the target mask picture as an intermediate frame comprises: Acquire a preset number of pure noise pictures arranged in sequence, and replace the pure noise picture located in the middle of the preset number of pure noise pictures arranged in sequence with the target RGB picture to obtain the RGB video sequence; and, A preset number of all-0 mask pictures arranged in sequence are obtained, and an all-0 mask picture located in the middle of the preset number of all-0 mask pictures arranged in sequence is replaced with the target mask picture to obtain the mask picture sequence.

4. The generation method according to claim 1, characterized in that: The intermediate summation element (X m ) and the other summation elements (X i ), generate transparent video, including: The diffusion model is based on the intermediate summation element (X m ) to obtain prior information (K m , V m ), and based on the prior information and the other summation elements (X i ) Get the intermediate attention matrix; Obtaining a second transparent latent feature matrix based on the intermediate attention matrix through the diffusion model; The transparent video is generated based on the second transparent latent feature matrix.

5. The generation method according to claim 4, characterized in that: The diffusion model is based on the intermediate summation element (X m ) to obtain prior information (K m , V m ),include: The prior information is obtained through the diffusion model based on the following formula: K m =X m W K ,V m =X m W V Among them, X m is the intermediate summation element, W K is the bond weight matrix of the diffusion model, W V is the value weight matrix of the diffusion model, K m and V m is the prior information.

6. The generation method according to claim 5, characterized in that: The method based on the prior information and the other summation elements (X i ) Get the intermediate attention matrix, including: The intermediate attention matrix is ​​calculated by the following formula: (Q i )′=X i W′ Q Among them, Out′ is the intermediate attention matrix, (Q i ) is the query feature matrix corresponding to the i-th other picture, X i is the other summed element corresponding to the i-th other picture, W′ O is the output weight matrix of the diffusion model, W Q is the query weight matrix of the diffusion model, d is the feature dimension, T Represents a transpose operation.

7. The generation method according to claim 4, characterized in that: The step of generating the transparent video based on the second transparent latent feature matrix includes: Inputting the second transparent latent feature matrix into a VAE decoder to obtain a decoded video sequence; The decoded video sequence and the second transparent latent feature matrix are input into a transparent decoder to obtain the decoded transparent video.

8. A transparent video generation device, characterized in that: include: A video sequence acquisition module is configured to acquire an RGB video sequence and a mask picture sequence, wherein the RGB video sequence uses a target RGB picture as an intermediate frame, and the mask picture sequence uses a target mask picture as an intermediate frame, and the target RGB picture and the target mask picture are obtained by decomposing a target transparent picture; a noise adding module configured to add noise to other pictures contained in the RGB video sequence except the target RGB picture, so as to obtain a noisy RGB video sequence; A transparent latent feature matrix acquisition module is configured to input the noisy RGB video sequence and the mask image sequence into a transparent encoder to obtain a first transparent latent feature matrix; An RGB latent feature matrix acquisition module is configured to input the noisy RGB video sequence into a VAE encoder to obtain an RGB latent feature matrix; A sum matrix calculation module is configured to calculate a sum matrix of the first transparent latent feature matrix and the RGB latent feature matrix, and input the sum matrix into a diffusion model, wherein the sum matrix includes intermediate sum elements (X m ) and other sum elements corresponding to other images that have been noised (X i ); The transparent video generation module is configured to generate a transparent video based on the intermediate sum element (X m ) and the other summation elements (X i ), generate transparent video.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method for generating a transparent video as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method for generating a transparent video as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generating a transparent video according to any one of claims 1 to 7 is implemented.

Images