Transparent video generation method and device, electronic equipment, storage medium and computer program product

Abstract

The present disclosure relates to a transparent video generation method and device, electronic equipment, storage medium and computer program product. The method comprises: obtaining an RGB video sequence and a mask picture sequence; adding noise to pictures other than a target RGB picture in the RGB video sequence; inputting the noise-added RGB video sequence and the mask picture sequence into a transparent encoder; inputting the noise-added RGB video sequence into a VAE encoder to obtain an RGB latent feature matrix; calculating a sum matrix of a first transparent latent feature matrix and the RGB latent feature matrix, and inputting the sum matrix into a diffusion model; and generating a transparent video based on an intermediate sum element (X m ) and other sum elements (X i ). In this way, the present disclosure can use an intermediate frame attention mechanism to provide prior information for the transparent video generation process, and can decouple the content information, transparent information and action information in the video frame, thereby ensuring that the generated transparent region is accurate and artifact-free.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to methods, apparatus, electronic devices, storage media, and computer program products for generating transparent video. Background Technology

[0002] Green screen technology is widely used in film, television, and video production. Currently, the mainstream method for obtaining green screen video is to shoot it in a professional studio, which often requires significant manpower, resources, and time, resulting in high production costs. In contrast, transparent video retains the transparency information within the frames, allowing for direct video overlay to achieve the same effect as green screen technology.

[0003] In related technologies, the main approach is to directly incorporate a temporal model into the transparent image generation model to extend it into a transparent video generation model. However, directly incorporating a temporal model can lead to inconsistent data distribution, resulting in inaccurate transparent areas and artifacts in the generated transparent video. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, storage medium, and computer program product for generating transparent video, to at least solve the problems of inaccurate transparent areas and artifacts in the generated transparent video in the aforementioned related technologies.

[0005] According to a first aspect of the present disclosure, a method for generating transparent video is provided, comprising: acquiring an RGB video sequence and a mask image sequence, wherein the RGB video sequence uses a target RGB image as an intermediate frame, and the mask image sequence uses a target mask image as an intermediate frame; obtaining the target RGB image and the target mask image by decomposing the target transparent image; adding noise to other images included in the RGB video sequence besides the target RGB image to obtain a noisy RGB video sequence; inputting the noisy RGB video sequence and the mask image sequence into a transparent encoder to obtain a first transparent latent feature matrix; inputting the noisy RGB video sequence into a VAE encoder to obtain an RGB latent feature matrix; calculating a summation matrix of the first transparent latent feature matrix and the RGB latent feature matrix, and inputting the summation matrix into a diffusion model, wherein the summation matrix contains intermediate summation elements (X) corresponding to the RGB intermediate frames that have not been noisily added. m ) and other additive elements (X) corresponding to other images that have already been noise-added. i Based on the intermediate summing element (X) m ) and the other summing elements (X) i ), generate transparent video.

[0006] Optionally, obtaining the RGB video sequence and the mask image sequence includes: inputting the target text into a transparent image generation model to obtain the target transparent image, wherein the image content of the target transparent image is the content indicated by the target text; decomposing the target transparent image to obtain the target RGB image and the target mask image; and obtaining the RGB video sequence with the target RGB image as the intermediate frame and the mask image sequence with the target mask image as the intermediate frame.

[0007] Optionally, obtaining the RGB video sequence with the target RGB image as the intermediate frame and the mask image sequence with the target mask image as the intermediate frame includes: obtaining a preset number of sequentially arranged pure noise images, and replacing the pure noise image located in the middle position of the preset number of sequentially arranged pure noise images with the target RGB image to obtain the RGB video sequence; and obtaining a preset number of sequentially arranged all-zero mask images, and replacing the all-zero mask image located in the middle position of the preset number of sequentially arranged all-zero mask images with the target mask image to obtain the mask image sequence.

[0008] Optionally, the addition based on the intermediate summing element (X) m ) and the other summing elements (X) i Generate a transparent video by: using the diffusion model based on the intermediate summing element (X) m Obtain prior information (K) m V m ), and based on the prior information and the other summing elements (X) i Obtain the intermediate attention matrix; obtain the second transparent latent feature matrix based on the intermediate attention matrix using the diffusion model; generate the transparent video based on the second transparent latent feature matrix.

[0009] Optionally, the diffusion model is based on the intermediate summing element (X). m Obtain prior information (K) m V m This includes: obtaining the prior information through the diffusion model based on the following formula:

[0010] K m =X m W K V m =X m W V

[0011] Among them, X m For the intermediate summation element, WK Let W be the bond weight matrix of the diffusion model. V Let K be the value weight matrix of the diffusion model. m and V m This refers to the prior information.

[0012] Optionally, the step of basing the prior information and the other summing elements (X) i Obtaining the intermediate attention matrix includes: calculating the intermediate attention matrix using the following formula:

[0013]

[0014] (Q i )′=X i W′ Q

[0015] Where Out′ is the intermediate attention matrix, (Q i Let X be the query feature matrix corresponding to the i-th other image. i For the other summation elements corresponding to the i-th other image, W′ O Let W′ be the output weight matrix of the diffusion model. Q Let d be the query weight matrix of the diffusion model, where d is the feature dimension and T represents the transpose operation.

[0016] Optionally, generating the transparent video based on the second transparent latent feature matrix includes: inputting the second transparent latent feature matrix into a VAE decoder to obtain a decoded video sequence; and inputting the decoded video sequence and the second transparent latent feature matrix into a transparent decoder to obtain the decoded transparent video.

[0017] According to a second aspect of the present disclosure, an apparatus for generating transparent video is provided, comprising: a video sequence acquisition module configured to acquire an RGB video sequence and a mask image sequence, wherein the RGB video sequence uses a target RGB image as an intermediate frame, and the mask image sequence uses a target mask image as an intermediate frame, and the target RGB image and the target mask image are obtained by decomposing the target transparent image; a noise addition module configured to add noise to other images included in the RGB video sequence besides the target RGB image to obtain a noisy RGB video sequence; a transparent latent feature matrix acquisition module configured to input the noisy RGB video sequence and the mask image sequence into a transparent encoder to obtain a first transparent latent feature matrix; an RGB latent feature matrix acquisition module configured to input the noisy RGB video sequence into a VAE encoder to obtain an RGB latent feature matrix; and a summation matrix calculation module configured to calculate a summation matrix of the first transparent latent feature matrix and the RGB latent feature matrix, and input the summation matrix into a diffusion model, wherein the summation matrix includes intermediate summation elements (X) corresponding to the RGB intermediate frames that have not been noisily added. m ) and other additive elements (X) corresponding to other images that have already been noise-added. i The transparent video generation module is configured to be based on the intermediate summing element (X). m ) and the other summing elements (X) i ), generate transparent video.

[0018] Optionally, the video sequence acquisition module is configured to: input the target text into a transparent image generation model to obtain the target transparent image, wherein the image content of the target transparent image is the content indicated by the target text; decompose the target transparent image to obtain the target RGB image and the target mask image; and acquire the RGB video sequence with the target RGB image as the intermediate frame and the mask image sequence with the target mask image as the intermediate frame.

[0019] Optionally, the video sequence acquisition module is configured to: acquire a preset number of sequentially arranged pure noise images, and replace the pure noise image located in the middle position of the preset number of sequentially arranged pure noise images with the target RGB image to obtain the RGB video sequence; and acquire a preset number of sequentially arranged all-zero mask images, and replace the all-zero mask image located in the middle position of the preset number of sequentially arranged all-zero mask images with the target mask image to obtain the mask image sequence.

[0020] Optionally, the transparent video generation module is configured to: generate the video based on the intermediate summing element (X) using the diffusion model. mObtain prior information (K) m V m ), and based on the prior information and the other summing elements (X) i Obtain the intermediate attention matrix; obtain the second transparent latent feature matrix based on the intermediate attention matrix using the diffusion model; generate the transparent video based on the second transparent latent feature matrix.

[0021] Optionally, the transparent video generation module is configured to obtain the prior information based on the diffusion model using the following formula:

[0022] K m =X m W K V m =X m W V

[0023] Among them, X m For the intermediate summation element, W K Let W be the bond weight matrix of the diffusion model. V Let K be the value weight matrix of the diffusion model. m and V m This refers to the prior information.

[0024] Optionally, the transparent video generation module is configured to calculate the intermediate attention matrix using the following formula:

[0025]

[0026] (Q i )′=X i W′ Q

[0027] Where Out′ is the intermediate attention matrix, (Q i Let X be the query feature matrix corresponding to the i-th other image. i For the other summation elements corresponding to the i-th other image, W′ O Let W′ be the output weight matrix of the diffusion model. Q Let d be the query weight matrix of the diffusion model, where d is the feature dimension and T represents the transpose operation.

[0028] Optionally, the transparent video generation module is configured to: input the second transparent latent feature matrix into the VAE decoder to obtain a decoded video sequence; and input the decoded video sequence and the second transparent latent feature matrix into the transparent decoder to obtain the decoded transparent video.

[0029] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a method for generating transparent video according to the present disclosure.

[0030] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform a method for generating transparent video according to the present disclosure.

[0031] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a method for generating transparent video according to the present disclosure.

[0032] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects:

[0033] In this disclosure, a diffusion model can be used to determine the intermediate summation element (X) corresponding to the RGB intermediate frame without noise. m ) and other additive elements (X) corresponding to other images that have already been noise-added. i This disclosure utilizes an intermediate frame attention mechanism to provide prior information for the generation of transparent video, thereby decoupling the content information, transparency information, and motion information within video frames. This ensures that the generated transparent video has accurate transparent areas and is free of artifacts.

[0034] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0035] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0036] Figure 1 This is a flowchart illustrating a method for generating transparent video according to exemplary embodiments of the present disclosure;

[0037] Figure 2 This is a schematic diagram illustrating 16 video frames contained in an RGB video sequence according to an exemplary embodiment of the present disclosure;

[0038] Figure 3 This is a system architecture diagram illustrating the generation of transparent video according to exemplary embodiments of the present disclosure;

[0039] Figure 4This is a block diagram illustrating an apparatus for generating transparent video according to an exemplary embodiment of the present disclosure;

[0040] Figure 5 This is a block diagram illustrating an electronic device according to exemplary embodiments of the present disclosure. Detailed Implementation

[0041] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0042] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0043] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. As another example, "performing at least one of step one and step two" indicates the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0044] Green screen technology is widely used in film, television, and video production. Currently, the mainstream method for obtaining green screen video is to shoot it in a professional studio, which often requires significant manpower, resources, and time, resulting in high production costs. In contrast, transparent video retains the transparency information within the frames, allowing for direct video overlay to achieve the same effect as green screen technology.

[0045] However, current models typically only generate standard videos without transparency information, and research on transparent video technology remains limited. Among related technologies, transparent image generation can inject transparency information into the latent feature distribution of a diffusion model. Building upon this, the simplest way to extend transparent image generation to transparent video generation is to integrate a pre-trained motion module into the layer diffusion model. However, since the pre-trained transparent image diffusion model is trained on transparent images, while the motion module is trained on standard video data, this leads to a mismatch in data distribution. This mismatch causes problems such as inaccurate transparent regions and severe distortion. Furthermore, due to the very large model size, training with a small amount of data prevents the model from learning the true distribution of transparent data, resulting in poor transparent video generation performance. However, training with large datasets is also very costly. Therefore, how to achieve transparent video generation while maintaining high video frame quality is a problem worthy of further research.

[0046] To address the aforementioned problems in related technologies, the transparent video generation method, apparatus, electronic device, storage medium, and computer program product provided in this disclosure can utilize a diffusion model based on the intermediate summing element (X) corresponding to the un-noiseed RGB intermediate frame. m ) and other additive elements (X) corresponding to other images that have already been noise-added. i This disclosure utilizes an intermediate frame attention mechanism to provide prior information for the generation of transparent video, thereby decoupling the content information, transparency information, and motion information within video frames. This ensures that the generated transparent video has accurate transparent areas and is free of artifacts.

[0047] Figure 1 This is a flowchart illustrating a method for generating transparent video according to an exemplary embodiment of the present disclosure.

[0048] Reference Figure 1 In step 101, an RGB video sequence and a mask image sequence can be obtained. The RGB video sequence can use the target RGB image as intermediate frames, and the mask image sequence can use the target mask image as intermediate frames. Additionally, the target RGB image and target mask image can be obtained by decomposing the target transparent image.

[0049] According to exemplary embodiments of this disclosure, target text can be input into a transparent image generation model to obtain a target transparent image, wherein the image content of the target transparent image can be the content indicated by the target text. That is, a text prompt can be input into the transparent image generation model to obtain a transparent image containing an alpha channel. For example, assuming the target text content is: "A little boy wearing a blue T-shirt is playing the violin," the transparent image generation model can generate a transparent image with the image content "A little boy wearing a blue T-shirt is playing the violin" based on the target text.

[0050] It should be noted that the transparent image can refer to an image containing RGBA four-channel information, with the last alpha channel representing transparency information. Furthermore, this transparent image can serve as prior information for subsequent processing. Additionally, the transparent image containing the alpha channel can be a grayscale image, where the pixel element values ​​range from 0 to 255. An element value of "0" represents complete transparency, meaning the corresponding pixel is completely invisible; an element value of "255" represents complete opacity.

[0051] Next, the target transparent image can be decomposed to obtain the target RGB image and the target mask image. That is, the transparent image can be decomposed into an RGB image with a gray background and a mask image containing only transparency information. The "mask image" M∈R(H×W), where H represents the height of the mask image, W represents the width of the mask image, and its element values ​​range from 0 to 255. An element value of "0" represents complete transparency, with the corresponding pixels being completely invisible; an element value of "255" represents complete opacity. R represents a two-dimensional real number space. The "RGB image with a gray background" I∈R(H×W×C), where H represents the height of the RGB image, W represents the width of the RGB image, C represents the number of channels in the RGB image, and R represents a three-dimensional real number space.

[0052] Then, an RGB video sequence with the target RGB image as intermediate frames and a mask image sequence with the target mask image as intermediate frames can be obtained. That is, the RGB image with a gray background and the target mask image can be used as intermediate frames to expand into a temporal video.

[0053] According to exemplary embodiments of this disclosure, a predetermined number of sequentially arranged pure noise images can be obtained, and the pure noise image located in the middle position of the predetermined number of sequentially arranged pure noise images can be replaced with a target RGB image, thereby obtaining an RGB video sequence. For example, 16 sequentially arranged pure noise images can be obtained, and the pure noise image located in the middle position, i.e., the 7th position, of the 16 sequentially arranged pure noise images can be replaced with a target RGB image, thereby obtaining an RGB video sequence.

[0054] Figure 2 This is a schematic diagram illustrating 16 video frames comprised of an RGB video sequence according to an exemplary embodiment of the present disclosure. (Refer to...) Figure 2 The video frame located in the middle position, i.e., the 7th position, is the target RGB image; the video frames located in the 0th to 6th positions are pure noise images; and the video frames located in the 8th to 15th positions are pure noise images.

[0055] Furthermore, a predetermined number of sequentially arranged all-zero mask images can be obtained, and the all-zero mask image located in the middle position of the predetermined number of sequentially arranged all-zero mask images can be replaced with the target mask image, thereby obtaining a mask image sequence. For example, 16 sequentially arranged all-zero mask images can be obtained, and the all-zero mask image located in the middle position, i.e., the 7th position, of the 16 sequentially arranged all-zero mask images can be replaced with the target mask image, thereby obtaining a mask image sequence.

[0056] In step 102, noise can be added to other images in the RGB video sequence besides the target RGB image to obtain a noisy RGB video sequence.

[0057] In step 103, the noisy RGB video sequence and the masked image sequence can be input into the transparent encoder to obtain the first transparent latent feature matrix. It should be noted that the transparent encoder is pre-trained on a large number of transparent images; therefore, it can ensure that adding transparent information to the features does not change the original distribution, thereby reducing artifacts caused by inconsistent data distribution. Thus, the transparent encoder can use the masked image sequence and the RGB video sequence as input, ensuring that the encoded features contain transparent information without altering the normal feature distribution.

[0058] In step 104, the noisy RGB video sequence can be input into the VAE encoder to obtain the RGB latent feature matrix. That is, the VAE encoder can be used to encode the noisy RGB video sequence to obtain normally distributed RGB latent features. It should be noted that a variational autoencoder (VAE) typically consists of an encoder and a decoder; the "VAE encoder" mentioned above can refer to the encoder included in a variational autoencoder.

[0059] It should be noted that the RGB latent features x∈R (B×F×H×W×C) In this context, B can represent the batch of transparent images. For example, the batch can be, but is not limited to, 2, 3, or 5, etc. In this case, a transparent video can be generated for each transparent image in the batch, and the process of generating transparent videos for different transparent images can be independent of each other. F can represent the number of video frames; H can represent the height of the RGB latent feature matrix; W can represent the width of the RGB latent feature matrix; C can represent the number of channels of the target RGB image; and R can represent a 5-dimensional real space.

[0060] In step 105, the sum of the first transparent latent feature matrix and the RGB latent feature matrix can be calculated, and the sum can be input into the diffusion model. This sum can contain the intermediate sum elements (X) corresponding to the RGB intermediate frames that have not been denoised. m ) and other additive elements (X) corresponding to other images that have already been noise-added. i ).

[0061] For example, as mentioned earlier, both the RGB video sequence and the masked image sequence can contain 16 video frames. In this case, the summation matrix can contain the RGB intermediate frames that have not been noise-added, that is, the intermediate summation element (X) corresponding to the 7th RGB video frame. m ), and the other summation elements (X) corresponding to each of the other images that have been noisy, from the 0th to the 6th and from the 8th to the 15th noisy images. i ).

[0062] It should be noted that the "diffusion model" in this disclosure can be composed of multiple spatial modules and temporal modules. The "spatial module" can be composed of convolutional layers and attention layers composed of multiple attention layers, and the "temporal module" can be composed of inter-frame attention layers.

[0063] Specifically, the attention layer composed of various attention mechanisms in the spatial module can include a "Self-Attention layer," a "Cross-Attention layer," and an "Inter-Frame Attention layer." The "Inter-Frame Attention layer" can obtain information from undisturbed intermediate frames by using them as keys and values, and other frames as query frames, thus providing prior information for the frames to be predicted. Furthermore, when processing features, the spatial module needs to incorporate the video length dimension into the batch dimension for computation.

[0064] The attention layer in the temporal module is mainly a self-attention layer. Furthermore, when processing features, the temporal module can perform attention calculations on the video frame dimension to obtain the relationship between different frames.

[0065] In step 106, the intermediate summation element (X) can be used as a basis. m ) and other summation elements (X) i ), generate transparent video.

[0066] According to exemplary embodiments of this disclosure, a diffusion model can be used based on intermediate additive elements (X). m Obtain prior information (K) m V m ), and can be based on prior information and other additive elements (X) i First, obtain the intermediate attention matrix Out′. Then, based on the intermediate attention matrix, a second transparent latent feature matrix can be obtained using a diffusion model. Next, a transparent video can be generated based on the second transparent latent feature matrix.

[0067] According to exemplary embodiments of this disclosure, prior information can be obtained using a diffusion model based on the following formula:

[0068] K m =X m W K V m =X m W V

[0069] Among them, X m For the middle element, W K W is the bond weight matrix of the diffusion model. V K is the value weight matrix of the diffusion model. m and V m This is prior information.

[0070] In this disclosure, the intermediate frame attention layer in the spatial module contained in the diffusion model can be used to take the undisturbed intermediate frames as keys and values, and the remaining frames can be used as query frames, thereby obtaining prior information from the undisturbed intermediate frames.

[0071] According to exemplary embodiments of this disclosure, the intermediate attention matrix can be calculated using the following formula, i.e., the intermediate frame attention mechanism can be formulated as follows:

[0072]

[0073] (Q i )′=X i W′ Q

[0074] Where Out′ is the intermediate attention matrix, (Q i X' is the query feature matrix corresponding to the i-th other image. i For the other summation elements corresponding to the i-th other image, W′ O Let W′ be the output weight matrix of the diffusion model. Q Let d be the query weight matrix of the diffusion model, where d is the feature dimension and T represents the transpose operation.

[0075] It should be noted that in the intermediate attention matrix Out′, only W′ is present. Q and W′ O It is trainable, while the other parameters can be set to non-trainable during the training phase. Since the intermediate frames contain high-quality detail information and transparency information, in this disclosure, an intermediate frame attention mechanism can be used to set the unnoised intermediate frames as keys and values, and the remaining frames as query frames. This allows the content information and transparency information of the intermediate frames to be obtained as prior information, enabling the model to decouple the content and transparency information from the action, thus allowing the model to focus more on learning the action.

[0076] Furthermore, in this disclosure, the intermediate frame attention mechanism and the self-attention mechanism can be executed in parallel, and their outputs can be added together. The result of the addition can then be input into the cross-attention layer to learn inter-frame information.

[0077] According to an exemplary embodiment of this disclosure, a second transparent latent feature matrix can be input into a VAE decoder to obtain a decoded video sequence. Then, the decoded video sequence and the second transparent latent feature matrix can be input into a transparent decoder to obtain a decoded transparent video. It should be noted that, as mentioned above, a variational autoencoder (VAE) typically consists of an encoder and a decoder, and the aforementioned "VAE decoder" can refer to the decoder included in the variational autoencoder.

[0078] Figure 3 This is a system architecture diagram illustrating the generation of transparent video according to an exemplary embodiment of the present disclosure. (Refer to...) Figure 3 When generating transparent videos, the main components required are a Layer Diffuse model, a Contrastive Language-Image Pre-training (CLIP) model (CLIP is primarily used to extract textual features from the target text; for example, the target text could be: "A little boy in a blue T-shirt is playing the violin"), a Tansparency Encoder, a Tansparency Decoder, a VAE Encoder, a VAE Decoder, an Adder, and a diffusion model. The diffusion model mainly consists of a Spatial Block and a Motion Module, with the Spatial Block primarily containing a self-attention layer, an intermediate frame attention layer, and a cross-attention layer.

[0079] In this disclosure, a diffusion model can be used to determine the intermediate summation element (X) corresponding to the RGB intermediate frame without noise. m ) and other additive elements (X) corresponding to other images that have already been noise-added. i This disclosure utilizes an intermediate frame attention mechanism to provide prior information for the generation of transparent videos. This decouples the content information, transparency information, and action information within video frames, allowing the model to focus more on action learning. Consequently, it ensures the generation of transparent videos with accurate transparent areas and no artifacts.

[0080] Figure 4 This is a block diagram illustrating a transparent video generation apparatus 400 according to an exemplary embodiment of the present disclosure.

[0081] Reference Figure 4 The transparent video generation device 400 may include a video sequence acquisition module 401, a noise addition module 402, a transparent latent feature matrix acquisition module 403, an RGB latent feature matrix acquisition module 404, a summation matrix calculation module 405, and a transparent video generation module 406.

[0082] The video sequence acquisition module 401 can acquire RGB video sequences and mask image sequences. The RGB video sequence can use the target RGB image as intermediate frames, and the mask image sequence can use the target mask image as intermediate frames. Furthermore, the target RGB image and target mask image can be obtained by decomposing the target transparent image.

[0083] According to an exemplary embodiment of this disclosure, the video sequence acquisition module 401 can input target text into a transparent image generation model to obtain a target transparent image, wherein the image content of the target transparent image can be the content indicated by the target text. That is, the prompt can be input into the transparent image generation model to obtain a transparent image containing an alpha channel. For example, assuming the text content of the target text is: "A little boy wearing a blue T-shirt is playing the violin", the transparent image generation model can generate a transparent image with the image content "A little boy wearing a blue T-shirt is playing the violin" based on the target text.

[0084] It should be noted that the transparent image can refer to an image containing RGBA four-channel information, with the last alpha channel representing transparency information. Furthermore, this transparent image can serve as prior information for subsequent processing. Additionally, the transparent image containing the alpha channel can be a grayscale image, where the pixel element values ​​range from 0 to 255. An element value of "0" represents complete transparency, meaning the corresponding pixel is completely invisible; an element value of "255" represents complete opacity.

[0085] Next, the video sequence acquisition module 401 can decompose the target transparent image to obtain the target RGB image and the target mask image. That is, the transparent image can be decomposed into an RGB image with a gray background and a mask image containing only transparency information. The "mask image" M∈R(H×W), where H represents the height of the mask image, W represents the width of the mask image, and its element value ranges from 0 to 255. The element value "0" represents complete transparency, and its corresponding pixels are completely invisible; the element value "255" represents complete opacity. R represents a two-dimensional real number space. The "RGB image with a gray background" I∈R(H×W×C), where H represents the height of the RGB image, W represents the width of the RGB image, C represents the number of channels in the RGB image, and R represents a three-dimensional real number space.

[0086] Then, the video sequence acquisition module 401 can acquire an RGB video sequence with the target RGB image as the intermediate frame, and a mask image sequence with the target mask image as the intermediate frame. That is, the RGB image with a gray background and the target mask image can be used as intermediate frames to expand the sequence into a temporal video.

[0087] According to an exemplary embodiment of this disclosure, the video sequence acquisition module 401 can acquire a preset number of sequentially arranged pure noise images, and can replace the pure noise image located in the middle position of the preset number of sequentially arranged pure noise images with a target RGB image, thereby obtaining an RGB video sequence. For example, it can acquire 16 sequentially arranged pure noise images, and can replace the pure noise image located in the middle position, i.e., the 7th position, of the 16 sequentially arranged pure noise images with a target RGB image, thereby obtaining an RGB video sequence.

[0088] Furthermore, the video sequence acquisition module 401 can acquire a preset number of sequentially arranged all-zero mask images, and can replace the all-zero mask image located in the middle position of the preset number of sequentially arranged all-zero mask images with the target mask image, thereby obtaining a mask image sequence. For example, it can acquire 16 sequentially arranged all-zero mask images, and can replace the all-zero mask image located in the middle position, i.e., the 7th position, of the 16 sequentially arranged all-zero mask images with the target mask image, thereby obtaining a mask image sequence.

[0089] The noise-adding module 402 can add noise to other images in the RGB video sequence besides the target RGB image to obtain a noise-adding RGB video sequence.

[0090] The transparent latent feature matrix acquisition module 403 can input the noisy RGB video sequence and the mask image sequence into the transparent encoder to obtain the first transparent latent feature matrix. It should be noted that the transparent encoder is pre-trained on a large number of transparent images; therefore, it can ensure that adding transparent information to the features does not change the original distribution, thereby reducing artifacts caused by inconsistent data distribution. Thus, the transparent encoder can use the mask image sequence and the RGB video sequence as input, ensuring that the encoded features contain transparent information without altering the normal feature distribution.

[0091] The RGB latent feature matrix acquisition module 404 can input the noisy RGB video sequence into the VAE encoder to obtain the RGB latent feature matrix. That is, the VAE encoder can be used to encode the noisy RGB video sequence to obtain the normally distributed RGB latent features.

[0092] It should be noted that the RGB latent features x∈R (B×F×H×W×C)In this context, B can represent the batch of transparent images. For example, the batch can be, but is not limited to, 2, 3, or 5, etc. In this case, a transparent video can be generated for each transparent image in the batch, and the process of generating transparent videos for different transparent images can be independent of each other. F can represent the number of video frames; H can represent the height of the RGB latent feature matrix; W can represent the width of the RGB latent feature matrix; C can represent the number of channels of the target RGB image; and R can represent a 5-dimensional real space.

[0093] The summation matrix calculation module 405 can calculate the summation matrix of the first transparent latent feature matrix and the RGB latent feature matrix, and can input the summation matrix into the diffusion model. This summation matrix can contain the intermediate summation elements (X) corresponding to the RGB intermediate frames that have not been denoised. m ) and other additive elements (X) corresponding to other images that have already been noise-added. i ).

[0094] For example, as mentioned earlier, both the RGB video sequence and the masked image sequence can contain 16 video frames. In this case, the summation matrix can contain the RGB intermediate frames that have not been noise-added, that is, the intermediate summation element (X) corresponding to the 7th RGB video frame. m ), and the other summation elements (X) corresponding to each of the other images that have been noisy, from the 0th to the 6th and from the 8th to the 15th noisy images. i ).

[0095] It should be noted that the "diffusion model" in this disclosure can be composed of multiple spatial modules and temporal modules. The "spatial module" can be composed of convolutional layers and attention layers composed of multiple attention layers, and the "temporal module" can be composed of inter-frame attention layers.

[0096] Specifically, the attention layer composed of various attention mechanisms in the spatial module can include a "Self-Attention layer," a "Cross-Attention layer," and an "Inter-Frame Attention layer." The "Inter-Frame Attention layer" can obtain information from undisturbed intermediate frames by using them as keys and values, and other frames as query frames, thus providing prior information for the frames to be predicted. Furthermore, when processing features, the spatial module needs to incorporate the video length dimension into the batch dimension for computation.

[0097] The attention layer in the temporal module is mainly a self-attention layer. Furthermore, when processing features, the temporal module can perform attention calculations on the video frame dimension to obtain the relationship between different frames.

[0098] The transparent video generation module 406 can be based on the intermediate summing element (X) m ) and other summation elements (X) i ), generate transparent video.

[0099] According to an exemplary embodiment of this disclosure, the transparent video generation module 406 can generate a video based on an intermediate additive element (X) using a diffusion model. m Obtain prior information (K) m V m ), and can be based on prior information and other additive elements (X) i The intermediate attention matrix Out′ is obtained. Then, the transparent video generation module 406 can obtain the second transparent latent feature matrix based on the intermediate attention matrix using a diffusion model. Next, the transparent video generation module 406 can generate a transparent video based on the second transparent latent feature matrix.

[0100] According to an exemplary embodiment of this disclosure, the transparent video generation module 406 can obtain prior information based on the following formula using a diffusion model:

[0101] K m =X m W K V m =X m W V

[0102] Among them, X m For the middle element, W K W is the bond weight matrix of the diffusion model. V K is the value weight matrix of the diffusion model. m and V m This is prior information.

[0103] In this disclosure, the intermediate frame attention layer in the spatial module contained in the diffusion model can be used to take the undisturbed intermediate frames as keys and values, and the remaining frames can be used as query frames, thereby obtaining prior information from the undisturbed intermediate frames.

[0104] According to an exemplary embodiment of this disclosure, the transparent video generation module 406 can calculate the intermediate attention matrix using the following formula, that is, the intermediate frame attention mechanism can be formulated as follows:

[0105]

[0106] (Q i )′=X i W′ Q

[0107] Where Out′ is the intermediate attention matrix, (Q i X' is the query feature matrix corresponding to the i-th other image. i For the other summation elements corresponding to the i-th other image, W′ O Let W′ be the output weight matrix of the diffusion model. Q Let d be the query weight matrix of the diffusion model, where d is the feature dimension and T represents the transpose operation.

[0108] It should be noted that in the intermediate attention matrix Out′, only W′ is present. Q and W′ O It is trainable, while the other parameters can be set to non-trainable during the training phase. Since the intermediate frames contain high-quality detail information and transparency information, in this disclosure, an intermediate frame attention mechanism can be used to set the unnoised intermediate frames as keys and values, and the remaining frames as query frames. This allows the content information and transparency information of the intermediate frames to be obtained as prior information, enabling the model to decouple the content and transparency information from the action, thus allowing the model to focus more on learning the action.

[0109] Furthermore, in this disclosure, the intermediate frame attention mechanism and the self-attention mechanism can be executed in parallel, and their outputs can be added together. The result of the addition can then be input into the cross-attention layer to learn inter-frame information.

[0110] According to an exemplary embodiment of this disclosure, the transparent video generation module 406 can input the second transparent latent feature matrix into the VAE decoder to obtain a decoded video sequence. Then, the transparent video generation module 406 can input the decoded video sequence and the second transparent latent feature matrix into the transparent decoder to obtain the decoded transparent video.

[0111] Figure 5 This is a block diagram illustrating an electronic device 500 according to an exemplary embodiment of the present disclosure.

[0112] Reference Figure 5 The electronic device 500 includes at least one memory 501 and at least one processor 502. The at least one memory 501 stores instructions that, when executed by the at least one processor 502, perform a method for generating transparent video according to an exemplary embodiment of the present disclosure.

[0113] As an example, electronic device 500 may be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned instructions. Here, electronic device 500 is not necessarily a single electronic device, but may be a collection of any devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 500 may also be part of an integrated control system or system manager, or may be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0114] In electronic device 500, processor 502 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0115] The processor 502 can execute instructions or code stored in the memory 501, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transmission protocol.

[0116] The memory 501 may be integrated with the processor 502, for example, by arranging RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 501 may include a separate device, such as an external disk drive, a storage array, or other storage device usable by any database system. The memory 501 and the processor 502 may be operatively coupled, or may communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 502 to read files stored in the memory.

[0117] In addition, electronic device 500 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of electronic device 500 can be interconnected via a bus and / or network.

[0118] According to exemplary embodiments of this disclosure, a computer-readable storage medium may also be provided, which, when executed by a processor of an electronic device, enables the electronic device to perform the aforementioned method for generating transparent video. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0119] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, including a computer program that, when executed by a processor, implements the method for generating transparent video according to the present disclosure.

[0120] According to the transparent video generation method, apparatus, electronic device, storage medium, and computer program product disclosed herein, a diffusion model can be used to generate transparent video based on the intermediate summation element (X) corresponding to the un-noiseed RGB intermediate frame. m ) and other additive elements (X) corresponding to other images that have already been noise-added. i This disclosure utilizes an intermediate frame attention mechanism to provide prior information for the generation of transparent video, thereby decoupling the content information, transparency information, and motion information within video frames. This ensures that the generated transparent video has accurate transparent areas and is free of artifacts.

[0121] According to exemplary embodiments of this disclosure, a transparent encoder can take a masked image sequence and an RGB video sequence as input, which can ensure that the encoded features contain transparent information without changing the normal feature distribution.

[0122] According to an exemplary embodiment of this disclosure, since intermediate frames contain high-quality detail information and transparency information, in this disclosure, an intermediate frame attention mechanism can be used to set the unnoised intermediate frames as keys and values, and the remaining frames can be set as query frames. In this way, the content information and transparency information of the intermediate frames can be obtained as prior information, so that the model can decouple the content and transparency information from the action, that is, the model can focus more on the learning of the action.

[0123] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0124] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for generating transparent video, characterized in that, include: Obtain an RGB video sequence and a mask image sequence, wherein the RGB video sequence uses the target RGB image as the intermediate frame, and the mask image sequence uses the target mask image as the intermediate frame. The target RGB image and the target mask image are obtained by decomposing the target transparent image. Noise is added to all images in the RGB video sequence except the target RGB image to obtain a noisy RGB video sequence; The noisy RGB video sequence and the masked image sequence are input into a transparent encoder to obtain a first transparent latent feature matrix; The noisy RGB video sequence is input into the VAE encoder to obtain the RGB latent feature matrix; Calculate the sum of the first transparent latent feature matrix and the RGB latent feature matrix, and input the sum of the matrix into the diffusion model, wherein the sum of the matrix contains the intermediate sum of the RGB intermediate frames that have not been denoised. ) and other additive elements corresponding to other images that have already been noise-added ( ); Based on the intermediate summing element ( ) and the other summing elements ( ), generate transparent video; Wherein, the element based on the intermediate summation element ( ) and the other summing elements ( Generate transparent video, including: Based on the intermediate additive element through the diffusion model ( Obtain prior information ( , ), and based on the prior information and the other summing elements ( Obtain the intermediate attention matrix; The second transparent latent feature matrix is ​​obtained based on the intermediate attention matrix using the diffusion model. The transparent video is generated based on the second transparent latent feature matrix; Wherein, the diffusion model is based on the intermediate summing element ( Obtain prior information ( , ),include: The prior information is obtained using the diffusion model based on the following formula: ， in, Add the intermediate element. Here is the bond weight matrix of the diffusion model. The value weight matrix of the diffusion model is... and The prior information; Among them, based on the prior information and the other summing elements ( Obtain the intermediate attention matrix, including: The intermediate attention matrix is ​​calculated using the following formula: in, The intermediate attention matrix, Let i be the query feature matrix corresponding to the i-th other image. For the other summation elements corresponding to the i-th other image, This is the output weight matrix of the diffusion model. Let d be the query weight matrix of the diffusion model, where d is the feature dimension and T represents the transpose operation; The diffusion model includes a temporal module, and the attention layer included in the temporal module is a self-attention layer. The step of obtaining the second transparent latent feature matrix based on the intermediate attention matrix using the diffusion model includes: The intermediate attention matrix is ​​input into the self-attention layer in the temporal module of the diffusion model to obtain the second transparent latent feature matrix; The step of generating the transparent video based on the second transparent latent feature matrix includes: The second transparent latent feature matrix is ​​input into the VAE decoder to obtain the decoded video sequence; The decoded video sequence and the second transparent latent feature matrix are input into the transparent decoder to obtain the decoded transparent video.

2. The generation method as described in claim 1, characterized in that, The acquisition of the RGB video sequence and the mask image sequence includes: The target text is input into the transparent image generation model to obtain the target transparent image, wherein the image content of the target transparent image is the content indicated by the target text; The target transparent image is decomposed to obtain the target RGB image and the target mask image; Obtain the RGB video sequence with the target RGB image as the intermediate frame, and the mask image sequence with the target mask image as the intermediate frame.

3. The generation method as described in claim 2, characterized in that, The step of acquiring the RGB video sequence with the target RGB image as the intermediate frame and the mask image sequence with the target mask image as the intermediate frame includes: Obtain a predetermined number of sequentially arranged pure noise images, and replace the middle pure noise image among the predetermined number of sequentially arranged pure noise images with the target RGB image to obtain the RGB video sequence; and, Obtain a predetermined number of zero-mask images arranged in sequence, and replace the zero-mask image located in the middle position of the predetermined number of zero-mask images with the target mask image to obtain the mask image sequence.

4. A device for generating transparent video, characterized in that, include: The video sequence acquisition module is configured to acquire an RGB video sequence and a mask image sequence, wherein the RGB video sequence uses the target RGB image as the intermediate frame, and the mask image sequence uses the target mask image as the intermediate frame. The target RGB image and the target mask image are obtained by decomposing the target transparent image. The noise-adding module is configured to add noise to other images in the RGB video sequence besides the target RGB image, to obtain a noise-adding RGB video sequence; The transparent latent feature matrix acquisition module is configured to input the noisy RGB video sequence and the mask image sequence into the transparent encoder to obtain the first transparent latent feature matrix; The RGB latent feature matrix acquisition module is configured to input the noisy RGB video sequence into a VAE encoder to obtain the RGB latent feature matrix. The summation matrix calculation module is configured to calculate the summation matrix of the first transparent latent feature matrix and the RGB latent feature matrix, and input the summation matrix into the diffusion model, wherein the summation matrix contains intermediate summation elements corresponding to the RGB intermediate frames that have not been noise-added ( ) and other additive elements corresponding to other images that have already been noise-added ( ); The transparent video generation module is configured to be based on the intermediate summing element ( ) and the other summing elements ( ), generate transparent video; The transparent video generation module is configured as follows: Based on the intermediate additive element through the diffusion model ( Obtain prior information ( , ), and based on the prior information and the other summing elements ( Obtain the intermediate attention matrix; The second transparent latent feature matrix is ​​obtained based on the intermediate attention matrix using the diffusion model. The transparent video is generated based on the second transparent latent feature matrix; The transparent video generation module is configured as follows: The prior information is obtained using the diffusion model based on the following formula: ， in, Add the intermediate element. Here is the bond weight matrix of the diffusion model. The value weight matrix of the diffusion model is... and The prior information; The transparent video generation module is configured as follows: The intermediate attention matrix is ​​calculated using the following formula: in, The intermediate attention matrix, Let i be the query feature matrix corresponding to the i-th other image. For the other summation elements corresponding to the i-th other image, This is the output weight matrix of the diffusion model. Let d be the query weight matrix of the diffusion model, where d is the feature dimension and T represents the transpose operation; The diffusion model includes a temporal module, and the attention layer included in the temporal module is a self-attention layer. The transparent video generation module is configured as follows: The intermediate attention matrix is ​​input into the self-attention layer in the temporal module of the diffusion model to obtain the second transparent latent feature matrix; The transparent video generation module is configured as follows: The second transparent latent feature matrix is ​​input into the VAE decoder to obtain the decoded video sequence; The decoded video sequence and the second transparent latent feature matrix are input into the transparent decoder to obtain the decoded transparent video.

5. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method for generating transparent video as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method for generating transparent video as described in any one of claims 1 to 3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for generating transparent video as described in any one of claims 1 to 3.

Images