Image matting method based on text prior and stable diffusion model

Abstract

The invention discloses an image matting method based on text prior and a stable diffusion model, which comprises the following steps of: firstly, acquiring an original image and a foreground description text thereof, and respectively extracting image hidden space representation and text features by using an image encoder and a text encoder; learning the text features by using a learnable text feature converter to obtain language features; then the image hidden space representation and the language features are sent to a teacher denoising network for pre-training; after the pre-training is completed, taking the internal features and the attention map as supervision information of a student denoising network, and performing distillation training on the student denoising network to identify transparency to obtain a soft grounding result; and finally, inputting the soft grounding result and the original image into a detail-matting encoder together to obtain a predicted transparency mask. According to the method, the soft grounding framework based on text guidance is introduced, the extended mode priori knowledge of the denoising network in the SD model is fully utilized, the matting precision in a complex scene is remarkably improved, and traditional visual priori guidance does not need to be relied on.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image processing, and particularly relates to an image matting method based on text prior and stable diffusion model. Background Art

[0002] Image Matting is a classic problem in the field of computer vision, aiming to accurately extract foreground objects from an image and estimate the transparency (alpha matte) of each pixel. The traditional image matting problem is usually modeled by the following formula: I = αF+(1 - α)B, where I is the input image, α is the transparency, F is the foreground color, and B is the background color. Since this problem is a highly ill-posed problem, researchers usually rely on various prior information to assist in solving, such as trimaps (segmentation maps annotating foreground, background, and unknown regions), background estimation, binary masks, and user interaction inputs (such as clicks, scribbles, etc.). Although these methods perform well in some scenarios, there are obvious limitations. For example, the generation of trimaps requires a large amount of manual annotation, background estimation and binary masks perform poorly in dynamic scenarios, and user interaction inputs are difficult to handle complex texture and multi-object scenarios in natural images. In recent years, with the development of Vision-Language Models, text-guided image processing technologies have made significant progress. Especially in image generation and segmentation tasks, text-to-image diffusion models (such as StableDiffusion, SD) have demonstrated powerful semantic discrimination capabilities and cross-modal attention mechanisms. These models can generate images or perform pixel-level segmentation tasks through natural language descriptions, providing new ideas for image matting. There are mainly three image matting methods in the prior art.

[0003] The first is a text-guided matting method, especially CLIPMat (Li, J., Zhang, J., & Tao, D. (2023). Referring image matting. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition (pp. 22448-22457).). CLIPMat uses the Contrastive Language-Image Pretraining (CLIP) model to fuse visual and text features, enabling text-description-based matting. It dynamically identifies the matting object through text description, avoiding the dependence on trimaps or user interaction in traditional methods. The second is a matting method based on Diffusion Models. Diffusion Models have shown great potential in generation tasks and dense prediction tasks. For example, the Context Matting method proposed by Guo H et al. utilizes the context correspondence prior in the Stable Diffusion model to perform matting in different scenarios by referring to an image (Guo, H., Ye, Z., Cao, Z., & Lu, H. (2024). In-Context Matting. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 3711-3720).). The Generative Matting algorithm proposed by Wang Z et al. models the matting task as a generation task and predicts the transparency mask through a multi-step denoising process (Wang, Z., Li, B., Wang, J., Liu, Y. L., Gu, J., Chuang, Y. Y., & Satoh, S. I. (2024, July). Matting by generation. In ACM SIGGRAPH 2024 Conference Papers (pp. 1-11).). The third is a diffusion matting method based on trimaps, which directly performs diffusion in the pixel space or latent space, starting from a perturbed trimap or pure noise to gradually generate the transparency mask.In addition, the Diffusion Priors Joint Prediction (DRIP) model proposed by Li X et al. introduces an improved self-attention mechanism in the latent space on this basis to better model the matting context (Li, X., Yang, Z., Quan, R., & Yang, Y. (2025). DRIP: Unleashing Diffusion Priors for Joint Foreground and Alpha Prediction in Image Matting. Advances in Neural Information Processing Systems, 37, 79868-79888.).

[0004] Although the above-mentioned existing technologies have made remarkable progress, there are still limitations. On the one hand, most existing image matting methods (such as methods based on trimaps, background estimation, or user interaction) rely on specific visual prior information to assist in solving the ill-posed problem of image matting. However, the acquisition of prior information is difficult and the accuracy is low. Therefore, they perform poorly when dealing with complex scenes, multiple objects, or dynamic textures, and it is difficult to achieve automation or large-scale applications. On the other hand, matting methods based on diffusion models (such as the Stable Diffusion model) usually require a multi-step denoising process, with a relatively high computational cost, and it is difficult to be applied to real-time scenarios or resource-constrained devices. Additionally, existing matting methods are usually optimized for specific tasks (such as portrait matting or natural image matting), and it is difficult to generalize to multiple matting tasks to meet diverse matting requirements. Therefore, there is an urgent need for a more flexible and general image matting method that can handle multiple matting tasks without relying on complex visual priors. Summary of the Invention

[0005] The main objective of the present invention is to overcome the shortcomings and deficiencies of the existing technologies, and provide an image matting method based on text priors and the Stable Diffusion model. By taking spatial localization and transparency estimation as independent learning objectives, and leveraging the semantic depth of the diffusion model, the dependence on rigid visual priors is reduced, thereby achieving a more accurate and flexible matting effect in complex scenes.

[0006] To achieve the above objective, the first objective of the present invention is to provide an image matting method based on text priors and the Stable Diffusion model, including the following steps:

[0007] Obtain the original image and its foreground description text, and use an image encoder and a text encoder to extract the image latent space representation and text features respectively; both the image encoder and the text encoder are pre-trained and their parameters are frozen;

[0008] The text features are learned using a learnable text feature converter to obtain language features;

[0009] A knowledge distillation framework is used to train a student denoising network. The training process is as follows: The latent space representation of the images in the set image training set and the language features are fed into the teacher denoising network for pre-training, and a semantic soft grounding result is output; After the pre-training of the teacher denoising network is completed, the internal features of the teacher denoising network and the attention map are used as the supervision information for the student denoising network, and the student denoising network is distilled and trained, and the student denoising network is allowed to perform transparency recognition on the latent space representation of the images in the set image training set and the text features input to obtain a soft grounding result; Both the teacher denoising network and the student denoising network use the denoising network of the stable diffusion model; The soft grounding result includes a transparency confidence map and a rough transparency mask; After the training is completed, the parameters of the trained student denoising network are frozen;

[0010] The latent space representation of the original image and the language features are input into the trained student denoising network to obtain a soft grounding result, which is upsampled to the same resolution as the original image and stitched with the original image and input into the detail encoder to obtain a fused feature; The matte context feature is obtained from the output of the last layer of the trained student denoising network, and after applying a group of learnable convolutional blocks to align the feature spaces, it is stitched with the fused feature and input into the matte decoder to obtain a predicted transparency mask; The learnable convolutional blocks are pre-trained and their parameters are frozen.

[0011] As a preferred technical solution, the image encoder uses a variational autoencoder; The text encoder uses a CLIP model with frozen parameters after pre-training; The text feature converter consists of two multi-layer perceptrons.

[0012] As a preferred technical solution, the last latent prediction head in the denoising network of the stable diffusion model is replaced by a new prediction head consisting of a two-group convolution-batch normalization-activation function module.

[0013] As a preferred technical solution, during the pre-training of the teacher denoising network, the classification probability of each channel in the semantic soft grounding result is learned by minimizing the cross-entropy loss, expressed as:

[0014]

[0015] where L CE is the cross-entropy loss, is the classification probability of the c-th channel in the semantic soft grounding result of the set image training set, is the reference region representation of the c-th channel generated by applying image morphological operations to the reference transparency mask of the set image training set.

[0016] As a preferred technical solution, the student denoising network is obtained by optimizing the denoising network of the StableDiffusion model. The optimization process is as follows:

[0017] Prune the redundant blocks of the denoising network of the StableDiffusion model;

[0018] Introduce a learnable downsampling operation to optimize the self-attention operation in the denoising network of the StableDiffusion model. Use the downsampling operation to calculate the key embedding feature and value embedding feature in the self-attention operation, and keep the query embedding feature unchanged. The downsampling operation is implemented through a convolutional layer with a kernel size and stride both equal to k×k.

[0019] As a preferred technical solution, the parameters of the teacher denoising network are frozen during the distillation training process of the student denoising network;

[0020] During the distillation training process, feature-level distillation loss and transparency mining loss are used for optimization learning;

[0021] The feature-level distillation loss is expressed as:

[0022]

[0023] where L SG is the feature-level distillation loss, represents the intermediate feature output by the teacher denoising network at the l-th layer, represents the intermediate feature output by the student denoising network at the l-th layer, z I is the image latent space representation of the set image training set, f T is the language feature of the set image training set; M is a binary mask, with the foreground and background regions set to 1 and other regions set to 0; ⊙ is the Hadamard product; Φ is a projection module used to align the intermediate features of the teacher denoising network and the student denoising network; is the squared L2 norm function;

[0024] The transparency mining loss is expressed as:

[0025]

[0026] where L STM is the transparency mining loss, is the average of the self-attention maps at different resolutions output by the teacher denoising network, is the average of the self-attention maps at different resolutions output by the student denoising network, represents matrix multiplication, α GTTo set the reference transparency mask of the image training set, a state matrix corresponding to the resolution of the self-attention map is directly downsampled by bilinear interpolation;

[0027] The projection module consists of three learnable convolutional layers and an intermediate LeakyReLU layer, and the formula is:

[0028] Φ(F) = Cv 3×3 (Cv 3×3 (LeakyReLU(Cv 1×1 (F)))),

[0029] where F is the input feature, and Cv 3×3 is a learnable 3×3 convolutional layer, and Cv 1×1 is a learnable 1×1 convolutional layer.

[0030] As a preferred technical solution, during the distillation training of the student denoising network, the cross-attention map distillation loss is also used to constrain the cross-attention map between the teacher denoising network and the student denoising network, the first norm loss is used to learn the rough transparency mask, and the binary cross-entropy loss is used to learn the transparency confidence map;

[0031] The cross-attention map distillation loss is expressed as:

[0032]

[0033] where L CTM is the cross-attention map distillation loss, is the average value of the cross-attention maps at different resolutions output by the teacher denoising network, is the average value of the cross-attention maps at different resolutions output by the student denoising network;

[0034] The first norm loss is expressed as:

[0035]

[0036] where α lr is the rough transparency mask in the soft grounding result of the set image training set, is the true transparency mask obtained by downsampling the reference transparency mask of the set image training set, and ‖‖1 is the L1 norm;

[0037] The binary cross-loss is expressed as:

[0038]

[0039] where R is the transparency confidence map in the soft grounding result of the set image training set, and e R is the exponential of all values in R with the natural constant as the base, is the binary mask for the transparent region;

[0040] The total loss function in the distillation training process of the student denoising network is expressed as:

[0041]

[0042] where λ STM , λ CTM , λ SG , λ R are the hyperparameters corresponding to the respective losses.

[0043] As a preferred technical solution, the detail encoder and the matte decoder are implemented using an encoder-decoder architecture, and both include four convolutional neural network layers; each convolutional neural network layer includes three operations: convolutional calculation - batch normalization - ReLU activation function;

[0044] The detail encoder has a downsampling operation with a set magnification between convolutional neural network layers;

[0045] The matte decoder has an upsampling operation with a set magnification between convolutional neural network layers;

[0046] The detail encoder and the matte decoder have skip connections at the same resolution;

[0047] The batch normalization and ReLU of the last layer of the matte decoder are removed, and the prediction result is directly output after convolutional calculation;

[0048] The detail decoder and the matte decoder are supervised and trained using the first norm loss and the Laplacian loss.

[0049] The second object of the present invention is to provide an image matte extraction system based on text prior and stable diffusion model, which is applied to the image matte extraction method based on text prior and stable diffusion model, and includes a feature representation module, a semantic acquisition module, a soft grounding implementation module, and a soft grounding processing module;

[0050] The feature representation is used to obtain the original image and its foreground description text, and the image encoder and the text encoder are used to extract the image latent space representation and the text features respectively; both the image encoder and the text encoder are pre-trained and the parameters are frozen;

[0051] The semantic acquisition module is used to learn the text features using a learnable text feature transformer to obtain language features;

[0052] The soft grounding implementation module is used to train the student denoising network using the knowledge distillation framework. The training process is as follows: the image latent space representation and language features of the set image training set are sent to the teacher denoising network for pre-training, and the semantic soft grounding result is output; after the pre-training of the teacher denoising network is completed, the internal features and attention map of the teacher denoising network are used as the supervision information of the student denoising network, and the student denoising network is distilled and trained, and the student denoising network is allowed to perform transparency recognition on the image latent space representation and text features of the input set image training set to obtain the soft grounding result; the teacher denoising network and the student denoising network both use the denoising network of the stable diffusion model; the soft grounding result includes a transparent confidence map and a rough transparency mask; after the training is completed, the parameters of the trained student denoising network are frozen;

[0053] The soft grounding processing module is used to input the image latent space representation and language features of the original image into the trained student denoising network to obtain a soft grounding result, interpolate to the same resolution as the original image and splice with the original image to input into the detail encoder to obtain a fused feature; obtain the cutout context feature from the last layer output of the trained student denoising network, apply a set of learnable convolution blocks to align the feature space and then splice with the fused feature, and input into the cutout decoder to obtain a predicted transparency mask; the learnable convolution block is obtained through pre-training and the parameters are frozen.

[0054] The third object of the present invention is to provide a computer-readable storage medium storing a program, which, when executed by a processor, implements the image cutout method based on text prior and stable diffusion model.

[0055] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0056] 1. This invention unifies multiple cutout tasks (such as animal cutout, natural image cutout, video instance cutout, and portrait cutout) by introducing a text-guided soft grounding framework, overcoming the traditional method's reliance on specific scenes (such as the need for tripartite maps or background estimation). This invention uses a multi-stage approach to fine-tune the denoising network in the stable diffusion model, and then fully utilizes the powerful cross-modal prior knowledge learned in the denoising network in large-scale pre-training, so that the fine-tuned model can flexibly handle soft grounding tasks in a variety of complex scenes based on text prompts, without relying on the traditional fixed visual level prior guidance.

[0057] 2. When solving the problem of text-guided matting, the present invention is not limited to simple text-guided matting (for example, there is only one prominent object in the picture, without multiple objects interfering with each other or objects with complex texture structures), but also has strong adaptability to complex scenes. Therefore, the present invention proposes an asymmetric knowledge distillation framework that fully decouples the two sub-problems of instance-level localization and transparency estimation. The teacher denoising network is used for semantic localization training, and then the student denoising network gradually refines the transparency estimation. Different from existing technologies based on visual priors or direct adaptation of StableDiffusion, this method significantly improves the matting accuracy in complex scenes (such as multi-object or complex transparency patterns). This high accuracy benefits from the design of phased learning objectives, effectively avoiding the complexity and stability problems of direct joint optimization.

[0058] 3. The present invention can effectively improve the model operation efficiency and fully reduce the time cost. The present invention makes full use of the accurate intermediate feature semantic supervision signals provided by the teacher denoising network, which can guide the training of other more lightweight student denoising networks to complete the entire soft grounding task. During inference, only these lightweight student denoising networks are needed to achieve text-guided matting, and the teacher denoising network is no longer required. Among these lightweight operations include pruning redundant network parameters and optimizing self-attention operations to improve the inference efficiency and running time. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0060] Figure 1 It is a schematic flowchart of the image matting method based on text prior and StableDiffusion model in the embodiments of the present invention.

[0061] Figure 2 It is a schematic framework diagram of the image matting method based on text prior and StableDiffusion model in the embodiments of the present invention.

[0062] Figure 3 It is a schematic structural diagram of the image matting system based on text prior and StableDiffusion model in the embodiments of the present invention.

[0063] Figure 4 It is a structural diagram of the computer storage medium in the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0064] To enable those skilled in the art to better understand the solution of this application, the following will clearly and completely describe the technical solution in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of this application.

[0065] In this application, the mention of "embodiment" means that the specific features, structures or characteristics described in combination with the embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described in this application can be combined with other embodiments.

[0066] Embodiment 1

[0067] First, the goal of the embodiment of this application is elaborated, that is: given the input image I ∈ R H×W×3 and the foreground text description T, make full use of the cross-modal prior of the Stable Diffusion (SD) model, and realize the prediction of the transparency mask α ∈ [0, 1] within a single time step through fine-tuning. H×W×1 . The process and framework of the method of this application are as Figure 1 、 2 shown. To fine-tune the SD model to the soft grounding task, the key idea of this application is to introduce an intermediate goal (i.e., semantic soft grounding) and decompose the soft grounding problem into two sub-goals. In the first sub-goal, the original diffusion model is taught to locate the semantic components of the target transparency mask; then the located semantic information is extracted from the resulting diffusion model, and at the same time, the transparent information is explored to complete the second sub-goal (i.e., transparency soft grounding). Therefore, this application proposes an asymmetric distillation framework and two specialized distillation goals to encourage the model to identify transparent details from the corresponding transparency masks while retaining the localization ability of the transparency masks learned in the first step; the strong semantic guidance in the first step of this framework can teach a more computationally friendly model to achieve the final soft grounding task and be applicable to more practical applications. Under the guidance of the soft grounding result and the output features of the well-trained diffusion model, the mask decoder can easily predict the final transparency mask.

[0068] Specifically, please refer to Figure 1 、 2 , in an embodiment of this application, an image matting method based on text prior and stable diffusion model is provided, including the following steps:

[0069] S1. Obtain the original image and its foreground description text, and use an image encoder and a text encoder to extract the image latent space representation and text features respectively; among them, both the image encoder and the text encoder are pre-trained and the parameters are frozen.

[0070] Specifically, in this application, the image encoder uses a variational autoencoder ε to compress the original image I into an image latent space representation z I : = ε(I); the text encoder uses a CLIP model (Contrastive Language–Image Pretraining) with frozen parameters after pre-training.

[0071] S2. Use a learnable text feature transformer to learn the text features to obtain language features.

[0072] Specifically, in this application, the learnable text feature transformer is a text adaptation module composed of two multi-layer perceptrons, which is used to further optimize the text features and extract language features.

[0073] S3. Use a knowledge distillation framework to train a student denoising network. The training process is as follows: input the image latent space representation and language features of the set image training set into the teacher denoising network for pre-training, and output a semantic soft grounding result; after the teacher denoising network is pre-trained, use the internal features and attention maps of the teacher denoising network as the supervision information for the student denoising network, perform distillation training on the student denoising network, and let the student denoising network perform transparency recognition on the input image latent space representation and text features of the set image training set to obtain a soft grounding result; among them, both the teacher denoising network and the student denoising network use the denoising network of the stable diffusion model; the soft grounding result includes a transparent confidence map and a rough transparency mask; after the training is completed, freeze the parameters of the trained student denoising network.

[0074] Furthermore, in this application, first fine-tune the SD model to locate each semantic component (foreground region, transparency region, background region) of the target transparency mask, that is: input the image latent space representation z I and the corresponding language features together into the denoising network of the diffusion model (a U-shaped neural network used to predict the noise at each time step in the diffusion algorithm, denoted as ∈), and let ∈ distinguish each semantic region (foreground, background, and transparent regions) of the transparency mask and predict the classification result of each pixel point, denoted as This function is called semantic soft grounding. To achieve this goal, first, the StableDiffusion model is modified by replacing the last latent prediction head in its denoising network with a new prediction head consisting of a two-group convolutional-batch normalization-activation function module (ConvBNReLU module). After achieving semantic soft grounding, it is necessary to explore transparency information based on semantic information, so there is also transparency soft grounding; to achieve the exploration of transparency information, this application proposes a soft grounding distillation framework, using the StableDiffusion model in the semantic soft grounding process as the teacher denoising network Using the StableDiffusion model in the transparency soft grounding process as the student denoising network Using the semantic knowledge distilled by the teacher denoising network to supervise the student denoising network to further identify the corresponding transparency details.

[0075] Furthermore, the teacher denoising network Is obtained by pre-training the denoising network of the StableDiffusion model; during the pre-training process, the classification probability of each channel in the semantic soft grounding result is learned by minimizing the cross-entropy loss, expressed as:

[0076]

[0077] where L CE Is the cross-entropy loss, Is the classification probability of the c-th channel in the semantic soft grounding result of the set image training set, Is the reference region representation of the c-th channel (such as foreground region, transparency region, background region) generated by applying image morphological operations to the reference transparency mask of the set image training set. In this embodiment, the kernel size of the morphological operation is 15.

[0078] Furthermore, the student denoising network Applies two optimization operations to the denoising network of the StableDiffusion model (the same as the teacher denoising network It is obtained by optimizing (with the same structure and parameters). On the one hand, pruning is performed on the redundant blocks of the denoising network of the StableDiffusion model. Some redundant blocks in the denoising network of the StableDiffusion model are directly pruned, thus obtaining a model with fewer parameters. Specifically, the denoising network includes three parts: an encoder, an intermediate block, and a decoder. In this embodiment, the pruning of redundant blocks refers to the solution proposed in the paper published by Kim B et al. (Kim, B.K., Song, H.K., Castells, T., & Choi, S. (2024, September). Bk-sdm: A lightweight, fast, and cheap version of StableDiffusion. In European Conference on Computer Vision (pp. 381 - 399).). The network layers numbered 2, 5, 8, 10, 11 in the encoder of the denoising network are deleted, the entire intermediate block in the denoising network is deleted, and the network layers numbered 0, 1, 4, 7, 10 in the decoder of the denoising network are deleted.

[0079] On the other hand, a large number of self-attention operations in the denoising network ∈ of the StableDiffusion model are key components for long-range dependence modeling. However, the matrix multiplication operation in self-attention has a high computational cost (resulting in an overall time complexity of O(n 2 d+nd 2 ))), especially in the case of higher resolutions. Nevertheless, the soft grounding task has relatively loose requirements for prediction accuracy because the soft grounding results (the rough transparency mask α lr and the transparent confidence map R) provide sufficient elastic tolerance space for the errors in the next transparency enhancement process. Based on this, this application proposes that the self-attention operation can be optimized by learning asymmetric sparse correspondence relationships to make it more computationally efficient. Specifically, this method believes that the dense affinity matrix given by the self-attention map A f can be simplified to unidirectional sparsity; that is, a smaller number of feature tokens can be learned as representatives, and the learned affinity relationships are used to replace the affinity relationships between the original feature tokens and other tokens in a relatively small area. Therefore, this application introduces a learnable downsampling operation φ, which is applied to the input feature f of self-attention to calculate the key embedding feature K and the value embedding feature V (K S =W K φ(f), V S =W V φ(f)), while the query embedding feature Q s =W Qf remains unchanged. Among them, the downsampling operation φ can be implemented by a convolutional layer with a kernel size and a stride both equal to k×k. In this embodiment, for the self-attention operations with resolutions of 16×16 and 32×32, let k 16×16 = k 32×32 = 2, and the self-attention operations with other resolutions are directly deleted.

[0080] Furthermore, after the teacher denoising network converges during pre-training, the student denoising network is subjected to distillation training to explore transparency information. The student denoising network uses the internal features and attention maps of the teacher denoising network as supervision information, and conducts distillation training based on the powerful semantic guidance provided by the teacher denoising network . During the distillation training process, the parameters of the teacher denoising network are frozen.

[0081] After distillation training, the student denoising network is expected to have good ability to solve the soft grounding problem. In implementation, the student denoising network is learned to predict a rough transparency map and a transparent confidence map which indicates the regions that may be transparent, so as to be further upsampled and enhance α lr to α. Next, this method introduces two key losses to better promote this distillation process; during the distillation training process, feature-level distillation loss and transparency mining loss are used for optimization learning; among them, the feature-level distillation loss constrains the semantic representation of the student denoising network to be consistent with that of the teacher denoising network , expressed as:

[0082]

[0083] where, L SG is the feature-level distillation loss, represents the intermediate feature output by the teacher denoising network at the l-th layer, represents the intermediate feature output by the student denoising network at the l-th layer, z I is the image latent space representation of the set image training set, f T is the language feature of the set image training set; M is a binary mask, with the foreground region and the background region set to 1 and other regions set to 0; ⊙ is the Hadamard product; is the squared L2 norm function; Φ is a projection module used to align the intermediate features of the teacher denoising network and the student denoising network; considering that due to different learning objectives, and There are differences in the feature space between them. This module consists of three learnable convolutional layers and an intermediate LeakyReLU layer, and the formula is:

[0084] Φ(F) = Cv 3×3 (Cv 3×3 (LeakyReLU(Cv 1×1 (F)))),

[0085] where F is the input feature, and Cv 3×3 is a learnable 3×3 convolutional layer, and Cv 1×1 is a learnable 1×1 convolutional layer.

[0086] Another loss is the transparency mining loss. To further encourage to explore more transparency details, this application continues to mine and discover potential transparency information from . Starting from the self-attention maps of the teacher denoising network , which is a key component for modeling the internal consistency of the object; since has learned the semantic knowledge of the transparency channel, the implicit transparency information used to identify these regions can also be reflected in the self-attention maps. Therefore, this application believes that the self-attention maps can provide transparency clues for to further explore transparency details. Therefore, given the self-attention map of the feature f This application introduces a self-attention-based transparency mining loss L STM , to only constrain the affinity related to the transparency mask. To achieve this, this application propagates the reference transparency mask α in all self-attention layers with the same resolution through matrix multiplication GT , and minimizes the difference between the propagation results of the teacher denoising network and the student denoising network. The principle behind this is that A SA can be regarded as a transition matrix, and α GT can be regarded as the current state. Their matrix multiplication transfers the current state to the result state. By constraining the result state, the affinity related to α in the self-attention map can be indirectly constrained only, which further helps to explore transparency details and avoids the significant computational cost of constraining the entire self-attention map. The transparency mining loss L STM is expressed as:

[0087]

[0088] where L STM is the transparency mining loss, is the average value of the self-attention maps at different resolutions output by the teacher denoising network, is the average value of the self-attention maps at different resolutions output by the student denoising network, represents matrix multiplication rather than element-wise multiplication, α GT is the state matrix obtained by directly downsampling the benchmark transparency mask of the set image training set through bilinear interpolation to the resolution corresponding to the self-attention map to meet the requirements of matrix multiplication.

[0089] Furthermore, for the distillation training process of the student denoising network, the present application also uses the cross-attention map distillation loss to constrain the cross-attention maps between the teacher denoising network and the student denoising network, uses the first norm loss to learn the rough transparency mask, and uses the binary cross-entropy loss to learn the transparency confidence map.

[0090] Among them, the cross-attention map distillation loss mines potential transparency clues from the text expression, and this loss directly constrains and the cross-attention maps between them, expressed as:

[0091]

[0092] Among them, L CTM is the cross-attention map distillation loss, is the average value of the cross-attention maps at different resolutions output by the teacher denoising network, is the average value of the cross-attention maps at different resolutions output by the student denoising network; in implementation, all cross-attention maps are grouped by resolution and averaged to calculate the loss, and then the average value of the losses in each resolution is taken as the final attention distillation loss.

[0093] The first norm loss (L1 loss) is used to learn the rough transparency mask α lr , expressed as:

[0094]

[0095] Among them, α lr is the rough transparency mask in the soft grounding result of the set image training set, is the true transparency mask obtained by downsampling the benchmark transparency mask of the set image training set, and ‖‖1 is the L1 norm;

[0096] The binary cross loss (BCE loss) is used to learn the transparency confidence map R, expressed as:

[0097]

[0098] Among them, R is the transparent confidence map in the soft grounding result of the set image training set, and e R is the exponential of all values in R with the natural constant as the base, is the binary mask of the transparent region;

[0099] Thus, the student denoising network The total loss function in the distillation training process is expressed as:

[0100]

[0101] Among them, λ STM 、λ CTM 、λ SG 、 λ R are the hyperparameters corresponding to the losses respectively. In this embodiment, the hyperparameters of the student denoising network are set to

[0102] During the distillation training process, 's L SG actually provides a very strong supervision signal in terms of localization knowledge retention, because it directly penalizes the internal feature differences in semantics. It is believed that these strong supervision signals can be fully utilized to teach the student denoising network. The strong supervision signal provided at the internal feature level can quickly enable the student denoising network to learn soft semantic localization knowledge, which is very difficult to achieve in an independent end-to-end training strategy. When the student denoising network converges, its parameters can be fixed for obtaining the soft grounding result of the input image.

[0103] S4. Input the image latent space representation and language features of the original image into the trained student denoising network to obtain a soft grounding result, upsample it to the same resolution as the original image and splice it with the original image, then input it into the detail encoder to obtain a fused feature; obtain the matting context feature from the output of the last layer of the trained student denoising network, apply a group of learnable convolutional blocks to align the feature space and then splice it with the fused feature, and input it into the matting decoder to obtain the predicted transparency mask; the learnable convolutional blocks, detail encoder, and matting encoder are pre-trained and their parameters are frozen.

[0104] Specifically, in order to upsample the rough transparency mask α lr from to the transparency mask of H×W, this application adopts a detail encoder-decoder structure. First, the detail encoder takes the soft grounding result spliced with the original image as the input to extract detail features. Before that, the rough transparency mask α in the soft grounding result lrThe transparency confidence map R is directly upsampled to the same resolution as the original image I, and the transparency confidence map R is transformed by a logistic function (i.e., the Sigmoid function); the detail features of the last layer of the detail encoder are concatenated with the features extracted from and then input into the matte decoder to predict the final transparency mask α.

[0105] Specifically, the detail encoder and the matte decoder are implemented using a common encoder-decoder architecture, both of which include four convolutional neural network layers; each convolutional neural network layer includes three operations: convolutional calculation-batch normalization-ReLU activation function; among them, the detail encoder has a downsampling operation with a set magnification between convolutional neural network layers; similarly, the matte decoder has an upsampling operation with a set magnification between convolutional neural network layers; at the same time, the detail encoder and the matte decoder have skip connections at the same resolution; in addition, the batch normalization and ReLU of the last layer of the matte decoder are removed, and the prediction result is directly output after convolutional calculation. In this embodiment, the set magnification is 2.

[0106] More specifically, the detail decoder and the matte decoder are supervised and trained using the first norm loss and the Laplacian loss; for more efficient training, the parameters of the student denoising network are frozen during the process, and only the parameters of the detail encoder and the matte decoder are trained, and a set of learnable convolutional blocks are applied to the output features of to align the feature space before concatenation.

[0107] It should be noted that for the foregoing method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously.

[0108] Based on the same idea as the image matte extraction method based on text prior and stable diffusion model in the above embodiments, the present invention also provides an image matte extraction system based on text prior and stable diffusion model, which can be used to execute the above image matte extraction method based on text prior and stable diffusion model. For the convenience of description, in the structural schematic diagram of the image matte extraction system embodiment based on text prior and stable diffusion model, only the parts related to the embodiments of the present invention are shown. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than illustrated, or combine some components, or different component arrangements.

[0109] Please refer to Figure 3, in another embodiment of the present application, an image matting system based on text prior and stable diffusion model is provided. The system includes a feature representation module, a semantic acquisition module, a soft grounding implementation module, and a soft grounding processing module;

[0110] Among them, the feature representation is used to obtain the original image and its foreground description text, and an image encoder and a text encoder are used to extract the image latent space representation and text features respectively; both the image encoder and the text encoder are pre-trained and their parameters are frozen;

[0111] The semantic acquisition module is used to learn the text features using a learnable text feature transformer to obtain language features;

[0112] The soft grounding implementation module is used to train a student denoising network using a knowledge distillation framework. The training process is as follows: The image latent space representation and language features of the set image training set are sent into the teacher denoising network for pre-training, and a semantic soft grounding result is output; after the teacher denoising network is pre-trained, the internal features and attention maps of the teacher denoising network are used as the supervision information for the student denoising network to perform distillation training on the student denoising network, and let the student denoising network perform transparency recognition on the image latent space representation and text features of the input set image training set to obtain a soft grounding result; both the teacher denoising network and the student denoising network use the denoising network of the stable diffusion model; the soft grounding result includes a transparent confidence map and a rough transparency mask; after the training is completed, the parameters of the trained student denoising network are frozen;

[0113] The soft grounding processing module is used to input the image latent space representation and language features of the original image into the trained student denoising network to obtain a soft grounding result, up-sample it to the same resolution as the original image and splice it with the original image and input it into the detail encoder to obtain a fused feature; obtain the matting context feature from the output of the last layer of the trained student denoising network, apply a set of learnable convolutional blocks to align the feature space and then splice it with the fused feature, and input it into the matting decoder to obtain a predicted transparency mask; the learnable convolutional blocks, the detail encoder, and the matting encoder are pre-trained and their parameters are frozen.

[0114] It should be noted that the image matting system based on text prior and stable diffusion model of the present invention corresponds one-to-one with the image matting method based on text prior and stable diffusion model of the present invention. The technical features and their beneficial effects described in the embodiments of the above image matting method based on text prior and stable diffusion model are applicable to the embodiments of the image matting based on text prior and stable diffusion model. For specific content, reference can be made to the description in the method embodiments of the present invention, which will not be repeated here. This is hereby declared.

[0115] In addition, in the implementation manner of the image matting system based on text prior and stable diffusion model in the above embodiments, the logical division of each program module is only an example. In practical applications, according to needs, for example, considering the configuration requirements of the corresponding hardware or the convenience of software implementation, the above functions can be assigned to different program modules to complete, that is, the internal structure of the image matting system based on text prior and stable diffusion model is divided into different program modules to complete all or part of the functions described above.

[0116] Please refer to Figure 4 , in one embodiment, a computer-readable storage medium is provided, in which a program is stored. When the program is executed by at least one processor, an image matting method based on text prior and stable diffusion model is implemented, specifically as follows:

[0117] Obtain the original image and its foreground description text, and use an image encoder and a text encoder to extract the image latent space representation and text features respectively; among them, both the image encoder and the text encoder are pre-trained and the parameters are frozen;

[0118] Apply a learnable text feature transformer to learn the text features to obtain language features;

[0119] Adopt a knowledge distillation framework to train a student denoising network. The training process is as follows: input the image latent space representation and language features of the set image training set into the teacher denoising network for pre-training, and output a semantic soft grounding result; after the pre-training of the teacher denoising network is completed, use the internal features and attention maps of the teacher denoising network as the supervision information for the student denoising network, perform distillation training on the student denoising network, and let the student denoising network perform transparency recognition on the input image latent space representation and text features of the set image training set to obtain a soft grounding result; both the teacher denoising network and the student denoising network adopt the denoising network of the stable diffusion model; the soft grounding result includes a transparent confidence map and a rough transparency mask; after the training is completed, freeze the parameters of the trained student denoising network;

[0120] Input the image latent space representation and language features of the original image into the trained student denoising network to obtain a soft grounding result, up-sample it to the same resolution as the original image and splice it with the original image, then input it into the detail encoder to obtain a fused feature; obtain the matting context feature from the output of the last layer of the trained student denoising network, apply a set of learnable convolutional blocks to align the feature spaces and then splice them with the fused feature, and input them into the matting decoder to obtain the predicted transparency mask; the learnable convolutional blocks, the detail encoder, and the matting encoder are pre-trained and the parameters are frozen.

[0121] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in this application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0122] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0123] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention should be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. An image matting method based on text prior and stable diffusion model, characterized in that It includes the following steps: Obtain the original image and its foreground description text, and use an image encoder and a text encoder to extract the image latent space representation and text features respectively; both the image encoder and the text encoder are pre-trained and their parameters are frozen; Apply a learnable text feature transformer to learn the text features to obtain language features; Use a knowledge distillation framework to train a student denoising network. The training process is as follows: Feed the image latent space representation and language features of the set image training set into the teacher denoising network for pre-training to output a semantic soft grounding result; after the pre-training of the teacher denoising network is completed, use the internal features and attention maps of the teacher denoising network as the supervision information for the student denoising network, perform distillation training on the student denoising network, and let the student denoising network perform transparency recognition on the input image latent space representation and text features of the set image training set to obtain a soft grounding result; both the teacher denoising network and the student denoising network adopt the denoising network of the stable diffusion model; the soft grounding result includes a transparent confidence map and a rough transparency mask; after the training is completed, freeze the parameters of the trained student denoising network; Input the image latent space representation and language features of the original image into the trained student denoising network to obtain a soft grounding result, up-sample it to the same resolution as the original image and splice it with the original image and input it into the detail encoder to obtain a fused feature; Obtain the matte context feature from the output of the last layer of the trained student denoising network, apply a set of learnable convolutional blocks to align the feature space and then splice it with the fused feature, and input it into the matte decoder to obtain the predicted transparency mask; the learnable convolutional blocks, detail encoder and matte encoder are pre-trained and their parameters are frozen.

2. The image matting method based on text prior and stable diffusion model according to claim 1, wherein The image encoder adopts a variational autoencoder; the text encoder adopts a CLIP model that is pre-trained and has its parameters frozen; the text feature transformer consists of two multi-layer perceptrons.

3. The image matting method based on text prior and Stable Diffusion model according to claim 1, wherein The last latent prediction head in the denoising network of the stable diffusion model is replaced by a new prediction head composed of a two-group convolution-batch normalization-activation function module.

4. The image matting method based on text prior and stable diffusion model according to claim 3, wherein During the pre-training process of the teacher denoising network, learn the classification probability of each channel in the semantic soft grounding result by minimizing the cross-entropy loss, expressed as: Among them, L CE is the cross-entropy loss, is the classification probability of the c-th channel in the semantic soft grounding result of the set image training set, is the reference region representation of the c-th channel generated by applying image morphological operations to the reference transparency mask of the set image training set.

5. The image matting method based on text prior and stable diffusion model according to claim 3, wherein, The student denoising network is obtained by optimizing the denoising network of the stable diffusion model. The optimization process is as follows: Prune the redundant blocks of the denoising network of the stable diffusion model; Introduce a learnable down-sampling operation to perform self-attention optimization on the self-attention operation in the denoising network of the stable diffusion model. Use the down-sampling operation to calculate the key embedding feature and value embedding feature in the self-attention operation, and keep the query embedding feature unchanged; the down-sampling operation is implemented by a convolutional layer, and its kernel size and stride are both equal to k×k.

6. The image matting method based on text prior and Stable Diffusion model according to claim 5, characterized in that, During the distillation training process of the student denoising network, the parameters of the teacher denoising network are frozen; During the distillation training process, use the feature-level distillation loss and the transparency mining loss for optimization learning; The feature-level distillation loss is expressed as: Among them, L SG is the feature-level distillation loss, represents the intermediate feature output by the teacher denoising network at the l-th layer, represents the intermediate feature output by the student denoising network at the l-th layer, z I is the image latent space representation of the set image training set, f T is the language feature of the set image training set; M is a binary mask, with the foreground and background regions set to 1 and other regions set to 0; ⊙ is the Hadamard product; Φ is a projection module for aligning the intermediate features of the teacher denoising network and the student denoising network; is the squared L2 norm function; The transparency mining loss is expressed as: Among them, L STM is the transparency mining loss, is the average value of the self-attention maps at different resolutions output by the teacher denoising network, is the average value of the self-attention maps at different resolutions output by the student denoising network, represents matrix multiplication, and α GT is the state matrix obtained by directly downsampling the benchmark transparency mask of the set image training set through bilinear interpolation to the resolution corresponding to the self-attention map; The projection module consists of three learnable convolutional layers and an intermediate LeakyReLU layer, and the formula is: Φ(F) = Cv 3×3 (Cv 3×3 (LeakyReLU(Cv 1×1 (F)))), Among them, F is the input feature, and Cv 3×3 is a learnable 3×3 convolutional layer, and Cv 1×1 is a learnable 1×1 convolutional layer.

7. The image matting method based on text prior and Stable Diffusion model according to claim 6, wherein During the distillation training process of the student denoising network, the cross-attention map distillation loss is also used to constrain the cross-attention map between the teacher denoising network and the student denoising network, the first norm loss is used to learn the rough transparency mask, and the binary cross-entropy loss is used to learn the transparent confidence map; The cross-attention map distillation loss is expressed as: Among them, L CTM is the cross-attention map distillation loss, is the average value of the cross-attention maps at different resolutions output by the teacher denoising network, is the average value of the cross-attention maps at different resolutions output by the student denoising network; The first norm loss is expressed as: Among them, α lr is the rough transparency mask in the soft grounding result of the set image training set, is the true transparency mask obtained by downsampling the reference transparency mask of the set image training set, and ‖‖1 is the L1 norm; The binary cross-loss is expressed as: where R is the transparency confidence map in the soft-grounding result of the set of predefined image training, and e R is the exponential function with the base of the natural constant for all values in R, is the binary mask of the transparent region; The total loss function during the distillation training process of the student denoising network is expressed as: Among them, λ STM , λ CTM , λ SG , λ R are respectively the hyperparameters corresponding to the losses.

8. The image matting method based on text prior and stable diffusion model according to claim 1, wherein The detail encoder and the matte decoder are implemented using the encoder-decoder architecture, and both contain four convolutional neural network layers; each convolutional neural network layer includes three operations: convolutional calculation-batch normalization-ReLU activation function; The detail encoder has a downsampling operation with a set magnification between convolutional neural network layers; The matte decoder has an upsampling operation with a set magnification between convolutional neural network layers; The detail encoder and the matte decoder have skip connections at the same resolution; The batch normalization and ReLU of the last layer of the matte decoder are removed, and the prediction result is directly output after convolutional calculation; The detail decoder and the matte decoder are supervised and trained using the first norm loss and the Laplacian loss.

9. An image matting system based on text prior and stable diffusion model, characterized in that, Applied to the image matte extraction method based on text prior and stable diffusion model described in any one of claims 1-8, including a feature representation module, a semantic acquisition module, a soft grounding implementation module, and a soft grounding processing module; The feature representation is used to obtain the original image and its foreground description text, and the image encoder and the text encoder are used to extract the image latent space representation and text features respectively; both the image encoder and the text encoder are pre-trained and the parameters are frozen; The semantic acquisition module is used to learn the text features using a learnable text feature transformer to obtain language features; The soft grounding implementation module is used to train the student denoising network using a knowledge distillation framework. The training process is as follows: The image latent space representation and language features of the set image training set are fed into the teacher denoising network for pre-training, and the semantic soft grounding result is output; after the pre-training of the teacher denoising network is completed, the internal features and attention map of the teacher denoising network are used as the supervision information for the student denoising network to perform distillation training on the student denoising network, and the student denoising network is made to perform transparency recognition on the input image latent space representation and text features of the set image training set to obtain the soft grounding result; both the teacher denoising network and the student denoising network use the denoising network of the stable diffusion model; the soft grounding result includes a transparent confidence map and a rough transparency mask; after the training is completed, the parameters of the trained student denoising network are frozen; The soft grounding processing module is used to input the image latent space representation and language features of the original image into the trained student denoising network to obtain the soft grounding result, upsample it to the same resolution as the original image and splice it with the original image and input it into the detail encoder to obtain the fused feature; Obtain the matting context features from the output of the last layer of the trained student denoising network, apply a set of learnable convolutional blocks to align the feature spaces, and then concatenate them with the fused features, and input them into the matting decoder to obtain the predicted transparency mask; the learnable convolutional blocks, the detail encoder, and the matting encoder are pre-trained and their parameters are frozen.

10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by a processor, it implements the image matting method based on text prior and stable diffusion model according to any one of claims 1-8.

Images