Natural image matting method and system based on text and boundary information aggregation

Abstract

The invention provides a natural image matting method and system based on text and boundary information aggregation, and relates to the technical field of image processing. Splicing the original color image and the corresponding ternary image, extracting initial features, and obtaining enhanced fusion features through multi-scale Laplacian high-frequency extraction and cosine similarity weighted fusion; based on the enhanced fusion feature and the ternary image, generating a gated ternary fusion feature fused with priori knowledge through a trans-attention mechanism; global coding modeling is carried out on the gated three-value fusion features to obtain deep features; text prompt and multi-scale boundary information are introduced based on deep features, adaptive up-sampling is guided through a cross-attention mechanism, semantic difference consistency constraint is adopted between decoding layers, consistency constraint is implemented from pixel appearance, high-level semantics and color component dimensions, and finally a transparency image is output through a prediction header to obtain an image matting result. And the fidelity and the boundary accuracy of high-frequency details in a matting result are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a natural image matting method and system based on the aggregation of text and boundary information. Background Technology

[0002] The core of image matting is to accurately estimate the transparency value of the foreground from a complex background. (The range of values ​​is [0,1]), and its commonly used observation model can be expressed as follows: ,in It is the input image. , They represent the foreground and background, respectively. But in natural scenes... and The inability to observe directly leads to the same It may correspond to multiple groups ( This solution makes the image cutout problem unpredictable, especially in semi-transparent or highly textured areas such as hair and gauze, where the difference between foreground and background details is subtle, making it more prone to problems such as boundary drift and artifacts.

[0003] This lack of certainty also presents many limitations to existing image matting methods: traditional sampling methods rely on the separability of foreground / background colors and textures, extrapolating unknown areas from known areas. If the colors of the foreground and background are similar and the lighting is complex, incorrect matching and boundary artifacts are likely to occur. Propagation / optimization methods spread information through prior knowledge, but they over-smooth in scenes with abrupt texture changes, erasing real high-frequency details and causing edge blurring.

[0004] While deep learning methods have made progress, they still have shortcomings: encoder-decoder convolutional networks are limited by receptive field and downsampling loss, making it difficult to balance long-range dependencies and small boundaries, and skip connections are prone to introducing background texture copy artifacts; although Transformer enhances global modeling, it dilutes local high-frequency responses and has high computational cost; at the same time, existing interpolation strategies in the decoding stage lack boundary priors, making it difficult to distinguish between the high-frequency components of real boundaries and background interference. Coupled with the domain offset problem of training on synthetic data, the performance metrics and subjective quality of the model will decline after deployment, making it particularly difficult to meet the comprehensive requirements of weak trima, real-time processing, and complex boundary fidelity in industrial scenarios. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a natural image matting method and system based on the aggregation of text and boundary information. This method effectively improves the fidelity of high-frequency details and the accuracy of boundaries in the matting results, and achieves clear separation of multiple instances in complex scenes.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a natural image matting method based on the aggregation of text and boundary information, comprising: The original color image and the corresponding ternary image are stitched together and the initial features are extracted. Enhanced fused features are obtained by multi-scale Laplacian high-frequency extraction and cosine similarity weighted fusion. The enhanced fusion features are projected as keys and values, and the ternary graph is mapped as a query vector. Gated ternary fusion features that fuse prior knowledge are generated through a cross-attention mechanism. The gated ternary fusion features are then globally encoded and modeled to obtain deep features. In the decoding stage, text prompts and multi-scale boundary information are introduced based on deep features. Adaptive upsampling is guided by a cross-attention mechanism. At the same time, semantic differential consistency constraints are adopted between each decoding layer. Consistency constraints are implemented from the dimensions of pixel appearance, high-level semantics and color components to obtain color semantic fusion features. The color semantic fusion features are used to output a transparency map through a prediction head to obtain a matted image of the original color image that meets predefined requirements for high-frequency detail fidelity and boundary accuracy.

[0007] Secondly, the present invention provides a natural image matting system based on the aggregation of text and boundary information, comprising: The feature extraction unit is used to stitch the original color image with the corresponding ternary image and extract the initial features. Enhanced fused features are obtained by multi-scale Laplacian high-frequency extraction and cosine similarity weighted fusion. The prior fusion unit projects the enhanced fusion features into keys and values, maps the ternary graph into a query vector, and generates gated ternary fusion features that fuse prior knowledge through a cross-attention mechanism. The gated ternary fusion features are then globally encoded and modeled to obtain deep features. The decoding constraint unit is used to introduce text prompts and multi-scale boundary information based on deep features during the decoding stage, guide adaptive upsampling through cross-attention mechanism, and adopt semantic differential consistency constraints between each decoding layer to implement consistency constraints from the dimensions of pixel appearance, high-level semantics and color components to obtain color semantic fusion features. The result output unit is used to output a transparency map through the color semantic fusion feature via the prediction head, so as to obtain a matted image of the original color image whose high-frequency detail fidelity and boundary accuracy meet the predefined requirements.

[0008] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the natural image matting method based on the aggregation of text and boundary information described in the first aspect.

[0009] Fourthly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the natural image matting method based on text and boundary information aggregation described in the first aspect.

[0010] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention first stitches together the original image and a ternary image to extract initial features. These features are then enhanced through multi-scale Laplacian high-frequency extraction and cosine similarity weighted fusion, strengthening details such as edges and textures while maintaining spatial consistency. Next, a cross-attention mechanism is used to fuse prior knowledge from the ternary image, generating gated ternary fusion features. Global encoding yields semantically rich deep features. In the decoding stage, textual prompts and multi-scale boundary information guide adaptive upsampling. Combined with semantic difference consistency constraints, feature coherence is ensured across pixel, semantic, and color dimensions, effectively avoiding boundary blurring, detail loss, and color drift. The final output transparency map accurately separates the foreground and background, making it particularly suitable for complex scenes such as hair and thin fabrics. This significantly improves the accuracy and stability of image matting, balancing detail fidelity and overall coherence.

[0011] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0012] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof.

[0013] Figure 1 This is a flowchart illustrating the main steps of a natural image matting method based on the aggregation of text and boundary information, as provided in an embodiment of the present invention. Figure 2 A flowchart illustrating a natural image matting method based on the aggregation of text and boundary information, provided in an embodiment of the present invention; Figure 3 A schematic diagram of the Laplace-cosine similarity enhancement module provided in an embodiment of the present invention; Figure 4 A schematic diagram of the structure of the Text-Boundary Guided Interpolation Module (T-EIB) provided in an embodiment of the present invention; Figure 5 A schematic diagram of the Semantic Differential Consistency Module (SDB) provided in an embodiment of the present invention; Figure 6 Example diagram of qualitative results provided for embodiments of the present invention. Detailed Implementation

[0014] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0015] Example 1 like Figure 1 As shown, this embodiment discloses a natural image matting method based on the aggregation of text and boundary information, including the following steps: S1: The original color image and the corresponding ternary image are stitched together and the initial features are extracted. Enhanced fusion features are obtained by multi-scale Laplacian high-frequency extraction and cosine similarity weighted fusion. S2: Project the enhanced fusion features as keys and values, map the ternary graph as a query vector, and generate gated ternary fusion features that fuse prior knowledge through a cross-attention mechanism; perform global encoding modeling on the gated ternary fusion features to obtain deep features; S3: In the decoding stage, text prompts and multi-scale boundary information are introduced based on deep features. Adaptive upsampling is guided through a cross-attention mechanism. At the same time, semantic differential consistency constraints are adopted between each decoding layer. Consistency constraints are implemented from the dimensions of pixel appearance, high-level semantics and color components to obtain color semantic fusion features. S4: The color semantic fusion features are output as a transparency map through the prediction head to obtain a matted image of the original color image whose high-frequency detail fidelity and boundary accuracy meet the predefined requirements.

[0016] Next, combined Figure 2 This embodiment provides a detailed description of a natural image matting method based on the aggregation of text and boundary information.

[0017] (a) Data Visualization Obtain the original color image and its corresponding ternary image provided by the user. The original image is the original color image, such as a common RGB format color image; the ternary image is the image obtained after processing the original color image, which has only three pixel values.

[0018] In this embodiment, the ternary image serves as prior knowledge, pre-classifying the pixels of an original image into three categories: foreground, background, and uncertain regions. By pre-dividing the pixels into three parts, the model's classification performance of pixels in the original image can be improved.

[0019] The original color image and its corresponding ternary image are first normalized to [0,1], then... This serves as the computational basis for the observation model. In this observation model, F represents the foreground and B represents the background. I represents the model's prediction result, and I represents the synthesized image; the model performs a process on an image. Value prediction can utilize foreground information, background information, and It can be used to synthesize any image.

[0020] (II) Three-valued graph information injection strategy In the input stage of the model, the ternary image Tri and the original image X are fused using the proposed Trimap Information Injection (TII) strategy.

[0021] First, Tri and X are concatenated by channels. Preliminary feature results are obtained through an encoder, a LapCos-LapCos similarity enhancement module, and channel mapping. and .

[0022] 1. Encoder Based on a CNN, initial features are extracted from the concatenated Tri and X. .

[0023] 2. Laplace-Cosine Similarity Enhancement Module Then, the initial features The input LapCos-cosine similarity enhancement block normalizes and converges the high-frequency responses obtained from the adjacent differences of the multi-scale Gaussian pyramid, and assigns weights to directionally consistent regions with channel cosine similarity, so that the features entering the global encoder (Transformer) have both rich detail and spatial consistency, thus alleviating the dilution of edges and textures caused by long-range dependency modeling.

[0024] The Laplacian-cosine similarity enhancement block utilizes a two-step approach—multi-scale Gaussian pyramids and channel cosine similarity—to ensure global attention integrates long-range dependencies without compromising high-frequency details. The core idea of ​​the Laplacian-cosine similarity enhancement block is to perform position-wise weighting of the cosine similarity between the "high-frequency features extracted by multi-scale Laplacian convolution" and the "current initial features," thereby amplifying local details such as edges and textures without disrupting spatial consistency, and feeding these richer representations into subsequent global self-attention.

[0025] The process involves first extracting high-frequency responses from the input features using multi-scale Laplacian convolution kernels to form a multi-scale Laplacian feature matrix. Then, the cosine similarity between this matrix and the input features in the channel dimension is calculated to obtain a similarity weight map. Finally, this weight is used to reweight the original input features to strengthen the representation of regions with consistent structural patterns (such as boundaries and subtextures), thereby improving detail fidelity and spatial coherence and providing sufficient detailed input for the subsequent Transformer.

[0026] Specifically, such as Figure 3 As shown, the Laplace-cosine similarity enhancement module includes two branches: the Laplace branch and the cosine similarity branch.

[0027] Laplace branch with initial features After being used as input, it is passed through three multi-scale LapK branches. i (i=1, 2, 3), Gaussian pyramids and Laplacian convolutions are applied to the features at different resolution scales to obtain three multi-scale high-frequency features. (s=1, 2, 3). Next, channel normalization is performed for each scale to obtain... (s=1, 2, 3).

[0028] Next, the outputs of adjacent Laplacian branches are differentially processed (subtracted pixel by pixel and channel by channel) to obtain two difference maps. and Then, the two are added together according to pixels to obtain the Laplacian feature matrix. .

[0029] Cosine similarity branch with initial features For input, first expand the operation pair through a local window. Perform image patch reshaping, rearranging each spatial location... The neighborhood is flattened into local patch vectors and then L2 normalized along the vector dimension.

[0030] Subsequently, to achieve weighted modeling of the correlation between local patches and templates in different directions, a model of size [size missing] was created. We use an empty weight tensor and initialize the weights by applying a normal distribution to it using Xavier initialization. Then, we flatten the convolutional kernel weights corresponding to each output channel in the spatial dimension to a length of... The weight vector is registered as a learnable parameter, which is then adaptively updated based on the loss gradient during training.

[0031] Next, for each spatial location, the response is calculated one by one using the normalized local patch vector and the normalized convolution kernel weight vector of each output channel through an Einstein-style vector inner product operation (einsum), resulting in a position-by-position cosine similarity map arranged in the channel-space dimension. The combination of the convolution kernel weight vector and the local patch vector jointly determines the weight distribution and directional selectivity of this similarity map. Based on this, the sign of the cosine similarity map is extracted using a sign function, and then a monotonically nonlinear transformation consisting of a sigmoid and power-law mapping is applied to its absolute value to amplify or suppress different response intensities, thus forming a weighted coefficient map that assigns greater weight to high-similarity regions and less weight to low-similarity regions.

[0032] Finally, compare the results with... Enhanced features are obtained by combining them through residual connections. .

[0033] Enhance features Map the vectors to three vectors, Q, K, and V, and use a self-attention mechanism to obtain a cosine similarity attention map. This ensures that subsequent global self-attention does not dilute local high-frequency details when integrating long-range dependencies.

[0034] For the output of the two branches and The initial output of LapCos is obtained by fusing the features of the two sources through channel addition. After that, and Multiplying and then adding the results allows LapCos, as a pre-block, to enhance high-frequency extraction and maintain spatial consistency through the combination of multi-scale Laplacian and cosine similarity matrices, mitigating edge rounding and detail loss issues. This ultimately yields the enhanced fusion features from the LapCos output. .

[0035] In this embodiment, high-frequency features are extracted by multi-scale Laplacian and weighted by cosine similarity, providing the encoding end with a detailed and structurally coherent prior representation, effectively reducing the loss of high-frequency information during the encoding and upsampling process, and balancing the preservation of local details with overall spatial consistency.

[0036] 3. Channel Mapping Output features of the LapCos module Perform a projection operation to obtain K and V. Specifically, first... By using patching and flattening operations The representation is divided into several non-overlapping patches, and each patch is expanded into a vector representation. Then, the vector representations are projected onto a unified representation space using a learnable embedding map (emb). Based on this, two independent linear projections (implemented by fully connected layers) are used to map the representation space, ultimately yielding vector representations K and V.

[0037] Meanwhile, Tri is used to reduce the feature map size through adaptive pooling, and the pooling result is mapped to a query vector using channel mapping. By employing a cross-attention approach to compensate for the semantic confusion caused by direct concatenation, the prior information in Tri (foreground / unknown / background, etc.) is robustly and deeply fused with the original image X to obtain prior fusion features. .

[0038] Among them, the cross-attention fusion method first obtains the query vector and The transpose of the result is multiplied, then aligned and concatenated, activated using the Softmax function, and then multiplied by the result. Multiply by, and you get The calculation process is as follows:

[0039]

[0040]

[0041]

[0042] in, Indicates 3 3. Convolution operation, This indicates a mapping. This indicates that the overall result is scaled by the square root of the input dimension.

[0043] Furthermore, to avoid directly incorporating Tri's noise into the backbone, the initial prior fusion features are obtained... Then, the initial injection result is defined as follows: Where P represents and The difference terms between them. Then define the gating function. We obtain a scalar or channel gate, where MLP represents a multilayer perceptron consisting of three fully connected layers, and gap represents the global average pooling operation. Then let... The final gated ternary fusion feature output by the ternary image information injection strategy is obtained. .

[0044] In this embodiment, the ternary map is injected across attention rather than directly concatenated, which effectively suppresses the interference of low-quality / noise on feature extraction and improves the extraction of unknown regions. Stability estimation: Based on the ternary graph information injection strategy, semantic aliasing caused by early splicing is significantly reduced, while the network's focus is shifted to unknown regions and sidebands, improving the stability and accuracy of subsequent boundary and detail recovery.

[0045] (III) Decoder In obtaining gating ternary fusion features Then, deep features are obtained by passing them through Transformer Blocks. The final prediction result is obtained through subsequent decoding stages. Currently, conventional decoding mostly relies on fixed upsampling or convolution to achieve prediction. The above process is based on spatial proximity for reconstruction, which easily leads to problems such as blurred boundaries, smoothed details, and adhesion of similar small structures when high-frequency details and semi-transparent areas such as hair, thin gauze, and glass edges are included. On the other hand, although global encoding integrates long-range dependencies, its emphasis on large-scale relationships further dilutes local high-frequency responses, resulting in a lack of clear structural guidance for boundaries during decoding and reconstruction. Furthermore, real-world scenarios often have multiple instances of the same type, making it impossible to dynamically select targets based on saliency. Relying solely on image features and ternary map priors makes it difficult to achieve controllable separation during the reconstruction stage.

[0046] To address the aforementioned issues, this embodiment, in addition to using convolution for feature extraction at the decoding end, introduces a text-boundary guided interpolation module (T-EIB) and the traditional convolution process. This approach uses boundary saliency as a spatial prior and "textual cues" as a semantic prior to guide the target class and orientation. Specifically, after the features are processed by the first convolutional layer, they are input into the first T-EIB for the first round of text-boundary guided interpolation enhancement. Then, the second convolutional layer further extracts features from this enhanced data, which are then input into the second T-EIB for the second round of enhancement. Both modules jointly generate adaptive interpolation weights during upsampling through cross-attention and gating residuals, thereby restoring resolution while suppressing blurring and ringing caused by fixed interpolation, avoiding confusion of similar instances, and achieving clearer and more coherent boundary reconstruction in complex transition regions.

[0047] Specifically, the T-EIB framework is as follows: Figure 4 As shown.

[0048] First, to maintain feature stability, the original input is... Obtained through parallel convolutions of 1×1 and 3×3 respectively. and .

[0049] Next, input features Gradient calculation and edge detection are performed in the horizontal and vertical directions through multiple convolution processes. To obtain multi-scale boundary maps Then, the boundary map results at different scales are stitched together to obtain the comprehensive edge features. .

[0050] Furthermore, the CLIP pre-trained model is introduced to combine text and Transform into vectorized embedding representation And linearly mapped to a condition vector of the same dimension as the decoded features. .

[0051] Finally, the features Key-value projection Subsequently, text-conditionally generated The query vector Q and the projected K and V are used to perform a cross-attention mechanism to obtain the attention weights. ,in This represents the region of unknown pixels in the previous ternary image Tri, thus limiting attention to the unknown region.

[0052] Ultimately, attention weights and and The features are then concatenated to obtain the final text-boundary guided features. This process is the final output of T-EIB.

[0053] In this embodiment, the Text-Boundary Guided Interpolation Block (T-EIB) uses boundary saliency as a spatial prior and textual prompts obtained by the multimodal encoder as a semantic prior. During the upsampling stage, it generates interpolation weights that take into account both structure and semantics, thereby achieving controllable separation of multiple instances of the same type and clear boundary reconstruction.

[0054] Meanwhile, during the decoding stage, to avoid pixel and chroma drift and semantic jumps that occur during upsampling and cross-layer fusion, an inter-layer semantic differential consistency (SDB) module is added during convolution and T-EIB processes to constrain the coherence of adjacent decoding layers along the three axes of "pixel appearance - higher-level semantics - color components". Its structure is as follows: Figure 5 As shown.

[0055] SDB includes three branches. First, for the... Layer features First, it is scaled up and then differentiated according to the channel dimension to obtain the pixel appearance features. .

[0056] Secondly, regarding the first Features obtained by T-EIB of the layer First, features are extracted from the data using multiple parallel convolutions with different kernel sizes, and finally, the features are merged to obtain the merged features. As a high-level semantic guide, it is combined with the obtained pixel appearance features. Perform a concatenation operation to obtain semantic appearance fusion features. .

[0057] Finally, to avoid deviations in the final prediction results caused by differences in color components across different decoding layers, the following will be implemented: and The color component of the corresponding feature is obtained by averaging the values ​​at the feature channel level using a color feature extractor. and Next, the difference between the two color components is calculated to obtain the color component difference characteristics. Finally, the color component difference characteristics are... Features fused with semantic appearance The final output of SDB, color semantic fusion features, is obtained through concatenation. The mathematical expression of the above process is as follows:

[0058]

[0059]

[0060]

[0061] Here, `cat` represents the concatenation operation. This represents a convolution with n different kernel sizes. and These represent the difference and summation operations, respectively. This indicates the scaled-up version. Differentiation is performed based on channels. m represents the mean, and CFE represents the color feature extractor.

[0062] The semantic differential consistency module performs residual correction on adjacent decoding layers to alleviate semantic drift and chroma deviation between layers, and achieves higher boundary clarity and coherence in complex transition areas.

[0063] This embodiment constrains color, semantics, and pixels during the decoding stage, ensuring consistency across different scales in the decoding layer and reducing boundary breaks and discontinuities in the prediction results. Simultaneously, the synergistic combination of SDB and T-EIB, while learning under semantic / structural dual priors, continuously maintains the boundary shape of the final prediction results across layers, preventing upper-layer errors from being amplified in lower layers and improving boundary consistency after controllable separation of multiple instances.

[0064] Meanwhile, visualization of cross-attention weights and boundary saliency maps helps to locate the module's area of ​​interest, facilitating optimization on the engineering side; the overall design adopts a unified encoding-decoding paradigm and lightweight module design, making it easy to port and accelerate on various inference platforms.

[0065] Finally, the color semantic fusion features are input into a lightweight prediction head (convolution, sigmoid) to output a transparency map, thus completing the image matting.

[0066] During training, this embodiment does not use any pre-trained model except for CLIP; the input image and ternary image are cropped into 512×512 training blocks, with a batch size of 16 during training and 1 during testing; the optimizer used is AdamW with a learning rate of 1×10⁻⁶. - ³; Experiments were conducted on a server equipped with four NVIDIA RTX 3090 GPUs. Under the same training / evaluation protocol, the comparative methods were uniformly retrained and tested on the AIM-500 and AM-2k datasets to ensure comparability and objectivity of conclusions.

[0067] To verify the effectiveness of the image matting method described in this embodiment, such as... Figure 6 As shown, using the original image as input, we first obtain manually annotated real results as a benchmark, and then compare the method of this embodiment (this example) with existing technologies such as guided contextual attention matting, real-time matting, semantically guided robust matting, matting based on simple visual transformers, and attention-guided hierarchical aggregation matting. The results show that this embodiment has clearer edges and more complete feature preservation in detailed areas such as the fur edges of leopards and alpacas, sofa textures, and beverage decorations. The synthesized results are more natural and accurate, and the matting accuracy is significantly better than existing methods.

[0068] As one implementation method, the training process is as follows: Step 1: Data Processing Acquisition and Reduction: Read the color image X (RGB, 8-bit) and the corresponding ternary image T (one-hot three channels: foreground / unknown / background), scale proportionally along the longest side, and crop the center to 512×512, normalizing the pixels to [0,1]. Boundary Prior: Calculate the Sobel / Scharr gradient for the unknown region boundary of X or T to obtain the initial boundary magnitude and direction map (for use by the boundary branch of T-EIB and the color difference modulation of SDB). Text Hint Preprocessing: If text conditions are used, unify lowercase, remove stop words, limit the maximum length (e.g., 32 tokens), and retain directional / positional phrases; mark empty hints when there are no hints to ensure that the model only calculates based on the boundary prior. Data Augmentation: Random horizontal flip, ±20° rotation, 0.9-1.1 scaling, ±10% color jitter, JPEG compression noise simulation (Q∈[60,95]), motion blur probability 0.1; apply slight morphological perturbation (1-3 pixels) to T to improve robustness to labeled boundary noise. Segmentation and caching: The training / validation / test sets are divided into 8:1:1, and LMDB / Record formats are generated to accelerate IO. Random seeds for samples are recorded to ensure reproducibility.

[0069] Step 2: Data Visualization Original visualization: Overlaying X and T, using pseudo-color annotation for before / unknown / background; Feature and attention visualization: Periodically exporting TII cross-attention heatmaps, LapCos weight maps (high-frequency response and cosine similarity), and residual heatmaps before and after SDB correction (pixel / semantic / color paths). Boundary quality replay: Predicting... Synchronously render to a unified background and compare edges before and after compositing; establish fixed visualization panels for difficult examples such as hair, gauze, and glass edges for easy vertical comparison; export a list of failed examples (including prompt text and boundary magnitude histograms) to provide a basis for subsequent fine-tuning. Logs and metrics dashboards: record the training / validation curves of SAD, MSE, Grad, and Conn, and overlay the learning rate, weight decay, and gradient norm.

[0070] Step 3: Model Training In one embodiment, the network parameters are first initialized, including all convolutional kernels at the encoder and decoder ends, layer normalization and feedforward layer weights, query / key / value and output matrices for multi-head attention, linear projection parameters in Ternary Graph Information Injection (TII) used to map the ternary graph to the query vector, multi-scale Laplacian convolutional kernels and similarity temperature coefficients in LapCos (LapCos), text encoder adaptation layers in the text prompting branch, key / value projections of the text-boundary guided interpolation block (T-EIB), boundary gating and kernel prediction subnetworks, and pixel / semantic / color projections and gating residual coefficients for inter-layer semantic difference consistency (SDB). Simultaneously, an optimizer and learning rate scheduler (e.g., AdamW, initial learning rate 1×10⁻⁶) are set. -4 Weight decay 1×10 -4 It supports mixed precision and gradient clipping. Each iteration reads a color image X and a ternary image T from the training set, normalizes them according to a unified standard, and clips them to 512×512, generating an unknown domain mask U and a narrow sideband mask. If text prompts are configured, they are simultaneously fed into the text encoder to obtain semantic embeddings. The forward propagation sequentially passes through shallow convolution and TII (generating a query with T and performing cross-attention with shallow image key / value pairs, robustly injecting priors / unpriorities / afterpriorities), and LapCos (multi-scale Laplacian for high-frequency enhancement and cosine similarity for directional weighting, outputting a detailed and spatially consistent encoding table). The decoder consists of a Transformer encoder (integrating long-range dependencies), a T-EIB at the decoder (using boundary saliency as a spatial prior and text embedding as a semantic prior, generating interpolation kernels through cross-attention and learnable upsampling to replace fixed interpolation, achieving controllable separation and clear boundary reconstruction for multiple instances of the same type), an SDB (gated residual correction of pixel / semantic / color differences between adjacent decoding layers to suppress cross-layer shift and chroma drift), and finally, a lightweight prediction head (convolution + sigmoid) outputs an opacity map. The loss function employs a composite design in the unknown region: pixel-level L1 is used to converge the main strength, binary cross-entropy is used for the separability of fine structures, gradient consistency masks the gradient with the ground truth and penalizes the gradient of the predicted result to reduce oversmoothing. Optional connectivity / sideband sparsity and text alignment regularization (e.g., maximizing the cosine similarity between the upsampled features and the text vector) are added to enhance boundary coherence and semantic alignment; the total loss is a weighted sum.

[0071] Gradients are backpropagated to each sub-module of TII, LapCos, Transformer, T-EIB, SDB, and the prediction head. Gradient pruning is used to stabilize training, and parameters are updated according to the learning rate. During training, iteration is performed with a batch size of 16. The optimal weights of the metrics are periodically evaluated on the validation set according to SAD, MSE, Grad, and Conn. If the performance no longer improves after several consecutive rounds or the total number of rounds reaches a preset threshold, the training stops early. To enhance generalization, augmentation is performed on the data side using random flipping / rotation / scaling, color jittering, and mild morphological perturbation of ternary images. The entire training process repeatedly executes the closed loop of "forward-loss-backward-update" under a unified protocol until convergence, finally obtaining a matting model with clear boundaries and high detail in complex textures and semi-transparent scenes.

[0072] This specific embodiment introduces multi-scale Laplace-cosine enhancement at the encoding end, which is beneficial to... Predicting high-frequency edges and textures is amplified; priors foreground / background and unknown regions are robustly incorporated into the initial features via Trimap cross-attention injection; a text-boundary guided interpolation block (T-EIB) is designed at the decoding end, using boundary saliency as a spatial prior and CLIP-encoded text cues as a semantic prior. Cross-attention is used to generate interpolation weights that balance structure and semantics during the upsampling stage, achieving controllable separation and clear boundary reconstruction for multiple instances of the same type; and lightweight semantic difference correction and boundary / gradient consistency loss are applied to produce higher fidelity output without significantly increasing computational overhead. Image and composite effect.

[0073] Example 2 This embodiment provides a natural image matting system based on the aggregation of text and boundary information, including: The feature extraction unit is used to stitch the original color image with the corresponding ternary image and extract the initial features. Enhanced fused features are obtained by multi-scale Laplacian high-frequency extraction and cosine similarity weighted fusion. The prior fusion unit projects the enhanced fusion features into keys and values, maps the ternary graph into a query vector, and generates gated ternary fusion features that fuse prior knowledge through a cross-attention mechanism. The gated ternary fusion features are then globally encoded and modeled to obtain deep features. The decoding constraint unit is used to introduce text prompts and multi-scale boundary information based on deep features during the decoding stage, guide adaptive upsampling through cross-attention mechanism, and adopt semantic differential consistency constraints between each decoding layer to implement consistency constraints from the dimensions of pixel appearance, high-level semantics and color components to obtain color semantic fusion features. The result output unit is used to output a transparency map through the color semantic fusion feature via the prediction head, so as to obtain a matted image of the original color image whose high-frequency detail fidelity and boundary accuracy meet the predefined requirements.

[0074] Example 3 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a natural image matting method based on the aggregation of text and boundary information as described in Embodiment 1 above.

[0075] Example 4 This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the natural image matting method based on the aggregation of text and boundary information as described in Embodiment 1 above.

[0076] The steps or modules involved in Embodiments 2 to 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0077] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A natural image matting method based on the aggregation of text and boundary information, characterized in that, include: The original color image and the corresponding ternary image are stitched together and the initial features are extracted. Enhanced fused features are obtained by multi-scale Laplacian high-frequency extraction and cosine similarity weighted fusion. The enhanced fusion features are projected as keys and values, the ternary graph is mapped as a query vector, and gated ternary fusion features that fuse prior knowledge are generated through a cross-attention mechanism. Global encoding modeling is performed on the gated ternary fusion features to obtain deep features; In the decoding stage, text prompts and multi-scale boundary information are introduced based on deep features. Adaptive upsampling is guided through a cross-attention mechanism. At the same time, semantic differential consistency constraints are adopted between each decoding layer. Consistency constraints are implemented from the dimensions of pixel appearance, high-level semantics and color components to obtain color semantic fusion features. The color semantic fusion features are used to output a transparency map through a prediction head to obtain a matted image of the original color image that meets predefined requirements for high-frequency detail fidelity and boundary accuracy.

2. The natural image matting method based on text and boundary information aggregation as described in claim 1, characterized in that, The ternary image includes three types of pixels: foreground, background, and uncertain region, which serve as prior knowledge.

3. The natural image matting method based on text and boundary information aggregation as described in claim 1, characterized in that, The enhanced fusion features obtained by multi-scale Laplacian high-frequency extraction and cosine similarity weighted fusion specifically include: The initial features are input into the Laplacian-cosine similarity enhancement module, which includes a Laplacian branch and a cosine similarity branch; The Laplacian branch performs multi-scale Gaussian pyramid processing and Laplacian convolution on the initial features to extract high-frequency features at different scales. After channel normalization, it performs differential operations on features at adjacent scales to obtain the Laplacian feature matrix. The cosine similarity branch performs local window expansion and normalization on the initial features, models the correlation between local patches and orientation templates through learnable weight tensors, calculates position-by-position cosine similarity and generates a weighted coefficient map through nonlinear transformation, then fuses it with the initial features through residual connection, and obtains a cosine similarity attention map based on the self-attention mechanism. The output features of the two branches are added together through channels and then multiplied and added with the initial features to obtain the enhanced fused features.

4. The natural image matting method based on text and boundary information aggregation as described in claim 3, characterized in that, The cosine similarity branch specifically includes: Using the initial features as input, the features are first rearranged into image patches through a local window expansion operation, flattening the neighborhood of each spatial location into a local patch vector and normalizing it; Create an empty weight tensor and initialize it with a normal distribution. Flatten the convolutional kernel weights corresponding to each output channel into vectors and register them as learnable parameters so that they can be updated adaptively during training. For each spatial location, the response between the normalized local patch vector and the normalized convolution kernel weight vector of each channel is calculated by the vector inner product operation, and the positional cosine similarity map of the channel-space dimension is obtained. After extracting symbols from the similarity map, a weighted coefficient map is generated by amplifying high-response regions and suppressing low-response regions through monotonic nonlinear transformation. The weighted coefficient map is combined with the initial features through residual connection to obtain the enhanced features; The enhanced features are mapped to three vectors, Q, K, and V, and a cosine similarity attention map is obtained using a self-attention mechanism.

5. The natural image matting method based on the aggregation of text and boundary information as described in claim 1, characterized in that, The process of projecting enhanced fusion features as keys and values, mapping ternary graphs to query vectors, and generating gated ternary fusion features that integrate prior knowledge through a cross-attention mechanism specifically includes: The enhanced fusion features are segmented and flattened, projected onto a unified representation space through a learnable embedding map, and then linearly projected to obtain the keys and values. After convolution processing of the ternary image, the query vector is obtained through mapping operations; The query vector is multiplied by the transpose of the key, activated by the Softmax function, and then multiplied by the value to obtain the initial prior fusion feature; The difference term between the initial prior fused features and the enhanced fused features is calculated, and a gating function is constructed through global average pooling and multilayer perceptron to generate gating coefficients; The difference terms are weighted based on the gating coefficients and then added to the enhanced fusion features to obtain the gated ternary fusion features that fuse prior knowledge.

6. The natural image matting method based on text and boundary information aggregation as described in claim 1, characterized in that, The method of introducing text prompts and multi-scale boundary information based on deep features, and guiding adaptive upsampling through a cross-attention mechanism, specifically includes: The deep features are input into the text-boundary guided interpolation module, which first extracts features at different scales through parallel convolution; Then, gradient calculations and edge detection are performed on the deep features in the horizontal and vertical directions to obtain multi-scale boundary maps, which are then stitched together to form comprehensive edge features. A pre-trained model is introduced to transform text and integrated edge features into vectorized embeddings, which are linearly mapped to text conditional vectors of the same dimension as the decoded features; Deep features are key-value projected, and the text condition vector is used as the query vector. Cross-attention operation is performed with the key and value to obtain the attention weight, while the attention is restricted to the unknown region of the ternary map. The attention weights are concatenated with the features extracted by parallel convolution to obtain text-boundary guided features.

7. The natural image matting method based on text and boundary information aggregation as described in claim 1, characterized in that, The process employs semantic differential consistency constraints across each decoding layer, implementing consistency constraints from the dimensions of pixel appearance, high-level semantics, and color components to obtain color semantic fusion features, specifically including: The previous layer's decoded features are scaled, and the differential results are calculated according to the channel dimension to obtain the pixel appearance features; For the current layer text-boundary guided features, features are extracted and merged in parallel using multiple convolutional kernels of different sizes to obtain high-level semantic features, which are then concatenated with pixel appearance features to generate semantic appearance fusion features; The color component mean values ​​of the previous layer decoded features and the current layer text-boundary guided features are calculated by the color feature extractor, and the difference between the two is calculated to obtain the color component difference features. The color component difference features are concatenated with the semantic appearance fusion features to obtain the color semantic fusion features.

8. A natural image matting system based on the aggregation of text and boundary information, characterized in that, include: The feature extraction unit is used to stitch the original color image and the corresponding ternary image and extract the initial features. Enhanced fused features are obtained by multi-scale Laplacian high-frequency extraction and cosine similarity weighted fusion. The prior fusion unit is used to project the enhanced fusion features into keys and values, map the ternary graph into query vectors, and generate gated ternary fusion features that fuse prior knowledge through a cross-attention mechanism. Global encoding modeling is performed on the gated ternary fusion features to obtain deep features; The decoding constraint unit is used to introduce text prompts and multi-scale boundary information based on deep features during the decoding stage, guide adaptive upsampling through cross-attention mechanism, and adopt semantic differential consistency constraints between each decoding layer to implement consistency constraints from the dimensions of pixel appearance, high-level semantics and color components to obtain color semantic fusion features. The result output unit is used to output a transparency map through the color semantic fusion feature via the prediction head, so as to obtain a matted image of the original color image whose high-frequency detail fidelity and boundary accuracy meet the predefined requirements.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the natural image matting method based on the aggregation of text and boundary information as described in any one of claims 1-7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the natural image matting method based on the aggregation of text and boundary information as described in any one of claims 1-7.

Images