Image anaphora segmentation method based on multilevel feature fusion and dual-channel information enhancement

Abstract

The invention aims to disclose an image anaphora segmentation method based on multilevel feature fusion and dual-channel information enhancement. The method comprises the following steps: obtaining and extracting visual features to be segmented and language features in an anaphora text; self-adaptively matching visual features of different levels by using language features, and performing spectrum enhancement on the visual features by using Fourier transform; through cross-modal fusion of visual and language features of different hierarchies, multi-level visual features with language perception are constructed; constructing visual language fusion features by using dual-channel information enhancement; and generating a segmentation mask. According to the method, the advantages of a large model, adaptive dynamic matching and feature enhancement are fused, automatic generation of the segmentation mask is realized, the efficiency and accuracy are improved, and the method has wide application and popularization values.

Description

Technical Field

[0001] This invention relates to image reference segmentation, specifically to an image reference segmentation method based on multi-level feature fusion and dual-channel information enhancement, belonging to the interdisciplinary field of computer vision and natural language processing. Background Technology

[0002] With the development of Internet technology, various types of image data are emerging in an endless stream. It is urgent to understand and utilize these image data correctly. Image reference segmentation, as a multimodal task of visual language, has received increasing attention.

[0003] The referential image includes the image to be segmented and the referential text. Image referential segmentation refers to using natural language descriptions to locate the target object being referred to, that is, generating a segmentation mask based on the given natural language description, and accurately segmenting the target region referred to by the referential text in the image.

[0004] With the emergence and development of deep learning, people have proposed a variety of image reference segmentation methods and achieved remarkable results. In patent 202510186603.3, Zhang Xiaobo et al. used image feature fusion to construct global and local features, but did not enhance the image features. They used the fused features to achieve image-text retrieval, but did not solve the image reference segmentation problem. In patent CN202510092856.4, Liu Ting et al. addressed the image reference segmentation problem by first generating candidate segmentation masks and then using spatial guided scoring and semantic alignment scoring to achieve reference segmentation, but the feature interaction between text and image was not sufficient. In patent CN202411979530.5, Li Qing et al. addressed the image reference segmentation problem by first extracting and fusing text features and image features, and then directly concatenating the two features to generate dual prompts, but the text features were not fully utilized. In patent CN202411914326.5, Yang Hang et al. used a neural network-based language dependency parser to pre-annotate language elements such as parts of speech and semantic roles when modeling the dependency relationships between words, which increased the implementation cost of the image reference segmentation task to some extent. Summary of the Invention

[0005] The purpose of this invention is to disclose an image referential segmentation method based on multi-level feature fusion and dual-channel information enhancement. This method utilizes language features to dynamically match visual features at different levels, and fully utilizes the overall detail information of the image through the enhanced visual features after multi-level fusion. It uses dual-channel enhancement to enhance the interaction between language features and visual features and constructs visual-language fusion features to enhance the role of relevant language features in referential image segmentation. Based on the visual-language fusion features, it generates query instance coefficients and candidate masks to improve the matching degree of segmentation masks. The specific process includes: (1) extracting referential image features: obtaining and extracting the visual features to be segmented and the language features in the referential text; (2) using language features to adaptively match visual features at different levels and using Fourier transform to enhance their spectrum; (3) constructing multi-level visual features with language perception by fusing visual and language features at different levels across modalities; (4) using dual-channel information enhancement to construct visual-language fusion features; and (5) generating segmentation masks.

[0006] Specifically, the method of the present invention includes the following steps:

[0007] A. Obtain and extract the visual features of the referential image and the linguistic features of the referential text. The specific steps are as follows:

[0008] A1: Obtain the image to be segmented and the text it refers to;

[0009] A2 uses a pre-trained text encoder model to extract two levels of linguistic features from the referential text. The specific steps are as follows:

[0010] A2.1 breaks down the referential text into subwords and adds special markers such as start [EOS] and end [SOS];

[0011] A2.2 Embed subwords into word vectors and add positional encoding;

[0012] A2.3 The pre-trained text encoder is used to process word vectors layer by layer and extract language features. The specific steps are as follows:

[0013] A2.3.1 Using the special markers mentioned above, extract sentence-level linguistic features. ;

[0014] A2.3.2 Skip the special markers above and extract word-level language features. ;

[0015] A3 uses a pre-trained visual encoder model to extract 12 layers of visual features from the image to be segmented. The specific steps are as follows:

[0016] A3.1 Adjust the resolution of the image to be segmented and divide it into non-overlapping image blocks;

[0017] A3.2 Map each image patch as a vector and concatenate the learnable class token and position code;

[0018] A3.3 uses a pre-trained visual encoder model to process image patch vectors layer by layer, and utilizes intermediate layer class tokens to extract 12 layers of candidate visual features from the image to be segmented. ;

[0019] B. Adaptively match visual features at different levels using linguistic features, and perform spectral enhancement using Fourier transform. The specific steps are as follows:

[0020] B1 Dynamic Selection and Sentence-Level Language Features The most relevant high, medium, and low-level visual features , The specific steps are as follows:

[0021] B1.1 Select high-level, medium-level, and low-level visual feature candidate layers from the 12 candidate visual feature layers respectively;

[0022] B1.2 Utilizing alternative visual features tokens Calculate its relationship with sentence-level language features cosine similarity ;

[0023] B1.3 Take the cosine similarity sets from the candidate layers of high, medium, and low-level visual features respectively, and use them as the affinity between the visual features and linguistic features of that layer. , They represent high, medium, and low levels, respectively.

[0024] B1.4 Utilizing visual and linguistic features to assess trustworthiness Calculate the trust score The specific steps are as follows:

[0025] B1.4.1 The confidence scores from the high, medium, and low-level visual feature candidate layers obtained above are... After being processed by two linear layers, a ReLU activation function, and a Sigmoid function, the confidence weights of the corresponding visual feature candidate layers are calculated. ;

[0026] B1.4.2 The confidence level of the visual feature candidate layers obtained above at each level. After passing through a linear layer, a normalized layer, and a ReLU activation function, the gating weights of the corresponding visual feature candidate layers are calculated. ;

[0027] B1.4.3 The gating weights of each visual feature candidate layer obtained above. Weighting by trust level After multiplying, summing over all columns yields the confidence score for the corresponding visual feature candidate layer. ;

[0028] B1.5 Visual features at different levels are dynamically selected using the confidence scores of the visual feature candidate layer. The specific steps are as follows:

[0029] B1.5.1 Select the layer with the highest confidence score among the candidate layers for advanced visual features.

[0030] The corresponding alternative visual features are used as high-level visual features. ;

[0031] B1.5.2 Select the layer with the highest confidence score among the candidate layers for intermediate visual features, and downsample its corresponding candidate visual features by a factor of 2 to calculate the intermediate visual features. ;

[0032] B1.5.3 Select the layer with the highest confidence score among the candidate low-level visual features, and then downsample its corresponding candidate visual features by a factor of 4 to calculate the low-level visual features. ;

[0033] B2 utilizes Fourier transform to perform spectral enhancement on high, medium, and low-level visual features, obtaining the enhanced high, medium, and low-level visual features. , The specific steps are as follows:

[0034] B2.1 High, medium, and low-level visual features , After performing Fourier transforms to obtain complex spectra, the low-frequency components are shifted to the center of the spectrum to obtain the centered complex spectrum.

[0035] B2.2 High-frequency and low-frequency filters are used to process high-level and medium- and low-level visual features respectively, and then the filtered complex spectrum is calculated by multiplying it element by element with the decentralized complex spectrum obtained in B2.1.

[0036] B2.3 Calculate using inverse Fourier transform to convert the filtered complex spectrum obtained above back to the spatial domain;

[0037] B2.4 The visual features after Fourier transform are summed with the original features through cross-layer connections and residual connections are then performed to calculate the enhanced high, medium, and low-level visual features. , ;

[0038] C constructs a multi-layered visual feature system with language perception by fusing visual and linguistic features at different levels across modalities. The specific steps are as follows:

[0039] C1 utilizes cross-modal fusion of sentence-level language features. and higher visual features after spectral enhancement Constructing high-level visual features with language awareness The specific steps are as follows:

[0040] C1.1 The sentence-level linguistic features obtained in A4.1 above Language mapping that includes a linear layer, batch normalization, and LeakyReLU activation function. The mapped sentence-level language features are calculated. :

[0041] ;

[0042] C1.2 High-level visual features enhanced by the spectrum obtained in B2 above. Visual projection consisting of a convolutional layer, batch normalization, and ReLU activation function. Calculate the high-level visual features after projection :

[0043] );

[0044] C1.3 Mapping sentence-level language features Advanced visual features after projection Multiplication yields higher-level visual features with accompanying language perception. :

[0045] = ;

[0046] C2 enhances low- and mid-level visual features through cross-level fusion spectrum. , Constructing composite visual features with hierarchical expressive capabilities. The specific steps are as follows:

[0047] C2.1 Low-level visual features after spectral enhancement Perform upsampling to obtain the same as Feature representation with the same scale ;

[0048] C2.2 Spectral Enhancement of Intermediate Visual Features After 1x1 convolution and ReLU activation, it is combined with the upsampled low-level visual features. By adding them together, we can calculate the composite visual features. ;

[0049] C3 integrates advanced visual features with accompanying language awareness. and complex visual features with hierarchical expressive power. Constructing multi-level visual features with language awareness The specific steps are as follows:

[0050] C3.1 builds upon C1 to include advanced visual features with language awareness. After passing through convolutional layers and the ReLU activation function, a more discriminative language-aware algorithm is calculated. ;

[0051] C3.2 Composite visual features constructed from C2 After upsampling, the result was calculated to be the same as Features of the same scale;

[0052] C3.3 Add the two results obtained above to calculate the multi-level visual features. ;

[0053] D utilizes dual-channel information enhancement to construct visual-language fusion features. The specific steps are as follows;

[0054] D1 refines the visual subdivisions of C3 to obtain multi-level visual features. Refine the details and construct a multi-layered visual feature set. The specific steps are as follows:

[0055] D1.1 with multi-layered visual features As a query vector (Q) and key-value pairs (KV), through After the layer normalization module, the data is fed into a multi-head self-attention mechanism. With the original multi-layered visual features Perform residual connections and compute the enhanced feature representation. :

[0056] ;

[0057] D1.2 Using a multilayer perceptron The result obtained from D1.1 above Using multilayer perceptron After undergoing a nonlinear transformation, compared with the original Residual connections are performed to calculate the refined multi-level visual features. :

[0058] ;

[0059] D2 Based on refined multi-level visual features and word-level linguistic features By enhancing features through language reinforcement branches and cross-modal interaction, visually perceptive language features are constructed. The specific steps are as follows:

[0060] D2.1 Language features at the word level As the query vector (Q), the refined multi-level visual features constructed above As key-value pairs (KV), after cross-attention Afterwards, language features at the original word level Perform residual connections to calculate enhanced word-level linguistic features. :

[0061] ;

[0062] D2.2 Utilizes a multilayer perceptron to refine the multilayer visual features obtained from D1. After undergoing a nonlinear transformation, compared with the original Perform residual connections to calculate language features with visual perception. :

[0063] ;

[0064] D3 utilizes refined multi-layered visual features Language features with visual perception To enable cross-modal interaction and construct visual-language fusion features The specific steps are as follows:

[0065] D3.1 Refined Multi-layered Visual Features Acting as Q, the visually perceptual language features obtained from D2 above. Serving as key-value pairs (KV), and processed through a cross-attention mechanism, combined with the original refined multi-layered visual features. Perform residual linking to calculate the visual features that best fit the meaning of the referential text. :

[0066] ;

[0067] D3.2 Using a multilayer perceptron, the result obtained in D3.1 above is processed. After undergoing a nonlinear transformation, compared with the original Perform residual connections to calculate visual-language fusion features. :

[0068] ;

[0069] E generates a segmentation mask, and the specific steps are as follows;

[0070] E1 Generates alternative segmentation masks based on visual reinforcement branching. The specific steps are as follows:

[0071] E1.1 The refined multi-layered visual features obtained from C1 above. After 3x3 convolutions and alternating upsampling operations The discriminatively enhanced multi-level visual feature representation is calculated. :

[0072] ;

[0073] E1.2 Using 1x1 convolution The above results The number of channels is adjusted to the number of candidate masks σ, and the discriminatively enhanced σ-channel multi-level visual features are calculated. :

[0074] ;

[0075] E1.3 The sigmoid activation function is used to calculate the probability value of each pixel, and candidate masks are generated based on these probability values. ;

[0076] E2 generates instance query coefficients based on language refinement branches. The specific steps are as follows:

[0077] E2.1 The visual language fusion features obtained from D above After two convolutions, features are further extracted, and discriminatively enhanced visual-language fusion features are calculated. ;

[0078] E2.2 Using 1*1 convolutional layers Perform channel transformation to align the number of channels with the number of candidate masks σ, and calculate the discriminatively enhanced σ-channel visual-language fusion features. ;

[0079] E2.3 using the Tanh activation function Normalization is performed to calculate the instance query coefficient. ;

[0080] E3 instance query coefficient and alternative masks Multiply to generate the final image segmentation mask. .

[0081] Compared with existing technologies, this invention has the following advantages: it implements a complete image referencing segmentation method, considers the imbalance between image and text features in image referencing segmentation tasks, extracts multi-level visual and linguistic features, and enhances visual features through multi-level fusion, making full use of the overall and detailed information of the image; it utilizes dual-channel further refined visual and linguistic features for cross-modal interaction, improving the matching degree between the referential text and the image to be segmented. This invention integrates the advantages of large models, adaptive dynamic matching, and feature enhancement, realizing the automatic generation of segmentation masks, improving efficiency and accuracy, and has broad application and promotion value. Attached Figure Description

[0082] Figure 1 Flowchart of an image referencing segmentation method based on global local features and dual-channel information enhancement Detailed Implementation

[0083] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0084] The purpose of this invention is to disclose an image referential segmentation method based on multi-level feature fusion and dual-channel information enhancement. This method utilizes a large model to extract visual and linguistic features and achieves dynamic matching of visual features at different levels. Through multi-level fusion and enhancement of visual features, it fully utilizes the overall detail information of the image. It also utilizes dual-channel enhancement to improve the interaction between linguistic and visual features and constructs visual-linguistic fusion features, thereby enhancing the role of relevant linguistic features in image referential segmentation. Finally, it generates query instance coefficients and candidate masks based on the visual-linguistic fusion features, improving the matching degree of the segmentation mask. The method of the present invention is divided into 5 stages: extracting referential image features, adaptively matching visual features and performing spectral enhancement, constructing hierarchical visual features with language awareness, constructing visual-language fusion features, and generating a segmentation mask; the specific process includes: (1) extracting referential image features: obtaining and extracting the visual features to be segmented and the language features in the referential text; (2) using language features to adaptively match visual features at different levels and performing spectral enhancement on them using Fourier transform; (3) constructing multi-level visual features with language awareness by fusing visual and language features at different levels across modalities; (4) constructing visual-language fusion features using dual-channel information enhancement; and (5) generating a segmentation mask.

[0085] The technical solution of this invention will be clearly and completely described below in accordance with the steps. The example of a segmented image is: image G.jpg, with the corresponding text description: green shirt touching frizbee;

[0086] 1. Obtain and extract the visual features of the referential image and the linguistic features of the referential text. The specific steps are as follows:

[0087] 1.1 Obtain the image to be segmented, G.jpg, with the text: green shirt touching frizbee;

[0088] 1.2 Using a CLIP-based Transformer as the language encoder, the linguistic features of "green shirttouching frizbee" are extracted. The specific steps are as follows:

[0089] 1.2.1 The referential text “green shirt touching frizbee” was tokenized using clip.tokenize(), split into sub-tokens, and marked with [SOS] and [EOS] tags and filled with [PAD]. The results after splitting are shown in Table 1 below.

[0090] 1.2.2 Embed the tokens as word vectors;

[0091] 1.2.3 The Transformer encoder is used to process word vectors layer by layer and extract language features. Each token outputs a hidden state, and the output at the [EOS] position is taken as the sentence-level feature. Irrelevant tokens such as [SOS] and [PAD] are skipped, and the hidden states of all tokens are extracted as word-level language features. Some intermediate states and their descriptions are shown in Table 1 below:

[0092] Table 1 Examples of Linguistic Features of Referential Texts

[0093] Token location Token value Features and uses 0 [SOS] Start mark 1 "green" Extractable word-level language features 2 "shirt" Extractable word-level language features 3 "touching" Extractable word-level language features 4 "frizbee" Extractable word-level language features 5 [EOS] Extractable sentence-level language features ≥5 [PAD] Fill mark

[0094] 1.3 Using a pre-trained visual encoder model, extract 12 layers of visual features from the image to be segmented. The specific steps are as follows:

[0095] 1.3.1 Adjust the resolution of the image to be segmented to 224*224*3, normalize the pixel values ​​and divide it into 16*16 blocks, resulting in (224*224) / (16*16)=196 blocks;

[0096] 1.3.2 Each block is flattened into a vector (16*16*3=768), positional encoding is introduced to preserve spatial information, and a learnable [CLS] token is inserted at the beginning of the sequence to obtain the output size (197*768).

[0097] 1.3.3 The encoder based on the clip architecture ViT-B consists of 12 Transformer blocks, which process the vectors above layer by layer, skipping [CLS] and extracting the output of the last layer as the feature of all patches as candidate visual features. The main steps and operating dimensions are shown in Table 2 below;

[0098] Table 2. Main steps and output dimensions of visual feature extraction

[0099] step operate Output size Original image RGB three-channel 3×224×224 Blocks 16×16 non-overlapping blocks 196 × 768 +[CLS] & Position Code Add class token [CLS] and location code 197 × 768 Transformer encoder 12 floors 197 × 768 Feature extraction Extraction tokens [CLS] 1 × 768

[0100] 2. Adaptively match visual features at different levels using linguistic features, and perform spectral enhancement using Fourier transform. The specific steps are as follows:

[0101] 2.1 Dynamic Selection and Sentence-Level Linguistic Features The most relevant high, medium, and low-level visual features , The specific steps are as follows:

[0102] 2.1.1 From the 12 candidate visual features, select high, medium and low level visual feature candidate layers respectively: select layers 1-5 as low level visual feature candidate layers; select layers 6-9 as medium level visual feature candidate layers; select layers 10-12 as high level visual feature candidate layers.

[0103] 2.1.2 Utilizing alternative visual features The token is used to calculate its relationship with sentence-level language features. cosine similarity :

[0104]

[0105] in, This indicates element-wise multiplication. Indicates will Projected onto a linear layer of the same dimension as the language features. Indicates the size of the vector;

[0106] 2.1.3 The cosine similarity sets from the candidate layers of high, medium, and low-level visual features are respectively taken as the affinity between the visual features and linguistic features of that layer. :

[0107]

[0108]

[0109] ;

[0110] 2.1.4 Utilizing visual and linguistic features to assess trustworthiness Calculate the trust score The specific steps are as follows:

[0111] (1) The confidence level in the high, medium and low level visual feature candidate layers obtained above , , After processing through two linear layers, a ReLU activation function, and a Sigmoid function, the confidence weights of the corresponding visual feature candidate layers are calculated. , :

[0112]

[0113]

[0114]

[0115] in, Table of visual feature candidate layer weight calculation module at different levels;

[0116] (2) The confidence levels of the high, medium and low-level visual feature candidate layers obtained above are calculated. , , After passing through a linear layer, a normalized layer, and a ReLU activation function, the gating weights of the corresponding visual feature candidate layers are calculated. , :

[0117]

[0118]

[0119] ;

[0120] in, Table of visual feature candidate layer gating weight calculation module at different levels;

[0121] (3) The gating weights of each visual feature candidate layer obtained above are... Weighting by trust level After multiplication, use Summing all columns yields the confidence score for the corresponding visual feature candidate layer. :

[0122]

[0123]

[0124] ;

[0125] The calculation process of adaptive matching visual features is shown in Table 3 below, with examples as a reference:

[0126] Table 3. Visual Feature Mapping Process Data and Selection Results

[0127] i Number of layers selected 1 Low-level feature candidate layer 128.5 0.52 [1.8, 0.6, 2.1, 1.2, 0.0, ...] 3 2 Low-level feature candidate layer 167.2 0.68 [2.3, 1.1, 2.8, 1.7, 1.2, ...] 3 3 Low-level feature candidate layer 203.8 0.84 [3.1, 1.4, 3.2, 2.2, 1.8, ...] 3 4 Low-level feature candidate layer 145.6 0.71 [2.7, 1.2, 2.5, 1.9, 1.5, ...] 3 5 Low-level feature candidate layer 98.4 0.45 [1.9, 0.8, 1.8, 1.1, 0.9, ...] 3 6 Intermediate feature candidate layer 146.3 0.61 [2.4, 1.3, 2.9, 1.8, 1.4, ...] 7 7 Intermediate feature candidate layer 198.7 0.79 [3.2, 1.7, 3.4, 2.5, 2.1, ...] 7 8 Intermediate feature candidate layer 172.4 0.75 [2.8, 1.5, 3.0, 2.2, 1.9, ...] 7 9 Intermediate feature candidate layer 132.8 0.58 [2.1, 1.1, 2.3, 1.6, 1.2, ...] 7 10 Advanced Feature Candidate Layer 215.6 0.83 [3.5, 2.1, 3.8, 2.7, 2.3, ...] 10 11 Advanced Feature Candidate Layer 168.3 0.72 [2.9, 1.8, 3.2, 2.1, 1.9, ...] 10 12 Advanced Feature Candidate Layer 125.7 0.54 [2.2, 1.2, 2.5, 1.7, 1.3, ...] 10

[0128] 2.1.5 Dynamically select visual features at different levels using the confidence scores of the visual feature candidate layer. The specific steps in this example are as follows:

[0129] (1) The layer with the highest confidence score among the candidate layers of advanced visual features is the 10th layer. The candidate visual features corresponding to the 10th layer are selected as advanced visual features. ;

[0130] (2) The layer with the highest confidence score among the candidate layers of intermediate visual features is the 7th layer. The candidate visual features corresponding to the 7th layer are selected and downsampled by 2 times to calculate the intermediate visual features. ;

[0131] (3) The layer with the highest confidence score among the candidate layers of low-level visual features is the 3rd layer. After selecting the candidate visual features corresponding to the 3rd layer and downsampling them by 4 times, the low-level visual features are calculated. ;

[0132] The dimensions of visual features at different levels are shown in Table 4 below:

[0133] Table 4 Visual feature dimensions at different levels

[0134] Visual features Class Tokens (CLS) Quantity Space Network Feature output shape Advanced visual features 197 (14×14 + 1) 14×14 [1, 197, 768] Intermediate visual features 50 (7×7 + 1) 7×7 [1, 50, 768] Low-level visual features 10 (3×3 + 1) 3×3 [1, 10, 768]

[0135] 2.2 Fourier transform is used to perform spectral enhancement on high, medium, and low-level visual features to obtain the enhanced visual features. , The specific steps are as follows:

[0136] 2.2.1 High, medium, and low-level visual features , After performing Fourier transforms to obtain complex spectra, the low-frequency components are shifted to the center of the spectrum to obtain the centered complex spectrum.

[0137] 2.2.2 High-frequency and low-frequency filters are used to process high-level, medium-level, and low-level visual features respectively, and then the filtered complex spectrum is calculated by multiplying it element-by-element with the decentralized complex spectrum obtained in 2.2.1.

[0138] 2.2.3 Using the inverse Fourier transform, the filtered complex spectrum obtained above is converted back to the spatial domain;

[0139] 2.2.4 The visual features after Fourier transform are summed with the original features through cross-layer connections and residual connections are then performed to calculate the enhanced high, medium, and low-level visual features. , The specific implementation process is as follows:

[0140]

[0141]

[0142] ;

[0143] in, Indicates Fourier transform, Indicates the inverse Fourier transform. Indicates a high-frequency filter. This indicates a low-frequency filter; the Gaussian filter set in this method... =7;

[0144] 3. Constructing Language-Aware Local-Global Features

[0145] 3.1 By fusing visual and linguistic features at different levels across modalities, a multi-layered visual feature model with linguistic awareness is constructed. The specific steps are as follows:

[0146] 3.1.1 The sentence-level linguistic features obtained in 1.4.1 above After linear layer, batch normalization, and LeakyReLU activation function, the mapped sentence-level language features are calculated. :

[0147] ;

[0148] in, Represents language feature mapping;

[0149] 3.1.2 High-level visual features enhanced by the spectrum obtained in 2.2 above The high-level visual features are obtained after passing through a convolutional layer, batch normalization, and ReLU activation function. :

[0150] ) ;

[0151] in, Represents the projection of visual features;

[0152] 3.1.3 Mapping sentence-level language features Advanced visual features after projection Multiplication yields higher-level visual features with accompanying language perception. :

[0153] = ;

[0154] 3.2 Low-level and mid-level visual features enhanced by cross-level fusion spectrum , Constructing composite visual features with hierarchical expressive capabilities. The specific steps are as follows:

[0155] 3.2.1 Low-level visual features after spectral enhancement Upsampling , obtain and Feature representation with the same scale :

[0156] = ;

[0157] 3.2.2 Intermediate visual features enhanced by spectral enhancement After 1x1 convolution and ReLU activation, it is combined with the upsampled low-level visual features. By adding them together, we can calculate the composite visual features. :

[0158] = + ;

[0159] in, It consists of a 1x1 convolution and a ReLU activation function;

[0160] 3.3 By fusing advanced visual features with accompanying language perception and complex visual features with hierarchical expressive power. Constructing multi-level visual features with language awareness The specific steps are as follows:

[0161] 3.3.1 High-level visual features with language awareness constructed in 3.1 After passing through convolutional layers and the ReLU activation function, a more discriminative language-aware algorithm is calculated. ;

[0162] 3.3.2 The composite visual features constructed in 3.2 After upsampling, the result was calculated to be the same as Features of the same scale;

[0163] 3.3.3 Add the two results obtained above to calculate the multi-level visual features. :

[0164] ;

[0165] 4. Constructing visual-language fusion features by leveraging dual-channel information enhancement. The specific steps are as follows;

[0166] 4.1 Visual refinement of branches to analyze multi-level visual features Refine the details and construct a multi-layered visual feature set. The specific steps are as follows:

[0167] 4.1.1 Using multi-layered visual features As a query vector (Q) and key-value pairs (KV), through After the layer normalization module, the data is fed into a multi-head self-attention mechanism. With the original multi-layered visual features Perform residual connections and compute the enhanced feature representation. The specific calculation process is as follows:

[0168] Q=

[0169] K=

[0170] V=

[0171]

[0172] ;

[0173] in, , The weight matrix is ​​a learnable matrix. The dimension of the key (K), For normalization;

[0174] 4.1.2 Using a multilayer perceptron to process the above results After performing a nonlinear transformation, a multilayer perceptron is used. With the original Residual connections are performed to calculate the refined multi-level visual features. :

[0175] ;

[0176] 4.2 Based on refined multi-level visual features and word-level linguistic features By enhancing features through language reinforcement branches and cross-modal interaction, visually perceptive language features are constructed. The specific steps are as follows:

[0177] 4.2.1 Language features at the word level As the query vector (Q), the refined multi-level visual features constructed above As key-value pairs (KV), after cross-attention, they are combined with the original word-level linguistic features. Perform residual connections to calculate enhanced word-level linguistic features. :

[0178]

[0179] ;

[0180] 4.2.2 Using a multilayer perceptron, the refined multilayer visual features obtained in 4.1 are processed. After undergoing a nonlinear transformation, compared with the original Perform residual connections to calculate language features with visual perception. :

[0181] ;

[0182] 4.3 Utilizing refined multi-layered visual features Language features with visual perception To enable cross-modal interaction and construct visual-language fusion features The specific steps are as follows:

[0183] 4.3.1 Refined Multi-layered Visual Features Acting as Q, the above-obtained language features with visual perception. Serving as key-value pairs (KV), and processed through a cross-attention mechanism, combined with the original refined multi-layered visual features. Perform residual linking to calculate the visual features that best fit the meaning of the referential text. :

[0184]

[0185] ;

[0186] 4.3.2 Using a multilayer perceptron, the result obtained in 4.3.1 above is... After undergoing a nonlinear transformation, compared with the original Perform residual connections to calculate visual-language fusion features. :

[0187] ;

[0188] 5. Obtain the segmentation mask, the specific steps are as follows;

[0189] 5.1 Generation of Alternative Segmentation Masks Based on Visual Reinforcement Branching The specific steps are as follows:

[0190] 5.1.1 The refined multi-layered visual features obtained in 3.1 above After three 3x3 convolutions and alternating upsampling operations, a discriminatively enhanced multi-level visual feature representation is calculated. :

[0191]

[0192]

[0193] ;

[0194] 5.1.2 Use 1*1 convolution to transform the above result The number of channels was adjusted to 32 (the number of candidate masks set in this method), and the discriminatively enhanced 32-channel multi-level visual features were calculated. :

[0195] ;

[0196] 5.1.3 Regarding The sigmoid activation function is used to calculate the probability value of each pixel, and candidate masks are generated based on these probability values. ;

[0197] 5.2 Generating Instance Query Coefficients Based on Language Refinement Branches The specific steps are as follows:

[0198] 5.2.1 Integrate the visual language features obtained in section 4 above After two 3x3 convolutions, features are further extracted, and discriminatively enhanced visual-language fusion features are calculated. ;

[0199]

[0200] ;

[0201] 5.2.2 Using a 1*1 convolutional layer to... The channel transformation is changed to 32, and discriminatively enhanced 32-channel visual-language fusion features are calculated. :

[0202] ;

[0203] 5.2.3 Using the Tanh activation function to... Normalization is performed to calculate the instance query coefficient. ;

[0204] 5.3 Generating the Segmentation Mask

[0205] Instance query coefficients and alternative masks Multiplication yields the final image segmentation mask. .

[0206] This invention implements a complete image referencing segmentation method, extracting multi-level visual and linguistic features, and enhancing visual features through multi-level fusion; it also utilizes dual-channel refinement of visual and linguistic features for cross-modal interaction. This invention integrates the advantages of large models, adaptive dynamic matching, and feature enhancement to achieve automatic generation of segmentation masks, improving efficiency and accuracy, and has broad application and promotion value.

Claims

1. An image referencing segmentation method based on multi-level feature fusion and dual-channel information enhancement, comprising the following steps: A. Obtain and extract the visual features of the referential image and the linguistic features of the referential text. The specific steps are as follows: A1: Obtain the image to be segmented and the text it refers to; A2 uses a pre-trained text encoder model to extract two levels of linguistic features from the referential text. The specific steps are as follows: A2.1 breaks down the referential text into subwords and adds special markers such as start [EOS] and end [SOS]; A2.2 Embed subwords into word vectors and add positional encoding; A2.3 The pre-trained text encoder is used to process word vectors layer by layer and extract language features. The specific steps are as follows: A2.3.1 Using the special markers mentioned above, extract sentence-level linguistic features. ; A2.3.2 Skip the special markers above and extract word-level language features. ; A3 uses a pre-trained visual encoder model to extract 12 layers of visual features from the image to be segmented. The specific steps are as follows: A3.1 Adjust the resolution of the image to be segmented and divide it into non-overlapping image blocks; A3.2 Map each image patch as a vector and concatenate the learnable class token and position code; A3.3 uses a pre-trained visual encoder model to process image patch vectors layer by layer, and utilizes intermediate layer class tokens to extract 12 layers of candidate visual features from the image to be segmented. ; B. Adaptively match visual features at different levels using linguistic features, and perform spectral enhancement using Fourier transform. The specific steps are as follows: B1 Dynamic Selection and Sentence-Level Language Features The most relevant high, medium, and low-level visual features , ; B2 utilizes Fourier transform to perform spectral enhancement on high, medium, and low-level visual features, obtaining the enhanced high, medium, and low-level visual features. , The specific steps are as follows: B2.1 High, medium, and low-level visual features , After performing Fourier transforms to obtain complex spectra, the low-frequency components are shifted to the center of the spectrum to obtain the centered complex spectrum. B2.2 High-frequency and low-frequency filters are used to process high-level and medium- and low-level visual features respectively, and then the filtered complex spectrum is calculated by multiplying it element by element with the decentralized complex spectrum obtained in B2.

1. B2.3 Calculate using inverse Fourier transform to convert the filtered complex spectrum obtained above back to the spatial domain; B2.4 The visual features after Fourier transform are summed with the original features through cross-layer connections and residual connections are then performed to calculate the enhanced high, medium, and low-level visual features. , ; C constructs a multi-layered visual feature system with language perception by fusing visual and linguistic features at different levels across modalities. The specific steps are as follows: C1 utilizes cross-modal fusion of sentence-level language features. and higher visual features after spectral enhancement Constructing high-level visual features with language awareness ; C2 enhances low- and mid-level visual features through cross-level fusion spectrum. , Constructing composite visual features with hierarchical expressive capabilities. ; C3 integrates advanced visual features with accompanying language awareness. and complex visual features with hierarchical expressive power. Constructing multi-level visual features with language awareness ; D utilizes dual-channel information enhancement to construct visual-language fusion features. The specific steps are as follows: D1 refines the visual subdivisions of C3 to obtain multi-level visual features. Refine the details and construct a multi-layered visual feature set. ; D2 Based on refined multi-level visual features and word-level linguistic features By enhancing features through language reinforcement branches and cross-modal interaction, visually perceptive language features are constructed. ; D3 utilizes refined multi-layered visual features Language features with visual perception To enable cross-modal interaction and construct visual-language fusion features The specific steps are as follows: D3.1 Refined Multi-layered Visual Features Acting as Q, the visually perceptual language features obtained from D2 above. Serving as key-value pairs (KV), and processed through a cross-attention mechanism, combined with the original refined multi-layered visual features. Perform residual linking to calculate the visual features that best fit the meaning of the referential text. : ； D3.2 Using a multilayer perceptron, the result obtained in D3.1 above is processed. After undergoing a nonlinear transformation, compared with the original Perform residual connections to calculate visual-language fusion features. : ； E generates a segmentation mask, and the specific steps are as follows: E1 Generates alternative segmentation masks based on visual reinforcement branching. The specific steps are as follows: The specific steps are as follows: E1.1 The refined multi-layered visual features obtained from C1 above. After 3x3 convolutions and alternating upsampling operations The discriminatively enhanced multi-level visual feature representation is calculated. : ； E1.2 Using 1x1 convolution The above results The number of channels is adjusted to the number of candidate masks σ, and the discriminatively enhanced σ-channel multi-level visual features are calculated. : ； E1.3 The sigmoid activation function is used to calculate the probability value of each pixel, and candidate masks are generated based on these probability values. . E2 generates instance query coefficients based on language refinement branches. The specific steps are as follows: E2.1 The visual language fusion features obtained from D above After two convolutions, features are further extracted, and discriminatively enhanced visual-language fusion features are calculated. ; E2.2 Using 1*1 convolutional layers Perform channel transformation to align the number of channels with the number of candidate masks σ, and calculate the discriminatively enhanced σ-channel visual-language fusion features. ; E2.3 using the Tanh activation function Normalization is performed to calculate the instance query coefficient. ; E3 instance query coefficient and alternative masks Multiply to generate the final image segmentation mask. .

2. The image referencing segmentation method based on multi-level feature fusion and dual-channel information enhancement as described in claim 1, dynamically selects language features at the sentence level. The most relevant high, medium, and low-level visual features , The specific steps are as follows: B1.1 Select high-level, medium-level, and low-level visual feature candidate layers from the 12 candidate visual feature layers respectively; B1.2 Utilizing alternative visual features tokens Calculate its relationship with sentence-level language features cosine similarity B1.3 Take the cosine similarity sets from the candidate layers of high, medium, and low-level visual features respectively, and use them as the affinity between the visual features and linguistic features of that layer. ; B1.4 Utilizing visual and linguistic features to assess trustworthiness Calculate the trust score The specific steps are as follows: B1.4.1 The confidence scores from the high, medium, and low-level visual feature candidate layers obtained above are... After processing through two linear layers, a ReLU activation function, and a Sigmoid function, the confidence weights of the corresponding visual feature candidate layers are calculated. ; B1.4.2 The confidence level of the visual feature candidate layers obtained above at each level. After passing through a linear layer, a normalized layer, and a ReLU activation function, the gating weights of the corresponding visual feature candidate layers are calculated. ; B1.4.3 The gating weights of each visual feature candidate layer obtained above. Weighting by trust level After multiplying, summing over all columns yields the confidence score for the corresponding visual feature candidate layer. ; B1.5 Visual features at different levels are dynamically selected using the confidence scores of the visual feature candidate layer. The specific steps are as follows: B1.5.1 Select the layer with the highest confidence score among the candidate layers for high-level visual features, and use its corresponding candidate visual features as high-level visual features. ; B1.5.2 Select the layer with the highest confidence score among the candidate layers for intermediate visual features, and downsample its corresponding candidate visual features by a factor of 2 to calculate the intermediate visual features. ; B1.5.3 Select the layer with the highest confidence score among the candidate low-level visual features, and then downsample its corresponding candidate visual features by a factor of 4 to calculate the low-level visual features. .

3. The image referencing segmentation method based on multi-level feature fusion and dual-channel information enhancement as described in claim 1, which fuses sentence-level language features across modalities. and higher visual features after spectral enhancement Constructing high-level visual features with language awareness The specific steps are as follows: C1.1 The sentence-level linguistic features obtained in A4.1 above Language mapping that includes a linear layer, batch normalization, and LeakyReLU activation function. The mapped sentence-level language features are calculated. : ； C1.2 High-level visual features enhanced by the spectrum obtained in B2 above. Visual projection consisting of a convolutional layer, batch normalization, and ReLU activation function. Calculate the high-level visual features after projection : ) ； C1.3 Mapping sentence-level language features Advanced visual features after projection Multiplication yields higher-level visual features with accompanying language perception. : = 。 4. The image referencing segmentation method based on multi-level feature fusion and dual-channel information enhancement as described in claim 1, which uses cross-level fusion of spectrally enhanced low-level and mid-level visual features... , Constructing composite visual features with hierarchical expressive capabilities. The specific steps are as follows: C2.1 Low-level visual features after spectral enhancement Perform upsampling to obtain the same as Feature representation with the same scale ; C2.2 Spectral Enhancement of Intermediate Visual Features After 1x1 convolution and ReLU activation, it is combined with the upsampled low-level visual features. By adding them together, we can calculate the composite visual features. .

5. The image referencing segmentation method based on multi-level feature fusion and dual-channel information enhancement as described in claim 1, which fuses high-level visual features with accompanying language awareness. and complex visual features with hierarchical expressive power. Constructing multi-level visual features with language awareness The specific steps are as follows: C3.1 builds upon C1 to include advanced visual features with language awareness. After passing through convolutional layers and the ReLU activation function, a more discriminative language-aware algorithm is calculated. ; C3.2 Composite visual features constructed from C2 After upsampling, the result was calculated to be the same as Features of the same scale; C3.3 Add the two results obtained above to calculate the multi-level visual features. .

6. The image referencing segmentation method based on multi-level feature fusion and dual-channel information enhancement as described in claim 1, through visual refinement branches, refines multi-level visual features. Refine the details and construct a multi-layered visual feature set. The specific steps are as follows: D1.1 with multi-layered visual features As a query vector (Q) and key-value pairs (KV), through After the layer normalization module, the data is fed into a multi-head self-attention mechanism. With the original multi-layered visual features Perform residual connections and compute the enhanced feature representation. : ； D1.2 Using a multilayer perceptron The result obtained from D1.1 above Using multilayer perceptron After undergoing a nonlinear transformation, compared with the original Residual connections are performed to calculate the refined multi-level visual features. : 。 7. The image referencing segmentation method based on multi-level feature fusion and dual-channel information enhancement as described in claim 1, based on refined multi-level visual features. and word-level linguistic features By enhancing features through language reinforcement branches and cross-modal interaction, visually perceptive language features are constructed. The specific steps are as follows: D2.1 Language features at the word level As the query vector (Q), the refined multi-level visual features constructed above As key-value pairs (KV), after cross-attention Afterwards, language features at the original word level Perform residual connections to calculate enhanced word-level linguistic features. : ； D2.2 Utilizes a multilayer perceptron to refine the multilayer visual features obtained from D1. After undergoing a nonlinear transformation, compared with the original Perform residual connections to calculate language features with visual perception. : 。

Images