Indicator segmentation method based on visual fine-grained semantic driving cross-modal collaboration

Abstract

The invention relates to the technical field of computers, in particular to an exponential segmentation method based on visual fine-grained semantic-driven cross-modal collaboration, which follows a basic normal form of research in the field of exponential segmentation, and designs a method for carrying out image fine-grained visual enhancement, then carrying out semantic-driven cross-modal collaboration and finally carrying out semantic-driven cross-modal collaboration. And finally, segmenting the mask prediction model. According to an input image, fine-grained visual understanding is enhanced, understanding on a complex spatial position is enhanced, semantic-driven cross-modal collaborative decoding is carried out on the image after fine-grained understanding is enhanced in combination with a text, and then the decoded image is used for final segmentation mask prediction. The method can solve the problems that the correlation between an image area and related language description cannot be fully mined through an existing representative segmentation method, so that fine-grained alignment is insufficient, and a model cannot understand fine-grained correlation such as space and position.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a finger expression segmentation method based on visual fine-grained semantic-driven cross-modal collaboration. Background Technology

[0002] Referring expression segmentation is an emerging research direction in the field of graph-text multimodal computing, aiming to generate segmentation masks for target objects referenced by a given natural language query expression. It typically combines segmentation techniques and natural language processing techniques and is considered one of the most fundamental and challenging multimodal tasks. Early referring expression segmentation methods were based on traditional feature fusion and target segmentation techniques, using LSTM to encode text expressions and generating segmentation masks through simple feature concatenation and convolution operations. With the development of deep learning, it gradually evolved into multimodal feature fusion and end-to-end optimization. This stage of research led to a shift in referring expression segmentation methods from coarse-grained alignment to fine-grained alignment. BRINet dynamically adjusts visual feature weights through a region-aware attention module, enabling the separation of entity and attribute modeling in language descriptions; MABP explicitly models the relationships between objects through a graph attention mechanism. In recent years, with the emergence of Transformer and its variant models, inspired by their powerful encoding capabilities and the compatibility of token sequences as input with various modalities, new referring expression segmentation methods based on Transformer have emerged.

[0003] The Transformer was first proposed in the field of Natural Language Processing (NLP) as a model specifically designed to solve the problem of natural language translation. Leveraging its internal attention mechanism, it enhances the ability to perceive global features. Subsequent research based on the Transformer has led to the design of the VisionTransformer (ViT) model for image feature extraction and the BERT model for text feature extraction. Because ViT can directly use the Transformer for image feature sequence prediction without relying on CNNs, while BERT can generate deep bidirectional language representations through a pre-training-fine-tuning paradigm using stacked Transformer encoders, the Transformer has gradually become the fundamental paradigm in index expression segmentation research due to the unique advantages of these models.

[0004] Existing indexing and expression segmentation methods often combine these network architectures, using ViT or CNN to extract image features and BERT or LSTM to extract text features. Then, they design feature fusion modules with encoder-decoder architectures or image-text matching attention modules to fuse image and text features, finally using an MLP to predict the segmentation mask. Another important advancement in transformer-based methods is the evolution of cross-modal alignment from static to dynamic. However, this approach often fails to fully exploit the correlation between image regions and related linguistic descriptions, leading to insufficient fine-grained alignment and an inability to accurately understand positional and spatial relationships in the indexing expression, resulting in oversegmentation or undersegmentation. For example, in complex scenes, linguistic descriptions may involve dynamic context or ambiguity, but the model struggles to accurately associate key attributes and visual features in the description, causing the model to overemphasize regions in the image unrelated to the expression. Furthermore, some methods (such as Transformer-based decoders), while introducing cross-modal feature fusion modules, fail to fully integrate multi-scale feature interactions guided by language. This insufficient correlation mining can cause the segmentation results generated by the model to deviate from the actual target, even leading to target indexing shift.

[0005] To address the issue that existing indexing expression segmentation methods fail to fully exploit the correlation between image regions and related linguistic descriptions, leading to insufficient fine-grained alignment and an inability to accurately understand positional and spatial relationships within indexing expressions, resulting in oversegmentation or undersegmentation, some studies have focused on designing better feature fusion modules or using text for recursive stepwise reasoning. However, these efforts have neglected the importance of fine-grained visual understanding, resulting in limited performance improvements. Considering the crucial role of understanding fine-grained information (spatial, positional relationships, etc.) in guiding indexing expressions within images, it is essential to design an indexing expression segmentation method that combines visual fine-grained enhancement with semantic-driven cross-modal collaboration. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and propose an indexed expression segmentation method based on visual fine-grained semantic-driven cross-modal collaboration. Following the basic paradigm of indexed expression segmentation research, this invention designs a model that first performs fine-grained visual enhancement of the image, then semantic-driven cross-modal collaboration, and finally segmentation mask prediction. Based on the input image, fine-grained visual understanding is enhanced to improve the understanding of complex spatial locations. The enhanced image, combined with text, is then used for semantic-driven cross-modal collaborative decoding, and finally used for the final segmentation mask prediction. This addresses the problem that existing indexed expression segmentation methods cannot fully exploit the correlation between image regions and related linguistic descriptions, resulting in insufficient fine-grained alignment and the model's inability to understand fine-grained spatial and positional relationships.

[0007] To achieve the above objectives, the present invention adopts the following specific technical solution: The index expression segmentation method based on visual fine-grained semantic-driven cross-modal collaboration provided by this invention includes the following steps, which are performed sequentially as follows: Step 1: Collect the dataset of pointer-based segmentation and construct image-text pairs. It is divided into training set, test set and validation set; Step 2: Design the index expression segmentation model architecture with visual fine-grained enhancement and semantic-driven cross-modal collaboration, including a visual encoder, a text encoder, a fine-grained visual enhancement module, a semantic-driven cross-modal collaboration module, and a segmentation head; The visual encoder uses VIT to extract image features, the text encoder uses BEIT-3 to extract text features, the fine-grained visual enhancement module dynamically learns fine-grained image information using a learnable parameter matrix, and the semantic-driven cross-modal collaboration module is guided by text features, uses global attention, fuses and aligns cross-modal features, and finally inputs them into the segmentation head to achieve pixel-level target segmentation. Step 3: Use the image and text pairs obtained in Step 1 As input to the model, the image and text information are fed into the visual encoder and text encoder in step two, respectively, to obtain their respective feature vectors; For text information, basic cleaning is first performed to remove irrelevant symbols. Then, SentencePiece is used as a word segmentation tool to divide the text into sub-word units. Finally, BEIT-3 is used to extract text features. For images, features are extracted by the VIT visual encoder to obtain the input image. Image features ; Step 4: For the image features obtained in Step 3 This data is then fed into the dynamic query fine-grained enhancement module for enhanced fine-grained understanding; the dynamic query fine-grained enhancement module uses a learnable dynamic query matrix. To enhance fine-grained visual understanding; dynamic query matrix First, initialize randomly, then use image features. and dynamic query matrix Perform cross-modal attention computation to generate a spatial attention map for each region. Dynamic query matrix Each row vector corresponds to a spatial attention distribution in a region, and then the attention map is used. From image features The features of each region are dynamically aggregated to obtain enhanced image features. ; Step 5: For the text features obtained in Step 3 First, text features Original image features extracted with VIT By splicing the features together, we can obtain the spliced ​​features. The splicing method is Then the spliced ​​features Simple feature fusion and alignment is achieved through self-attention. Specifically, the concatenated features first pass through a multi-head self-attention mechanism, followed by residual connections, layer normalization, and then a fully connected feedforward neural network to obtain the simply fused features. ; Step six: Through global attention, the simply fused features are... Compared with image features after fine-grained visual enhancement Perform deep cross-modal alignment fusion; where the features after simple fusion are... As the query for global attention, the enhanced visual features output by the fine-grained visual enhancement module serve as the key and value for global attention, resulting in the deeply fused features. ; Step seven is a semantically driven cross-modal collaboration, accomplished through a global attention mechanism; the original text features are extracted by BEIT-3. As the key and value of global attention, the features are derived from deep fusion. As a query, it is then processed through a feedforward neural network to obtain the final semantically driven cross-modal output features. ; Step 8: Semantic-driven cross-modal output features Features of the original image The corresponding positions are weighted and summed, and the sum is input into the segmentation head to obtain the final segmentation mask; the segmentation head consists of a lightweight convolutional module, which contains two... Convolutional layers, each followed by batch normalization and ReLU activation function, the last one... The convolutional layer compresses the number of channels to 1, mapping each pixel to a scalar value; finally, the probability mask is output through the sigmoid activation function, completing the pixel-level accurate segmentation of the target of interest in the finger representation.

[0008] Furthermore, in step one, the datasets used to collect the specified segmentation data can be RefCOCO, RefCOCO+, RefCOCOg, or gRefCOCO datasets.

[0009] Furthermore, in step three, the image and text pairs obtained in step one are used... As input to the model, respectively and The vectors are fed into the visual encoder and text encoder in step two to obtain their respective feature vectors, as follows: Given an input image The feature map of the input image is obtained by using ViT as the backbone network of the visual encoder. For the input image First, it is linearly projected and flattened into N non-overlapping image blocks to obtain a flat image block sequence. ,in This indicates the number of image blocks. This indicates the resolution size of each image patch. Representative input image Height and width; ViT's Transformer encoder receives a sequence of flattened image patches. Perform global context reasoning and output image feature maps. ; Given input text First, the sentence is segmented into subwords using the SentencePiece tokenizer. Then, each token is mapped to a 1408-dimensional vector using a learnable word embedding matrix, and a learnable positional encoding matrix with dimensions of [missing information]. L represents the number of tokens, which is added element-wise to the token embeddings. A learnable [CLS] tag is then added before the token sequence. The dimension is... This is used to capture global semantics, and then the sequence is fed into BEIT-3 for text feature extraction. The BEIT-3 text encoder consists of 40 layers of MultiwayTransformer, each layer containing a shared self-attention module, a language expert feedforward network, residual connections, and layer normalization. The final output dimension is... Text features The output vector corresponding to the [CLS] token contains the global semantic representation of the text and is a global feature, while the output vector of all tokens, including [CLS], is a local feature.

[0010] Furthermore, in step four, the visual features obtained in step three are... The output vectors corresponding to the [CLS] labels, whose dimensions have been adjusted by a linear layer, are added element-wise and then fed into the dynamic query fine-grained enhancement module. Guided by global semantic features, this enhances the model's fine-grained visual understanding, as follows: For the text features output by BEIT-3, the output vector corresponding to the [CLS] marker has global semantic features. This allows it to pass through a linear layer, transforming its dimensions into... Then Add to For each row, the corresponding elements are summed element by element to obtain image features with global semantic information of the text. Then, pass this feature through a linear layer to restore its dimension to . This dimension is the same as the input image dimension, and then a learnable dynamic query matrix is ​​randomly initialized. Calculate the spatial attention for each region The calculation formula is: ; in, For a learnable parameter matrix, The GeLU activation function is used; the dimensions of the global text features are adjusted to match those of the visual features, and global attention is applied to both the visual and global text features. The global text features serve as the query for global attention, while the visual features serve as the key and value. This is achieved using an attention graph. From visual features The features of each region are dynamically aggregated, and the calculation formula is as follows: ; in, For a learnable parameter matrix, These are the aggregated regional features, and the features of each region are obtained by weighted summation of its corresponding attention maps.

[0011] Furthermore, in step five, the text features obtained in step three are... First, the text features are compared with the original visual features extracted by VIT. Perform a concatenation operation to obtain the borrowed features. The splicing method is Then, the concatenated features are fused and aligned using a multi-head attention mechanism, as follows: First, regarding text features After linear adjustment, the dimension is... Then combine it with image features splicing, splicing to dimensions splicing features Then, the concatenated features are fed into an 8-head multi-head self-attention mechanism for cross-modal feature fusion; the dimension is... The concatenated feature vector is split into 8 parts for use in the 8-head multi-head self-attention mechanism, and the dimension of each part is... The formula for calculating multi-head self-attention is: ; After calculating the attention for each head, the results from the multiple heads are concatenated and output. The concatenation method is as follows: ; Obtain the features after simple fusion .

[0012] Furthermore, in step six, the aggregated region features obtained in step four, enhanced with fine-grained visual understanding, are processed through a global attention mechanism. Features fused with multi-head self-attention mechanism Further integration will be carried out to enhance the model's understanding of complex spatial relationships, as follows: The purpose of this fusion is to aggregate regional features. For the model to function effectively, firstly... Mean pooling is performed to compress the information and adjust its dimensions. Then it goes through an L-layer global attention, where As the key and value of global attention, after dimensional adjustment As a query, the final fused features are obtained. .

[0013] Furthermore, in step seven, cross-modal collaboration is achieved through a global attention mechanism. This module's global attention mechanism is semantically driven, enabling full cross-modal interaction, as detailed below: First, the output of the global attention in step six is ​​augmented with a linear layer to increase its dimensionality, expanding it to match the dimensionality of the text features extracted in step three. Same dimensions, then text features As the key and value of semantic-driven attention, the output of global attention in step six, after dimensionality boosting, serves as the query. After passing through semantic-driven attention, it is then processed by a feedforward neural network, its dimensionality adjusted, and added to the text features. This is followed by a transposed convolution, upsampling to obtain fused features with the same dimensions as the input image, thus yielding the final fused features. .

[0014] Furthermore, step eight is detailed below: First, the fusion features are processed through two... Convolutional layers, each followed by batch normalization and ReLU activation functions, are used to enhance the discriminative power of features while maintaining spatial resolution; then... Compress the channel to Then the Sigmoid activation function is applied to the channel with The features are used to generate the target probability for each pixel, resulting in a probability map. Then with a threshold right Binarization enables pixel-level target segmentation.

[0015] The present invention can achieve the following technical effects: This invention provides a pointer expression segmentation method based on fine-grained visual enhancement and semantic-driven cross-modal collaboration. Following the fundamental paradigm of pointer expression segmentation research, it designs a model that first performs fine-grained visual enhancement of the image, then semantic-driven cross-modal collaboration, and finally segmentation mask prediction. Based on the input image, fine-grained visual understanding is enhanced to improve the understanding of complex spatial locations. The enhanced image, combined with text, is then used for semantic-driven cross-modal collaborative decoding, and finally used for the final segmentation mask prediction. This method addresses the problem that existing pointer expression segmentation methods fail to fully exploit the correlation between image regions and related linguistic descriptions, resulting in insufficient fine-grained alignment and the model's inability to understand fine-grained spatial and positional relationships. Attached Figure Description

[0016] Figure 1 This is a flowchart of the index expression segmentation method based on visual fine-grained enhancement and semantic-driven cross-modal collaboration provided according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of the dynamic query fine-grained enhancement module provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a semantically driven cross-modal collaboration module provided in an embodiment of the present invention. Detailed Implementation

[0017] In the following description, embodiments of the invention will be described with reference to the accompanying drawings. In the description below, the same modules are denoted by the same reference numerals. Where the same reference numerals are used, their names and functions are also the same. Therefore, their detailed description will not be repeated.

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0019] This invention provides an indexed expression segmentation method based on visual fine-grained enhancement and semantic-driven cross-modal collaboration. Following the basic paradigm of indexed expression segmentation, it designs an indexed expression segmentation model that first encodes text and images, then aligns features, and finally performs segmentation prediction. The framework diagram of the indexed expression segmentation method based on visual fine-grained enhancement and semantic-driven cross-modal collaboration is shown below. Figure 1As shown, BEIT-3 and VIT first extract image text features respectively, and then the extracted image text features are sent to the dynamic query fine-grained enhancement module. The structure of the dynamic query fine-grained enhancement module is as follows. Figure 2 As shown, a learnable parameter matrix is ​​used to enhance fine-grained visual understanding of images. The enhanced visual features are then fed together with the text into a designed semantic-driven cross-modal collaborative module for thorough cross-modal feature alignment. Finally, the segmentation head outputs a segmentation mask. Through the synergistic effect of the two modules and the large-scale model BEIT-3, the problem of insufficient fine-grained alignment caused by the inability of existing indexing expression segmentation methods to fully exploit the correlation between image regions and related language descriptions is solved, resulting in better indexing expression segmentation performance.

[0020] Specifically, the indexed expression segmentation method based on visual fine-grained enhancement and semantic-driven cross-modal collaboration includes the following steps, which are executed sequentially: Step 1: Collect commonly used datasets for indexed representation segmentation, such as RefCOCO, RefCOCO+, RefCOCOg, and gRefCOCO, construct image-text pairs, and divide them into training, test, and validation sets before training.

[0021] The purpose of descriptive segmentation is to find objects described by input descriptive text within input images, thus involving data from both text and image modalities. Relevant datasets in this domain are collected, containing a large number of image-text pairs. For example, a photo containing a yellow cup and a red cup on a table, or an image text pair such as "Yellow cup on the left side of the tabletop". .

[0022] Step 2: Design an indexed expression segmentation model architecture that combines fine-grained visual enhancement with semantic-driven cross-modal collaboration. The model includes a visual encoder, a text encoder, a fine-grained visual enhancement module, a semantic-driven cross-modal collaboration module, and a segmentation head.

[0023] The visual encoder uses ViT to extract image features, the text encoder uses BEIT-3 to extract text features, the fine-grained visual enhancement module dynamically learns fine-grained image information using a learnable parameter matrix, and the semantically driven cross-modal collaborative module has the following structure: Figure 3 As shown, the semantically driven cross-modal collaboration module is guided by text features, utilizes global attention, fully integrates and aligns cross-modal features, and finally inputs them into the segmentation head to achieve pixel-level target segmentation.

[0024] Step 3: Use the image and text pairs obtained in Step 1 As input to the model, respectively and The vectors are fed into the visual encoder and text encoder in step two to obtain their respective feature vectors.

[0025] For text information, basic cleaning is first performed to remove irrelevant symbols. Then, SentencePiece is used as a word segmentation tool to divide the text into sub-word units. Finally, BEIT-3 is used to extract full-text features. For images, features are extracted by the VIT visual encoder to obtain the input image. Feature map .

[0026] Given an input image The feature map of the input image is obtained by using ViT as the backbone network of the visual encoder. Specifically, for the input image First, it is linearly projected and flattened into N non-overlapping image blocks to obtain a flat image block sequence. ,in This indicates the number of image blocks. This indicates the resolution size of each image patch. Representative input image The height and width. ViT's Transformer encoder receives a sequence of flattened image patches. Perform global context reasoning and output image feature maps. .

[0027] Given input text First, the sentence is segmented into subwords using the SentencePiece tokenizer. For example, "the cat sat on the cushion" might be split into "cat", "sat", "cushion", and "on". The maximum sequence length is set to 40; any excess is truncated, and any insufficient length is padded with [PAD]. Then, each token is mapped to a 1408-dimensional vector using a learnable word embedding matrix. Finally, a learnable positional encoding matrix with dimensions of [missing information] is generated. , The number of tokens is summed element-wise with the token embeddings, and a learnable [CLS] tag is added before the token sequence. The dimension is... This is used to capture global semantics. The sequence is then fed into BEIT-3 for text feature extraction. BEIT-3's text encoder consists of a 40-layer Multiway Transformer, each layer containing a shared self-attention module, a language expert feedforward network, residual connections, and layer normalization. The final output dimension is... Text features The output vector corresponding to the [CLS] token contains the global semantic representation of the text and is a global feature, while the output vectors of all tokens (including [CLS]) are local features.

[0028] Step four, process the visual features obtained in step three. The output vectors corresponding to the [CLS] labels whose dimensions have been adjusted by the linear layer are added element by element, and then fed into the dynamic query fine-grained enhancement module. Under the guidance of global semantic features, the model's fine-grained visual understanding is enhanced.

[0029] For the image features obtained in step three This data is then fed into a designed dynamic query fine-grained enhancement module for enhanced fine-grained understanding. This module consists of a learnable dynamic query matrix. To enhance fine-grained visual understanding. The matrix is ​​first randomly initialized, and then visual features are... and dynamic query matrix Perform cross-modal attention computation to generate a spatial attention map for each region. Each row vector of this matrix corresponds to a spatial attention distribution within a region. Then, the attention map is used to analyze visual features. The features of each region are dynamically aggregated to obtain enhanced visual features. .

[0030] Specifically, for the text features output by BEIT-3, the output vector corresponding to the [CLS] marker (which has global semantic features) This allows it to pass through a linear layer, transforming its dimensions into... Then Add to For each row, the corresponding elements are summed element by element to obtain image features with global semantic information of the text. Then, this feature is passed through a linear layer to restore its dimension to . This dimension is the same as the input image dimension, and then a learnable dynamic query matrix is ​​randomly initialized. Then calculate the spatial attention for each region. The calculation formula is: ; (1) in, For a learnable parameter matrix, The GeLU activation function is applied, and then the dimensions of the global text features are adjusted to match those of the visual features. Global attention is then applied to both the visual and global text features, with the global text features serving as the query and the visual features as the key and value. This is then performed using an attention graph. From visual features The features of each region are dynamically aggregated, and the calculation formula is as follows: ; (2) in For a learnable parameter matrix, These are the aggregated regional features, and the features of each region are obtained by weighted summation of its corresponding attention maps.

[0031] Step 5: For the text features obtained in Step 3 First, the text features are compared with the original visual features extracted by VIT. Perform a concatenation operation to obtain the borrowed features. The splicing method is Then, the concatenated features are fused and aligned using a multi-head attention mechanism.

[0032] For the text features obtained in step three First, the text features are compared with the original visual features extracted by VIT. Perform a concatenation operation to obtain the borrowed features. The splicing method is The concatenated features are then fused and aligned using self-attention. Specifically, the concatenated features first pass through a multi-head self-attention mechanism, followed by residual connections, layer normalization, and then a simple fully connected feedforward neural network. This yields the simply fused features. .

[0033] The specific process is as follows: first, for text features... First, it is linearly adjusted to have a dimension of 1. Then combine it with image features splicing, splicing to dimensions splicing features Then, the concatenated features are fed into an 8-head multi-head self-attention mechanism for cross-modal feature fusion. First, the dimension is... The concatenated feature vector is split into 8 parts for use in the 8-head multi-head self-attention mechanism, and the dimension of each part is... The formula for calculating multi-head self-attention is: ; (3)

[0034] After calculating the attention for each head, the results from the multiple heads are concatenated and output. The concatenation method is as follows: ; (4) Obtain the features after simple fusion .

[0035] Step six: Using a global attention mechanism, aggregate the region features obtained in step four that have been enhanced with fine-grained visual understanding. Features fused with multi-head self-attention mechanism Further integration is needed to enhance the model's understanding of complex spatial relationships.

[0036] By using global attention, the features after simple fusion are... Image features enhanced with fine-grained visual enhancement Perform deep cross-modal alignment fusion. The features after simple fusion are... As the query for global attention, the enhanced visual features output by the fine-grained visual enhancement module serve as the key and value for global attention. This results in the deeply fused features. .

[0037] The specific process is as follows: the purpose of this fusion part is to aggregate the features of the region. To ensure the model functions effectively, the first step is to... Mean pooling is performed to compress the information and adjust its dimensions. Then it goes through an L-layer global attention, where As the key and value of global attention, after dimensional adjustment As a query, the final fused features are obtained. .

[0038] Step seven is semantically driven cross-modal collaboration, which also achieves cross-modal collaboration through a global attention mechanism. This module's global attention mechanism is semantically driven, enabling full cross-modal interaction. The original text features extracted by BEIT-3 are used in this step. As the key and value of global attention, the features output by the previous global attention are... As a query, it is then passed through a feedforward neural network to obtain the final semantically driven cross-modal output features. .

[0039] The specific implementation process is as follows: First, the output of the global attention in step six is ​​subjected to dimensionality enhancement through a linear layer, expanding its dimension to match the text features extracted in step three. Same dimensions, then text features As the key and value of semantic-driven attention, the output of global attention in step six, after dimensionality boosting, serves as the query. After passing through semantic-driven attention, it is then processed by a feedforward neural network, its dimensionality adjusted, and added to the text features. This is followed by a transposed convolution, upsampling to obtain fused features with the same dimensions as the input image, thus yielding the final fused features. .

[0040] Step 8: The final fusion features are obtained. A weighted fusion is performed between the original visual features and the data, which is then input into the segmentation head to obtain a segmentation mask, leading to the final segmentation result. Specifically, the fused features are first processed through two... Convolutional layers, each followed by batch normalization and ReLU activation functions, are used to enhance the discriminative power of features while maintaining spatial resolution. Then, after... Compress the channel to Then the Sigmoid activation function is applied to the channel with The features are used to generate the target probability for each pixel, resulting in a probability map. Then, using a threshold... right Binarization enables pixel-level target segmentation.

[0041] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0042] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

[0043] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A pointer expression segmentation method based on visual fine-grained semantic-driven cross-modal collaboration, characterized in that, Includes the following steps: Step 1: Collect the dataset of pointer-based segmentation and construct image-text pairs; Step 2: Design the representation segmentation model architecture, including a visual encoder, a text encoder, a fine-grained visual enhancement module, a semantic-driven cross-modal collaboration module, and a segmentation head; Step 3: Input the image information and text information of the image-text pair into the visual encoder and text encoder respectively to obtain image features. and text features ; Step 4: Extract image features Feed it into the dynamic query fine-grained enhancement module, using a learnable dynamic query matrix. Enhanced fine-grained visual understanding yields enhanced image features. ; Step 5: Extract text features Image features By splicing the features together, we can obtain the spliced ​​features. Then, feature fusion and alignment are performed using self-attention to obtain a simple fused feature. ; Step six: Through global attention, the simply fused features are... Enhanced image features as a query for global attention As the keys and values ​​for global attention, the features obtained after deep fusion are obtained. ; Step 7, Text Features Features after deep fusion serve as both keys and values ​​for global attention. As a query, the semantically driven cross-modal output features are obtained through a feedforward neural network. ; Step 8: Semantic-driven cross-modal output features Image features Perform a weighted sum at the corresponding positions, input it into the segmentation head to obtain the segmentation mask, and complete the segmentation of the target of interest in the instruction representation.

2. The index expression segmentation method based on visual fine-grained semantic-driven cross-modal collaboration according to claim 1, characterized in that, In step one, the datasets for collecting the specified segmentation can be RefCOCO, RefCOCO+, RefCOCOg, or gRefCOCO datasets.

3. The index expression segmentation method based on visual fine-grained semantic-driven cross-modal collaboration according to claim 1, characterized in that, In step two, the visual encoder uses VIT to extract image features, the text encoder uses BEIT-3 to extract text features, the fine-grained visual enhancement module dynamically learns fine-grained image information using a learnable parameter matrix, and the semantic-driven cross-modal collaboration module is guided by text features, uses global attention to fuse and align cross-modal features, and finally inputs them into the segmentation head to achieve pixel-level target segmentation.

4. The index expression segmentation method based on visual fine-grained semantic-driven cross-modal collaboration according to claim 1, characterized in that, In step three, the image and text pairs obtained in step one are used. As input to the model, respectively and The images are fed into the visual encoder and text encoder in step two to obtain image features. and text features The details are as follows: For text information, basic cleaning is first performed to remove irrelevant symbols. Then, SentencePiece is used as a word segmentation tool to divide the text into sub-word units. Finally, BEIT-3 is used to extract text features. For images, features are extracted by the VIT visual encoder to obtain the input image. Image features ; Given an input image The feature map of the input image is obtained by using VIT as the backbone network of the visual encoder. For the input image First, it is linearly projected and flattened into N non-overlapping image blocks to obtain a flat image block sequence. ,in This indicates the number of image blocks. This indicates the resolution size of each image patch. Representative input image Height and width; ViT's Transformer encoder receives a sequence of flattened image patches. Perform global context reasoning and output image feature maps. ; Given input text First, the sentence is segmented into sub-words using the SentencePiece tokenizer. Then, each token is mapped to a 1408-dimensional vector using a learnable word embedding matrix, and a learnable positional encoding matrix with dimensions of [missing information]. L represents the number of tokens, which is added element-wise to the token embeddings. A learnable CLS marker is then added before the token sequence. The dimension is... This is used to capture global semantics, and then the sequence is fed into BEIT-3 for text feature extraction. BEIT-3's text encoder consists of a 40-layer Multiway Transformer, each layer containing a shared self-attention module, a language expert feedforward network, residual connections, and layer normalization. The final output dimension is... Text features The output vector corresponding to the CLS token contains the global semantic representation of the text and is a global feature, while the output vectors of all tokens, including CLS, are local features.

5. The index expression segmentation method based on visual fine-grained semantic-driven cross-modal collaboration according to claim 4, characterized in that, In step four, the dynamic query matrix is ​​performed. First, initialize randomly, then use image features. and dynamic query matrix Perform cross-modal attention computation to generate a spatial attention map for each region. Dynamic query matrix Each row vector corresponds to a spatial attention distribution in a region, and then the attention map is used. From image features The features of each region are dynamically aggregated to obtain enhanced image features. ; The visual features obtained in step three The output vectors corresponding to the CLS tags whose dimensions have been adjusted by the linear layer are added element-wise, and then fed into the dynamic query fine-grained enhancement module. Guided by global semantic features, the module enhances the model's fine-grained visual understanding, as follows: For the text features output by BEIT-3, the output vector corresponding to the CLS marker has global semantic features. This allows it to pass through a linear layer, transforming its dimensions into... Then Add to For each row, the corresponding elements are summed element by element to obtain image features with global semantic information of the text. Then, pass this feature through a linear layer to restore its dimension to . This dimension is the same as the input image dimension, and then a learnable dynamic query matrix is ​​randomly initialized. Calculate the spatial attention for each region The calculation formula is: ； in, For a learnable parameter matrix, The GeLU activation function is used; the dimensions of the global text features are adjusted to match those of the visual features, and global attention is applied to both the visual and global text features. The global text features serve as the query for the global attention function, while the visual features serve as the key and value. This is achieved using an attention graph. From visual features The features of each region are dynamically aggregated, and the calculation formula is as follows: ； in, For a learnable parameter matrix, These are the aggregated regional features, and the features of each region are obtained by weighted summation of its corresponding attention maps.

6. The index expression segmentation method based on visual fine-grained semantic-driven cross-modal collaboration according to claim 5, characterized in that, In step five, the text features obtained in step three are... First, the text features are compared with the original visual features extracted by VIT. Perform a concatenation operation to obtain the borrowed features. The splicing method is Then, the concatenated features are fused and aligned using a multi-head attention mechanism, as follows: First, regarding text features After linear adjustment, the dimension is... Then combine it with image features splicing, splicing to dimensions splicing features Then, the concatenated features are fed into an 8-head multi-head self-attention mechanism for cross-modal feature fusion; the dimension is... The concatenated feature vector is split into 8 parts for use in the 8-head multi-head self-attention mechanism, and the dimension of each part is... The formula for calculating multi-head self-attention is: ； After calculating the attention for each head, the results from the multiple heads are concatenated and output. The concatenation method is as follows: ； Obtain the features after simple fusion .

7. The index expression segmentation method based on visual fine-grained semantic-driven cross-modal collaboration according to claim 6, characterized in that, In step six, the aggregated region features obtained in step four, enhanced with fine-grained visual understanding, are processed through a global attention mechanism. Features fused with multi-head self-attention mechanism Further integration will be carried out to enhance the model's understanding of complex spatial relationships, as follows: The purpose of this fusion is to aggregate regional features. For the model to function effectively, firstly... Mean pooling is performed to compress the information and adjust its dimensions. Then it goes through an L-layer global attention, where As the keys and values ​​for global attention, after dimensional adjustment As a query, the final fused features are obtained. .

8. The index expression segmentation method based on visual fine-grained semantic-driven cross-modal collaboration according to claim 7, characterized in that, In step seven, cross-modal collaboration is achieved through a global attention mechanism. This module's global attention mechanism is semantically driven, enabling full cross-modal interaction, as detailed below: First, the output of the global attention in step six is ​​augmented with a linear layer to increase its dimensionality, expanding it to match the dimensionality of the text features extracted in step three. Same dimensions, then text features As the key and value of semantically driven attention, the output of global attention in step six, after dimensionality boosting, serves as the query. After passing through semantically driven attention, it is then processed by a feedforward neural network, its dimensionality adjusted, and added to the text features. This is followed by a transposed convolution, upsampling to obtain fused features with the same dimensionality as the input image, resulting in the final fused features. .

9. The index expression segmentation method based on visual fine-grained semantic-driven cross-modal collaboration according to claim 8, characterized in that, Step eight is as follows: The segmentation head consists of a lightweight convolutional module, which contains two convolutional modules in sequence. Convolutional layers, each followed by batch normalization and ReLU activation function, the last one... Convolutional layers compress the number of channels to Each pixel is mapped to a scalar value, and finally a probability mask is output through the Sigmoid activation function. First, the fusion features are processed through two... Convolutional layers, each followed by batch normalization and ReLU activation functions, are used to enhance the discriminative power of features while maintaining spatial resolution; then... Compress the channel to Then the Sigmoid activation function is applied to the channel with The features are used to generate the target probability for each pixel, resulting in a probability map. Then with a threshold right Binarization enables pixel-level target segmentation.

Images