Information-driven target segmentation method based on modal fusion and self-evolution fine-tuning
By introducing a modal fusion adapter and a lightweight decoder, the high cost of information-driven target segmentation methods when deployed across multiple scenarios is solved, achieving efficient multimodal feature interaction and performance improvement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NORTHWESTERN POLYTECHNICAL UNIV
- Filing Date
- 2024-07-15
- Publication Date
- 2026-05-29
AI Technical Summary
Existing information-driven target segmentation methods are costly when deployed across multiple scenarios and lack effective multimodal interaction mechanisms, which limits their versatility.
A modal fusion adapter is introduced, which includes a spatial prior module and a multimodal attention module. Combined with a lightweight task-specific decoder, it enhances the interactivity of the visual language encoder and achieves performance improvement with only a few parameter updates.
With only 1.61% to 3.38% of the parameters updated, it achieves performance comparable to existing fully fine-tuned methods, improving the model's cross-scenario applicability and efficiency.
Smart Images
Figure CN118799576B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, specifically relating to an information-driven target segmentation method based on modality fusion and self-evolutionary fine-tuning. Background Technology
[0002] Information-driven object segmentation aims to predict the mask of a target object based on a given natural language text and an input image. This task differs from semantic segmentation, which labels each pixel in an image with a label from a fixed set of words. Instead, information-driven object segmentation requires identifying the object indicated by the linguistic representation, which increases the task's complexity due to its arbitrary context length and involvement of open-world vocabulary (such as object names, attributes, locations, etc.).
[0003] Recent research has demonstrated the effectiveness of fine-tuning general pre-trained models for information-driven object segmentation tasks. However, these methods require a separate copy of the fine-tuned model parameters for each dataset, making model deployment across multiple scenarios costly. This problem is particularly pronounced for large-scale pre-trained models containing hundreds of millions to trillions of parameters. Although various training methods for efficient parameter fine-tuning have been proposed, most are limited to unimodal tasks or simple classification tasks, with few studies focusing on dense prediction tasks and interactions between different modalities, thus limiting their generalizability. Summary of the Invention
[0004] To overcome the shortcomings of existing technologies, this invention provides an information-driven target segmentation method based on modality fusion and self-evolutionary fine-tuning. First, a modality fusion adapter that does not require pre-training is introduced and integrated into the original architecture of the pre-trained model to facilitate interaction between the visual and language encoders. The modality fusion adapter has two customized modules: (1) a spatial prior module for capturing the local semantics (spatial prior) of the intermediate layer feature maps; and (2) a multimodal attention module to facilitate information interaction between visual and language modalities. Second, a lightweight task-specific decoder is designed for information-driven target segmentation to further align visual and textual features. This invention can be seamlessly integrated into any pre-trained visual-language model to enhance and interact with their intermediate features, and achieves performance comparable to existing fully fine-tuned methods with only 1.61% to 3.38% parameter updates.
[0005] The technical solution adopted by this invention to solve its technical problem is as follows:
[0006] Step 1: Image and text feature extraction module; given an image and text, extract features using the image encoder and text encoder respectively;
[0007] Step 2: Modal fusion adapter;
[0008] Step 3: Customize the decoder for the task;
[0009] Step 4: Training objective.
[0010] Preferably, step 1 specifically comprises:
[0011] Step 1-1: Image encoder;
[0012] For a given image The image is fed into an image encoder to extract visual features. These represent the height and width of the image, respectively; for CNNs, the final... Stage visual characteristics For the visual Transformer-ViT, the ViT Transformer encoder is evenly divided into... Blocks, each block contains One coding layer, Represents the number of layers in the feature extractor; use the last one. The output of the blocks is used for feature interaction; the multi-level visual features of different blocks in ViT or the visual features of different stages in CNN are used as inputs to the modality fusion adapter and decoder in the framework for multimodal feature alignment.
[0013] Steps 1-2: Text Encoder;
[0014] For the input text information Similar to the image encoder, the Transformer encoder is divided into evenly distributed sections. Blocks, and extract text features from different blocks of the Transformer. in It is the feature dimension. The length of the expression; the input text sequence is surrounded by [SOS] and [EOS] markers; the representation of the last layer of the Transformer at the [EOS] markers is used as a global text feature. , Represents the feature dimension.
[0015] Preferably, step 2 specifically comprises:
[0016] Given visual features and text features First, the dimensions of visual and textual features are adjusted through scaling layers. This process can be formalized as follows:
[0017]
[0018]
[0019] in This indicates the zoom-in operation of the scaling layer. Represents a linear mapping layer; Indicates the first Visual characteristics of the stage Indicates the first Textual features of the stage;
[0020] The feature fusion via interactors can be expressed mathematically as follows:
[0021]
[0022]
[0023] in Represents an interactor; Representing the The adapter at each stage fuses visual features. Representing the The adapter of the stage integrates text features. Indicates the first The adapter at each stage fuses visual features. Indicates the first The adapter of the first stage integrates text features; where the first stage is... and The initial value is and The dimensionality is restored through scaling layers and linear projection, and residual connections are made with the original features of the next stage of the backbone network. This can be expressed mathematically as follows:
[0024]
[0025]
[0026]
[0027]
[0028] in This indicates the scaling operation of the scaling layer. Represents a linear mapping layer. Indicates the first Visual characteristics of the stage Indicates the first Textual features of the stage; Indicates the th after residual connection Phase-based fusion visual features Indicates the th after residual connection The fusion text features of the stages;
[0029] Step 2-1: Scaling the layer;
[0030] Features of the ViT intermediate layer from Plastic And convolutions are used to construct scaling layers; 2x2 convolutions with a stride of 2 are used to reduce the size of the feature maps; when extracting features from the intermediate layers of the backbone network, scaling layers are used to adjust the feature map size of the visual encoder. This process is formalized as follows:
[0031]
[0032] in , These represent the height and width of the feature maps in the visual backbone network, respectively. and These represent convolution and deconvolution operations, respectively. Indicates the first Stage feature map height, Indicates the first The width of the stage feature map; before adding the features back to the backbone network, a scaling layer is used to shrink them, which is the reverse of the scaling process;
[0033] Step 2-2: Interactor;
[0034] The interactor, based on an attention mechanism and a feedforward network, uses the original modality feature as the query for each feature from different modalities and retrieves the key and value from the other modality. The specific formula is as follows:
[0035]
[0036]
[0037]
[0038]
[0039] in Representing the The adapter at each stage fuses visual features. Representing the The adapter at each stage integrates text features. Indicates the first The adapter at each stage fuses visual features. Indicates the first The adapter integrates text features at each stage; and These represent the visual and textual features output by the interactor, respectively; MHSA represents multi-head self-attention, MHCA represents multi-head cross-attention, and FFN represents a fully connected layer.
[0040] Preferably, step 3 specifically comprises:
[0041] Step 3-1: Align the layers of the module;
[0042] Given multiple adapters from different stages, the visual features are fused. and global text features The feature dimensions are aligned through convolution, and then hierarchical fusion of multimodal features at different stages is obtained through a cross-attention mechanism. For the hierarchical fusion features, they are concatenated and aggregated using a 1×1 convolutional layer.
[0043]
[0044]
[0045]
[0046] [,] represents the concatenation operation. This represents the fused visual features obtained by formula (7). This represents the scaled-down visual features. Multimodal features representing the output of cross-attention. This represents the hierarchical fusion of multimodal features after concatenation;
[0047] A 2D spatial coordinate feature and The images are stitched together and then fused using a 3×3 convolution to obtain visual features. The specific formula is as follows:
[0048]
[0049] Then merge visual features Flatten them into a sequence;
[0050] Step 3-2: Global alignment module;
[0051] After obtaining multimodal features through hierarchical alignment, the attention mechanism of Transformer is used to fuse textual information and corresponding visual features, thereby fusing visual features. and global text features As input, first send to and A fixed sinusoidal spatial location code is added; subsequently, multimodal features are generated through self-attention and cross-attention modules. To capture global context information:
[0052]
[0053]
[0054] in This represents the features generated after using the self-attention mechanism. Representing global text features, MHSA represents multi-head self-attention, MHCA represents multi-head cross-attention, and FFN represents fully connected layers; the final multimodal features are obtained. It was used in subsequent information-driven target segmentation tasks;
[0055] Step 3-3: Output mapping layer;
[0056] Use an output mapping layer for multimodal features and global text features The transformation is as shown in the following equation:
[0057]
[0058]
[0059]
[0060] UpSample means Double upsampling, then convolution to Convert to , where N= and will Transformed into a 2D convolutional layer through linear projection Then Divided into weights and bias K represents the kernel size of the convolutional layer, and D represents the feature dimension, thus making As a two-dimensional convolutional layer, it is used to represent multimodal image pixels. This is converted into the final mask prediction.
[0061] Preferably, step 4 specifically comprises:
[0062] The text-to-pixel contrast loss is used, as shown in the following formula:
[0063]
[0064] in, Represents multimodal pixel features; and This represents the number of categories with labels of 1 and 0. The definition is as follows:
[0065]
[0066] Here It's the sigmoid function; the segmentation result is adjusted to... And it is upsampled to the original image size;
[0067] After obtaining the loss function, backpropagation is performed using the loss function. Multiple models are then trained, and the optimal model is selected for application. The final output of the model is the information-driven target segmentation result.
[0068] Preferably, the Both are 416.
[0069] The beneficial effects of this invention are as follows:
[0070] This invention proposes a modality fusion adapter that can be seamlessly integrated into any pre-trained vision-language model, enhancing the model's intermediate feature interaction capabilities and achieving performance comparable to existing full-scale fine-tuning methods with only 1.61% to 3.38% of the parameters updated. Attached Figure Description
[0071] Figure 1 This is a framework diagram of the method of the present invention.
[0072] Figure 2 The result is the target region segmentation on a denoted image segmentation dataset using the method of this invention. Detailed Implementation
[0073] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0074] This invention proposes an information-driven target segmentation method based on modality fusion and self-evolutionary fine-tuning. First, a modality fusion adapter that does not require pre-training is introduced, which can be seamlessly integrated into the original architecture of a pre-trained model to facilitate interaction between visual and language encoders. The modality fusion adapter has two customized modules: (1) a spatial prior module for capturing the local semantics (spatial prior) of intermediate layer feature maps; and (2) a multimodal attention module to facilitate information interaction between visual and language modalities. Second, a lightweight task-specific decoder is designed for information-driven target segmentation to further align visual and textual features. This invention can be seamlessly integrated into any pre-trained visual-language model to enhance and interact with their intermediate features. Furthermore, it achieves performance comparable to existing fully fine-tuned methods with only 1.61% to 3.38% parameter updates.
[0075] like Figure 1 As shown, the technical solution adopted by the present invention to solve its technical problem includes the following steps:
[0076] Step 1: Image and Text Feature Extraction Module. Given an image and text, extract their features using an image encoder and a text encoder, respectively.
[0077] Step 1-1: Image encoder;
[0078] For a given image The image is fed into an image encoder to extract visual features. These represent the height and width of the image, respectively; for CNNs, the final... Stage visual characteristics For the visual Transformer-ViT, the ViT Transformer encoder is evenly divided into... Blocks, each block contains One coding layer, Represents the number of layers in the feature extractor; use the last one. The output of the blocks is used for feature interaction; the multi-level visual features of different blocks in ViT or the visual features of different stages in CNN are used as inputs to the modality fusion adapter and decoder in the framework for multimodal feature alignment.
[0079] Steps 1-2: Text Encoder;
[0080] For the input text information Similar to the image encoder, the Transformer encoder is divided into evenly distributed sections. Blocks, and extract text features from different blocks of the Transformer. in It is the feature dimension. The length of the expression; the input text sequence is surrounded by [SOS] and [EOS] markers; the representation of the last layer of the Transformer at the [EOS] markers is used as a global text feature. , Represents the feature dimension;
[0081] Step 2: Modal fusion adapter;
[0082] Given visual features and text features First, the dimensions of visual and textual features are adjusted through scaling layers. This process can be formalized as follows:
[0083]
[0084]
[0085] in This indicates the zoom-in operation of the scaling layer. Represents a linear mapping layer; Indicates the first Visual characteristics of the stage Indicates the first Textual features of the stage;
[0086] The feature fusion via interactors can be expressed mathematically as follows:
[0087]
[0088]
[0089] in Represents the interactive element. Representing the The adapter at each stage fuses visual features. Representing the The adapter of the stage integrates text features. Indicates the first The adapter at each stage fuses visual features. Indicates the first The adapter of the first stage integrates text features; where the first stage is... and The initial value is and Finally, the dimensions are restored through scaling layers and linear projection, and residual connections are made with the original features of the next stage (block) of the backbone network. This process can be expressed mathematically as follows:
[0090]
[0091]
[0092]
[0093]
[0094] in This indicates the scaling operation of the scaling layer. Represents a linear mapping layer. Indicates the first Visual characteristics of the stage Indicates the first Textual features of the stage; Indicates the th after residual connection Phase-based fusion visual features Indicates the th after residual connection The text features of the phases are integrated; next, the architecture of the scaling layer and the interactor will be described in detail.
[0095] Step 2-1: Scaling the layer;
[0096] After extracting features from the image and text encoders, a module was designed to perform dimensionality transformation on the visual and text features, taking into account time complexity and spatial priority.
[0097] For ViT, the simple architectural design performs poorly due to the lack of vision-specific inductive biases. Recent research has shown that convolution can help the Transformer capture the local spatial context of an image. Therefore, features from the intermediate layers of ViT can be extracted from... Plastic Furthermore, convolutions are used to construct scaling layers. Additionally, for ResNet, as the input resolution increases, the feature maps in the first two stages can become very large, resulting in excessive length that cannot be processed as input to the attention algorithm. Therefore, 2x2 convolutions with a stride of 2 are used to reduce the size of the feature maps. To unify the size and enlarge the smaller feature maps, 2x2 deconvolutions with a stride of 2 are used to enrich the information. In short, scaling layers are used to adjust the feature map size of the visual encoder when extracting features from the intermediate layers of the backbone network. This process can be formalized as:
[0098]
[0099] in , It is one of the height and width of a feature map in the visual backbone network. Indicates the first Stage feature map height, Indicates the first Stage feature map width; and These represent convolution and deconvolution operations, respectively. With...
[0100] This involves interactive operations between feature maps of different modalities. Then, before adding the features back to the backbone network, a scaling layer is used to shrink them—the inverse of the scaling process.
[0101] Step 2-2: Interactor;
[0102] After processing features through a scaling layer, an interactor was designed to facilitate the interaction of modal information between the visual encoder and the text encoder. This enhances the features in the middle of the pre-trained backbone network while keeping the original parameters fixed. Specifically, the interactor is based on an attention mechanism and a feedforward network. For each feature from different modalities, the original modal feature is used as the query, and the key and value are obtained from the other modality. The specific formula is as follows:
[0103]
[0104]
[0105]
[0106]
[0107] in Representing the The adapter at each stage fuses visual features. Representing the The adapter at each stage integrates text features. Indicates the first The adapter at each stage fuses visual features. Indicates the first The adapter integrates text features at each stage; and These represent the visual and textual features output by the interactor, respectively. MHSA stands for Multi-head Self-Attention, MHCA stands for Multi-head Cross-Attention, and FFN stands for Fully Connected Layer.
[0108] Step 3: Task-Specific Decoder. A lightweight task-specific decoder was designed for information-driven object segmentation to further align visual and textual features. It consists of a hierarchical alignment module, a global alignment module, and an output mapping module.
[0109] Step 3-1: Align the layers of the module;
[0110] Given multiple visual features from different stages and global text features The feature dimensions are aligned through convolution, and then a cross-attention mechanism is used to obtain hierarchical fusion of multimodal features at different stages. For hierarchical fusion features, they are simply concatenated and aggregated using a 1×1 convolutional layer:
[0111]
[0112]
[0113]
[0114] in represents a convolution operation used to unify the dimensionality of features from different stages; MHCA stands for Multi-Head Cross Attention; and [,] is a concatenation operation. This represents the fused visual features obtained by formula (7). This represents the scaled-down visual features. Multimodal features representing the output of cross-attention. The hierarchical fusion of multimodal features after concatenation. Finally, a 2D spatial coordinate feature is generated. and The images are stitched together and then fused using a 3×3 convolution to obtain the fused visual features. The specific formula is as follows:
[0115]
[0116] Then, visual features are integrated. It is flattened into a sequence for later use.
[0117] Step 3-2: Global alignment module;
[0118] After obtaining multimodal features through hierarchical alignment, the attention mechanism of Transformer is used to fuse rich textual information and corresponding visual features. This is to achieve the fusion of visual features. and global text features As input, first send to and A fixed sinusoidal spatial location code is added. Subsequently, multimodal features are generated using self-attention and cross-attention modules. To capture global context information:
[0119]
[0120]
[0121] in This represents the features generated after using the self-attention mechanism. Representing global text features, MHSA represents multi-head self-attention, MHCA represents multi-head cross-attention, and FFN represents fully connected layers; the final multimodal features are obtained. It was used in subsequent information-driven target segmentation tasks.
[0122] Step 3-3: Output mapping layer;
[0123] To obtain mask predictions for each pixel based on the corresponding semantic information, an output mapping layer is used to process the multimodal features. and global text features The transformation is performed as shown in the following equation.
[0124]
[0125]
[0126]
[0127] UpSample means Double upsampling, then convolution to Convert to image pixel representation , where N= and will Transformed into a 2D convolutional layer through linear projection Then Divided into weights and bias Here, K represents the kernel size of the convolutional layer, and D represents the feature dimension. This makes... As a two-dimensional convolutional layer, it is used to represent multimodal image pixels. This is converted into the final mask prediction.
[0128] Step 4: Training Objective;
[0129] Considering that the CLIP pre-training strategy primarily aligns text representations with image-level representations, this approach may not be optimal for directional image segmentation tasks. Therefore, a novel training objective—text-to-pixel contrastive loss—is introduced to finer-tune the relationship between the two modalities. This method enables a more accurate capture of the subtle connections between text descriptions and image content. The formula for the text-to-pixel contrastive loss is as follows:
[0130]
[0131] Here and This represents the number of categories with labels of 1 and 0. Represents multimodal features; This represents the two-dimensional convolutional layer obtained through formula (22). The definition is as follows:
[0132]
[0133] Here It's the sigmoid function; the segmentation result is adjusted... for And it is upsampled to the original image size.
[0134] After obtaining the loss function, backpropagation is performed using the loss function. Multiple models can be obtained through training. The optimal model (the one with the best performance on the test set) is selected for application. Finally, information-driven target segmentation can be performed based on the output of the model.
[0135] Example:
[0136] This invention provides an information-driven target segmentation method based on modality fusion adapter and self-evolutionary parameter fine-tuning, the specific process of which is as follows:
[0137] 1. Image feature extraction;
[0138] Given an image of a natural scene, first adjust the input image to... Then, visual features are obtained through the backbone network ViT. .
[0139] 2. Text feature extraction;
[0140] Given a text, fix it to 17 words, truncate any words exceeding this limit, and use markers to delimit any words below this limit. <mask>The text is then completed and fed into a text encoder to obtain text features. .
[0141] 3. Dimensional alignment;
[0142] To enable feature-based interaction in the interactor, visual and textual features need to be aligned. Specifically, the visual features are fed into the scaling layer described in formula (1) to obtain... Then the text features are fed into the linear layer in formula (2) to obtain... .
[0143] 4. Feature interaction;
[0144] After obtaining visual and textual features, the aligned features are fused. Specifically, the fused features are... and The data is fed into the feature interactor (Equations (3) and (4)) and finally outputs the image fusion features.
[0145] Text fusion features .
[0146] Next, we will fuse features. and Perform dimensional adjustments to restore it to its original size; this is the inverse operation of the dimensional alignment described above, ultimately yielding... and Then the features are added back to the backbone network. Visual features from the three stages are extracted and fused to obtain the final visual features. and and global text features .
[0147] 5. Layer alignment module;
[0148] After obtaining visual features and global text features at different stages, they are fed into the hierarchical alignment module of the task-customized decoder for alignment. First, the visual features are transformed using convolutional layers to obtain... The three convolutional output channels are 256, 512, and 1024 respectively, and then the output channels of different stages are... and Multi-head cross attention Finally, splice it together. And send 1 Convolution is used to aggregate the results .get Next, a 2D spatial coordinate feature and The images are stitched together and then fused using a 3×3 convolution to obtain visual features. ,Then It is flattened into a sequence for later use.
[0149] 6. Global feature alignment;
[0150] get Next, it will be combined with global text features. The input is fed into the Transformer decoder (Equations (18)-(19)) to obtain the multimodal features used for segmentation. .
[0151] 7. Target mask generation;
[0152] To obtain mask predictions for each pixel based on the corresponding semantic information, an output mapping layer is used to process the multimodal features. and global text features The transformation is performed. Specifically, first, according to formula (20), the transformation is performed... conduct Double upsampling Then Transformed by linear projection And divided into weights and bias Finally, As a two-dimensional convolutional layer, it is used to represent multimodal images by pixels. Transform into the final mask prediction .
[0153] 8. Model training;
[0154] The text and image encoders are initialized using CLIP. The input images are resized to 416×416 according to CRIS settings. To accommodate additional [SOS] and [EOS] tokens, input sentences for RefCOCO and RefCOCO+ are limited to 17 words. The Transformer decoder has three layers, each with 8 heads, and a feedforward hidden dimension of 512. The output mapping layer is located in the last layer. The convolutional layers use 3×3 kernels. The network is trained for 50 epochs using the Adam optimizer with a learning rate of λ = 0.0001. The modality fusion adapter has a learning rate set to λ = 0.001. The learning rate is reduced by 0.1 at epoch 35, and the model is trained on two NVIDIA A100 processors with a batch size of 32. During inference, predictions are upsampled to the original image size and binarized with a threshold of 0.35 to produce the final result without any additional post-processing.
[0155] 9. Model application;
[0156] Through the training process described above, the model with the smallest loss function value is selected for testing. With the model parameters fixed, the input image and corresponding text description, after feature extraction, modality fusion adapter, and task-customized decoder, directly yields the mask for the target region. This experiment was conducted on RefcCOCO and RefCOCO+, and the generated results are as follows. Figure 2 As shown. The results show that the method of the present invention can achieve performance comparable to existing fully fine-tuning methods by updating only 1.61% to 3.38% of the parameters.< / mask>
Claims
1. An information-driven target segmentation method based on modality fusion and self-evolutionary fine-tuning, characterized in that, Includes the following steps: Step 1: Image and text feature extraction module; Given an image and text, extract features using an image encoder and a text encoder, respectively. Step 2: Modal fusion adapter; Given visual features and text features First, the dimensions of visual and textual features are adjusted through scaling layers. This process can be formalized as follows: ； ； in This indicates the zoom-in operation of the scaling layer. Represents a linear mapping layer; Indicates the first Visual characteristics of the stage Indicates the first Textual features of the stage; The feature fusion via interactors can be expressed mathematically as follows: ； ； in Represents an interactor; Representing the The adapter at each stage fuses visual features. Representing the The adapter at each stage integrates text features. Indicates the first The adapter at each stage fuses visual features. Indicates the first The adapter integrates text features at each stage; The first phase and The initial value is and The dimensionality is restored through scaling layers and linear projection, and residual connections are made with the original features of the next stage of the backbone network. This can be expressed mathematically as follows: ； ； ； ； in This indicates the scaling operation of the scaling layer. Represents a linear mapping layer. Indicates the first Visual characteristics of the stage Indicates the first Textual features of the stage; Indicates the th after residual connection Phase-based fusion visual features Indicates the th after residual connection The fusion text features of the stages; Step 2-1: Scaling the layer; Features of the ViT intermediate layer from Plastic And convolutions are used to construct scaling layers; 2x2 convolutions with a stride of 2 are used to reduce the size of the feature maps; when extracting features from the intermediate layers of the backbone network, scaling layers are used to adjust the feature map size of the visual encoder. This process is formalized as follows: ； in , These represent the height and width of the feature maps in the visual backbone network, respectively. and These represent convolution and deconvolution operations, respectively. Indicates the first Stage feature map height, Indicates the first The width of the stage feature map; before adding the features back to the backbone network, a scaling layer is used to shrink them, which is the reverse of the scaling process; Step 2-2: Interactor; The interactor is based on an attention mechanism and a feedforward network. For each feature from different modalities, it uses the original modal feature as the query and obtains the key and value from the other modality. The specific formula is as follows: ； ； ； ； in Representing the The adapter at each stage fuses visual features. Representing the The adapter at each stage integrates text features. Indicates the first The adapter at each stage fuses visual features. Indicates the first The adapter integrates text features at each stage; and These represent the visual and textual features output by the interactor, respectively; MHSA represents multi-head self-attention, MHCA represents multi-head cross-attention, and FFN represents a fully connected layer. Step 3: Customize the decoder for the task; Step 4: Training objective.
2. The information-driven target segmentation method based on modality fusion and self-evolutionary fine-tuning according to claim 1, characterized in that, Step 1 specifically involves: Step 1-1: Image encoder; For a given image The image is fed into an image encoder to extract visual features. These represent the height and width of the image, respectively; for CNNs, the final... Stage visual characteristics For the visual Transformer-ViT, the ViT Transformer encoder is evenly divided into... Blocks, each block contains One coding layer, Represents the number of layers in the feature extractor; use the last one. Block outputs are used for feature interaction; Vi Multi-level visual features from different blocks in T or visual features from different stages in CNN are used as inputs to the modality fusion adapter and decoder in the framework for multi-modal feature alignment. Steps 1-2: Text Encoder; For the input text information Similar to the image encoder, the Transformer encoder is divided into evenly distributed sections. Blocks, and extract text features from different blocks of the Transformer. in It is the feature dimension. The length of the expression; the input text sequence is surrounded by [SOS] and [EOS] markers; the representation of the last layer of the Transformer at the [EOS] markers is used as a global text feature. , Represents the feature dimension.
3. The information-driven target segmentation method based on modality fusion and self-evolutionary fine-tuning according to claim 2, characterized in that, Step 3 specifically involves: Step 3-1: Align the layers of the module; Given multiple adapters from different stages, the visual features are fused. and global text features The feature dimensions are aligned through convolution, and then hierarchical fusion of multimodal features at different stages is obtained through a cross-attention mechanism. For the hierarchical fusion features, they are concatenated and aggregated using a 1×1 convolutional layer. ； ； ； [,] represents the concatenation operation. This represents the fused visual features obtained by formula (7). This represents the scaled-down visual features. Multimodal features representing the output of cross-attention. This represents the hierarchical fusion of multimodal features after concatenation; A 2D spatial coordinate feature and The images are stitched together and then fused using a 3×3 convolution to obtain visual features. The specific formula is as follows: ； Then merge visual features Flatten them into a sequence; Step 3-2: Global alignment module; After obtaining multimodal features through hierarchical alignment, the attention mechanism of Transformer is used to fuse textual information and corresponding visual features, thereby fusing visual features. and global text features As input, first send to and Add a fixed sinusoidal spatial position code; Subsequently, multimodal features are generated using self-attention and cross-attention modules. To capture global context information: ； ； in This represents the features generated after using the self-attention mechanism. Representing global text features, MHSA represents multi-head self-attention, MHCA represents multi-head cross-attention, and FFN represents fully connected layers; the final multimodal features are obtained. It was used in subsequent information-driven target segmentation tasks; Step 3-3: Output mapping layer; Use an output mapping layer for multimodal features and global text features The transformation is as shown in the following equation: ； ； ； UpSample means Double upsampling, then convolution to Convert to , where N= and will Transformed into a 2D convolutional layer through linear projection Then Divided into weights and bias K represents the kernel size of the convolutional layer, and D represents the feature dimension, thus making As a two-dimensional convolutional layer, it is used to represent multimodal image pixels. This is converted into the final mask prediction.
4. The information-driven target segmentation method based on modality fusion and self-evolutionary fine-tuning according to claim 3, characterized in that, Step 4 specifically involves: The text-to-pixel contrast loss is used, as shown in the following formula: ； in, Represents multimodal pixel features; and This represents the number of categories with labels of 1 and 0. The definition is as follows: ； Here It's the sigmoid function; the segmentation result is adjusted to... And it is upsampled to the original image size; After obtaining the loss function, backpropagation is performed using the loss function. Multiple models are then trained, and the optimal model is selected for application. The final output of the model is the information-driven target segmentation result.
5. The information-driven target segmentation method based on modality fusion and self-evolutionary fine-tuning according to claim 4, characterized in that, The Both are 416.