Multimodal pre-training directivity segmentation method

Abstract

The invention belongs to the field of deep learning technology in computer vision, and discloses a multi-modal pre-training directivity segmentation method. According to the method, a frozen contrast language image pre-training model is adopted to construct an overall framework, and a sample initialization module, a language encoder module, a visual encoder module and a multi-image feature weighted fusion module are included, so that precise segmentation of a specified object in an image is realized cooperatively. The sample initialization module generates text descriptions of positive and negative samples, eliminates interferents through negative words, prepositions and new sentences generated by using a large language model, and ensures the accuracy of the model. The image is processed through a visual encoder, deep features are extracted, incomplete back propagation is carried out, and model parameter freezing is ensured. And finally, in a multi-image feature weighted fusion module, the positive and negative text features and the multi-level features are fused to form unified feature representation to guide image segmentation. According to the method, the understanding ability of the model for the key region and the key object is effectively improved, and the segmentation effect is enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology in computer vision, and relates to a multimodal pre-trained directional segmentation method, in particular to a zero-sample directional image segmentation method based on a pre-trained multimodal model. Background Art

[0002] Traditional zero-shot learning aims to predict unseen classes by transferring knowledge from seen classes to unseen classes. Early studies mainly relied on pre-trained word embeddings of class names or attributes to achieve zero-shot predictions by mapping between the visual representation of images and word embeddings. These research methods laid the foundation for achieving prediction tasks that do not rely on a large amount of labeled data. However, with the development of deep learning and the complexity of application scenarios, traditional zero-shot learning methods have gradually revealed their limitations.

[0003] In recent years, the contrastive language-image pre-training model (CLIP) and the language-image alignment model (ALIGN) have brought new inspiration to zero-shot learning through large-scale image-text pre-training. These methods not only utilize large-scale datasets, but also adopt contrastive learning methods, so that the model can learn more general and robust features in multimodal data. Through zero-shot knowledge transfer, CLIP and ALIGN have shown significant performance improvements in a variety of downstream tasks, such as image description, image-text retrieval, etc. Compared with traditional zero-shot learning, the advantage of zero-shot transfer is that it avoids fine-tuning the pre-trained model on a specific task dataset, which can save a lot of time and resources, especially reducing the cost of dataset collection and annotation. In terms of specific applications, existing studies have directly applied the CLIP encoder with only a small amount of structural modification without additional training for semantic segmentation, directional expression understanding and target positioning. These studies demonstrate the wide adaptability and strong zero-shot transfer capabilities of the CLIP encoder. Based on this research field, the present invention aims to further explore and optimize the application of CLIP in different tasks.

[0004] Recently, with the success of pre-training models using large-scale images and text, some methods have tried to use CLIP for dense prediction tasks. For example, object detection, semantic segmentation, etc. However, these dense prediction tasks are challenging because CLIP mainly learns image-level features rather than fine-grained features at the pixel level. To address this challenge, Gu Xiuye, Lin Tsung-Yi, Kuo Weicheng, and Cui Yin proposed an innovative method in their paper "Open vocabulary object detection via vision and language knowledge distillation" to crop the image into a region containing only the bounding box, and then use CLIP to extract visual features of the cropped region to classify unseen objects. This method has been applied to multiple dense prediction tasks that require CLIP's zero-shot transfer capability. Although this method solves the problem of pixel-level feature extraction to a certain extent, its limitation of only considering the cropped region is also obvious. In addition, there are several methods that take into account the global context in the image, not just the cropped region. In their paper "Adapting CLIP for Phrase Localization Without Further Training", Li Jiahao, Shakhnarovich Greg, and Yeh Raymond A proposed a method to modify CLIP to generate high-resolution spatial feature maps using superpixels for phrase localization. Zhou Chong, Loy Chen Change, and Dai Bo introduced the MaskCLIP method in their paper "Extract Free Dense Labels from CLIP", which handles pixel-level prediction by converting the value embedding layer and the last linear layer in CLIP's image encoder into two 1×1 convolutional layers. These methods have made significant progress in processing global and local features of images, making CLIP more widely and effectively used in dense prediction tasks. Existing methods still have the problem of poor alignment in terms of pixel feature extraction and fusion.

[0005] This paper focuses on using CLIP to extract visual features of context at each level to further improve the performance and adaptability of the model. We designed a positive and negative sample mode, and by combining the positive and negative sample information of global and local context, we better address the challenges in dense prediction tasks and improve the accuracy and robustness of the model. Summary of the invention

[0006] The technical problem to be solved by the present invention is: in view of the problem that traditional image segmentation models perform poorly for directional segmentation in complex scenes, a directional segmentation method based on image-text pre-training is proposed, which utilizes an image-text joint pre-training model to improve the positioning and segmentation effects of target objects.

[0007] The present invention firstly adopts the frozen contrastive language-image pre-training model (CLIP) framework, and fully exploits the potential of the text-image pre-training model of the CLIP model without introducing an additional mask generator. The text and image features are deeply fused through the multi-image feature weighted fusion module to achieve accurate segmentation of the specified target in the image. The present invention adopts a negative sample generation mechanism in the feature extraction process, which effectively enhances the model's ability to identify similar interferers.

[0008] The technical solution of the present invention:

[0009] A multimodal pre-trained directional segmentation method, whose overall architecture is mainly composed of a sample initialization module, a CLIP language encoder, a CLIP visual encoder, and a multi-image feature weighted fusion module; the sample initialization module is responsible for generating positive sample text descriptions and negative sample text descriptions; the CLIP language encoder is responsible for encoding the input positive sample text descriptions and negative sample text descriptions to generate language features; the CLIP visual encoder is used to extract multi-level visual features of the image; the multi-image feature weighted fusion module is responsible for fusing features from different sources in a multi-level and multi-method combination to guide the final image segmentation. The specific steps are as follows:

[0010] Step (1) Build the sample initialization module

[0011] The sample initialization module is designed to generate positive sample text descriptions and negative sample text descriptions for training. This module contains the following key components:

[0012] Positive sample description: The input target text is used as a positive sample. For example, for a cat, the positive sample description can be "This is a black cat sitting by the window."

[0013] Generate negative sample descriptions: In order to enhance the generalization ability of the model, negative sample descriptions are generated in different ways, including 1) using negative words: adding negative words to the positive sample description, such as "this is not a black cat", to form a description that is unrelated to the target object; 2) changing prepositions: for example, changing "by the window" to "on the floor" to generate new negative sample descriptions to help the model learn to distinguish between targets and backgrounds; 3) generating complex descriptions: using natural language processing technology to generate more complex negative sample descriptions and increase sample diversity.

[0014] Sample labeling: Label each positive and negative sample generated to ensure that they can be correctly identified and distinguished during subsequent training.

[0015] The text descriptions generated through the above steps will be input into the CLIP language encoder for further feature extraction.

[0016] Step (2) Build the CLIP language encoder

[0017] The CLIP language encoder is responsible for converting the generated text descriptions of positive and negative samples into corresponding feature representations. The specific steps are as follows:

[0018] Text preprocessing: Preprocess the input text description, including removing irrelevant symbols, word segmentation and stem extraction, to ensure the clarity and accuracy of the input text.

[0019] Word embedding: Use the pre-trained word embedding model that comes with the CLIP model language encoder to convert each word in the text description into a high-dimensional vector, ensuring that the vector of each word can capture its semantic information in the context.

[0020] CLIP model language encoder: Build a Transformer-based encoder consisting of multiple identical Transformer layers. Each Transformer layer of the CLIP model language encoder will perform multiple transformations on the input text description. The transformation mainly captures the contextual relationship between words through a multi-head self-attention mechanism, and enhances the expressiveness of words through nonlinear transformations through a feedforward neural network. Residual connections and layer normalization are used to maintain information flow and the stability of the CLIP language encoder, thereby gradually generating semantically rich text representations and capturing potential semantic relationships in the text.

[0021] Feature extraction: Extract features from the output of the CLIP model language encoder to form the final text feature vector. This feature vector will be used as input in subsequent tasks to help the model understand the meaning of the text.

[0022] Positive and negative sample feature separation: After feature extraction, the positive sample text features and negative sample text features are separated to facilitate subsequent feature fusion and training.

[0023] Step (3) Build the CLIP visual encoder

[0024] The CLIP visual encoder is mainly responsible for extracting deep visual features from the input image. The steps are as follows:

[0025] Image preprocessing: Standardize the input image of the target object specified by the corresponding text, including scaling to the specified size, normalizing pixel values, etc., to ensure the consistency and accuracy of the input.

[0026] Feature extraction: A visual transformer is used to extract multi-level features of the standardized image. The visual transformer contains multiple Transformer layers. Specifically, the input image is divided into patches of fixed size, each patch is flattened and projected into an embedding vector of fixed dimension, and position embedding is added to capture position information. The image is processed through multiple Transformer layers, each of which contains a multi-head self-attention mechanism and a feedforward neural network to capture global and local relationships and enhance feature representation.

[0027] Multi-level feature output: The features of the last layer are output in the visual transformer, and visual features are output in each transformer layer for feature fusion at different levels of the multi-image feature weighted fusion module; the extracted features include:

[0028] Visual feature maps of the middle layer: obtained through the forward propagation process of the CLIP visual encoder, extracting output feature maps of multiple layers; providing detailed information to help understand the local structure in the image;

[0029] Attention map: generated by the attention weight matrix in the multi-head self-attention mechanism in each Transformer layer, showing how much the CLIP visual encoder pays attention to a specific area;

[0030] Gradient map: generated by computing and visualizing the gradient of the CLIP visual encoder output with respect to the input image;

[0031] Activation map: This shows the response of the CLIP visual encoder to different regions of the image by extracting the output feature map of the layer corresponding to the visual feature map of the middle layer in the CLIP visual encoder;

[0032] Feature output: The processed visual features are output as the input of the multi-image feature weighted fusion module; Step (4) constructs the multi-image feature weighted fusion module

[0033] The input features of the multi-image feature weighted fusion module are: attention map, gradient map, activation map, visual feature map of the middle layer of CLIP, as well as positive sample text features and negative sample text features. The above features are obtained from the visual encoder and language encoder of CLIP. The core of the directional segmentation task is how to fuse visual features and language features. The core of this module is also to fuse attention maps, gradient maps, activation maps, visual feature maps and positive and negative sample text features. The fusion process is as follows:

[0034] Feature input: Get positive sample text features and negative sample text features from the CLIP language encoder, and get attention maps, gradient maps, activation maps, and visual feature maps of the middle layer from the CLIP visual encoder;

[0035] Feature weighting strategy: Design a feature weighting strategy based on task requirements and perform weighted fusion of features from different sources. The basis for weighting is mainly the similarity and importance of each feature.

[0036] Normalization: Before fusion, the attention map needs to discard the category token of the last dimension, and then normalize the rows and columns; in the gradient map, the gradient generated by the negative sample text features is negative, summed with the gradient generated by the positive sample text features, and fused after the ReLU activation function;

[0037] Cosine similarity weighting: The cosine similarity between the visual feature map of the middle layer and the text feature of the positive sample is calculated to obtain a weighted visual feature map; this process also includes similarity calculation and normalization of the activation map, which is weighted into the activation map;

[0038] Feature fusion and enhancement: The visual feature map, activation map, gradient map and attention map of the input intermediate layer are subjected to multiple weighted fusions using matrix multiplication and matrix connection strategies, combined with normalization functions, and combined with the visual feature maps, activation maps, gradient maps and attention maps extracted in multiple Transformer layers to further find the feature areas most relevant to the positive sample text features in the input image of the target object specified by the corresponding text; the negative sample text features use back propagation to generate gradient maps, combined with the visual feature map, activation map, gradient map, attention map of the CLIP visual encoder and the negative sample text features extracted by the CLIP text encoder, and their cosine similarity is calculated for optimization; the weighted summation method is used to fuse the positive sample text features and the negative sample text features; the formula is as follows:

[0039] F=α·F positive -(1-α)·F negative

[0040] Among them, F is the fused feature, α is the weight of the positive sample feature, and F negative is the fusion feature generated based on the negative sample features, F positive It is the fusion feature generated based on the positive sample features;

[0041] Step (5) Construct image segmentation module

[0042] The image segmentation module is responsible for further processing the fused features to convert them into segmentation results, as follows:

[0043] Multi-level feature processing: In the image segmentation module, the multi-level fused features extracted from the multi-image feature weighted fusion module are averaged to capture the details and context information in the image;

[0044] Input feature upsampling: The fused features are input into the image upsampling network, and the upsampling is performed by bilinear interpolation.

[0045] Classification and positioning: Classify each pixel through the classification layer to determine whether it belongs to the target object;

[0046] Post-processing: Post-process the segmentation results by smoothing and denoising to improve the visual effect and accuracy of the segmentation;

[0047] Output results: Output the final segmentation results for subsequent application and evaluation.

[0048] Beneficial effects of the invention: The invention combines the CLIP image and text pre-training model to fully explore the guiding role of text information in image segmentation and proposes a new directional segmentation model. This model not only performs well in segmentation tasks, but also effectively improves the model's directional segmentation ability in complex scenes through multi-level feature fusion and negative sample generation mechanism, and has extremely high application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is the overall architecture of the directional segmentation model based on image and text pre-training.

[0050] Figure 2 It is the detailed structure of the multi-feature map weighted fusion module. DETAILED DESCRIPTION

[0051] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0052] (1) Overall architecture

[0053] like Figure 1 As shown in the figure, we show the overall architecture of a directional segmentation model based on image-text pre-training. The input image is a picture of multiple elephants and a sentence about a baby elephant between the elephants. Our framework is used to locate all pixels related to the baby elephant in the image. The entire framework is built based on frozen CLIP, without introducing a mask generator for auxiliary segmentation, and is committed to more fully unleashing the image-text pre-training potential of CLIP.

[0054] The overall framework consists of multiple modules, mainly including:

[0055] Sample initialization module: Generate text descriptions of positive and negative samples.

[0056] CLIP language encoder: processes the input text and generates positive text features and negative text features.

[0057] CLIP Visual Encoder: Extracting deep features from images based on the visual transformer architecture.

[0058] Multi-feature map weighted fusion module: fuses visual features and text features at different levels to guide image segmentation.

[0059] The sample initialization module is the first step of the network, which aims to generate text descriptions of positive and negative samples. The original input text is assumed to be a positive sample, and negative samples are generated in the following three ways:

[0060] Negate the original sentence through negative words: Add negative words to the original sentence to generate a sentence with the opposite meaning, lock other objects in the image except the segmentation target we want to lock, and eliminate some interference.

[0061] Changing prepositions in sentences: By changing the position of prepositions in sentences, the semantics of sentences can be changed and similar distractors can be located. This is because the same objects in different positions in the image have a great impact on the segmentation target, and the model often cannot distinguish them well.

[0062] Generate new sentences using ChatGPT: Extract nouns from sentences, input a large number of example sentences from the RefCOCO dataset into ChatGPT, use the large language model to generate prompt templates, and match the extracted nouns to the templates as negative samples for the model.

[0063] The generated text descriptions are input into multiple CLIP language encoders. Each encoder processes the input text and generates positive and negative text features. These text features are used to guide image segmentation in subsequent steps.

[0064] The input image is first processed by the CLIP visual encoder. The CLIP visual encoder is based on the visual transformer architecture and can extract deep features in the image. We extracted the visual feature map of the middle layer, the attention map in the Transformer, and the gradient map and activation map of the middle layer obtained by backpropagation. The calculated gradient is an incomplete backpropagation, and the parameters of the model are not updated, which meets the premise of freezing the parameters of the model and not training. These features extracted from multiple aspects can help the model understand the key areas and objects in the image more comprehensively and improve the positioning and segmentation capabilities.

[0065] Next, in the multi-feature graph weighted fusion module, the positive and negative text features and multi-level features from different encoders are fused. Through weighted fusion, these features are finally integrated into a unified feature representation to guide image segmentation.

[0066] (2) Multi-feature map weighted fusion module

[0067] like Figure 2 As shown in the figure, the input features of the multi-feature map weighted fusion module are: the attention map of the middle layer of CLIP, the gradient map of the middle layer, the activation map of the middle layer, the visual feature map of the middle layer, as well as the positive sample text features and the negative sample text features.

[0068] The core feature fusion structure is to fuse the weighted or specially normalized activation map, gradient map and attention map of the CLIP intermediate layer. This is the core of the entire module.

[0069] Attention map of the middle layer of CLIP: Before fusion, the category token of the last dimension is first discarded, and then the rows and columns are normalized to optimize the attention map. Because the attention map is also a structure similar to the similarity feature, there is actually a lot of information hidden in the attention map of each layer of the visual Transformer. The purpose of extracting the attention feature map of the middle layer of CLIP here is to further explore the hidden information in the attention feature map and enhance the feature extraction capability of the network architecture.

[0070] Gradient map: First, generate its gradient according to the sample. Then, by default, the gradient generated by the positive sample is positive and the gradient generated by the negative sample is negative. Then they are summed and finally fused after the ReLU activation function. Since our negative samples at the text level target objects other than our correct target that may interfere with it, we should reduce the focus of the key areas of the gradient generated by the negative samples and design a structure in which the gradient generated by the negative samples is negative and summed with the gradient generated by the positive samples.

[0071] Visual feature map of the middle layer: First, calculate the cosine similarity with the positive sample feature and weight the visual feature map, then introduce the feature of the activation map of the middle layer to calculate the cosine similarity again and weight the activation map at this time, and finally normalize it with the Sigmoid function and weight it to the activation map feature. This branch mainly draws on the method of using cosine similarity in the CLIP network to calculate the similarity between two modalities, and uses cosine similarity to find the areas in the visual feature map and the activation map of the middle layer that are more related to the text features, and strengthen the attention to these areas.

[0072] Activation map of the middle layer: It is mainly based on the weighted fusion of multiple cosine similarity feature values ​​generated by the visual feature map branch of the middle layer, and finally fused with the gradient feature map. The purpose is to combine the features of multiple layers and branches to find more relevant feature areas in the network.

[0073] Negative sample features: Based on the method of calculating back-propagation gradients in deep learning networks, we use 0 and 1 of the cosine similarity of text features and visual features as labels, and obtain the gradient map features generated by negative sample features through an incomplete back-propagation process.

[0074] (3) Image segmentation module

[0075] The image segmentation module is an important part of the entire model and is responsible for converting the fused features into segmentation results. The specific steps are as follows:

[0076] Multi-level feature processing: In the segmentation network model, the multi-level fused features extracted from the multi-image feature weighted fusion module are averaged to capture the details and contextual information in the image;

[0077] Input feature upsampling: The fused features are input into the image upsampling network, and the upsampling is performed by bilinear interpolation.

[0078] Classification and positioning: Each pixel is classified through the classification layer to determine whether it belongs to the target object. The classification layer is implemented using convolutional layers and fully connected layers.

[0079] Post-processing: Perform post-processing on the segmentation results, such as smoothing and denoising, to improve the visual effect and accuracy of the segmentation.

[0080] Through the above steps, the present invention provides a novel and efficient zero-sample directional image segmentation method, which can accurately segment user-specified objects without the need for additional training, greatly reducing the dependence on labeled data and improving the practical application value of the model.

Claims

1. A multimodal pre-trained directional segmentation method, characterized in that: The overall model architecture based on this multimodal pre-trained directional segmentation method is mainly composed of a sample initialization module, a CLIP language encoder, a CLIP visual encoder, and a multi-image feature weighted fusion module; the sample initialization module is responsible for generating positive sample text descriptions and negative sample text descriptions; the CLIP language encoder is responsible for encoding the input positive sample text descriptions and negative sample text descriptions to generate language features; the CLIP visual encoder is used to extract multi-level visual features of the image; the multi-image feature weighted fusion module is responsible for fusing features from different sources in a multi-level and multi-method combination to guide the final image segmentation; the specific steps are as follows: Step (1) Build the sample initialization module The sample initialization module contains the following components: Positive sample description: Take the input target text as the positive sample; Negative sample description: including 1) using negative words: adding negative words to the positive sample description; 2) changing prepositions: changing the prepositions in the positive sample description to generate new negative sample descriptions to help the model learn to distinguish between targets and backgrounds; 3) generating complex descriptions: using natural language processing technology to generate more complex negative sample descriptions and increase sample diversity; Sample annotation: Annotate each positive and negative sample generated; The text descriptions generated above are respectively input into the CLIP language encoder for further feature extraction; Step (2) Build the CLIP language encoder The details are as follows: Text preprocessing: preprocess the input text description, including removing irrelevant symbols, word segmentation and stem extraction; Word embedding: Use the pre-trained word embedding model provided by the CLIP model language encoder to convert each word in the text description into a high-dimensional vector, ensuring that the vector of each word can capture its semantic information in the context; CLIP model language encoder: Build a Transformer-based encoder consisting of multiple identical Transformer layers. Each Transformer layer of the CLIP model language encoder will perform multiple transformations on the input text description. The transformation mainly captures the contextual relationship between words through a multi-head self-attention mechanism, enhances the expressiveness of words through nonlinear transformations through a feedforward neural network, and maintains information flow and stability of the CLIP language encoder through residual connections and layer normalization, thereby gradually generating semantically rich text representations and capturing potential semantic relationships in the text. Feature extraction: extract features from the output of the CLIP model language encoder to form the final text feature vector; Positive and negative sample feature separation: After feature extraction, the positive sample text features and negative sample text features are separated to facilitate subsequent feature fusion and training; Step (3) Build the CLIP visual encoder The details are as follows: Image preprocessing: Standardize the input image of the target object specified by the corresponding text, including scaling to the specified size and normalizing the pixel values ​​to ensure the consistency and accuracy of the input; Feature extraction: A visual transformer is used to extract multi-level features of the standardized image. The visual transformer contains multiple Transformer layers. Specifically, the input image is divided into patches of fixed size, each patch is flattened and projected into an embedding vector of fixed dimension, and position embedding is added to capture position information. The image is processed through multiple Transformer layers, each of which contains a multi-head self-attention mechanism and a feedforward neural network to capture global and local relationships and enhance feature representation. Multi-level feature output: The features of the last layer are output in the visual transformer, and visual features are output in each transformer layer for feature fusion at different levels of the multi-image feature weighted fusion module; the extracted features include: Visual feature maps of the middle layer: obtained through the forward propagation process of the CLIP visual encoder, extracting output feature maps of multiple layers; providing detailed information to help understand the local structure in the image; Attention map: generated by the attention weight matrix in the multi-head self-attention mechanism in each Transformer layer, showing how much the CLIP visual encoder pays attention to a specific area; Gradient map: generated by computing and visualizing the gradient of the CLIP visual encoder output with respect to the input image; Activation map: This shows the response of the CLIP visual encoder to different regions of the image by extracting the output feature map of the layer corresponding to the visual feature map of the middle layer in the CLIP visual encoder; Feature output: The processed visual features are output as the input of the multi-image feature weighted fusion module; Step (4) constructs the multi-image feature weighted fusion module The input features of the multi-image feature weighted fusion module are: attention map, gradient map, activation map, visual feature map of the middle layer, positive sample text features and negative sample text features; the fusion process of the multi-image feature weighted fusion module is as follows: Feature input: Get positive sample text features and negative sample text features from the CLIP language encoder, and get attention maps, gradient maps, activation maps, and visual feature maps of the middle layer from the CLIP visual encoder; Normalization: Before fusion, the attention map needs to discard the category token of the last dimension, and then normalize the rows and columns; in the gradient map, the gradient generated by the negative sample text features is negative, summed with the gradient generated by the positive sample text features, and fused after the ReLU activation function; Cosine similarity weighting: The cosine similarity between the visual feature map of the middle layer and the text feature of the positive sample is calculated to obtain a weighted visual feature map; this process also includes similarity calculation and normalization of the activation map, which is weighted into the activation map; Feature fusion and enhancement: The visual feature map, activation map, gradient map and attention map of the input intermediate layer are subjected to multiple weighted fusions using matrix multiplication and matrix connection strategies, combined with normalization functions, and combined with the visual feature maps, activation maps, gradient maps and attention maps extracted in multiple Transformer layers to further find the feature areas most relevant to the positive sample text features in the input image of the target object specified by the corresponding text; the negative sample text features use back propagation to generate gradient maps, combined with the visual feature map, activation map, gradient map, attention map of the CLIP visual encoder and the negative sample text features extracted by the CLIP text encoder, and their cosine similarity is calculated for optimization; the weighted summation method is used to fuse the positive sample text features and the negative sample text features; the formula is as follows: F=α·F positive -(1-a)·F negative Among them, F is the fused feature, α is the weight of the positive sample feature, and F negative is the fusion feature generated based on the negative sample features, F positive It is the fusion feature generated based on the positive sample features; Step (5) Construct image segmentation module The image segmentation module is responsible for further processing the fused features to convert them into segmentation results, as follows: Multi-level feature processing: In the image segmentation module, the multi-level fused features extracted from the multi-image feature weighted fusion module are averaged to capture the details and context information in the image; Input feature upsampling: The fused features are input into the image upsampling network, and the upsampling is performed by bilinear interpolation. Classification and positioning: Classify each pixel through the classification layer to determine whether it belongs to the target object; Post-processing: Post-process the segmentation results by smoothing and denoising to improve the visual effect and accuracy of the segmentation; Output results: Output the final segmentation results for subsequent application and evaluation.

Images