Double-decoder integrated indication image segmentation algorithm and device based on SAM completion

Abstract

The invention discloses a dual-decoder integrated indication image segmentation algorithm and a dual-decoder integrated indication image segmentation device based on SAM (Secure Anything Model) completion, a novel dual-branch decoder framework is introduced, and the SAM (Secure Anything Model) is combined to be used for RIS (Remote Information System). According to the framework, an MLP decoder and a KAN decoder are combined, and a multi-scale feature fusion module is added, so that the capability of identifying fine details in an image by a model is enhanced. Through an ensemble learning strategy, the robustness of the framework is further enhanced, and the insight of MLP and KAN decoder branches is integrated. More importantly, edge coordinates and bounding box coordinate information of the segmented target are used as input prompts of the SAM model. According to the strategy, the zero sample learning capability of the SAM is utilized to perfect and optimize the segmentation result. According to the method disclosed by the invention, the effectiveness of the method is proved on the basis of experimental results of widely accepted RefCOCO, RefCOCO + and RefCOCOg data sets. The result not only reaches the most advanced level in segmentation performance, but also is supported by ablation studies, which highlight the contribution of each component to the improvement of the overall performance.

Description

Technical Field

[0001] The invention relates to a dual-decoder integrated indication image segmentation algorithm and device based on SAM completion. Background Art

[0002] In today's digital age, the growing demand for human-computer interaction has promoted the rapid development of cross-modal fusion technology in the field of vision. As one of the key technologies in the field of vision, image segmentation is becoming increasingly important. In particular, referring image segmentation (RIS), as a basic and challenging multimodal task, has attracted extensive research attention. RIS technology enables the system to identify and segment specific objects or regions in an image based on natural language descriptions by integrating visual-language understanding and instance segmentation technology. These descriptions may include information such as the target's action, category, color, and position, providing great application potential for a variety of application scenarios, such as human-computer interaction and interactive image segmentation.

[0003] Existing image segmentation techniques mainly focus on segmentation tasks of predefined categories, while the RIS task generalizes the concept of segmentation to any object defined by natural language description. This generalization not only greatly expands the application scope of segmentation tasks, but also significantly increases the complexity of the task. In existing technologies, achieving RIS tasks requires the system to have a deep understanding of both image content and language semantics, which is a technically challenging goal.

[0004] The core goal of the image segmentation task is to use natural language description to achieve accurate pixel-level positioning of the target object in the image. To achieve this goal, the existing technology is mainly divided into the following two key steps:

[0005] 1) Visual and language feature extraction: The system first needs to extract features from the input image and natural language description. Currently, a variety of models have been applied to this purpose, including but not limited to convolutional neural networks (CNN), recurrent neural networks (RNN), and Transformer. These models act as feature extractors, processing image and text data separately to capture their respective modal features.

[0006] 2) Multimodal feature fusion and reasoning: After feature extraction, the next challenge is how to effectively fuse these features and use them for inference and prediction of image segmentation. In the existing technology, researchers have adopted a variety of fusion strategies, such as feature splicing, attention mechanism, and multimodal Transformer, to achieve in-depth integration of different modal features.

[0007] In practice, some researchers use a combination of VGG and LSTM to extract visual and language features respectively, and then flatten and attach the language semantic information to the visual features. The target image is segmented accurately at the pixel level through a fully convolutional network (FCN) style segmentation head. In addition, some researchers use a recursive way to fuse word embeddings into visual features, or use a query attention module to identify the language semantics most relevant to the image region. Other researchers have developed a keyword-aware visual context module to simulate the contextual semantics of keywords in the visual context, and used a multi-layer perceptron (MLP) as a segmentation head to combine keyword-aware visual context features and weighted query features to predict the final segmentation mask. However, although pixel-to-pixel prediction methods are very popular in the field of image processing, this method often ignores the structural relationship between the prediction results. To solve this problem, researchers proposed Polyformer based on a sequence-to-sequence (seq2seq) framework. Polyformer successfully achieved state-of-the-art segmentation results on datasets such as RefCOCO, RefCOCO+, and RefCOCOg by predicting the coordinates of the target bounding box instead of individual pixels. The success of Polyformer lies in its ability to capture the structural features of objects and use sparse polygonal vertices to represent segmentation masks. This representation method is not only more economical, but also more in line with the annotation format of instance segmentation datasets.

[0008] Although the Polyformer framework has achieved remarkable results in the field of index image segmentation and demonstrated excellent segmentation performance, its unique prediction mechanism is not without limitations. Specifically, the Polyformer framework is difficult to effectively integrate with pixel-to-pixel prediction methods that operate directly at the image level. This limitation restricts its wide applicability in a variety of application scenarios and limits the flexibility and universality of the technology.

[0009] In addition, the prediction model of the Polyformer framework involves a large number of parameters due to its complex structure. This feature requires a large training dataset to support its learning process to ensure the accuracy and robustness of the model. However, with limited resources, especially for ordinary users who can only use consumer-grade graphics cards, obtaining and processing such a large dataset poses a major challenge. This not only increases the cost of model training and deployment, but also prolongs the time required for the model to achieve the expected performance, affecting the practicality and accessibility of the model.

[0010] Therefore, it is necessary to design an image segmentation algorithm and device. Summary of the invention

[0011] The technical problem to be solved by the present invention is

[0012] The purpose of the present invention is to provide an integrated learning strategy that is particularly suitable for resource-constrained environments, especially for ordinary users who can only use consumer-grade graphics cards. Through this strategy, the present invention aims to optimize the Polyformer model structure to achieve significant performance improvement under limited resources. Therefore, the present invention provides a dual-decoder integrated indication image segmentation algorithm and device based on SAM completion.

[0013] A dual decoder integrated indicator image segmentation algorithm based on SAM completion, wherein the corresponding framework of the algorithm includes three modules: (1) a multimodal information encoder module, (2) a dual-branch decoder module, and (3) a SAM-based segmentation completion module.

[0014] The multimodal information encoder module includes: an image encoder based on SwinTransformer, a text encoder based on Bert, and a multimodal information encoder based on Transformer;

[0015] The workflow of the multimodal information encoder module is:

[0016] Target image I∈R H×W×3 Input to the image encoder composed of SwinTransformer Base, and obtain image features from the fourth stage of the encoder Where H is the height of the input image, which is 512 here; W is the width of the input image, which is 512 here; C v The number of channels set for the fourth-stage SwinTransformerBase module, here 768;

[0017] Target text T∈R L Input the text encoder composed of Bert_12 and obtain text features through the language embedding model Where L is the length of the text, which is determined by the longest text in a set of inputs; C l The number of channels set for the Bert_12 module, here 768;

[0018] The image features are then flattened into And by using the fully connected layer to project the image features and text into the same embedding space and then splice them, we get the features Input a multimodal encoder consisting of 6 layers of Transformer to obtain multimodal features Among them, C is the number of channels of the spliced ​​multimodal features, which is 768 here.

[0019] The function of this module is to extract different modal information from the input, combine them, and extract multimodal features; the output of this module will serve as the input of the dual-branch decoder module.

[0020] The dual-branch decoder module includes: an MLP-based multimodal decoder, a KAN-based multimodal decoder, and a decoder prediction result integration module;

[0021] The workflow of the dual-branch decoder module is:

[0022] Step 2.1: The multimodal features output by the multimodal information encoder module are respectively input into the MLP-based multimodal decoder and the KAN-based multimodal decoder to obtain two sets of segmentation coordinate results predicted by MLP and KAN respectively. The result is not a predicted image mask, but a continuous coordinate point sequence consisting of the coordinates of the upper left corner and the lower right corner of the segmentation target prediction box and the coordinates of the continuous edge points of the segmentation target;

[0023] Step 2.2: The two sets of segmentation coordinate results are input into the decoder prediction result integration module, with the MLP prediction result as the main one. If the average distance of the segmentation target edge point coordinates in the MLP prediction result is less than N, the KAN prediction result is compared. If the average distance of the segmentation target edge point coordinates in the KAN prediction result is greater than the average distance of the MLP result, the KAN prediction result is used. Otherwise, the MLP prediction result is used as the decoder output result. It can be seen from experimental data that when the average distance of the segmentation target edge point coordinates is less than 4, the obtained prediction results are mostly wrong. Therefore, N is set to 4.

[0024] The function of this module is to form a preliminary segmentation target mask. After obtaining the segmentation target edge point coordinate sequence predicted by the dual-branch decoder module, the segmentation target contour can be outlined according to the edge coordinate data, thereby forming a preliminary segmentation target mask. The output of this module will serve as the input of the SAM-based segmentation completion module.

[0025] The SAM-based segmentation completion module includes: a SAM input selection module, a SAM prediction module, and a SAM prediction result denoising module;

[0026] SAM-based segmentation and completion module process:

[0027] Step 3.1 The module obtains the segmentation target prediction box coordinates and preliminary segmentation target mask integrated by the dual-branch decoder, and determines the prompt data type input to SAM according to the coordinate results output by the decoder; Prompt refers to the prompt information input to SAM, which is used to guide or stimulate the model to generate specific outputs:

[0028] If the average distance of edge point coordinates is less than N, only the segmentation target prediction box coordinates integrated and output by the dual-branch decoder are used as prompt input to SAM; otherwise, n positive sample points are selected from the preliminary segmentation target mask formed by the decoder; that is, the prompt is used to prompt SAM that the coordinates of the input point belong to the segmentation target, and the points associated with it are likely to belong to the segmentation target. Theoretically, the larger n is, the more prompts SAM obtains, but the current positive sample point is only a decoder prediction point, not an exact internal coordinate point of the segmentation target. Therefore, in order to avoid introducing erroneous information, n is set to 5. The positive sample point coordinates and the target box coordinates are simultaneously input into SAM as prompts;

[0029] Step 3.2 The SAM prediction module accepts prompt and target image as input, obtains the embedding vector of the target image through the image encoder composed of VisionTransformer, and encodes the input prompt through the prompt encoder to obtain the corresponding embedding vector. After the SAM image decoder receives the image embedding vector and the prompt embedding vector, it outputs the segmentation target mask predicted by SAM and inputs it into the denoising module;

[0030] Step 3.3 The SAM prediction result denoising module obtains the segmentation target mask predicted by SAM as input, and uses the breadth-first learning algorithm to count the number of segmentation blocks contained in the SAM output result. When the number is greater than Block_N, it indicates that the prediction result contains too much noise, and it is discarded. Only the prediction result integrated and output by the decoder's dual-branch decoder is used as the final result; here, because SAM usually outputs some tiny noise blocks, in order to ensure the robustness of the framework, Block_N is set to 16; otherwise, the segmentation target mask′ predicted by SAM is used as a supplement to the decoder prediction result, and the SAM prediction mask′ is combined with the decoder prediction mask, that is, mask final =maskANDmask′, AND is a logical OR operation; thus forming the final segmentation target mask final ;

[0031] The function of this module is to use SAM to complete the mask of the segmentation target based on the decoder prediction results.

[0032] A dual decoder integrated indication image segmentation device based on SAM completion adopts the aforementioned dual decoder integrated indication image segmentation algorithm based on SAM completion to realize image segmentation.

[0033] Beneficial effects:

[0034] In the field of human-computer interaction, the fusion of visual and language cues is becoming increasingly important. This paper focuses on the challenges and progress of referential image segmentation (RIS), a task of image segmentation based on text descriptions. Traditional RIS methods focus on pixel-level classification. Although effective, these methods often ignore the interconnectedness between pixels, which is crucial for interpreting complex visual scenes. In addition, although the Polyformer model performs well in RIS, its large number of parameters and high training data requirements pose significant challenges. These factors limit its adaptability and optimization on standard consumer hardware, hindering further enhancements in subsequent research. To address these issues, this paper introduces a novel dual-branch decoder framework combined with SAM (Segment Anything Model) for RIS. The framework combines an MLP decoder and a KAN decoder, and adds a multi-scale feature fusion module to enhance the model's ability to recognize subtle details in images. The robustness of the framework is further enhanced by an ensemble learning strategy, integrating insights from the MLP and KAN decoder branches. More importantly, the present invention utilizes the segmentation target edge coordinates and bounding box coordinate information as input cues for the SAM model. This strategy leverages the zero-shot learning capability of SAM to refine and optimize the segmentation results. The effectiveness of this approach is demonstrated by experimental results on the widely recognized RefCOCO, RefCOCO+, and RefCOCOg datasets. The results not only achieve state-of-the-art segmentation performance, but are also supported by ablation studies that highlight the contribution of each component to the overall performance improvement.

[0035] The method of the present invention can not only fully utilize the computing power of existing hardware and reduce the dependence on high-end computing resources, but also effectively overcome the bottlenecks of the Polyformer framework in terms of parameter quantity and data requirements through innovative learning mechanisms. This optimization not only improves the efficiency and practicality of the model, but also enables a wider range of users to access and utilize advanced image segmentation technology, promoting the popularization and application of technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Wrong image segmentation for Polyformer;

[0037] Figure 2 This is the structure diagram of Polyformer;

[0038] Figure 3 This is the SAM structure diagram;

[0039] Figure 4 Indicates the image segmentation algorithm framework for dual decoder integration based on SAM completion;

[0040] Figure 5 This is a dual decoder structure diagram;

[0041] Figure 6 It is the KAN decoder branch;

[0042] Figure 7 Completion module for SAM-based segmentation targets;

[0043] Figure 8 The prediction result with noise information output by SAM (1);

[0044] Fig. 9 The prediction result (2) output by SAM with noise information. DETAILED DESCRIPTION

[0045] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. However, this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the content of the present invention belong to the scope of the present invention.

[0046] This study first adopted the frozen fine-tuning method and combined it with the Kolmogorov-Arnold Network (KAN) to develop a new classification head for segmentation target edge prediction. This innovative structure not only improves the accuracy of segmentation target edge point prediction, but also enhances the generalization ability of the model. As an innovative neural network architecture, KAN is inspired by the Kolmogorov-Arnold theorem. Compared with the traditional multi-layer perceptron (MLP), the activation function of KAN is transformed from a fixed function to a learnable univariate spline function. These functions are moved from nodes to edges, which significantly improves the flexibility and adaptability of the model.

[0047] Furthermore, the present invention integrates a multi-scale feature fusion module to form a new decoder for predicting edge coordinates of image segmentation. This fusion strategy can capture features of different scales and improve the model's ability to recognize image details. The input and output forms of KAN are consistent with MLP, so that the present invention can efficiently replace the MLP structure in the Polyformer model decoder with KAN, and only a small amount of parameter fine-tuning is required to train a prediction branch that is different from the original model output.

[0048] In order to further optimize the KAN decoder branch and increase its difference from the MLP decoder branch, the present invention introduces a multi-scale feature fusion module at the input stage of the KAN decoder. This module effectively fuses the features output by the image encoder and the text encoder, as well as the features processed by the multimodal encoder, and then inputs the fused features into the KAN decoder branch. By freezing the encoder part and only fine-tuning the KAN decoder, the present invention can obtain a decoder branch with complementary performance, thereby enhancing the diversity of the prediction results of the boundary coordinate points of the segmented target.

[0049] Although the KAN decoder branch can fine-tune Polyformer without adding a lot of computing resources, obtain a new indicator image segmentation model decoder, and improve the segmentation performance of the model under the action of the ensemble learning strategy, the performance improvement is limited because the input feature sources of the MLP decoder branch and the KAN decoder branch are the same. In order to maximize the advantages of the ensemble learning strategy, the present invention combines the output data characteristics of Polyformer with the input data characteristics of the visual large model SAM, and uses the powerful zero-sample learning ability of SAM to further improve the segmentation performance of the model when processing difficult-to-segment images.

[0050] Dual decoder integrated indication image segmentation algorithm framework and device based on SAM completion

[0051] Problem definition:

[0052] Among the current indicative image segmentation models, the Polyformer framework has successfully achieved excellent segmentation performance with its innovative target edge coordinate point serialization prediction mechanism. This mechanism enables the model to perform well in image segmentation tasks, especially in the precise positioning of target edges. However, Polyformer encounters challenges when faced with some image scenes with complex structures or numerous visual elements. Specifically, the model has difficulty accurately capturing the edges of the target in these scenes, which leads to confusion in the prediction of edge coordinate points and affects the final segmentation effect, such as Figure 1 This confusion in prediction not only affects the accuracy of segmentation, but also limits the application potential of the model in a wider range of scenarios.

[0053] In the structural design of the Polyformer model, the prediction of the edge coordinates of the segmentation target is a serialized process, in which the prediction of each coordinate point may be affected by the previous prediction result. Figure 1The results shown show that although the image's previous coordinate predictions are accurate, there may be a gradual increase in the prediction of subsequent points in the sequence. This cumulative error may eventually lead to errors in the prediction results of the entire image. In this process, the classification head of the model plays a crucial role. The quality of its performance is directly related to the accuracy of the coordinate point prediction, which in turn affects the overall image segmentation effect.

[0054] Traditionally, the classification head of a deep learning model is mainly composed of a multi-layer multilayer perceptron (MLP) network. However, recently, inspired by the Kolmogorov-Arnold representation theorem, a new type of neural network, the Kolmogorov-Arnold Network (KAN), has been proposed. While maintaining the same input and output data format as MLP, KAN has innovated the design strategy of the activation function, making it potentially an effective alternative to MLP. The introduction of KAN enables the present invention to improve the model segmentation performance under limited computing resources by fine-tuning part of the model architecture and its parameters without changing the existing model architecture of Polyformer.

[0055] Although the addition of the KAN classification head and the application of the dual-branch decoder module can improve the segmentation accuracy of the model to a certain extent, if the two decoder branches obtain the same features in the encoder processing stage, their prediction results may not be significantly different, thereby limiting the performance improvement potential of the ensemble learning strategy. In order to fully utilize the advantages of ensemble learning, the present invention proposes the need to introduce a new segmentation model that should be able to complement the output results of Polyformer, especially when predicting image areas that are difficult for Polyformer to process, it can provide accurate segmentation results.

[0056] like Figure 2 As shown, the output data format of the Polyformer model focuses on providing the coordinates of the predicted box and the precise location of the object edge. This output method reflects the performance characteristics of the model to a certain extent. Figure 1 By analyzing the incorrect segmentation cases in , this paper can summarize three typical problems that Polyformer may encounter during the segmentation process:

[0057] 1) Accuracy of prediction boxes: Although Polyformer is usually able to give basically correct prediction boxes when predicting the overall position of the target object, this is only the first step to a successful segmentation task.

[0058] 2) The problem of edge coordinate precision: When the predicted edge coordinates are wrong, it often manifests as a reduction in the coordinate spacing, which may be due to the model’s inability to capture target details.

[0059] 3) Partial segmentation problem: In some cases, Polyformer can correctly predict the edge coordinates of a part of the target, but fails to completely segment the entire target object, resulting in incomplete segmentation results.

[0060] In addition, although the special output data format of Polyformer has advantages in some aspects, it also brings certain limitations. This format may be difficult to directly compare and integrate with the prediction results of other models with superior performance, limiting the further improvement of model performance and the expansion of its application scope.

[0061] To overcome these limitations, it is necessary to explore new model architectures and integration strategies so that the prediction results of Polyformer can be fully utilized and combined with the advantages of other models to improve the overall segmentation performance and robustness.

[0062] However, recent breakthroughs in large-scale model research have brought new development opportunities for image segmentation technology. In April 2023, Meta launched an innovative segmentation model called Segment Anything Model (SAM). As a prompt model, the SAM model has the ability to segment any image without any manual annotation. The model can accept a variety of input prompts including points, boxes, or masks to clearly indicate the target object that needs to be segmented.

[0063] Compared with traditional segmentation models, the core advantage of SAM lies in its versatility. It does not require customized training or fine-tuning for specific tasks or datasets, but can directly apply pre-trained models to perform segmentation tasks, which greatly improves the flexibility and application scope of the model. The architectural details of SAM are in Figure 3 Detailed display in .

[0064] In view of these characteristics of the SAM model, the present invention proposes an innovative ensemble learning method. This method combines the output data of the Polyformer model and the input prompt characteristics of the SAM model. By using the prediction results of the Polyformer as the input prompts of the SAM, the present invention can fully utilize the zero-sample learning ability of the SAM. In addition, through the ensemble learning strategy, the present invention effectively integrates the prediction results of the SAM with the dual decoder prediction results of the Polyformer, forming a dual-head integrated indicator image segmentation algorithm framework based on SAM completion. This framework significantly improves the segmentation performance of the model in processing complex scenes, especially in the indicator image segmentation task, showing excellent segmentation accuracy and robustness.

[0065] Dual decoder integrated indicator image segmentation algorithm framework based on SAM completion

[0066] Overall framework composition: see Figure 4 The algorithm framework proposed in this invention mainly consists of three modules: 1) multimodal information encoder module; 2) dual-branch decoder module; 3) SAM-based segmentation and completion module.

[0067] Among them, the multimodal information encoder module includes: 1) image encoder; 2) text encoder; 3) multimodal information encoder.

[0068] The dual-branch decoder module includes: 1) a multimodal decoder based on MLP; 2) a multimodal decoder based on KAN; and 3) a decoder prediction result integration module.

[0069] The SAM-based segmentation and completion module includes: 1) a SAM input selection module; 2) a SAM prediction module; and 3) a SAM prediction result denoising module.

[0070] Overall framework operation process:

[0071] The multimodal information encoder module process: 1) The target image is input into the image encoder to obtain image features; 2) The target text is input into the text encoder to obtain text features; 3) The image features and text features are concatenated and input into the multimodal encoder to obtain multimodal features.

[0072] The main function of this module is to extract different modal information from the input, combine them, and extract multimodal features. The output of this module will be used as the input of the dual-branch decoder module.

[0073] Dual-branch decoder module process: 1) The multimodal features output by the multimodal information encoder module are respectively input into the dual-branch decoder to obtain two sets of segmentation coordinate results predicted by MLP and KAN respectively. 2) The two sets of results are input into the integration module, with the MLP prediction result as the main one. If the average distance of the segmentation target edge point coordinates in the MLP prediction result is less than N, the KAN prediction result is compared. If the average distance of the segmentation target edge point coordinates in the KAN prediction result is greater than the average distance of the MLP result, the KAN prediction result is used. Otherwise, the MLP prediction result is used as the decoder output result.

[0074] The main function of this module is to preliminarily predict the segmentation target mask. After obtaining the segmentation target edge point coordinate sequence predicted by the dual-branch decoder module, the segmentation target contour can be outlined according to the edge coordinate data to form a preliminary segmentation target mask. The output of this module will serve as the input of the SAM-based segmentation completion module.

[0075] SAM-based segmentation completion module process: 1) The module obtains the prediction results of the dual-branch decoder, and determines the prompt data type input to SAM according to the coordinate results output by the decoder: if the average distance of the edge point coordinates is less than N, only the target box coordinates are used as the prompt input to SAM; otherwise, n positive sample points are selected from the preliminary segmentation target mask formed by the decoder, and the coordinates of the positive sample points and the target box coordinates are simultaneously input as prompts to SAM; 2) The SAM prediction module accepts the prompt and the target image as input, outputs the segmentation target mask predicted by SAM, and inputs it into the denoising module; 3) The SAM prediction result denoising module obtains the segmentation target mask predicted by SAM as input, and uses the breadth learning priority algorithm to count the number of segmentation blocks contained in the SAM output result. When the number is greater than Block_N, it indicates that the prediction result contains too much noise, and it is discarded, and only the decoder prediction result is used as the final result; otherwise, the segmentation target mask predicted by SAM is used as a supplement to the decoder prediction result, and the SAM prediction mask is combined with the decoder prediction mask to form the final segmentation target mask.

[0076] The main function of this module is to use SAM to complete the mask of the segmentation target based on the decoder prediction results.

[0077] Encoder architecture: The core of the algorithm framework proposed in this study lies in the design of its encoder, which consists of three main parts: visual encoder, text encoder and multimodal Transformer encoder.

[0078] 1) Visual Encoder: The present invention uses Swin Transformer as the image encoder, which is a Transformer-based architecture designed for image processing tasks. Swin Transformer effectively extracts image features through a hierarchical structure and a window mechanism, capturing both local and global information. In particular, the present invention uses the fourth layer features extracted by Swin Transformer as the visual representation, and the size of the feature map is H×W×C v , where H and W are the height and width of the feature map, respectively, and C v is the number of feature channels.

[0079] 2) Text Encoder: For text encoding, the present invention uses the BERT model, which is a pre-trained deep bidirectional Transformer widely used in natural language processing tasks. BERT can deeply understand the semantic content of the text and provide the model with an accurate grasp of the language description. For the input language description, the language embedding feature size generated by BERT is L×C l , where L is the length of the text, Cl is the embedding dimension.

[0080] 3) Multimodal feature fusion: In order to achieve effective fusion of image and text features, the present invention serializes the image features and maps them to the same embedding space with the text features through a fully connected layer. The mapping matrix W obtained by learning v and W l and the bias vector b v and b l , the present invention converts the visual and text features into a unified dimension and obtains the fused feature representation F′ v and F′ l These features are then concatenated to form multimodal features, which provide input to the subsequent Transformer encoder.

[0081] 4) Multimodal Transformer Encoder: The fused features are then fed into a multimodal Transformer encoder, which is composed of N stacked Transformer layers. Each layer contains a multi-head self-attention mechanism, layer normalization, and a feed-forward network to process multimodal features. In order to preserve the position information in the features, the present invention adds absolute position encoding and relative position bias to the image and text features.

[0082] Through this carefully designed encoder architecture, the algorithm of the present invention can effectively process and integrate information from different modalities, laying a solid foundation for accurate image segmentation tasks.

[0083] Decoder architecture (see Figure 5 ): In the algorithm framework proposed in the present invention, the decoder part consists of two main branches: a decoder with a multi-layer perceptron (MLP) as the classification head, and a decoder with a Kolmogorov-Arnold Network (KAN) as the classification head, forming a dual-branch decoder structure.

[0084] This decoder uses a regression-based Transformer decoder, which directly predicts continuous floating-point coordinates, avoiding the quantization errors that may occur when quantizing coordinates to discrete values. The key to the decoder is to predict the coordinate sequence of the edge vertices of the target polygon, which define the outline of the target object. The decoder consists of N Transformer decoder layers, each of which contains a multi-head self-attention layer, a multi-head cross-attention layer, and a feedforward network to capture the relationship between multimodal features and 2D coordinate embeddings. The PolyFormer model innovatively constructs a 2D coordinate codebook to obtain precise coordinate embeddings for any floating-point coordinates from the height and width dimensions and the embedding dimension. The decoder obtains these precise coordinate embeddings from neighboring grid points through bilinear interpolation. The formula is as follows:

[0085]

[0086] 1) MLP decoder branch: Based on the output of the last Transformer decoder layer, two lightweight MLP classification heads are built to generate the coordinate prediction results of the edge points of the segmentation target. The category prediction head consists of two layers of MLP to determine whether the output is a coordinate marker, a separator marker, or a sequence end marker. The coordinate prediction head is a three-layer MLP feed-forward network that predicts the coordinates of the upper left and lower right corners of the prediction box of the segmentation target and the 2D coordinates of the edge vertices.

[0087] 2) KAN decoder branch: The KAN decoder is basically the same as the MLP decoder branch in structure, but with two major differences. First, it replaces the prediction head from MLP to KAN to introduce the advanced activation function design of the KAN network. Second, a multi-scale feature fusion module is added to the input of the decoder, which fuses the initial multimodal features extracted by the visual encoder and the text encoder with the features processed by the multimodal Transformer encoder. The KAN branch decoder is designed to effectively supplement the output results of the MLP branch decoder using consumer-grade graphics cards without increasing the computational burden. The introduction of the multi-scale feature fusion module is intended to enable the decoder to accept and process unfused visual features and text features, thereby obtaining richer coarse-grained information, and forming a difference with the prediction results of the MLP decoder branch, thereby enhancing the overall performance of the model.

[0088] Dual-branch decoder output integration strategy: One of the key components of this algorithm framework is the dual-branch decoder output integration module. This module is responsible for fusing the segmentation results generated by the MLP and KAN branch decoders to improve the overall segmentation accuracy.

[0089] The integration process first calculates the average distance between the two branches for the predicted boundary points of the segmentation target independently. This step is to quantify the accuracy of the predicted boundary points and provide a basis for subsequent integration decisions. Figure 1 Based on the characteristics of the segmentation error images shown in , the present invention screens out those images with smaller average point distances. This screening criterion is based on the assumption that a smaller average point distance may mean that the prediction result is closer to the actual situation. On the screened image set, the present invention further compares the average point distances of the prediction results of the two branches of MLP and KAN. The purpose of this comparison is to determine which branch performs better on a specific image. Finally, the present invention selects the prediction result with a larger average point distance as the final prediction result of the image. The advantage of this strategy is that it allows the model to flexibly select more reliable outputs in different situations, thereby improving the segmentation performance overall.

[0090] Through this integrated learning strategy, the present invention can effectively combine the advantages of the two branch decoders while reducing their respective limitations to achieve better segmentation results. This method not only improves the accuracy of segmentation, but also enhances the adaptability and robustness of the model for different types of images. Figure 6 .

[0091] The relevant code is as follows.

[0092]

[0093] Segmentation object completion module based on SAM (see Figure 7 ): Due to the limitation of not being able to fine-tune all parameters of the model, the dual-branch decoder integration of the present invention occasionally faces challenges on certain datasets, resulting in unsatisfactory segmentation results. To address this issue, the present invention integrates the Segment Anything Model (SAM) and uses its powerful zero-shot learning capability to improve the segmentation effect of the model.

[0094] The implementation process involves a series of modules: judgment module, positive sample point generation module, SAM module itself, and the final noise processing module. The SAM-based segmentation target supplementation module first obtains the output of the decoder to form the prompt data required for the SAM operation. The type of prompt data is determined based on the average point distance of the input result:

[0095] 1) If the average point distance is small, only the predicted box coordinates are used as hint data for SAM.

[0096] 2) If the average point distance is large, positive sample points are randomly selected within the segmented object and used together with the predicted box coordinates as hint data for SAM.

[0097] However, the supplementary results of SAM may contain errors or noise, such as Figure 8-9 As shown. These errors usually manifest as multiple prediction blocks. To solve this problem, the present invention develops a block counting module, which uses a breadth-first search algorithm to count the number of segmented blocks contained in the SAM output results. The decision process is as follows:

[0098] 1) If the number of segmented blocks exceeds a predefined threshold, the SAM result is discarded and the prediction of the dual-branch decoder is used as the main segmentation result.

[0099] 2) If the number of blocks does not exceed the threshold, the output of SAM is merged with the prediction of the dual-branch decoder to form a new, more refined segmentation result.

[0100] This ensemble approach ensures that the segmentation ability of the model is not only enhanced by the powerful learning capability of SAM, but also refined by a careful selection and ensemble process that minimizes noise and maximizes accuracy.

[0101] The relevant code is as follows.

[0102]

[0103]

[0104] 4 Experiments

[0105] 4.1 Experimental parameters

[0106] The model code in this invention is written in Python and constructed using the Pytorch framework. The experimental data used are the public datasets RefCOCO, RefCOCO+, and RefCOCOg. The model input image size is 512×512 to ensure consistent processing.

[0107] For fine-tuning of the KAN decoder branch, the present invention sets the learning rate to 5×10 -5 , the training rounds are 100 rounds. During the training process, the present invention only activates the parameters of the KAN decoder branch for updating, and freezes the parameters of the rest of the model to improve the training efficiency and stability. In addition, in the testing phase, the present invention sets the average point distance threshold to 4 to evaluate the accuracy of the predicted boundary points. The positive sample point generation value is set to 5 to enhance the model's ability to recognize positive samples. The discard threshold of the SAM prediction noise module is set to 15 to filter out possible noise interference and ensure the accuracy of the final segmentation result.

[0108] 4.2 Experimental Results

[0109] In this section, we present the experimental results of different methods on the datasets RefCOCO, RefCOCO+, and RefCOCOg. Table 1 lists the performance data including Polyformer and several other baseline models, as well as the results of the KAN decoder, the dual-branch decoder ensemble, and the ensemble method combined with SAM completion.

[0110] The results show that the Polyformer model achieves excellent performance on all datasets. Furthermore, by introducing the KAN decoder, the present invention observed a slight improvement in performance. When the dual decoder is integrated, the model performance is further enhanced. Most notably, after combining the SAM module, the model achieves the best performance on all evaluation indicators, especially on the RefCOCO dataset, where mIoU and oIoU reach 77.03% and 75.31% respectively, showing that the complementary role of SAM on difficult-to-segment images greatly improves the segmentation accuracy.

[0111] Table 1 Experimental data of 10 networks

[0112]

[0113] 4.3 Analysis

[0114] Table 1 summarizes the performance of several classic image segmentation models including Polyformer on the three datasets of RefCOCO, RefCOCO+ and RefCOCOg. These data not only show the basic performance of each model, but also reflect the performance improvement achieved by the improvements made by the present invention based on the Polyformer model.

[0115] When the present invention only uses the KAN decoder branch for image segmentation prediction, it is observed that the performance of the model on the RefCOCO and RefCOCO+ datasets is improved. However, on the more challenging RefCOCOg dataset, the performance slightly decreases. This indicates that the KAN decoder may have limitations when processing more complex image scenes. Further, by applying an ensemble learning strategy, the present invention fuses the prediction results of the two decoder branches. This strategy improves the performance of the model on all three datasets, showing the effectiveness of the ensemble method in improving segmentation accuracy. The most significant performance improvement comes from the SAM-based completion module introduced on the basis of the dual-branch decoder ensemble results. This improvement enables the model to achieve 77.03% mIoU and 75.31% oIoU on the RefCOCO dataset, and 71.63% and 68.07% mIoU and 70.65% and 67.80% oIoU on the two more challenging datasets RefCOCO+ and RefCOCOg, respectively. This result demonstrates that our model can achieve state-of-the-art segmentation performance even when fine-tuned using only a consumer-grade graphics card.

[0116] These results highlight the strength of our model design, especially when dealing with complex images and hard-to-segment objects, where significant performance gains are achieved by combining predictions from different decoder branches and the powerful zero-shot learning capability of SAM.

[0117] 4. Technical Effects of the Invention

[0118] With respect to the “problems existing in the prior art” in the above “background technology” section, the effects achieved are specifically described in combination with the improvements in the technical solution of the present invention, that is, how each improvement brings about these advantages is specifically analyzed in a reasoning way, so that the description of the advantages is reasonable and well-founded; it can be low cost, simple structure, good performance, easy to use and high production quality.

[0119] For example:

[0120] The dual-head integrated indication image segmentation algorithm framework based on Segment Anything Model (SAM) proposed in the present invention aims at the problems existing in the prior art and achieves significant performance improvement and many advantages through a series of innovative technical solutions.

[0121] First, the framework of the present invention realizes more accurate pixel-level positioning of targets in images by combining the collaborative work of encoders and decoders. The encoder uses Swin Transformer as a visual encoder, BERT model as a text encoder, and multimodal Transformer encoder for feature fusion, effectively extracting and integrating feature information from different modalities. Secondly, the innovative design of the decoder part, including two classification heads, MLP and KAN, not only improves the accuracy of segmentation, but also enhances the model's ability to capture features of different scales. In particular, the introduction of the multi-scale feature fusion module in the KAN decoder enables the model to better handle complex image structures. Further, the present invention adopts an integrated learning strategy to comprehensively consider the output results of the dual-branch decoder, and selects more accurate prediction results by calculating the average distance of the predicted boundary points, thereby further improving the accuracy of segmentation. Most importantly, the introduction of SAM greatly enhances the model's ability to handle difficult-to-segment images. The zero-sample learning ability of SAM enables the model of the present invention to effectively segment new scenes without additional training data, which is difficult to achieve in the prior art.

[0122] Specifically, the model of the present invention first prepares the prompt data required by SAM through the judgment module and the positive sample point generation module. Then, these data are input into the SAM module to generate the supplementary segmentation results. Finally, the results are optimized through the noise processing module to ensure the accuracy of the final output.

[0123] Through these technical improvements, the model of the present invention not only shows higher accuracy in the task of indicating image segmentation, but also has a significant improvement in robustness, especially when processing images with complex scenes and difficult-to-segment objects. These advantages make the model of the present invention more potential and valuable in practical applications.

[0124] Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A dual decoder integrated indication image segmentation algorithm based on SAM completion, characterized in that: The framework corresponding to the algorithm includes three modules: (1) a multimodal information encoder module, (2) a dual-branch decoder module, and (3) a SAM-based segmentation completion module.

2. The dual decoder integrated indication image segmentation algorithm based on SAM completion according to claim 1 is characterized in that: The multimodal information encoder module includes: an image encoder based on SwinTransformer, a text encoder based on Bert, and a multimodal information encoder based on Transformer; The workflow of the multimodal information encoder module is: Target image I∈R H×W×3 Input to the image encoder composed of SwinTransformer Base, and obtain image features from the fourth stage of the encoder Where H is the height of the input image; W is the width of the input image, C v The number of channels set for the fourth stage SwinTransformerBase module; Target text T∈R L Input the text encoder composed of Bert_12 and obtain text features through the language embedding model Where L is the length of the text, which is determined by the longest text in a set of inputs; C l The number of channels set for the Bert_12 module; The image features are then flattened into And by using the fully connected layer to project the image features and text into the same embedding space and then splice them, we get the features Input a multimodal encoder consisting of 6 layers of Transformer to obtain multimodal features Among them, C is the number of channels of the concatenated multimodal features. The function of this module is to extract different modal information from the input, combine them, and extract multimodal features; the output of this module will serve as the input of the dual-branch decoder module.

3. The dual decoder integrated indication image segmentation algorithm based on SAM completion according to claim 1, characterized in that: The dual-branch decoder module includes: a multimodal decoder based on MLP, a multimodal decoder based on KAN, and a decoder prediction result integration module; The workflow of the dual-branch decoder module is: Step 2.1: The multimodal features output by the multimodal information encoder module are respectively input into the MLP-based multimodal decoder and the KAN-based multimodal decoder to obtain two sets of segmentation coordinate results predicted by MLP and KAN respectively. The result is not a predicted image mask, but a continuous coordinate point sequence consisting of the coordinates of the upper left corner and the lower right corner of the segmentation target prediction box and the coordinates of the continuous edge points of the segmentation target; Step 2.2: The two sets of segmentation coordinate results are input into the decoder prediction result integration module, with the MLP prediction result as the main one. If the average distance of the segmentation target edge point coordinates in the MLP prediction result is less than N, the KAN prediction result is compared. If the average distance of the segmentation target edge point coordinates in the KAN prediction result is greater than the average distance of the MLP result, the KAN prediction result is used. Otherwise, the MLP prediction result is used as the decoder output result. The function of this module is to form a preliminary segmentation target mask. After obtaining the segmentation target edge point coordinate sequence predicted by the dual-branch decoder module, the segmentation target contour can be outlined according to the edge coordinate data, thereby forming a preliminary segmentation target mask. The output result of this module will be used as the input of the SAM-based segmentation completion module.

4. The dual decoder integrated indication image segmentation algorithm based on SAM completion according to claim 3 is characterized in that: The SAM-based segmentation completion module includes: a SAM input selection module, a SAM prediction module, and a SAM prediction result denoising module; SAM-based segmentation and completion module process: Step 3.1 The module obtains the segmentation target prediction box coordinates and the preliminary segmentation target mask integrated and output by the dual-branch decoder, and determines the prompt data type of the input SAM according to the coordinate results output by the decoder; Prompt refers to the prompt information input into SAM, which is used to guide or stimulate the model to generate specific output: If the average distance of edge point coordinates is less than N, only the segmentation target prediction box coordinates integrated and output by the dual-branch decoder are used as prompt input to SAM; otherwise, n positive sample points are selected from the preliminary segmentation target mask formed by the decoder, and the coordinates of the positive sample points and the target box coordinates are simultaneously used as prompt input to SAM; Step 3.2 The SAM prediction module accepts prompt and target image as input, obtains the embedding vector of the target image through the image encoder composed of VisionTransformer, and encodes the input prompt through the prompt encoder to obtain the corresponding embedding vector. After the SAM image decoder receives the image embedding vector and the prompt embedding vector, it outputs the segmentation target mask predicted by SAM and inputs it into the denoising module; Step 3.3 The SAM prediction result denoising module obtains the segmentation target mask predicted by SAM as input, and uses the breadth-first learning algorithm to count the number of segmentation blocks contained in the SAM output result. When the number is greater than Block_N, it indicates that the prediction result contains too much noise, so it is discarded and only the prediction result integrated by the decoder dual-branch decoder is used as the final result; otherwise, the SAM predicted segmentation target mask′ is used as a supplement to the decoder prediction result, and the SAM prediction mask′ is combined with the decoder prediction mask, that is, mask final =maskANDmask′, AND is a logical OR operation; thus forming the final segmentation target mask final ; The function of this module is to use SAM to complete the mask of the segmentation target based on the decoder prediction results.

5. A dual decoder integrated indication image segmentation device based on SAM completion, characterized in that: Image segmentation is achieved by using the dual decoder integrated indication image segmentation algorithm based on SAM completion as described in any one of claims 1 to 4.

Images