Dynamic decision image segmentation method based on self-prompt guidance

Abstract

The invention discloses a dynamic decision image segmentation method based on self-prompt guidance, and belongs to the technical field of computer vision. The method comprises the following steps: acquiring a data set and preprocessing data; an SAM model is constructed and optimized by combining specific domain knowledge, so that the SAM model can better adapt to complex scenes such as remote sensing images and medical images; adaptive prompts are automatically generated through a guide model, and the dependence of a traditional method on artificial points, frames or coarse-grained masks is eliminated; a dynamic decision-making mechanism is designed, and a decision-making strategy is dynamically adjusted according to image contents, so that the decision-making process is more flexible, and complex and changeful image contents are effectively handled; by optimizing the decision process, the model reasoning process is more transparent, and the interpretability and credibility of the labeling result are enhanced, so that the field requirements with strict requirements on high precision and high credibility are met.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a dynamic decision image segmentation method based on self-prompt guidance. Background Art

[0002] With the rapid development of artificial intelligence technology, especially breakthroughs in deep learning and computer vision, image segmentation technology has gradually become one of the key technologies in many industries, including remote sensing images, medical imaging, autonomous driving, industrial inspection and other fields. Image segmentation is not only the core link of computer vision tasks, but also the basis of various practical applications. However, existing image segmentation methods still face many bottlenecks when facing large-scale data annotation needs.

[0003] Traditional image segmentation methods rely on manual annotation or early algorithmic techniques, such as template matching and region growing. These methods work well in simple scenes, but they often face great challenges when dealing with complex image content (such as multi-target scenes, occlusion, and blurred boundaries). The segmentation effect is limited, the accuracy is not high, and it is difficult to handle large-scale data requirements. In addition, the computational efficiency of traditional methods is low, and it is difficult to achieve flexible adaptive adjustments, which limits their scalability in practical applications.

[0004] In recent years, the rapid development of deep learning technology, especially the introduction of generative adversarial networks, self-attention mechanisms and large-scale pre-trained models, has significantly improved image segmentation technology. Pre-trained models represented by the SAM model have become an important breakthrough in the field of image segmentation by virtue of their powerful instance segmentation capabilities in natural images and excellent performance in zero-sample conditions. However, despite the excellent performance of SAM in natural images, it still faces several problems when applied to fields such as remote sensing images and medical imaging. First, the SAM model is not optimized for the needs of specific fields, and its segmentation effect is often unsatisfactory in scenes such as complex backgrounds, target deformation or occlusion. In addition, as a category-independent interactive segmentation model, the SAM model lacks the ability to distinguish object categories, and its segmentation results are heavily dependent on manually provided prompts (such as points, boxes or coarse-grained masks). This artificial dependence limits the application of the model in automated segmentation tasks, especially when large-scale data processing, the efficiency of artificial prompts is low, and fast batch processing cannot be achieved.

[0005] In addition, the existing segmentation methods based on the SAM model lack an effective dynamic decision-making mechanism. Many methods rely on fixed local areas for segmentation and lack flexible dynamic decision-making capabilities. When the target morphology changes or is occluded, it is difficult for existing technologies to adaptively adjust the segmentation strategy, resulting in insufficient annotation accuracy and stability, and unable to cope with the diversity and variability of image content. Finally, the lack of automatic segmentation accuracy is still a problem that needs to be solved urgently. Although existing technologies can provide more accurate segmentation results in simple scenes, in complex image scenes, especially remote sensing images and medical images, there are still problems such as mis-segmentation, over-segmentation or non-segmentation, which seriously affect the quality of the segmentation results and restrict its promotion and application effect in practical applications. Summary of the invention

[0006] The purpose of the present invention is to propose a dynamic decision-making image segmentation method based on self-prompt guidance in view of the shortcomings of existing automatic segmentation methods based on the SAM model in terms of model adaptability, interactive dependence and dynamic decision-making ability.

[0007] The object of the present invention is achieved through the following technical solution: a dynamic decision image segmentation method based on self-prompt guidance, comprising the following steps:

[0008] Obtaining original images of the target area as samples, taking out some samples for manual annotation and making corresponding labels to construct a training data set, wherein the training data set includes a training set and a validation set;

[0009] Constructing a SAM model guided by a task self-prompter and a SAM model guided by a spatial position information driven self-prompter according to the training data set;

[0010] Train and fine-tune two SAM models;

[0011] After fusing the two fine-tuned SAM models, the fused model can dynamically adjust the reasoning strategy according to the specific image content and task requirements of the image to be labeled, thereby generating multiple mask regions;

[0012] Through the synthesis of multiple mask regions, noise removal, region screening and similarity-based optimization, the mask image is refined and semantically consistent, the quality and visualization of the segmentation results are improved, and finally a semantic-level instance segmentation mask map is obtained.

[0013] Furthermore, the construction of the training data set includes: first, ensuring that the extracted samples cover the diversity of the target objects; then, for each original image, using a manual segmentation and annotation tool to generate a segmentation mask of the target object, and annotating the bounding box and category label of the target; finally, based on the original image and the generated segmentation mask, bounding box and category label, constructing a training data set for training and verification, and randomly dividing it into a training set and a verification set in a ratio of 9:1.

[0014] Further, the SAM model guided by the task self-prompter includes: an image encoder module, a feature aggregator module, a feature separator module, a task-based prompter module and a mask decoder module;

[0015] The image encoder module uses a pre-trained SAM visual encoder to extract image features;

[0016] The feature aggregator module fuses and processes the input features through convolution and normalization operations to extract and integrate information of different scales;

[0017] The feature separator module is used to integrate features of different scales through upsampling, downsampling, convolution and lateral connection operations to enhance the multi-scale perception ability of the model;

[0018] The task-based prompter module includes: a transformer encoder for performing self-attention operations on four different scales of input feature maps from the feature separator module, extracting global semantic information and generating feature maps of four scales; a transformer decoder for interacting with zero-initialized task-related query tags and fusing with feature maps of four scales; and a mask filter, a prompt embedding and a semantic category for each instance generated through a mask projection layer, a prompt projection layer and a category inference layer, and finally generating a coarse mask based on the maximum size feature map;

[0019] The mask decoder module generates high-precision image segmentation masks through image embedding, mask labeling, 2 two-way transformer modules, 4 multi-layer perceptron modules and 2 upsampling transposed convolutional layers.

[0020] Further, the SAM model driven by spatial position information and guided by a self-prompter includes: an image encoder module, a feature aggregator module, a feature separator module, a prompter module driven by spatial position information, and a mask decoder module;

[0021] The image encoder module uses a pre-trained SAM visual encoder to extract image features;

[0022] The feature aggregator module fuses and processes the input features through convolution and normalization operations to extract and integrate information of different scales;

[0023] The feature separator module is used to integrate features of different scales through upsampling, downsampling, convolution and lateral connection operations to enhance the multi-scale perception ability of the model;

[0024] The prompter module driven by spatial position information includes: a lightweight region proposal network, which is used to generate category scores and bounding box regression values ​​from multi-scale input feature maps, remove redundant boxes through non-maximum suppression, and output multiple region proposal boxes; a position encoding module, which is used to apply sinusoidal position encoding to the input feature map to enhance its spatial position information; a region of interest alignment layer, which is used to extract features corresponding to each proposal box from the multi-scale feature map; a shared feature extractor, which is used to process the extracted features and generate shared features; a category inference layer and a boundary inference layer, which are used to generate semantic categories and bounding boxes for each instance, respectively; a region of interest feature extraction module, which is used to extract target features; a spatial information prompting layer, which is used to generate sparse target features and pass them to a mask decoder to generate a final multi-semantic segmentation mask;

[0025] The mask decoder module generates high-precision image segmentation masks through image embedding, mask labeling, 2 two-way transformer modules, 4 multi-layer perceptron modules and 2 upsampling transposed convolutional layers.

[0026] Furthermore, the training and fine-tuning of the two SAM models includes:

[0027] Preprocessing the training data set, including image loading, standardization, data enhancement, target screening, and image size unification to ensure the consistency and validity of image and mask information;

[0028] The preprocessed data is fed as input to the SAM model guided by the task self-cue and the SAM model guided by the spatial position information self-cue, freezing the parameters of the image encoder and fine-tuning the parameters of the feature aggregator, feature separator, cue encoder and mask decoder modules.

[0029] The AdamW optimizer is used to fine-tune the training parameters, including weight decay, initial learning rate, cosine annealing scheduler, linear warm-up strategy, batch size and number of iterative updates; the total loss function is calculated for the two SAM models respectively, where for the SAM model guided by the task self-prompter, the Hungarian algorithm is used to match the predicted mask with the real instance mask, and the category matching loss, mask cross entropy loss and mask Dice loss are calculated to ensure that each predicted instance is paired with the most appropriate real instance, and supervised learning of multi-category classification loss and mask classification loss is performed; for the SAM model guided by the self-prompter driven by spatial location information, the total loss includes region proposal loss, classification loss, regression loss and mask loss; hyperparameter optimization is performed through the training set, model performance is evaluated using the validation set, the best model configuration is selected based on the mean average precision (mAP), and finally the model with the best performance on the validation set is selected.

[0030] Furthermore, the two fine-tuned SAM models are fused, and the fused model dynamically adjusts the reasoning strategy according to the specific image content and task requirements of the image to be annotated, including:

[0031] Preprocess the image to be annotated, including loading the image, normalizing it, scaling it to the target size, and performing necessary padding to ensure that the image meets the model input requirements;

[0032] The preprocessed image is sent to the fused model for reasoning; the fused model includes two guiding mechanisms: the SAM model guided by the task self-prompter is used to adaptively select the most appropriate feature extraction and reasoning method according to the task requirements to ensure that the task characteristics are accurately processed; the SAM model driven by spatial position information is used to improve the model's modeling ability of the target spatial features by analyzing the distribution and layout of the target in space; a set of instance segmentation masks are generated by the fused model to represent the area of ​​each instance in the image; each mask is stored together with its corresponding label, score and other information as part of the reasoning result;

[0033] After generating multiple instance segmentation masks, each mask is processed through the fused model to dynamically select the optimal mask.

[0034] Furthermore, the processing of each mask by the fused model includes:

[0035] The area of ​​each mask is calculated through the fused model, and the effective masks with larger areas are screened out;

[0036] If there are overlapping areas of multiple masks, the fused model dynamically decides which masks to keep based on the area of ​​the overlapping area and the score of each mask. For overlapping areas with an overlapping area smaller than the set threshold, the mask with a larger area is selected; for larger overlapping areas, the mask with a higher score is selected first.

[0037] The selected masks, labels, and scores are stored in the final result array for subsequent analysis and processing.

[0038] Furthermore, the method of achieving precision and semantic consistency of the mask image through synthesis of multiple mask regions, noise removal, region screening and similarity-based optimization includes:

[0039] Extract multiple masks and category labels from the model fusion process; according to each category label, obtain the corresponding color from the palette and apply the color to the corresponding mask area; by assigning different colors to each binary mask, multiple masks are drawn on the same image to generate an instance mask map containing semantic information. The semantic category of the map is represented by the corresponding color, ensuring that each area can clearly identify its semantic category;

[0040] For the synthesized mask image, the connected areas of each color mask are marked to identify different areas in the image; an area threshold is set and connected areas smaller than the threshold are filtered out to remove the noise area; this process ensures that only larger valid areas are retained in the image; the filtered mask areas are merged into the final image to optimize the visualization effect and ensure the accuracy and effectiveness of the image;

[0041] Merge the overlapping areas between different masks.

[0042] Furthermore, the merging of overlapping areas between different masks is specifically as follows:

[0043] Define the expansion threshold of the adjacent area to determine the expansion range of the mask area during synthesis, ensuring that adjacent areas can be merged to avoid omissions;

[0044] Performing a connected region analysis on each color mask to extract statistical information of the region; based on the statistical information, expanding the bounding box of each connected region to cover more adjacent regions using a set expansion threshold;

[0045] Check the overlap between each connected region and other color masks, and determine whether the overlap belongs to an adjacent region by calculating the intersection. At the same time, apply the minimum pixel threshold to exclude the noise region that is too small. For adjacent regions that meet the merging conditions, merge them into one region and replace the color label of the adjacent region with the label of the current region.

[0046] The synthesized mask image is updated to ensure that the image only contains important and coherent regions, and remove noise and irrelevant small regions, thereby generating a concise and semantically consistent final mask image.

[0047] The present invention also provides a dynamic decision image segmentation device based on self-prompt guidance, comprising one or more processors for implementing the dynamic decision image segmentation method based on self-prompt guidance.

[0048] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention proposes a fusion and dynamic decision method combining the SAM model guided by the task self-prompter and the SAM model guided by the spatial position information self-prompter. The method can dynamically adjust the reasoning strategy according to the specific image content and task requirements, thereby achieving more accurate and efficient automatic image segmentation and significantly improving the instance segmentation accuracy.

[0049] (2) In view of the shortcomings of the existing technology that requires manual provision of prior guidance information (such as points, boxes or rough masks), this invention introduces a heuristic prompt learning mechanism, which enables the model to automatically generate adaptive prompts and complete accurate segmentation of the target without manual intervention. This innovation greatly improves the segmentation efficiency, reduces labor costs, and enhances the practicality of the method.

[0050] (3) The present invention inherits the zero-sample generalization capability of the SAM model and further introduces semantic category information on this basis. In the process of automatic image segmentation, the addition of semantic information makes the segmentation results not only more accurate in morphology, but also provides stronger differentiation capabilities at the semantic level. This innovation improves the overall accuracy of image segmentation results, especially in specific fields (such as object classification in remote sensing images or organ recognition in medical images), and can provide more accurate and reliable support, thereby broadening the application scenarios and practical value of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flow chart of the present invention;

[0052] Figure 2 A schematic diagram of the SAM model guided by the task self-prompter of the present invention;

[0053] Figure 3 A schematic diagram of a SAM model driven by a spatial position information self-prompter guidance according to the present invention;

[0054] Figure 4 It is a schematic diagram of the device of the present invention. DETAILED DESCRIPTION

[0055] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0056] like Figure 1 As shown, an embodiment of the present invention provides a dynamic decision image segmentation method based on self-prompt guidance, comprising the following steps:

[0057] Step 1: Dataset acquisition and annotation

[0058] Obtain raw image data from specific fields (such as remote sensing images, medical images, etc.), select about 1,000 sample images, and ensure that these images can cover the diversity of target objects. For each image containing a target object, use the manual segmentation and annotation tool to generate a segmentation mask for the target object, and at the same time generate the target's bounding box and category label.

[0059] Step 2: Model construction and design

[0060] 2.1 Assumptions Indicates the training data set contains training data. Indicates training data. is the 3D image volume data, Indicates the height of the volume data, Indicates the width of the volume data, Indicates the number of layers of volume data; is the true value annotated for the corresponding training data, including The coordinates of the object's bounding box ( ), its associated semantic category ( ) and binary mask ( ). The main goal is to train two self-prompt guided SAM models to handle the For any image, we can infer its semantic category and instance mask.

[0061] 2.2、Construct the SAM model guided by the task self-prompter: Figure 2 As shown in Figure 3, the model contains five main parts, namely, the image encoder module, the feature aggregator module, the feature separator module, the task-based prompter module and the SAM mask decoder module.

[0062] The image encoder module uses a pre-trained SAM visual encoder to extract image features. First, the input image is processed by the patch embedding module, which uses convolution operations to split the image into small patches and maps each patch to a high-dimensional feature space of 1280 dimensions to extract rich local information. Next, the image features are processed by 32 layers of transformer encoding layers, each of which contains a 32-head multi-head self-attention layer, two layers of feedforward neural networks, layer normalization, and residual connections to capture global context information and enhance the expressiveness of feature representation.

[0063] The feature aggregator module fuses and processes the input features through a series of convolution and normalization operations, aiming to extract and integrate information at different scales. First, the input 256-channel features are expanded to 512 channels through 1x1 convolution, and then layer normalization is performed to stabilize the training process. Next, the features are further processed using 3x3 convolutions, keeping the number of channels at 512, and layer normalization is performed again. Finally, another layer of 3x3 convolution reduces the feature dimension to 256 channels, and layer normalization is performed again to ensure the stability and consistency of the features. This series of operations enhances the feature expression ability through multiple convolutions and normalizations, helping the model to effectively utilize the fused multi-scale information in subsequent tasks.

[0064] The feature separator module aims to integrate features of different scales through a series of sophisticated upsampling, downsampling, convolution and lateral connection operations to enhance the multi-scale perception ability of the model. First, the module uses two layers of upsampling operations to process low-resolution feature maps. Specifically, the first layer uses transposed convolution to upsample the 256-channel feature map to 128 channels, then applies the Gaussian Error Linear Unit (GELU) activation function, and then upsamples it to 64 channels through transposed convolution; the second layer transposes the 256-channel feature map to 128 channels. At the same time, for high-resolution feature maps, the module uses maximum pooling operations to downsample and reduce their size. These operations create a hierarchical structure for feature maps of different scales, so that features of each scale can be effectively processed and utilized. In order to achieve the unification of features of different scales, the module further maps the number of channels between features of different scales through lateral connection operations. Specifically, the module uses four 1x1 convolutions to adjust the number of channels of the feature map to a unified 256 channels: first, the 64-channel feature is mapped to 256 channels through convolution; then, the 128-channel feature is mapped to 256 channels through convolution; for the 256-channel feature, the number of channels is kept unchanged through 1x1 convolution. Next, the module uses four 3x3 convolution layers to further process all the features obtained through lateral connections to ensure that the fused features are consistent in the number of channels, and enhance their stability and expression ability through layer normalization. The operation of each convolution layer will unify the number of channels to 256, further improving the representation ability and quality of the feature map. Specifically, the four 3x3 convolution layers ensure the consistency of the number of channels in the process of processing the feature map, and enable the smooth fusion and optimization of feature information of different scales. Through these upsampling, downsampling, convolution and lateral connection operations, the features from different scales are effectively separated and fused, providing a more stable and high-quality feature representation with multi-scale information for subsequent tasks, thereby enhancing the model's perception and understanding of complex image structures.

[0065] The task-based prompter module combines the encoder and decoder structure of the transformer, aiming to accurately generate multiple semantic segmentation masks through task-related query mechanisms. The module first concatenates the input feature maps of four different scales from the feature separator after position encoding and level encoding, flattens them and inputs them into the transformer encoder composed of 3 layers of attention layers and feedforward neural networks for self-attention operation to extract global semantic information. Next, the feature map output by the encoder is split into a set of feature maps of 4 scales , , , , these feature maps are consistent with the input feature maps of the original feature separator in terms of spatial size, where is the maximum size feature map, used to generate a rough mask The transformer decoder is then initialized with a zero-initialized learnable task-related query token. Interact with the 4 scale feature maps from the encoder , , , Fusion. The decoder passes through the mask projection layer Generate a per-instance mask filter , by hinting the projection layer Generate per-instance hint embeddings , and through the category inference layer Generate semantic categories for each instance The formulas for these operations are: , , , , ,in is a transformer decoder consisting of a criss-cross attention and feed-forward neural network, and is the linear projection layer, It is a hint projection layer composed of two layers of multi-layer perceptrons. Indicates the number of prompt groups, Indicates the number of embeddings for each cue.

[0066] The SAM mask decoder module generates high-precision image segmentation masks through image embedding, mask labeling, two two-way transformer modules, four multi-layer perceptron modules, and two upsampled transposed convolutional layers. First, the high-dimensional image features from the image encoder module are processed by the image embedding module, where the feature dimension is reduced from 1280 to 256 through point-by-point convolution and standard convolution, and layer normalization is combined to improve the stability and usability of the features. Next, the mask labeling part is a rough mask generated by the task-based prompter module. The proposed method starts with a hint embedding layer and further generates hint embedding features of the mask through the hint embedding layer. These hint embedding features, together with the hint features from the hint projection layer, are processed through a two-way transformer module. In the two-way transformer module, the self-attention layer and the cross-modal attention layer enable the mask tags to interact deeply with the image features, thereby capturing rich spatial and semantic information in the image. The module includes a self-attention layer, a cross-modal tag-image cross-attention layer, a multimodal perceptron layer, and two cross-modal image-tag cross-attention layers, which extract and transform image features layer by layer to promote information fusion and representation. After these attention layers, the multi-layer perceptron module further transforms the features to generate mask filters. In order to solve the resolution problem in image segmentation, the decoder also includes two upsampling convolutional layers, which gradually restore the low-resolution feature maps to higher-resolution fine-grained feature maps, thereby ensuring the refinement and accuracy of the segmentation mask. In addition, the SAM mask decoder uses position encoding and category embedding layers to provide spatial position information and semantic category guidance to further enhance segmentation accuracy. Finally, after these multi-level processing, the SAM mask decoder is able to generate accurate and detailed image segmentation masks.

[0067] 2.3. Constructing a SAM model driven by spatial location information and guided by a self-prompter: Figure 3 As shown in Figure 3, the model contains five core modules, namely, an image encoder module, a feature aggregator module, a feature separator module, a prompter module driven by spatial position information, and a mask decoder module.

[0068] Among them, the image encoder module, feature aggregator module, feature separator module and mask decoder module correspond to the task self-prompter guided SAM model constructed in step 2.2.

[0069] The spatial location information-driven prompter module combines a lightweight region proposal network and region of interest pooling, aiming to accurately generate multiple semantic segmentation masks through accurately generated candidate boxes and features related to spatial location information extracted from them. The workflow of this module can be divided into multiple stages, which are carried out in the following order: First, the input feature maps of four different scales from the feature separator are input into the lightweight proposal network, which consists of 1 convolutional layer, 1 classification layer and 1 regression layer to generate category scores and bounding box regression values ​​for each anchor point. By combining the prior box, the network converts the classification and regression features extracted from the multi-scale input feature map into the final bounding box, and removes redundant high-overlapping boxes through non-maximum suppression, and finally obtains 1000 region proposal boxes, denoted as , . Secondly, in order to enhance the model's ability to perceive the spatial position of objects, the four input feature maps of different scales from the feature separator are applied with sinusoidal position encoding (PE). Each feature map is added to the corresponding sinusoidal position encoding to obtain a multi-scale feature map , This enhances the location features and helps improve the spatial positioning capability of the model. Next, the extracted region proposal box Then, the target region feature extraction module consisting of 4 different scale region alignment layers is used to extract the features corresponding to each region of interest from the 4 different scale multi-layer feature maps. , the operation formula is , where the extracted features The size is Then each The feature map is flattened into a one-dimensional vector and extracted through a shared feature extractor consisting of two fully connected layers to obtain the shared feature Then, it passes through a fully connected layer of category information derivation layer. , generating the semantic category of each instance , and is derived from the boundary information of another fully connected layer Generate bounding boxes for each instance The formulas are ))), ))). Next, based on the features extracted by the target region feature extraction module, non-maximum suppression is applied to each type of target frame to remove overlapping low-scoring frames and obtain the corresponding features. These features are then fed into another region of interest feature extraction module, which further extracts features corresponding to each region of interest from feature maps of four different scales. , The operation formula is , extracted The size of the feature map is .Then It is fed into a spatial information hint layer consisting of 1 convolutional layer and 3 fully connected layers to generate sparse target features. These target features are further passed to the mask decoder for processing. Through the above steps, the module can accurately generate multiple semantic segmentation masks and effectively handle complex target recognition and segmentation tasks.

[0070] Step 3: Model training and fine-tuning

[0071] The present invention trains and fine-tunes two different SAM models, namely the SAM model guided by the task self-prompter and the SAM model guided by the spatial position information driven self-prompter. The specific steps of the training and fine-tuning process are as follows:

[0072] 3.1. Preprocessing operations are performed on the manually annotated training set and validation set: First, load the image from the file and convert it into a floating point format. Next, the image is standardized and normalized using a fixed mean and standard deviation to ensure data consistency. Subsequently, the image annotation information is loaded, including the target bounding box and segmentation mask of each image, to ensure that each image corresponds to accurate instance segmentation data. In order to improve the generalization ability of the model and enhance data diversity, the present invention adopts a variety of data enhancement techniques. These enhancement operations include image rotation, horizontal flipping, upside down flipping, scaling, translation, random cropping, sharpening, smoothing, and adjusting the grayscale, contrast and brightness of the image. In addition, the present invention also introduces technologies such as random occlusion and image denoising. These enhancement operations ensure that the image and the corresponding mask information remain aligned during the transformation process, thereby ensuring the accuracy of the mask. During the preprocessing process, the targets are also screened and too small targets are filtered out to ensure that the targets in the training set are sufficiently effective and representative, and to avoid unnecessary noise interfering with the training process. Ultimately, all images and their annotation information will be packaged into the input format required by the model, and padding operations will be used to ensure that images and masks in the batch have uniform sizes, thereby ensuring data consistency during training.

[0073] 3.2. Data input and model initialization: First, use the preprocessed image data as input and input it into the SAM model guided by the task self-prompter and the SAM model guided by the spatial position information driven self-prompter respectively. During the fine-tuning process, the present invention freezes the parameters of the image encoder to ensure that it is not updated during the training process. In order to retain the features that the image encoder has learned, the loss of valid image features during the fine-tuning process is avoided. The focus of fine-tuning is to update the parameters of the following modules: feature aggregator, feature separator, prompt encoder and mask decoder.

[0074] 3.3. Optimizer settings and training parameters: The AdamW optimizer is used in the fine-tuning process, which has good convergence performance and stability. During training, the present invention sets the following key parameters: weight decay, initial learning rate, cosine annealing scheduler and linear warm-up strategy, batch size and number of iterations.

[0075] 3.4. Calculation of the total loss function: During the fine-tuning process, the present invention calculates the total loss function to guide model optimization and parameter updating. The total loss functions of different models are slightly different. For the SAM model guided by the task self-prompter, its total loss function consists of classification loss and mask loss. The classification loss uses cross entropy to calculate the difference between the predicted category and the true category, and the mask loss uses binary cross entropy to calculate the difference between the predicted mask and the true mask, specifically including coarse mask loss and fine-grained mask loss. The loss function is in the form of:

[0076] ;

[0077] in, is the classification loss, The mask loss.

[0078] For the SAM model driven by spatial location information and guided by the self-prompter, the loss function includes region proposal loss, classification loss, regression loss and mask loss. The region proposal loss evaluates the performance of the region proposal network, the classification loss calculates the difference between the predicted category and the true category through cross entropy, the regression loss calculates the smoothed absolute error loss based on the difference between the predicted coordinate offset and the target offset, and the mask loss calculates the difference between the predicted mask and the true instance mask through binary cross entropy. The form of the loss function is:

[0079] ;

[0080] in Propose a loss for the region, is the classification loss, is the regression loss, The mask loss.

[0081] 3.5. Matching process and supervised learning: When training the SAM model guided by the task self-prompter, the predicted mask is first matched with the real instance mask. The matching process uses the Hungarian algorithm to find the best matching relationship to ensure that each predicted instance is paired with the most appropriate real instance. The matching loss includes three parts: category matching loss, which is used to calculate the difference between the predicted category and the real category; mask cross entropy loss, which is used to calculate the cross entropy difference between the predicted mask and the real mask; mask Dice loss, which evaluates the similarity between the predicted mask and the real mask through the Dice coefficient. After the matching is completed, the supervised learning stage begins. In this stage, the supervision items in the training process include multi-category classification loss and mask classification loss. The multi-category classification loss is used to calculate the classification error of targets of different categories, while the mask classification loss is used to calculate the error between the predicted mask and the matched real instance mask, thereby further optimizing the performance of the model.

[0082] 3.6. Hyperparameter adjustment and model selection: During the model fine-tuning process, the present invention uses the training set for parameter optimization and learning, improves the generalization ability of the model through data enhancement, and the validation set is used to regularly evaluate the performance of the model to ensure that it does not overfit and can perform well on unseen data. The hyperparameter adjustment process is based on the mean average precision (mAP) on the validation set. By monitoring the changes in mAP, hyperparameters such as learning rate, batch size, and training cycle are optimized to select the best model configuration. Finally, based on the comprehensive evaluation of the training set and the validation set, the present invention selects the model that performs best on the validation set to ensure its robustness and accuracy in practical applications.

[0083] Step 4: Model fusion and dynamic decision making

[0084] In practical applications, factors such as the scale, position, and shape of the target often change, so it is crucial to have the ability to make adaptive adjustments based on these changes. To this end, the present invention proposes a fusion method that combines a SAM model guided by a task self-prompter with a SAM model guided by a spatial position information driven self-prompter. Through model fusion and dynamic decision-making, this method can dynamically adjust the reasoning strategy according to the specific image content and task requirements, thereby significantly improving the accuracy of instance segmentation. The specific process of this process is as follows:

[0085] 4.1. Preprocessing of samples to be labeled: The images to be labeled are first loaded from the file and subjected to a series of standardization processes. Specifically, the images are first normalized and processed using the same mean and standard deviation as the training data to ensure that the distribution of the image data is consistent with the training data. Next, the image is scaled to the target size, keeping the aspect ratio unchanged to prevent image deformation or distortion. If the size of the image does not match the target input requirements, necessary padding will be performed to ensure that the image meets the size requirements of the model input. The preprocessed images and related metadata (such as image ID, path, and original size) will be packaged into the model input format for subsequent reasoning.

[0086] 4.2. Model reasoning and fusion: The preprocessed image will be sent to the fused model for reasoning. The fusion model consists of two guidance mechanisms: the SAM model guided by the task self-prompter and the SAM model guided by the spatial position information driven self-prompter. The SAM model guided by the task self-prompter can adaptively select the most appropriate feature extraction and reasoning method according to the task requirements to ensure that the task characteristics are accurately processed; while the SAM model driven by spatial position information improves the model's modeling ability of the target's spatial features by analyzing the distribution and layout of the target in space. Through the combination of these two guidance mechanisms, the model can generate a set of instance segmentation masks that represent the area of ​​each instance in the image. Each mask will be stored together with its corresponding label, score and other information as part of the reasoning result.

[0087] 4.3. Dynamically select the best mask: After generating multiple instance masks, the model processes each mask to dynamically select the best mask. First, the model calculates the area of ​​each mask and filters out valid masks with larger areas (for example, masks with an area smaller than a set threshold will be discarded). Next, if there are overlapping areas of multiple masks, the model dynamically decides which masks to keep based on the area of ​​the overlapping area and the score value of each mask. The specific operations include: for overlapping areas with an overlapping area smaller than a set threshold, select a mask with a larger area; and for larger overlapping areas, give priority to masks with higher scores. Finally, according to this rule, the model will retain the most appropriate masks, labels, and scores and store them in the final result array for subsequent analysis and processing.

[0088] Step 5: Multi-mask region optimization and synthesis

[0089] This module achieves precision and semantic consistency of mask images through synthesis of multiple mask regions, noise removal, region screening, and similarity-based optimization, thus improving the quality and visualization of segmentation results. The specific steps are as follows:

[0090] 5.1. Multi-mask region synthesis and semantic information presentation: In the multi-mask region optimization process, multiple masks and instance labels are first extracted from the model fusion and dynamic decision steps. According to each label value, the corresponding color is obtained from the palette and applied to the corresponding mask region. By assigning different colors to each binary mask, multiple masks are drawn on the same image to generate an instance mask map containing semantic information. The semantic categories in this map are represented by corresponding colors to ensure that each region can clearly identify its semantic category.

[0091] 5.2. Small area noise removal and mask area screening: For the synthesized mask image, first mark the connected areas of each color mask to identify different areas in the image. Next, set an area threshold and filter out connected areas smaller than the threshold to remove the noise area. This process ensures that only larger valid areas are retained in the image to avoid affecting the clarity of the final result due to the presence of small areas. Finally, merge the screened mask area into the final image to optimize the visualization effect and ensure the accuracy and effectiveness of the image.

[0092] 5.3 Mask synthesis and optimization based on regional similarity and minimum pixel threshold: In the process of superimposing multiple masks, there may be overlapping areas between different masks. In order to reduce color label errors, remove noise and enhance regional coherence, it is necessary to merge overlapping areas. First, define the expansion threshold of the adjacent area to determine the expansion range of the mask area during synthesis, so as to ensure that adjacent areas can be merged and avoid omissions. Then, connective region analysis is performed on each color mask to extract statistical information of the area, such as area and bounding box. Based on this information, the bounding box of each connected area is expanded to cover more adjacent areas using the set expansion threshold. Next, check the overlapping parts of each connected area with other color masks, and determine whether they belong to adjacent areas by calculating the intersection. At the same time, apply the minimum pixel threshold to exclude too small noise areas. For adjacent areas that meet the merging conditions, merge them into one area, and replace the color label of the adjacent area with the label of the current area. In order to improve processing efficiency, multi-threading is used to process multiple color masks in parallel to quickly complete mask synthesis and optimization. Finally, the synthesized mask image is updated to ensure that the image only contains important and coherent areas, remove noise and irrelevant small areas, and generate a concise and semantically consistent final mask map.

[0093] Through the above steps, the final image obtained is a high-quality semantic-level instance segmentation mask map generated by the automated segmentation method.

[0094] Corresponding to the aforementioned embodiment of a method for dynamic decision image segmentation based on self-prompt guidance, the present invention also provides an embodiment of a device for dynamic decision image segmentation based on self-prompt guidance.

[0095] like Figure 4 As shown, an embodiment of the present invention provides a dynamic decision image segmentation device based on self-prompt guidance, including one or more processors, which are used to implement a dynamic decision image segmentation method based on self-prompt guidance in the above embodiment.

[0096] The embodiment of the dynamic decision image segmentation device based on self-prompt guidance of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From the hardware level, if Figure 4 As shown, it is a hardware structure diagram of any device with data processing capability where the dynamic decision image segmentation device based on self-prompt guidance of the present invention is located, except Figure 4In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0097] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0098] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

Claims

1. A dynamic decision image segmentation method based on self-prompt guidance, characterized in that: The steps include: Obtaining original images of the target area as samples, taking out some samples for manual annotation and making corresponding labels to construct a training data set, wherein the training data set includes a training set and a validation set; Constructing a SAM model guided by a task self-prompter and a SAM model guided by a spatial position information driven self-prompter according to the training data set; Train and fine-tune two SAM models; After fusing the two fine-tuned SAM models, the fusion model dynamically adjusts the reasoning strategy according to the specific image content and task requirements of the image to be labeled, thereby generating multiple mask regions; Through the synthesis of multiple mask regions, noise removal, region screening and similarity-based optimization, the mask image is refined and semantically consistent, and finally a semantic-level instance segmentation mask image is obtained.

2. The method for dynamic decision image segmentation based on self-prompt guidance according to claim 1, characterized in that: The construction of the training data set includes: ensuring that the extracted samples cover the diversity of the target objects; for each original image, using a manual segmentation and annotation tool to generate a segmentation mask of the target object, and annotating the bounding box and category label of the target; constructing a training data set for training and verification based on the original image and the generated segmentation mask, bounding box and category label, and randomly dividing it into a training set and a verification set in a ratio of 9:

1.

3. The method for dynamic decision-making image segmentation based on self-prompt guidance according to claim 1, characterized in that: The SAM model guided by the task self-prompter includes: an image encoder module, a feature aggregator module, a feature separator module, a task-based prompter module and a mask decoder module; The image encoder module uses a pre-trained SAM visual encoder to extract image features; The feature aggregator module fuses and processes the input features through convolution and normalization operations to extract and integrate information of different scales; The feature separator module is used to integrate features of different scales through upsampling, downsampling, convolution and lateral connection operations to enhance the multi-scale perception ability of the model; The task-based prompter module includes: a transformer encoder for performing self-attention operations on four different scales of input feature maps from the feature separator module, extracting global semantic information and generating feature maps of four scales; a transformer decoder for interacting with zero-initialized task-related query tags and fusing with feature maps of four scales; and a mask filter, prompt embedding and semantic category for each instance are generated through a mask projection layer, a prompt projection layer and a category inference layer, and a coarse mask is generated according to the maximum size feature map; The mask decoder module generates high-precision image segmentation masks through image embedding, mask labeling, 2 two-way transformer modules, 4 multi-layer perceptron modules and 2 up-sampling transposed convolutional layers.

4. The method for dynamic decision image segmentation based on self-prompt guidance according to claim 1, characterized in that: The SAM model driven by spatial position information and guided by a self-prompter includes: an image encoder module, a feature aggregator module, a feature separator module, a prompter module driven by spatial position information, and a mask decoder module; The image encoder module uses a pre-trained SAM visual encoder to extract image features; The feature aggregator module fuses and processes the input features through convolution and normalization operations to extract and integrate information of different scales; The feature separator module is used to integrate features of different scales through upsampling, downsampling, convolution and lateral connection operations to enhance the multi-scale perception ability of the model; The prompter module driven by spatial position information includes: a lightweight region proposal network, which is used to generate category scores and bounding box regression values ​​from multi-scale input feature maps, remove redundant boxes through non-maximum suppression, and output multiple region proposal boxes; a position encoding module, which is used to apply sinusoidal position encoding to the input feature map to enhance its spatial position information; a region of interest alignment layer, which is used to extract features corresponding to each proposal box from the multi-scale feature map; a shared feature extractor, which is used to process the extracted features and generate shared features; a category inference layer and a boundary inference layer, which are used to generate semantic categories and bounding boxes for each instance, respectively; a region of interest feature extraction module, which is used to extract target features; a spatial information prompting layer, which is used to generate sparse target features and pass them to a mask decoder to generate a final multi-semantic segmentation mask; The mask decoder module generates high-precision image segmentation masks through image embedding, mask labeling, 2 two-way transformer modules, 4 multi-layer perceptron modules and 2 upsampling transposed convolutional layers.

5. The method for dynamic decision-making image segmentation based on self-prompt guidance according to claim 1, characterized in that: The training and fine-tuning of the two SAM models includes: Preprocessing the training data set, including image loading, standardization, data enhancement, target screening, and image size unification to ensure the consistency and validity of image and mask information; The preprocessed data is fed as input to the SAM model guided by the task self-cue and the SAM model guided by the spatial position information self-cue, freezing the parameters of the image encoder and fine-tuning the parameters of the feature aggregator, feature separator, cue encoder and mask decoder modules. The AdamW optimizer is used to fine-tune the training parameters, including weight decay, initial learning rate, cosine annealing scheduler, linear warm-up strategy, batch size, and number of iterations. The total loss function is calculated for the two SAM models respectively. For the SAM model guided by the task self-prompter, the Hungarian algorithm is used to match the predicted mask with the real instance mask, and the category matching loss, mask cross entropy loss and mask Dice loss are calculated to ensure that each predicted instance is paired with the most appropriate real instance, and supervised learning of multi-category classification loss and mask classification loss is performed; for the SAM model guided by the self-prompter driven by spatial location information, the total loss includes region proposal loss, classification loss, regression loss and mask loss; hyperparameter optimization is performed through the training set, model performance is evaluated using the validation set, the best model configuration is selected based on the average accuracy, and finally the model with the best performance on the validation set is selected.

6. The method for dynamic decision-making image segmentation based on self-prompt guidance according to claim 1, characterized in that: The two fine-tuned SAM models are fused, and the fused model dynamically adjusts the reasoning strategy according to the specific image content and task requirements of the image to be labeled, including: Preprocess the image to be annotated, including loading the image, normalizing it, scaling it to the target size, and performing necessary padding to ensure that the image meets the model input requirements; The preprocessed image is sent to the fused model for reasoning; the fused model includes two guiding mechanisms: the SAM model guided by the task self-prompter is used to adaptively select the most appropriate feature extraction and reasoning method according to the task requirements to ensure that the task characteristics are accurately processed; the SAM model driven by spatial position information is used to improve the model's modeling ability of the target spatial characteristics by analyzing the distribution and layout of the target in space; a set of instance segmentation masks are generated by the fused model to represent the area of ​​each instance in the image; each mask is stored together with its corresponding label and score information as part of the reasoning result; After generating multiple instance segmentation masks, each mask is processed through the fused model to dynamically select the optimal mask.

7. The method for dynamic decision image segmentation based on self-prompt guidance according to claim 6, characterized in that: The processing of each mask by the fused model includes: The area of ​​each mask is calculated through the fused model, and the effective masks with larger areas are screened out; If there are overlapping areas of multiple masks, the fused model dynamically decides which masks to keep based on the area of ​​the overlapping area and the score of each mask. For overlapping areas whose overlapping area is smaller than the set threshold, the mask with a larger area is selected; for overlapping areas with a larger area, the mask with a higher score is selected first. The selected masks, labels, and scores are stored in the final result array.

8. The method for dynamic decision-making image segmentation based on self-prompt guidance according to claim 1, characterized in that: The method of achieving precision and semantic consistency of the mask image by synthesizing multiple mask regions, removing noise, selecting regions, and optimizing based on similarity includes: Extract multiple masks and category labels from the model fusion process; according to each category label, obtain a corresponding color from the color palette, and apply the color to the corresponding mask area; by assigning different colors to each mask, draw multiple masks on the same image to generate an instance mask map containing semantic information, and the semantic category of the instance mask map is represented by the corresponding color to ensure that each area can clearly identify its semantic category; For the synthesized mask image, mark the connected areas of each color mask to identify different areas in the image; set an area threshold and filter out connected areas smaller than the threshold to remove noise areas; merge the filtered mask areas into the final image to ensure the accuracy and effectiveness of the image; Merge the overlapping areas between different masks.

9. The method for dynamic decision image segmentation based on self-prompt guidance according to claim 8, characterized in that: The merging of overlapping areas between different masks is specifically as follows: Define the expansion threshold of the neighboring area to determine the expansion range of the mask area during synthesis; Performing a connected region analysis on each color mask to extract statistical information of the region; based on the statistical information, expanding the bounding box of each connected region to cover more adjacent regions using a set expansion threshold; Check the overlap between each connected region and other color masks, and determine whether the overlap belongs to an adjacent region by calculating the intersection. At the same time, apply the minimum pixel threshold to exclude the noise region that is too small. For adjacent regions that meet the merging conditions, merge them into one region and replace the color label of the adjacent region with the label of the current region. The synthesized mask image is updated to ensure that the image only contains important and coherent areas, and removes noise and irrelevant small areas to generate the final mask image.

10. A dynamic decision image segmentation device based on self-prompt guidance, characterized in that: It comprises one or more processors for implementing a dynamic decision image segmentation method based on self-prompt guidance as described in any one of claims 1-9.

Images