An Image Segmentation Method, Device, Storage Medium and Electronic Device Based on SAM Model with Dynamic Prompt Constraint and Edge Anchor Point Guidance

Abstract

The present application provides an image segmentation method, device, storage medium, and electronic device based on dynamic Prompt constraint and edge anchor guidance of the SAM model, which relates to the technical field of image segmentation. The method includes: designing and training a lightweight dynamic adjustment network using an MLP structure; using the lightweight dynamic adjustment network to control the intensity and spatial range of the interactive prompt provided by the user introduced into the image encoder; training the image encoder based on the edge anchor guidance mechanism; inputting the introduction result of the interactive prompt and the image into the image encoder to obtain an image embedding vector; inputting the interactive prompt into the prompt encoder to obtain a prompt embedding vector; inputting the image embedding vector and the prompt embedding vector into the mask decoder for interaction, and obtaining a segmented mask image after multiple layers of cross-attention and upsampling. This method effectively solves the problems of fuzzy segmentation boundaries and inconsistent targets under traditional weak annotation conditions, and demonstrates extremely strong medical image segmentation capabilities and generalization.

Description

Technical Field

[0001] This application relates to the field of image segmentation technology. Specifically, it relates to an image segmentation method, device, storage medium, and electronic device based on dynamic Prompt constraint and edge anchor point guidance of the SAM model. Background Art

[0002] The typical architecture of the SAM model is as follows: The prompt information such as points and boxes provided by the user is first encoded by the Prompt Encoder, but these encoded Prompt signals only interact with the image features in the Mask prediction head of the Decoder to guide the generation of the segmentation result. At the same time, the input image extracts multi-scale features through multiple Transformer Blocks (Stage1 - Stage4) of the Image Encoder, and the features at each stage aggregate global context information through the self-attention mechanism.

[0003] Although this design can establish long-range dependencies, due to the lack of explicit constraints on the Prompt signal, it leads to two key defects: On the one hand, in difficult samples such as low-contrast medical images, the semantic representation of the deep network (Stage3 - 4) will over-diffuse, and the attention distribution shows a global diffusion characteristic. On the other hand, when dealing with hard samples, the full-level attention defocus phenomenon starts from Stage2, resulting in the model almost completely losing the ability to locate the target boundary. The root cause of these defects is that the existing architecture fails to establish an effective cross-level Prompt constraint mechanism, making the ImageEncoder unable to maintain continuous attention to the prompt area, and ultimately leading to a significant deterioration of the feature expression consistency as the network depth increases. In addition, since the deep network needs to learn a large number of redundant background features to identify edge details, this directly leads to a significant decline in the performance of the SAM model in the segmentation task of weak-edge targets (such as early tumors). Therefore, there is an urgent need for an image segmentation method to make up for the defects existing in the prior art. Summary of the Invention

[0004] Embodiments of this application provide an image segmentation method, device, storage medium, and electronic device based on dynamic Prompt constraint and edge anchor point guidance of the SAM model to solve the technical problems existing in the prior art.

[0005] Other features and advantages of this application will become apparent through the following detailed description, or be learned in part through the practice of this application.

[0006] According to the first aspect of the embodiments of the present application, there is provided an image segmentation method based on dynamic Prompt constraint and edge anchor point guidance of the SAM model. In the SAM model, an image encoder, a prompt encoder, and a mask decoder are provided. The method includes:

[0007] Design and train a lightweight dynamic adjustment network using an MLP structure;

[0008] Use the lightweight dynamic adjustment network to control the intensity and spatial range of the interactive prompt provided by the user introduced into the image encoder;

[0009] Train the image encoder based on the edge anchor point guidance mechanism;

[0010] Input the introduction result of the interactive prompt and the image provided by the user into the trained image encoder, and process to obtain an image embedding vector;

[0011] Input the interactive prompt provided by the user into the prompt encoder to process and obtain a prompt embedding vector;

[0012] Input the image embedding vector and the prompt embedding vector into the mask decoder for interaction, and obtain a segmented mask image after multi-layer cross-attention and upsampling.

[0013] In some embodiments of the present application, based on the foregoing solution, designing a lightweight dynamic adjustment network using an MLP structure includes:

[0014] Design a lightweight dynamic adjustment network using a lightweight architecture of MLP + multi-head output.

[0015] In some embodiments of the present application, based on the foregoing solution, the process of training the lightweight dynamic adjustment network includes:

[0016] Train the lightweight dynamic adjustment network using a three-stage progressive training scheme, including:

[0017] In the first stage, freeze the backbone network and only train the modules related to the interactive prompt;

[0018] In the second stage, unfreeze the high-level network for adjustment;

[0019] In the third stage, adjust the parameters of the entire model.

[0020] In some embodiments of the present application, based on the foregoing solution, the use of the lightweight dynamic adjustment network to control the intensity and spatial range of the interactive prompt provided by the user introduced into the image encoder includes:

[0021] The interactive prompts provided by the user and the feature maps output by different layers in the image encoder are input into the lightweight dynamic adjustment network for processing, generating a set of fusion weights and diffusion coefficients that adaptively change with the layer level.

[0022] In some embodiments of the present application, based on the foregoing solution, training the image encoder based on the edge anchor point guidance mechanism includes:

[0023] Using depthwise separable convolution to extract edge responses from the shallow features of the image encoder, and generating an anchor mask in cooperation with a batch normalization layer and a Gaussian error linear unit activation;

[0024] Establishing a dual alignment mechanism, using the anchor mask to shield the gradients of non-anchor regions during backpropagation in the spatial dimension, and only updating the weights corresponding to the anchor points, and calculating the L2 alignment loss function between the deep features and the anchor features in the semantic dimension;

[0025] Adopting an alternating training strategy, only allowing the anchor regions to participate in gradient updates during odd training cycles, and restoring the full-image normal training during even training cycles;

[0026] During the entire training process, dynamically balance the interaction between the shallow and deep layer features in the image encoder through learnable gating parameters.

[0027] In some embodiments of the present application, based on the foregoing solution, inputting the introduction result of the interactive prompt and the image provided by the user into the trained image encoder to process and obtain an image embedding vector includes:

[0028] Encoding the interactive prompt provided by the user into a Gaussian heat map;

[0029] Combining the fusion weights, the diffusion coefficients and the Gaussian heat map, and inputting the combination result and the image provided by the user into the image encoder to process and obtain an image embedding vector.

[0030] According to the second aspect of the embodiments of the present application, there is provided an image segmentation device based on SAM model with dynamic Prompt constraint and edge anchor point guidance, including:

[0031] A first generation unit for designing and training a lightweight dynamic adjustment network using an MLP structure;

[0032] A fusion unit for using the lightweight dynamic adjustment network to control the intensity and spatial range of the interactive prompt provided by the user introduced into the image encoder;

[0033] A training unit for training the image encoder based on the edge anchor point guidance mechanism;

[0034] A second generating unit is configured to input the interactive prompt introduction result and the image provided by the user into the trained image encoder to obtain an image embedding vector;

[0035] a third generating unit, configured to input the interactive prompt provided by the user into the prompt encoder for processing to obtain a prompt embedding vector;

[0036] A segmentation processing unit is used to input the image embedding vector and the prompt embedding vector into the mask decoder for interaction, and obtain a segmented mask image after multiple layers of cross attention and upsampling.

[0037] According to a third aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the storage medium stores computer instructions. When the computer instructions are executed on a computer, the computer executes the method according to the first aspect.

[0038] According to a fourth aspect of the embodiments of the present application, there is provided an electronic device, including: a memory and a processor;

[0039] The memory is used to store computer instructions;

[0040] The processor is configured to call the computer instructions stored in the memory so that the electronic device executes the method according to the first aspect.

[0041] The technical solution of the present application solves the problem of attention dispersion in deep networks by establishing a hierarchical interactive prompt constraint mechanism in the image encoder, so that the model maintains stable segmentation performance in low-contrast medical images, and alleviates the semantic dispersion problem and improves the model training efficiency through the edge geometry anchor guidance mechanism. The overall technical solution inherits the efficient interactive characteristics of the image segmentation method based on the SAM model while effectively solving the problems of blurred segmentation boundaries and target inconsistency under traditional weak annotation conditions, demonstrating extremely strong medical image segmentation capabilities and generalization.

[0042] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are incorporated into and constitute a part of the specification, illustrating embodiments consistent with the present application and, together with the specification, explaining the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0044] Figure 1Shows a flowchart of an existing method for image segmentation using the SAM model according to an embodiment of the present application;

[0045] Figure 2 Shows a schematic flowchart of an image segmentation method based on dynamic Prompt constraint and edge anchor point guidance using the SAM model according to an embodiment of the present application;

[0046] Figure 3 Shows a block diagram of an image segmentation device based on dynamic Prompt constraint and edge anchor point guidance using the SAM model according to an embodiment of the present application;

[0047] Figure 4 Shows a block diagram of an electronic device according to an embodiment of the present application;

[0048] Figure 5 Shows a schematic diagram of the structure of a computer system of an electronic device suitable for implementing the embodiments of the present application.

[0049] Description of reference numerals

[0050] 300 - Image segmentation device based on dynamic Prompt constraint and edge anchor point guidance using the SAM model, 301 - First generation unit, 302 - Fusion unit, 303 - Training unit, 304 - Second generation unit, 305 - Third generation unit, 306 - Segmentation processing unit, 400 - Electronic device, 410 - Memory, 411 - Computer program 411, 420 - Processor, 500 - Computer system, 501 - Central processing unit, 502 - Read-only memory, 503 - Random access memory, 504 - Bus, 505 - Input / output interface, 506 - Output part, 507 - Output part, 508 - Storage part, 509 - Communication part, 510 - Driver, 511 - Removable medium. Detailed implementation manners

[0051] Now, example embodiments will be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more complete and comprehensive, and will fully convey the concept of the example embodiments to those skilled in the art.

[0052] In addition, the described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, numerous specific details are provided to give a thorough understanding of the embodiments of the present application. However, those skilled in the art will realize that the technical solutions of the present application can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the present application.

[0053] The block diagrams shown in the drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0054] The flowcharts shown in the drawings are only illustrative and not necessarily include all the contents and operations / steps, nor are they necessarily executed in the described order. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined, so the actual execution order may change according to the actual situation.

[0055] It should be noted that the terms "first", "second", etc. in the specification, claims, and above-mentioned drawings of the present application are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the objects used in this way can be interchanged under appropriate circumstances so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described.

[0056] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0057] Hereinafter, some embodiments of the present application will be described in detail in conjunction with the drawings. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0058] As Figure 1 shown, the process of the existing method for image segmentation using the SAM model is as follows:

[0059] 1. The most primitive SAM has two inputs: an image and an interactive prompt (including points / box / text / mask);

[0060] 2. The image vector is first sent to the image encoder to obtain the image embedding vector; at the same time, the interactive prompt is also sent to the prompt encoder to obtain the prompt embedding vector.

[0061] 3. Finally, the image embedding vector obtained by the image encoder and the prompt embedding vector obtained by the prompt encoder are sent into the mask decoder for interaction. After multi-layer cross-attention and upsampling, the segmented mask image is finally obtained (by default, three masks with high confidence are generated for the user to choose, or the user can manually set to generate only one).

[0062] Among them, mask represents the mask, points represents the point prompt, box represents the box prompt, and text represents the text prompt.

[0063] The existing image segmentation technology using the SAM model has the following problems and defects.

[0064] (1) The problem of unbalanced hierarchical features in the image encoder of the SAM model. This problem is mainly reflected in that the shallow network can effectively capture local edge features. However, as the network depth increases, due to the effect of the global attention mechanism, the position signal of the interactive prompt in the deep network tends to gradually decay. This decay is particularly obvious in low-contrast images. When the target and the background have similar gray levels (such as tumors and normal tissues), the deep features are easily interfered by the background, resulting in blurred target boundaries or even complete missed segmentation. In extremely difficult hard samples (target-background similarity > 90%), this problem is even more serious, and the attention distribution of each layer from the shallow layer is difficult to effectively focus on the prompt area. The root cause of this phenomenon is that the existing methods lack a cross-layer Prompt constraint mechanism. The interactive prompt information only acts on the Decoder end and is not injected into the image encoder, so it cannot correct the inherent attention distribution characteristics of the Transformer network. At the same time, the traditional inter-layer feature transfer completely relies on the self-attention mechanism and lacks the ability to adaptively strengthen the signals in key areas.

[0065] (2) The problem of loss of edge geometric information. This problem stems from the fact that the deep network needs to learn a large amount of redundant background features to identify edge details, which directly leads to a significant decline in performance in the segmentation task of weak-edge targets (such as early tumors). The root cause of this phenomenon is that the existing methods fail to make full use of the edge features extracted by the shallow network to explicitly constrain the distribution of deep attention. At the same time, the fixed interactive prompt injection weight cannot adapt to the differences in feature requirements of different network layers. For example, the shallow layer requires stronger edge constraints while the deep layer requires more flexible guidance. The lack of this dynamic adaptability makes it difficult for the network to complete the understanding of high-level semantics while maintaining geometric details.

[0066] Current improved research based on SAM mainly focuses on three technical directions. In terms of Decoder optimization, typical methods such as MedSAM enhance interactive prompt interaction by introducing cross-attention mechanisms in the Mask prediction head. However, such improvements are limited to the Decoder side and are difficult to solve the problem of feature dispersion inside the Encoder. The parameter fine-tuning scheme adjusts model parameters by introducing an adaptation layer or low-rank weight update. Although it improves the model's adaptability, it significantly increases the training burden. In the architecture expansion direction, researchers attempt to enhance the model's capabilities through 3D convolution expansion or automatic parameter adjustment mechanisms. However, these methods often face the problem of balancing computational efficiency and segmentation accuracy.

[0067] These improvement methods generally have three key limitations: First, there is a lack of an effective cross-level interactive prompt signal transmission mechanism; second, a reliable cross-level alignment path for geometric features has not been established; finally, there is insufficient adaptability to static prompts. Especially when dealing with low-contrast regions or tiny lesions in medical images, the performance of existing solutions remains unstable.

[0068] To address the above defects and technical problems, this application provides an image segmentation method based on dynamic Prompt constraint and edge anchor guidance of the SAM model. The objectives of this method are as follows:

[0069] 1. Establish a hierarchical Prompt constraint mechanism in the image encoder to solve the problem of attention dispersion in deep networks;

[0070] 2. Through the edge geometric anchor alignment mechanism, achieve dual technical effects: on the one hand, use shallow edge features to constrain the deep attention distribution and slow down the problem of semantic dispersion; on the other hand, improve the training efficiency through gradient control in the anchor area.

[0071] 3. Adopt a lightweight dynamic weight design to achieve precise adjustment of the Prompt intensity at each level with only a small increase in parameters.

[0072] See Figure 2 , which shows a schematic flowchart of an image segmentation method based on dynamic Prompt constraint and edge anchor guidance of the SAM model according to an embodiment of this application.

[0073] As Figure 2 shown, an image segmentation method based on dynamic Prompt constraint and edge anchor guidance of the SAM model is presented, which specifically includes steps S100 to S600.

[0074] Refer to Figure 2 , in step S100, a lightweight dynamic adjustment network is designed and trained using the MLP structure.

[0075] It can be understood that in order to achieve efficient parameter utilization, a minimalist dynamic weight adjustment network is designed in this embodiment. This network adopts a lightweight structure design. By analyzing the global information of the input features, it adaptively generates the interactive prompt injection intensity coefficients at each level. With almost no increase in computational overhead, it realizes precise control of the hierarchical interactive prompt intensity, enabling the model to automatically adjust the constraint intensity at each level according to different input features.

[0076] In some feasible embodiments, based on the foregoing solution, a lightweight dynamic adjustment network is designed using an MLP structure, including:

[0077] A lightweight dynamic adjustment network is designed using a lightweight architecture of MLP + multi-head output.

[0078] It should be noted that MLP refers to a multi-layer perceptron, which consists of an input layer, one or more hidden layers, and an output layer.

[0079] Exemplarily, the design process of the lightweight dynamic adjustment network is as follows:

[0080] First, the feature map (dimension C, with additional coordinate information, as shown in the example in the figure as 768 + n) is input into the shared layer, and it is compressed to dimension D (recommended D = C / 3, such as 256; it can be flexibly adjusted within the range of C / 4 to C / 2) through Linear(C→D)→GELU→LayerNorm; subsequently, a lightweight Head: Linear(D→2)→GELU is configured for each stage s, mapping the D-dimensional feature to a binary vector [w_s, α_s], and activating them through the function Sigmoid(w_s) and the function Softplus(α_s) respectively to obtain the stage weight w_s∈(0,1) and the adjustment coefficient α_s>0. The total number of parameters of the entire network mainly comes from C×D of the shared layer and 2D×S of S Heads (S is the number of stages). By selecting D≈C / 3 and restricting the number of stages, the total number of parameters can be controlled within 1.5C + 1, achieving extreme lightweight.

[0081] Among them, Linear represents a fully connected layer, GELU represents a Gaussian error linear unit, and LayerNorm represents layer normalization.

[0082] In some feasible embodiments, based on the foregoing solution, the process of training the lightweight dynamic adjustment network includes:

[0083] The lightweight dynamic adjustment network is trained using a three-stage progressive training scheme, including:

[0084] In the first stage, the backbone network is frozen, and only the modules related to the interactive prompt are trained;

[0085] In the second stage, thaw the high-level network for adjustment;

[0086] In the third stage, adjust the parameters of the entire model.

[0087] It should be noted that in the actual application process, the above training order is crucial for performance realization and cannot be adjusted, but the specific parameters of each stage can be appropriately adjusted.

[0088] For different application scenarios, the present invention provides the following optional implementation schemes:

[0089] Computationally sensitive scenarios: The dynamic weight network can be omitted and fixed hierarchical weights can be used instead;

[0090] Accuracy-first scenarios: Non-maximum suppression can be added after edge detection to improve the boundary accuracy by 1.2%;

[0091] Medical imaging scenarios: The heatmap intensity (σ = 3) and edge detection threshold need to be specially adjusted;

[0092] For the anchor-guided semantic boundary learning network, the present invention recommends the following alternating training scheme:

[0093] For the 1st to Nth epochs: The anchor regions participate in backpropagation and other regions are frozen;

[0094] For the (N + 1)th to 2Nth epochs: The entire image participates in backpropagation and normal training is resumed;

[0095] It can be repeated several rounds (Anchor ⇄ Full) for alternating refinement.

[0096] Continue to refer to Figure 2 , step S200, using the lightweight dynamic adjustment network to control the intensity and spatial range of the interactive prompt provided by the user introduced into the image encoder.

[0097] It can be understood that in this embodiment, the interactive prompt is innovatively injected into the image feature encoder to solve problems such as deep semantic diffusion and target blurring when the SAM model processes medical images, and improve the model's attention and feature focusing ability on the prompt area.

[0098] In some feasible embodiments, based on the foregoing scheme, the using the lightweight dynamic adjustment network to control the intensity and spatial range of the interactive prompt provided by the user introduced into the image encoder includes:

[0099] Input the interactive prompt provided by the user and the feature maps output by different layers in the image encoder into the lightweight dynamic adjustment network for processing, and generate a set of fusion weights and diffusion coefficients that adaptively change with the hierarchy.

[0100] Exemplarily, the specific process of the lightweight dynamic adjustment network for processing the feature maps output by different layers in the interactive prompt and the image encoder is as follows:

[0101] First, the image features representing the feature maps output by different layers are concatenated with the prompt features representing the interactive prompt and then input into a shared MLP module; global features are extracted through a linear layer, GELU activation, and LayerNorm, and then output to four Stage Heads respectively; each Stage Head further predicts the Prompt injection intensity coefficients (weight W and sigma coefficient σ) corresponding to each level. Among them, W controls the injection intensity through Sigmoid, and σ ensures non-negativity through Softplus, so as to achieve fine adjustment of the Prompt injection weights for each layer, enhancing the dynamic adaptability of the guidance ability while maintaining computational efficiency.

[0102] It should be noted that different from the traditional method that only acts on the Decoder, this solution uses a dynamic weight control mechanism to enable the shallow network to obtain stronger Prompt guidance to capture edge details, while the deep network maintains moderate constraints to retain semantic flexibility, effectively solving the problem of deep attention diffusion. Without changing the backbone network structure, this mechanism significantly improves the model's continuous attention ability to the prompt area.

[0103] Continue to refer to Figure 2 , step S300, training the image encoder based on the edge anchor point guidance mechanism.

[0104] It should be noted that the edge anchor point guidance mechanism means that the edge response map extracted from the shallow features extracts key geometric anchor points through a lightweight depthwise separable convolution module and generates an edge anchor point map; this map is sent into the deep feature map as a cross-layer guidance signal to explicitly modulate it, guiding the deep attention to focus on the edge key area, thereby enhancing the perception ability of weak boundary targets. Through this mechanism, while maintaining the semantic expression ability, the model effectively avoids overfitting of the deep layer to background redundancy, realizing the efficient integration of edge perception and semantic expression.

[0105] In some feasible embodiments, based on the foregoing solution, training the image encoder based on the edge anchor point guidance mechanism includes:

[0106] Using depthwise separable convolution to extract edge responses from the shallow features of the image encoder, and generating an anchor point mask in cooperation with a batch normalization layer and a Gaussian error linear unit activation;

[0107] Establish a dual alignment mechanism. In the spatial dimension, use the anchor mask to shield the gradients of non-anchor regions during backpropagation, and only update the weights corresponding to the anchors. In the semantic dimension, calculate the L2 alignment loss function between the deep features and the anchor features;

[0108] Adopt an alternating training strategy. In odd training epochs, only the anchor regions participate in gradient update, and in even training epochs, the whole image resumes normal training;

[0109] During the entire training process, dynamically balance the interaction between shallow and deep features in the image encoder through learnable gating parameters.

[0110] It can be understood that for the problem of geometric information loss, the present invention proposes to extract edge anchors from shallow features to guide the deep network. Specifically, identify the high-response geometric key points in the shallow features through an edge detection algorithm, and establish a cross-level attention distribution constraint, so that the deep network can maintain sensitivity to key edge regions. This constraint method based on geometric priors not only reduces the learning requirement of the deep network for redundant background features, but also significantly improves the segmentation accuracy of weak-edge targets.

[0111] It should be noted that the edge anchor guidance mechanism is only enabled during the model training phase and does not require additional calculation during the inference phase. However, because it optimizes the structure perception ability of deep semantics during training, the model can also maintain good boundary segmentation results during inference, and is particularly suitable for the accurate segmentation of weak boundaries and low-contrast targets (such as lesions or tumors) in medical images.

[0112] Continue to refer to Figure 2 , step S400, input the introduction result of the interactive prompt and the image provided by the user into the trained image encoder, and process to obtain an image embedding vector.

[0113] In some feasible embodiments, based on the foregoing solution, the inputting the introduction result of the interactive prompt and the image provided by the user into the trained image encoder to process and obtain an image embedding vector includes:

[0114] Encode the interactive prompt provided by the user into a Gaussian heat map;

[0115] Combine the fusion weight, the diffusion coefficient and the Gaussian heat map, and input the combination result and the image provided by the user into the image encoder to process and obtain an image embedding vector.

[0116] Continue to refer to Figure 2 , step S500, input the interactive prompt provided by the user into the prompt encoder to process and obtain a prompt embedding vector.

[0117] Continue to refer to Figure 2, step S600, input the image embedding vector and the prompt embedding vector into the mask decoder for interaction, and obtain the segmented mask image after multiple layers of cross-attention and upsampling.

[0118] In summary, while inheriting the efficient interaction characteristics of the image segmentation method based on the SAM model, the technical solution of this application effectively solves the problems of fuzzy segmentation boundaries and inconsistent targets under traditional weak annotation conditions, demonstrating extremely strong medical image segmentation capabilities and generalization.

[0119] This method shows significant technical advantages compared with the prior art. Experimental data shows that: on the ISIC skin lesion dataset, the Dice coefficient of this method reaches 0.74 when using point prompts, an 8.8% improvement compared to the baseline SAM (0.68); it reaches 0.89 when using box prompts, an 18.7% improvement compared to SAM (0.75). In the Kvasir polyp segmentation task, the Dice values under point prompts and box prompts reach 0.86 and 0.94 respectively, a 4.9% and 2.2% improvement compared to SAM. For the BUSI breast ultrasound image segmentation, this method achieves Dice values of 0.85 and 0.89 under point prompts and box prompts respectively, a 6.3% and 3.5% improvement compared to SAM. In the REFUGE fundus image segmentation task, the Dice values under point prompts and box prompts reach 0.59 and 0.88 respectively, a 6.1% and 13.2% improvement compared to SAM.

[0120] These improvements stem from three core technologies: the hierarchical Prompt injection mechanism solves the problem of deep attention dispersion, enabling the model to maintain stable segmentation performance in low-contrast medical images; the edge geometric anchor constraint significantly improves the boundary localization accuracy, while reducing the parameter update amount by 18% through gradient mask control; the lightweight dynamic adjustment network only increases the number of parameters by 0.5%. Especially for difficult samples with high target-background similarity, this method shows stronger robustness, providing a more reliable solution for medical image analysis.

[0121] The device embodiments of this application are introduced below, which can be used to execute a dynamic Prompt constraint and edge anchor-guided image segmentation method based on the SAM model in the above embodiments of this application. For details not disclosed in the device embodiments of this application, please refer to the method embodiments of this application above.

[0122] Refer to Figure 3 As shown, an image segmentation device 300 based on a dynamic Prompt constraint and edge anchor guidance of the SAM model according to an embodiment of this application includes:

[0123] The first generation unit 301 is used to design and train a lightweight dynamic adjustment network using an MLP structure;

[0124] The fusion unit 302 is configured to use the lightweight dynamic adjustment network to control the intensity and spatial range of the interactive prompt provided by the user introduced into the image encoder;

[0125] The training unit 303 is configured to train the image encoder based on the edge anchor point guiding mechanism;

[0126] The second generation unit 304 is configured to input the introduction result of the interactive prompt and the image provided by the user into the trained image encoder, and process to obtain an image embedding vector;

[0127] The third generation unit 305 is configured to input the interactive prompt provided by the user into the prompt encoder to process and obtain a prompt embedding vector;

[0128] The segmentation processing unit 306 is configured to input the image embedding vector and the prompt embedding vector into the mask decoder for interaction, and obtain a segmented mask image after multi-layer cross-attention and upsampling.

[0129] As Figure 4 shown, an embodiment of the present application further provides an electronic device 400, including a memory 410, a processor 420, and a computer program 411 stored on the memory 410 and executable on the processor. When the processor 420 executes the computer program 411, the steps of the above-mentioned image segmentation method based on the SAM model with dynamic Prompt constraint and edge anchor point guidance are implemented.

[0130] Since the electronic device introduced in this embodiment is the device adopted by the image segmentation device based on the SAM model with dynamic Prompt constraint and edge anchor point guidance in the embodiment of the present application, based on the method introduced in the embodiment of the present application, those skilled in the art can understand the specific implementation manners and various variations of the electronic device in this embodiment. Therefore, the specific implementation of how this electronic device implements the method in the embodiment of the present application will not be described in detail here. As long as the device adopted by those skilled in the art to implement the method in the embodiment of the present application belongs to the scope to be protected by the present application.

[0131] In the specific implementation process, when the computer program 411 is executed by the processor, it can implement any implementation manner in the corresponding embodiment of the first aspect.

[0132] Figure 5 shows a schematic structural diagram of a computer system of an electronic device suitable for implementing the embodiment of the present application.

[0133] It should be noted that Figure 5The computer system 500 of the illustrated electronic device is merely an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0134] As Figure 5 shown, the computer system 500 includes a central processing unit 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory 502 or the program loaded from the storage section 508 into the random access memory 503, such as executing the methods described in the above embodiments. In the random access memory 503, various programs and data required for system operation are also stored. The central processing unit 501, the read-only memory 502, and the random access memory 503 are connected to each other via a bus 504. The input / output interface 505 is also connected to the bus 504.

[0135] The following components are connected to the input / output interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed so that a computer program read from it can be installed into the storage portion 508 as needed.

[0136] Specifically, according to the embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments of the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from the network through the communication portion 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit 501, various functions defined in the system of the present application are executed.

[0137] It should be noted that the computer-readable medium shown in the embodiments of the present application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device. In the present application, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and this computer-readable medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. Among them, each block in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks shown can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0139] The units involved in the embodiments described in this application can be implemented in software or in hardware, and the described units can also be provided in a processor. In some cases, the names of these units do not constitute a limitation on the units themselves.

[0140] As another aspect, the present application also provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes a method for image segmentation based on dynamic Prompt constraint and edge anchor point guidance in the above embodiments.

[0141] As another aspect, the present application also provides a computer-readable medium. The computer-readable medium can be included in the electronic device described in the above embodiments; it can also exist separately without being assembled into the electronic device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by an electronic device, the electronic device implements a method for image segmentation based on dynamic Prompt constraint and edge anchor point guidance in the above embodiments.

[0142] It should be noted that although several modules or units of a device for action execution are mentioned in the above detailed description, such a division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of the two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0143] Through the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software or by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0144] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include well-known common general knowledge or conventional technical means in the technical field not disclosed in the present application. It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. An image segmentation method based on dynamic Prompt constraint and edge anchor point guidance of the SAM model, wherein, The SAM model is provided with an image encoder, a prompt encoder, and a mask decoder, and is characterized in that the method includes: Design and train a lightweight dynamic adjustment network using an MLP structure; Use the lightweight dynamic adjustment network to control the intensity and spatial range of the interactive prompt provided by the user introduced into the image encoder; Train the image encoder based on an edge anchor point guidance mechanism; Input the introduction result of the interactive prompt and the image provided by the user into the trained image encoder, and process to obtain an image embedding vector; Input the interactive prompt provided by the user into the prompt encoder for processing to obtain a prompt embedding vector; Input the image embedding vector and the prompt embedding vector into the mask decoder for interaction, and obtain a segmented mask image after multi-layer cross-attention and upsampling; Design a lightweight dynamic adjustment network using an MLP structure, including: Design a lightweight dynamic adjustment network using a lightweight architecture of MLP + multi-head output; The process of training the lightweight dynamic adjustment network includes: Train the lightweight dynamic adjustment network using a three-stage progressive training scheme, including: In the first stage, freeze the backbone network and only train the modules related to the interactive prompt; In the second stage, unfreeze the high-level network for adjustment; In the third stage, adjust the parameters of the entire model; The using the lightweight dynamic adjustment network to control the intensity and spatial range of the interactive prompt provided by the user introduced into the image encoder includes: Input the interactive prompt provided by the user and the feature maps output by different layers in the image encoder into the lightweight dynamic adjustment network for processing, and generate a set of fusion weights and diffusion coefficients that adaptively change with the layer; The training the image encoder based on an edge anchor point guidance mechanism includes: Use depthwise separable convolution to extract edge responses from the shallow features of the image encoder, and cooperate with a batch normalization layer and a Gaussian error linear unit activation to generate an anchor mask; Establish a dual alignment mechanism, use the anchor mask to shield the gradients of non-anchor regions in backpropagation in the spatial dimension, and only update the weights corresponding to the anchor points, and calculate the L2 alignment loss function between the deep features and the anchor features in the semantic dimension; Adopt an alternating training strategy, only involve the anchor region in gradient update in odd training cycles, and restore the full image to normal training in even training cycles; During the entire training process, dynamically balance the interaction between the shallow and deep layer features in the image encoder through learnable gating parameters.

2. The method according to claim 1, wherein The inputting the introduction result of the interactive prompt and the image provided by the user into the trained image encoder to process and obtain an image embedding vector includes: Encode the interactive prompt provided by the user into a Gaussian heatmap; Combine the fusion weights, the diffusion coefficients, and the Gaussian heatmap, and input the combination result and the image provided by the user into the image encoder to process and obtain an image embedding vector.

3. An image segmentation device based on SAM model with dynamic Prompt constraint and edge anchor point guidance, applied to the method as described in Claim 1, wherein, The SAM model is provided with an image encoder, a prompt encoder, and a mask decoder, and is characterized in that the device includes: A first generation unit for designing and training a lightweight dynamic adjustment network using an MLP structure; A fusion unit, configured to use the lightweight dynamic adjustment network to control the intensity and spatial range of the interactive prompt provided by the user introduced into the image encoder; A training unit, configured to train the image encoder based on an edge anchor point guiding mechanism; A second generation unit, configured to input the introduction result of the interactive prompt and the image provided by the user into the trained image encoder to process and obtain an image embedding vector; A third generation unit, configured to input the interactive prompt provided by the user into the prompt encoder to process and obtain a prompt embedding vector; A segmentation processing unit, configured to input the image embedding vector and the prompt embedding vector into the mask decoder for interaction, and obtain a segmented mask image after multi-layer cross-attention and upsampling; 4. A computer-readable storage medium, characterized in that, Computer instructions are stored in the storage medium, and when the computer instructions run on a computer, the computer is caused to execute the method according to any one of claims 1-2.

5. An electronic device, characterized in that, Comprising: A memory and a processor; The memory is configured to store computer instructions; The processor is configured to call the computer instructions stored in the memory, so that the electronic device executes the method according to any one of claims 1-2.

Images