A medical image segmentation method and related device based on semi-supervised adapter fine-tuning and prompt learning based on SAM

Abstract

This application discloses a medical image segmentation method and related device based on semi-supervised adapter fine-tuning and prompt learning based on SAM. The method includes: importing and preprocessing a medical image dataset, loading a pre-trained SAM model and a YOLO prompt model, inserting dual adapters into each visual transformer block of the SAM encoder, and embedding a super-prompt adapter, an MLP alignment adapter, and a cross-attention adapter into the decoder; training the prompt model based on labeled data, screening qualified unlabeled data through KL divergence, generating click prompts, and performing pseudo-label segmentation using the SAM model; constructing a joint loss function containing a contrast manifold regularization term, and performing adapter fine-tuning based on the pseudo-label and labeled data while freezing the SAM backbone. The device includes a data import unit, a model loading unit, and a prompt training unit. The present invention effectively reduces labeling dependency, enhances model generalization and segmentation accuracy, and retains the original performance advantages of the SAM backbone.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a medical image segmentation method and related devices based on SAM semi-supervised adapter fine-tuning and prompt learning. Background Art

[0002] In the field of medical image segmentation, accurate and efficient tumor segmentation is crucial for diagnosis, treatment planning, and robot-assisted surgery. However, existing segmentation methods face many challenges. On the one hand, fully supervised learning methods rely on large-scale labeled datasets, while medical image labeling is time-consuming and requires expertise, resulting in data scarcity. On the other hand, although semi-supervised learning alleviates this problem by combining a small amount of labeled data with a large amount of unlabeled data, its application in medical image segmentation is still limited by differences in data distribution and insufficient model generalization capabilities. In addition, although existing adapter fine-tuning methods can effectively adjust pre-trained models to adapt to specific tasks, their dependence on labeled data and the efficiency of utilizing unlabeled data when processing medical images still need to be improved. Therefore, there is an urgent need for a semi-supervised adapter fine-tuning method that can efficiently utilize unlabeled data and enhance the generalization ability of the model to improve the accuracy and efficiency of medical image segmentation while reducing the dependence on large amounts of labeled data. Summary of the Invention

[0003] In order to solve the above technical problems, the present invention relates to a medical image segmentation method and related devices based on SAM semi-supervised adapter fine-tuning and prompt learning, which include but are not limited to a medical image segmentation device, electronic device, computer-readable storage medium and computer program product based on SAM semi-supervised adapter fine-tuning and prompt learning.

[0004] In a first aspect, a medical image segmentation method based on semi-supervised adapter fine-tuning and prompt learning of SAM is provided, the method comprising:

[0005] Importing and preprocessing a medical image dataset, wherein the medical image dataset includes labeled data and unlabeled data;

[0006] Load the pre-trained SAM model and the YOLO-based hint model, insert two adapters in each visual transformer block of the encoder of the SAM model; insert the super hint adapter, MLP alignment adapter and cross attention adapter in the decoder of the SAM model;

[0007] The prompt model is trained using the labeled data, the KL divergence between the unlabeled data and the labeled data is calculated, qualified unlabeled data is screened out according to the KL divergence, and the trained prompt model is used to perform target detection on the qualified unlabeled data to obtain a click prompt;

[0008] Using the SAM model to perform prompt image segmentation on the qualified unlabeled data according to the click prompt to obtain pseudo-label data;

[0009] A contrast manifold regularization term is introduced to construct a loss function; under the premise of freezing the backbone weight of the SAM model, the adapter is fine-tuned based on the loss function using the pseudo-labeled data and the annotated data to obtain the adapter weight;

[0010] Based on the adapter weight, the SAM model and the adapter are used to perform image segmentation to obtain a segmentation result.

[0011] In combination with any embodiment of the present application, the step of inserting two adapters into each visual transformer block of the encoder of the SAM model includes:

[0012] The first adapter is inserted after the multi-head attention of the encoder of the SAM model and before the residual connection, and the second adapter is inserted in the MLP residual path of the encoder of the SAM model.

[0013] In combination with any embodiment of the present application, the inserting of the super hint adapter, the MLP alignment adapter and the cross attention adapter into the decoder of the SAM model includes:

[0014] The super hint adapter is inserted before each visual attention block of the decoder of the SAM model, the MLP alignment adapter is inserted in the MLP residual path of the decoder of the SAM model, and the criss-cross attention adapter is inserted after the criss-cross attention residual connection of the decoder of the SAM model.

[0015] In combination with any embodiment of the present application, the step of screening out qualified unlabeled data according to the KL divergence includes:

[0016] Calculate the characteristic distribution of the labeled data and the unlabeled data using a kernel density estimation method;

[0017] A threshold is set based on the calculated KL divergence value, and the unlabeled data having the KL divergence value lower than the threshold is filtered, wherein the threshold is less than or equal to 1.2 times the mean KL divergence value of the unlabeled data.

[0018] In combination with any embodiment of the present application, the step of performing prompt image segmentation on the qualified unlabeled data according to the click prompt using the SAM model to obtain pseudo-label data includes:

[0019] Inputting the qualified unlabeled data;

[0020] Inputting the click prompt into the prompt encoder of the SAM model to perform image segmentation to obtain a segmentation mask;

[0021] The segmentation mask is integrated with the qualified unlabeled data to obtain pseudo-labeled data.

[0022] In combination with any embodiment of the present application, the introduction of the contrast manifold regularization term and the construction of the loss function include:

[0023] During the training of the adapter, calculating a manifold regularization loss and a contrastive learning loss based on the pseudo-labeled data and the labeled data;

[0024] The manifold regularization loss and the contrastive learning loss are weightedly summed to obtain a loss function.

[0025] In a second aspect, a medical image segmentation device based on semi-supervised adapter fine-tuning and prompt learning using SAM is provided, characterized by comprising:

[0026] A data input unit, configured to import and preprocess a medical image dataset, wherein the medical image dataset includes annotated data and unannotated data;

[0027] A model loading unit, configured to load a pre-trained SAM model and a YOLO-based hint model, insert two adapters into each visual transformer block of the encoder of the SAM model; and insert a super hint adapter, an MLP alignment adapter, and a cross attention adapter into the decoder of the SAM model;

[0028] a prompt generating unit, configured to train the prompt model using the labeled data; calculate the KL divergence between the unlabeled data and the labeled data, screen out qualified unlabeled data based on the KL divergence, perform target detection on the qualified unlabeled data using the trained prompt model, and obtain a click prompt;

[0029] a pseudo-label generating unit, configured to perform prompt image segmentation on the qualified unlabeled data according to the click prompt using the SAM model to obtain pseudo-label data;

[0030] A fine-tuning unit is configured to introduce a contrast manifold regularization term and construct a loss function; under the premise of freezing the backbone weight of the SAM model, fine-tune the adapter using the pseudo-label data and the annotated data based on the loss function to obtain an adapter weight;

[0031] An image segmentation unit is configured to perform image segmentation using the SAM model and the adapter based on the adapter weight to obtain a segmentation result.

[0032] In a third aspect, an electronic device is provided, comprising: a processor, a communication module, a sensor, a user interface, and a storage unit, wherein the storage unit is configured to store computer program code, wherein the program code comprises computer instructions. When the processor executes these instructions, the electronic device performs the method described in the second aspect and any embodiment thereof.

[0033] In a fourth aspect, another electronic device is provided, comprising: a processor, a wireless communication module, a touch screen, a speaker, and a storage unit, wherein the storage unit is configured to store computer program code, wherein the program code comprises computer instructions. When the processor executes these instructions, the electronic device performs the method described in the second aspect and any embodiment thereof.

[0034] In a fifth aspect, a computer-readable storage medium is provided, wherein a computer program is stored, wherein the program includes program instructions. When these instructions are executed by a processor, the processor will perform the method described in the second aspect and any embodiment thereof.

[0035] In a sixth aspect, a computer program product is provided, wherein the computer program product comprises a computer program or instructions. When the computer program or instructions are run on a computer, the computer will execute the method described in the second aspect and any embodiment thereof.

[0036] It should be understood that the above general description and the following detailed description are only used as examples and explanations and do not limit the present application in any way.

[0037] Compared to the prior art, this invention proposes a medical image segmentation method and related device based on semi-supervised adapter fine-tuning and cue learning using SAM. This method first imports and preprocesses a medical image dataset, then loads a pretrained SAM model and a YOLO-based cue model. Two adapters are inserted into each visual transformer block of the SAM model's encoder, and a super-cue adapter, an MLP alignment adapter, and a cross-attention adapter are inserted into the decoder. Next, the cue model is trained using labeled data, the KL divergence between unlabeled and labeled data is calculated, qualified unlabeled data is screened, and object detection is performed on the qualified unlabeled data to obtain click cues. The SAM model is then used to perform cue image segmentation on the qualified unlabeled data based on the click cues, generating pseudo-labeled data. While freezing the backbone weights of the SAM model, a contrastive manifold regularization term is introduced to construct a loss function. Adapter fine-tuning is performed using pseudo-labeled and labeled data to obtain adapter weights. Finally, image segmentation is performed based on the adapter weights to obtain a segmentation result. Through semi-supervised learning and adapter fine-tuning, the present invention effectively reduces the dependence on a large amount of labeled data, improves the performance of medical image segmentation, and maintains the stability of the model's performance, with high practicality and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0039] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0040] Figure 1 A schematic diagram of a medical image segmentation method for semi-supervised adapter fine-tuning and prompt learning based on SAM provided in an embodiment of the present disclosure;

[0041] Figure 2 Schematic diagram of adapter details of a medical image segmentation method for semi-supervised adapter fine-tuning and prompt learning based on SAM provided in an embodiment of the present disclosure;

[0042] Figure 3 A schematic diagram of a medical image segmentation device for semi-supervised adapter fine-tuning and prompt learning based on SAM provided in an embodiment of the present disclosure;

[0043] Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] In order to allow professionals in this technical field to more fully understand the technical solution of the present application, the technical solution of the present application will be explained in detail and clearly with the help of the accompanying drawings. It should be noted that the described embodiments are only some examples of the present application and do not represent all. Based on these embodiments, those skilled in the art can directly deduce all other possible implementation plans without engaging in creative thinking, and these are also included in the scope of protection of the present application.

[0045] In the specification, claims, and related drawings of this application, the terms "first," "second," and the like are used solely to distinguish between different elements and do not imply any particular order. Furthermore, the use of "including," "having," and their variations denotes non-exclusive inclusion. This means that if a process, method, system, product, or apparatus includes a series of steps or components, the process, method, system, product, or apparatus is not limited to the enumerated steps or components and may include other steps or components not listed, or other steps or units inherent to the process, method, system, product, or apparatus.

[0046] The “embodiment” mentioned in this document refers to any instance in which a particular feature, structure or characteristic is combined, and these instances may belong to at least one embodiment of the present application. The “embodiment” mentioned in this document does not necessarily refer to the same specific case, nor does it mean that they are independent or exclusive alternatives. It should be understood by those skilled in the art that the embodiments described herein can be used in conjunction with other embodiments. It should be understood that in this application, “at least one” includes one or more instances, “a plurality” means two or more instances, and “at least two” means two or more instances.

[0047] It should be understood that the method embodiment of the present application can also be implemented by a processor executing computer program code. The embodiment of the present application is described below in conjunction with the drawings in the embodiment of the present application.

[0048] See also Figure 1 , Figure 1 A schematic diagram of a medical image segmentation method for semi-supervised adapter fine-tuning and prompt learning based on SAM is provided in an embodiment of the present disclosure. The specific steps are as follows:

[0049] 101. Import and preprocess data.

[0050] A medical image dataset is imported and preprocessed, where the medical image dataset includes labeled data and unlabeled data.

[0051] In this embodiment, the preprocessing includes at least image cropping and image normalization.

[0052] In this embodiment, sources of medical images include but are not limited to importing from hospital databases, importing from medical devices, etc.

[0053] 102. Load the model and adapter.

[0054] Load the pre-trained SAM model and the YOLO-based hint model, insert two adapters into each visual transformer block of the SAM model's encoder; insert the super hint adapter, MLP alignment adapter, and cross attention adapter into the SAM model's decoder. For details, please refer to Figure 2 , Figure 2 Schematic diagram of adapter details of a medical image segmentation method for semi-supervised adapter fine-tuning and prompt learning based on SAM provided in an embodiment of the present disclosure.

[0055] The specific steps are: the first adapter is inserted after the multi-head attention and before the residual connection of the encoder of the SAM model, and the second adapter is inserted into the MLP residual path of the encoder of the SAM model;

[0056] The super hint adapter is inserted before each visual attention block of the decoder of the SAM model, the MLP alignment adapter is inserted into the MLP residual path of the decoder of the SAM model, and the criss-cross attention adapter is inserted after the criss-cross attention residual connection of the decoder of the SAM model.

[0057] In this embodiment, the version of YOLO is not lower than YOLOv5.

[0058] In another possible implementation, the SAM model may be a lightweight version that has been distilled from specific medical domain knowledge.

[0059] In another possible implementation, the SAM model may be a pre-trained version in the medical field.

[0060] In another possible implementation, the SAM model may be a fine-tuned version in the medical field.

[0061] In another possible implementation, other SAM-related adapters may be used.

[0062] 103. Filter unlabeled data to train the prompt model.

[0063] The prompt model is trained with the labeled data, the KL divergence between the unlabeled data and the labeled data is calculated, qualified unlabeled data is screened out according to the KL divergence, and the trained prompt model is used to perform target detection on the qualified unlabeled data to obtain a click prompt.

[0064] In another possible implementation, other similarity measurement methods, such as cosine similarity or JS divergence, may be considered to evaluate the similarity between unlabeled data and labeled data.

[0065] 104. Tip segmentation to obtain pseudo labels.

[0066] The SAM model is used to perform prompt image segmentation on the qualified unlabeled data according to the click prompt to obtain pseudo-label data.

[0067] In another possible implementation, the step may consider using other interactive image segmentation models.

[0068] 105. Adapter fine-tuning.

[0069] A contrast manifold regularization term is introduced to construct a loss function. Under the premise of freezing the backbone weight of the SAM model, the adapter is fine-tuned based on the loss function using the pseudo-label data and the annotated data to obtain the adapter weight.

[0070] In this embodiment, the method for constructing the contrast manifold regularization term can refer to a semi-supervised medical image segmentation method based on contrast manifold regularization previously applied by the applicant (patent number 202510866998.1).

[0071] In another possible implementation, the contrast manifold regularization term may be combined with other loss terms, including but not limited to weighted summation, to construct a loss function.

[0072] 106. Image segmentation.

[0073] Based on the adapter weight, the SAM model and the adapter are used to perform image segmentation to obtain a segmentation result.

[0074] In another possible implementation, the segmentation result may be further post-processed.

[0075] See also Figure 3 , Figure 3 Schematic diagram of a medical image segmentation device 1 for semi-supervised adapter fine-tuning and prompt learning based on SAM provided by an embodiment of the present disclosure. The device 1 specifically includes a data input unit 11, a model loading unit 12, a prompt generation unit 13, a pseudo-label generation unit 14, a fine-tuning unit 15, and an image segmentation unit 16. Specifically:

[0076] A data input unit 11 is used to import and preprocess a medical image dataset, wherein the medical image dataset includes annotated data and unannotated data;

[0077] A model loading unit 12 is configured to load a pre-trained SAM model and a YOLO-based hint model, insert two adapters into each visual transformer block of the encoder of the SAM model, and insert a super hint adapter, an MLP alignment adapter, and a cross attention adapter into the decoder of the SAM model.

[0078] The prompt generating unit 13 is configured to train the prompt model using the labeled data; calculate the KL divergence between the unlabeled data and the labeled data, screen out qualified unlabeled data based on the KL divergence, perform target detection on the qualified unlabeled data using the trained prompt model, and obtain a click prompt;

[0079] a pseudo-label generating unit 14, configured to perform prompt image segmentation on the qualified unlabeled data according to the click prompt using the SAM model to obtain pseudo-label data;

[0080] A fine-tuning unit 15 is configured to introduce a contrast manifold regularization term and construct a loss function; and under the premise of freezing the backbone weights of the SAM model, fine-tune the adapter using the pseudo-labeled data and the annotated data based on the loss function to obtain an adapter weight.

[0081] The image segmentation unit 16 is configured to perform image segmentation based on the adapter weights using the SAM model and the adapter to obtain a segmentation result.

[0082] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0083] The above describes in detail the method of the embodiment of the present application, and the following provides an apparatus of the embodiment of the present application.

[0084] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0085] See also Figure 4 , Figure 4 The following is a schematic diagram of the hardware architecture of an electronic device described in an embodiment of the present application. The electronic device 2 is primarily composed of a processor 21 and a memory 22. In addition, the device may also include an input device 23 and an output device 24. The processor 21, memory 22, input device 23, and output device 24 are interconnected via connecting components, which may be various interfaces, data cables, or communication buses, and are not specifically specified in the present embodiment.

[0086] Processor 21 may be one or more graphics processing units (GPUs). If processor 21 is a GPU, the GPU may be single-core or multi-core. Optionally, processor 21 may comprise a processor group consisting of multiple GPUs, interconnected via one or more buses. Furthermore, the processor may be other types of processors, which are not specifically limited in this embodiment of the present application.

[0087] Memory 22 is designed to store computer program instructions and various program codes required to execute the present invention. Optionally, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), which are used to store relevant instructions and data.

[0088] The input device 23 is used to input data and / or signals, and the output device 24 is used to output data and / or signals. The input device 23 and the output device 24 can be independent devices or an integrated device.

[0089] It should be appreciated that in the embodiment of the present application, the memory 22 can store not only relevant instructions but also relevant data. The embodiment of the present application does not specify the specific data content stored in the memory.

[0090] You should understand that Figure 4 Only a simplified design of an electronic device is shown. In actual use, the electronic device may also include other necessary components, such as different numbers of input / output devices, processors, memories, etc. All electronic devices that can implement the embodiments of this application are within the scope of protection of this application.

[0091] Those skilled in the art will recognize that, according to the components and algorithm steps of each example described in the embodiments disclosed herein, these functions can be implemented by electronic hardware or by combining computer software and electronic hardware. Whether these functions are performed by hardware or software will be determined based on the specific application requirements and design limitations of the technical solution. Technicians can adopt different implementation methods according to the requirements of each specific application, but such implementation methods should not be considered to exceed the scope of protection of this application.

[0092] Professionals should understand that, for the sake of ease of description and simplification, the specific operating procedures of the above-mentioned systems, devices, and components can refer to the corresponding steps in the previous method embodiments and will not be repeated here. At the same time, professionals should also understand that each embodiment in this application has its own focus. For the sake of ease of description and simplification, the same or similar content may not be repeated in different embodiments. Therefore, if a part is not mentioned or not explained in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0093] In the several embodiments provided in this application, it should be recognized that the disclosed systems, devices and methods can also be implemented in other ways. For example, the device embodiments described are only exemplary, in which the division of the units is only a division of logical functions, and there may be different division methods in actual implementation. For example, multiple units or components may be merged or integrated into another system, or certain features may be omitted, or certain steps may not be performed. In addition, the connections between each other shown or discussed, whether direct or indirect, whether coupling or communication connection, may be implemented in electrical, mechanical or other forms through interfaces, devices or units.

[0094] Units described as independent components may or may not actually be physically separate; parts presented as units may or may not be physical entities; that is, they may be centralized in one location or distributed across multiple network nodes. Depending on actual needs, some or all of these units may be selected to achieve the objectives of this embodiment.

[0095] Furthermore, in the various embodiments of the present application, the various functional units may be integrated into a single processing unit, physically exist independently, or two or more units may be combined into a single unit. In the aforementioned embodiments, the relevant functions may be implemented in whole or in part through software, hardware, firmware, or any combination thereof. If software implementation is chosen, it may be implemented in whole or in part in the form of a computer program product. This computer program product comprises one or more computer instructions. When these instructions are loaded and executed on a computer, they will generate, in whole or in part, the processes or functions described in the embodiments of this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. These computer instructions may be stored in a computer-readable storage medium or transmitted via such a medium. The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic cable, DSL) or wireless (e.g., infrared, wireless, microwave, etc.) means. A computer-readable storage medium may be any computer-accessible, usable medium, or a data storage facility such as a server or data center that integrates one or more usable media. These available media may include magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., DVDs), semiconductor media (e.g., SSDs), etc. Those skilled in the art will appreciate that all or part of the process steps for implementing the above-described method embodiments can be accomplished through hardware associated with computer program instructions. These programs can be stored on computer-readable storage media. When executed, these programs will contain the processes for each of the above-described method embodiments. These storage media include, but are not limited to, various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A medical image segmentation method based on semi-supervised adapter fine-tuning and prompt learning based on SAM, characterized by: The method comprises: Importing and preprocessing a medical image dataset, wherein the medical image dataset includes labeled data and unlabeled data; Load the pre-trained SAM model and the YOLO-based hint model, insert the first adapter after the multi-head attention of each visual transformer block of the encoder of the SAM model and before the residual connection, and insert the second adapter in the MLP residual path of the encoder of the SAM model; insert the super hint adapter before each visual attention block of the decoder of the SAM model, insert the MLP alignment adapter in the MLP residual path of the decoder of the SAM model, and insert the cross attention adapter after the cross attention residual connection of the decoder of the SAM model; The prompt model is trained using the labeled data, and a kernel density estimation method is used to calculate the characteristic distribution of the labeled data and the unlabeled data to obtain KL divergence. The unlabeled data having a KL divergence value lower than a threshold is screened to obtain qualified unlabeled data. The trained prompt model is used to perform target detection on the qualified unlabeled data to obtain a click prompt. Using the SAM model to perform prompt image segmentation on the qualified unlabeled data according to the click prompt to obtain pseudo-label data; During the adapter training process, a manifold regularization loss and a contrastive learning loss are calculated based on the pseudo-labeled data and the annotated data, and a loss function is constructed by weighted summing the manifold regularization loss and the contrastive learning loss. Under the premise of freezing the backbone weight of the SAM model, the adapter is fine-tuned using the pseudo-labeled data and the annotated data based on the loss function to obtain an adapter weight. Based on the adapter weight, the SAM model and the adapter are used to perform image segmentation to obtain a segmentation result.

2. The method according to claim 1, characterized in that The preprocessing includes image cropping and image normalization.

3. The method according to claim 1, characterized in that The YOLO version used in the prompt model is YOLOv11.

4. The method according to claim 1, wherein The threshold is less than or equal to 1.2 times the mean KL divergence of the unlabeled data.

5. The method according to claim 1, wherein The step of performing prompt image segmentation on the qualified unlabeled data according to the click prompt using the SAM model to obtain pseudo-label data includes: Inputting the qualified unlabeled data; Inputting the click prompt into the prompt encoder of the SAM model to perform image segmentation to obtain a segmentation mask; The segmentation mask is integrated with the qualified unlabeled data to obtain pseudo-labeled data.

6. A medical image segmentation device based on semi-supervised adapter fine-tuning and prompt learning based on SAM, characterized in that The device comprises: A data input unit, configured to import and preprocess a medical image dataset, wherein the medical image dataset includes annotated data and unannotated data; A model loading unit is used to load a pre-trained SAM model and a YOLO-based hint model, insert a first adapter after the multi-head attention of each visual transformer block of the encoder of the SAM model and before the residual connection, and insert a second adapter into the MLP residual path of the encoder of the SAM model; insert a super hint adapter before each visual attention block of the decoder of the SAM model, insert an MLP alignment adapter into the MLP residual path of the decoder of the SAM model, and insert a cross attention adapter after the cross attention residual connection of the decoder of the SAM model; a prompt generating unit, configured to train the prompt model using the labeled data, calculate the characteristic distribution of the labeled data and the unlabeled data using a kernel density estimation method to obtain a KL divergence, screen the unlabeled data having a KL divergence value lower than a threshold to obtain qualified unlabeled data, perform target detection on the qualified unlabeled data using the trained prompt model, and obtain a click prompt; a pseudo-label generating unit, configured to perform prompt image segmentation on the qualified unlabeled data according to the click prompt using the SAM model to obtain pseudo-label data; A fine-tuning unit is configured to calculate, during the adapter training process, a manifold regularization loss and a contrastive learning loss based on the pseudo-labeled data and the annotated data, and perform a weighted summation of the manifold regularization loss and the contrastive learning loss to construct a loss function; and, under the premise of freezing the backbone weights of the SAM model, fine-tune the adapter using the pseudo-labeled data and the annotated data based on the loss function to obtain an adapter weight; An image segmentation unit is configured to perform image segmentation using the SAM model and the adapter based on the adapter weight to obtain a segmentation result.

7. An electronic device, characterized in that: include: A processor and a storage unit, the storage unit is used to store computer program code, the code includes computer instructions, when the processor executes these instructions, the electronic device performs the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 5.

9. A computer program product, characterized in that The computer program product comprises a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is caused to perform the method according to any one of claims 1 to 5.

Images