Medical image segmentation method and device based on multi-model cooperation and active learning

Abstract

The invention relates to the technical field of medical image segmentation and diagnosis, in particular to a medical image segmentation method and device based on multi-model collaboration and active learning, and the method comprises the steps: firstly building a medical text-vision joint prompt based on an OFA (One ForALL) multi-mode model and a BiomedBERT language model collaboratively; fusing multi-modal input through a GLIP model to generate a target positioning frame; then, medical image segmentation guided by a target frame is realized based on the MedSAM architecture, and an initial pseudo mask is generated; and finally, carrying out quality screening by adopting a pseudo-label area sampling strategy, carrying out segmentation model iteration training by combining real labeling and an optimized pseudo-label set, and according to the process, through the synergistic effect of four stages of cross-modal representation alignment, target area positioning, refined segmentation and dynamic label optimization, carrying out segmentation model iteration training. And efficient medical image segmentation under the condition of few samples is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image segmentation, in particular to a medical image segmentation method and device based on multi-model cooperation and active learning. BACKGROUND

[0002] As a key tool in modern medical field, medical image segmentation technology shows irreplaceable clinical value in precise disease diagnosis, personalized surgical planning, and dynamic evaluation of treatment effect. Taking the detection of digestive tract lesions as a typical application scenario, accurate lesion segmentation can not only provide quantitative basis for tumor staging, but also assist in judging key pathological parameters such as tumor infiltration depth, thereby directly affecting the selection of treatment plan and prognosis evaluation. However, this field has long been plagued by the problem of high data annotation cost: on the one hand, high-precision medical image annotation requires professional physicians with rich clinical experience to carefully outline each pixel of the image, which is a labor-intensive mode of work that seriously hinders the construction of large-scale standardized data sets; on the other hand, digestive tract medical images generally have fuzzy tissue boundaries, low contrast between lesion areas and normal tissues, and small lesions are easy to be overlooked, which further increases the complexity and subjectivity of annotation work. Existing research has confirmed that the performance of traditional fully supervised segmentation models will show a cliff-like decline when the amount of labeled data is insufficient, making it difficult to meet the dual requirements of segmentation accuracy and robustness in clinical practice.

[0003] To solve the core problem of limited annotation resources, current technical routes mainly focus on active learning and semi-supervised learning. Active learning selects the most informative samples for annotation through iteration, trying to maximize model performance with minimal annotation cost. However, traditional methods rely too much on the confidence indicators (such as entropy, uncertainty, etc.) of model prediction, and in the unstable performance stage of early model training, they are prone to select redundant samples or misjudge high-value samples, resulting in low learning efficiency in the cold start stage. Semi-supervised learning can generate pseudo-labels from unlabeled data to expand the training set, but when the number of labeled samples is too small, the quality of pseudo-labels is limited by the limited generalization ability of the initial model, which can easily cause the "snowball effect" of error accumulation. Especially in challenging tasks such as segmentation of small lesions in the digestive tract, conventional confidence threshold strategies are difficult to effectively distinguish between high-difficulty samples with real learning value and low-quality samples containing annotation noise, ultimately leading the model optimization process to a local optimal solution.

[0004] In view of this, the present application aims to provide a medical image segmentation method and device based on multi-model collaboration and active learning. This method effectively integrates the prior knowledge transfer ability of large models and the intelligent sample screening mechanism of active learning by constructing a multi-model collaboration framework, significantly reduces the dependence on labeled data, and innovatively designs a pseudo-label quality optimization strategy, thereby systematically improving the segmentation performance of the model in complex medical image scenarios. Better meet the demand for high-precision and high-efficiency segmentation in clinical practice. SUMMARY

[0005] To solve the above problems, the present application provides a medical image segmentation method and device based on multi-model collaboration and active learning, which is based on attribute description generation, cross-modal prompt optimization, large model pseudo-label, active learning optimization, and improves the prediction and diagnosis accuracy and comprehensiveness of diseases.

[0006] The technical scheme adopted by the present application is:

[0007] A medical image segmentation method based on multi-model collaboration and active learning, comprising the following steps:

[0008] S1. By inputting the medical image and the preset prompt template, the common medical attributes of the image are extracted using the BiomedBERT model based on biomedical field text pre-training, and the common medical attributes include lesion location;

[0009] S2. By inputting the medical image and the customized visual question answering statement, the unique medical attributes of the image are extracted using the large language model One For All, and the unique medical attributes include lesion color and shape;

[0010] S3. The medical attributes extracted in steps S1 and S2 and the original medical image are input into the cross-modal object detection large model GLIP to generate a lesion object box; if the object position cannot be determined, a default object box covering 80% of the image area is generated;

[0011] S4. The object box generated in step S3 is input into the medical image segmentation large model MedSAM to generate an initial pseudo-label mask image;

[0012] S5. During the training process of the segmentation model, the pseudo-label area screening strategy is used to optimize the quality of the initial pseudo-label, the area of the pseudo-label binary image is sorted, the pseudo-label with larger area is preferentially selected, and the low-quality pseudo-label sample is screened out by the active learning algorithm and is marked by a professional physician to obtain a true label;

[0013] S6. Based on the optimized label data, the DSC loss function is used to train the special segmentation model, and the random gradient descent method is used to update the model parameters to improve the segmentation performance.

[0014] Further, the BiomedBERT model realizes performance breakthrough in biomedical NLP tasks by being pre-trained from scratch based on biomedical field text and adopting a field-specific vocabulary.

[0015] Further, the visual question answering mode inputs the large language model OFA (One For All) by combining a customized question sentence with a medical image, and the question sentence is designed for specific lesion properties of a single image.

[0016] Further, the GLIP model reconstructs target detection into a phrase positioning task under a text prompt, extracts regional features by using an image encoder, generates text features by using a language encoder, aligns image and text features by using cross-modal deep fusion, and finally predicts the position of the target frame matched with the text semantic by combining a regressor.

[0017] Further, the MedSAM model guides the image encoder, prompt encoder and mask decoder architecture by the target frame, fuses image features and prompt information by using the cross-attention mechanism, and generates an accurate segmentation mask of the target region.

[0018] Further, the pseudo-label area screening strategy calculates the pixel area ratio of the pseudo-label binary image, preferentially retains the pseudo-label with a low area ratio, and actively selects samples with a high area ratio for manual annotation.

[0019] Further, the special segmentation model includes any one of U-net, CompNet, SegFormer or Transattunet.

[0020] Based on the same inventive concept, the application also provides a medical image segmentation device based on multi-model collaboration and active learning, comprising a processor and a memory, the memory stores executable instructions, when the instructions are executed by the processor, the steps of the above method are realized.

[0021] The beneficial effects of the application are as follows:

[0022] The segmentation method provided by the application comprehensively utilizes the advantages of different models by constructing a multi-model collaborative framework of BiomedBERT+OFA→GLIP→MedSAM, multi-model collaboration and active learning strategy, extracts rich and accurate medical attribute information from medical images, generates lesion target frame and initial pseudo-label mask image, optimizes the quality of pseudo-labels through pseudo-label area screening and active learning algorithm, and finally trains a special segmentation model based on the optimized label data, effectively improves the accuracy and performance of medical image segmentation, reduces the dependence on a large amount of manually annotated data, and improves the segmentation efficiency and practicability. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 A schematic diagram of four core modules in the embodiment of the present application;

[0024] Figure 2 A visualization result diagram of the method in the embodiment of the present application and other advanced active learning methods combined with pseudo labels under different segmentation model architectures in the HyperKvasir-SEG dataset; wherein the other advanced active learning methods are: (Least Conffdence (LC); Entropy Sampling (En); Margin Sampling (MS); BatchActive learning by Diverse Gradient Embeddings (BADGE); BayesianActive Learning by Disagreement (BALD)); and the different segmentation model architectures are: (U-Net; CompNet; SegFormer; Transattunet);

[0025] Figure 3 A visualization result diagram of the method in the embodiment of the present application and other advanced active learning methods combined with pseudo labels under different segmentation model architectures in the ISIC 2017 dataset; wherein the other advanced active learning methods are: (Least Conffdence (LC); Entropy Sampling (En); Margin Sampling (MS); BatchActive learning by Diverse Gradient Embeddings (BADGE); BayesianActive Learning by Disagreement (BALD)); and the different segmentation model architectures are: (U-Net; CompNet; SegFormer; Transattunet);

[0026] Figure 4The figure shows the visualization results of the method in the embodiment of the present invention and other advanced active learning methods combined with pseudo labels in the ISIC 2018 dataset under different segmentation model architectures; among them, the other advanced active learning methods are: LeastConffdence (LC); Entropy Sampling (En); Margin Sampling (MS); BatchActive learning by Diverse Gradient Embeddings (BADGE); BayesianActive Learning by Disagreement (BALD); and the different segmentation model architectures are: (U-Net; CompNet; SegFormer; Transattunet). DETAILED DESCRIPTION

[0027] To facilitate understanding of the present invention, the present invention will be described more fully below through examples, with preferred embodiments of the present invention provided below. However, the present invention can be implemented in a variety of different forms and is not limited to the embodiments described herein. Any other embodiments obtained by modifying or equivalently replacing the technical solution of the present invention without inventive results are within the scope of protection of the present invention.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for describing specific embodiments and are not intended to limit the present invention.

[0029] The numerical values ​​disclosed in the embodiments of the present invention are approximate values, not definite values. Where errors or experimental conditions permit, all values ​​within the error range may be included without being limited to the specific numerical values ​​disclosed in the embodiments of the present invention.

[0030] like Figure 1 As shown, this embodiment provides a medical image segmentation method based on multi-model collaboration and active learning, which includes four core modules: ① Collaborative construction of a medical text-visual joint prompt based on the OFA multimodal model and the BiomedBERT language model; ② Generate a target positioning frame by fusing multimodal inputs through the GLIP model; ③ Implement target frame-guided medical image segmentation based on the MedSAM architecture to generate an initial pseudo mask; ④ Use a pseudo label area sampling strategy for quality screening, and iteratively train the segmentation model by combining the real annotations with the optimized pseudo label set. This method achieves efficient medical image segmentation under few-sample conditions through the synergy of four stages: cross-modal representation alignment, target area positioning, refined segmentation, and dynamic label optimization.

[0031] The method comprises the following steps:

[0032] S1. By inputting a medical image and a preset prompt template, a common medical attribute of the image is extracted by using a BiomedBERT model pre-trained based on biomedical field text, and the common medical attribute comprises a lesion position;

[0033] The specific steps are as follows:

[0034] Taking a medical image as input, a BiomedBERT model is used for structured feature extraction. For a target category (such as a polyp), an attribute prediction template is constructed:

[0035] 'location':f'The locationof{cls_name}is at[MASK].',

[0036] Where {cls_name} needs to be replaced with the class name of the target data set (for example, polyps). As shown in the upper part of Figure 1 , this step gives a target category c and a pre-defined attribute set A = {a1, a2, …, a N}, for each attribute a i , a template sentence is constructed. Through a mask position probability prediction mechanism, the words with the optimal conditional probability are selected from a medical professional vocabulary as attribute values, and the semantic analysis of core features such as lesion positioning is completed:

[0037]

[0038] Where is a medical field vocabulary.

[0039] S2. By inputting a medical image and a customized visual question and answer statement, a large language model One ForAll is used to extract specific medical attributes of the image, and the specific medical attributes comprise lesion color and shape;

[0040] The specific steps are as follows:

[0041] A question and answer pair (such as color, shape, and other visual attributes) oriented to image characteristics is constructed, and an OFA visual question and answer model is used for feature decoding. An interactive question template is designed, and for each input image x and its visual attribute set B = b1, b2, …, b M , a question is designed. For example

[0042] 'What is the color of these c?',

[0043] 'What is the shape of these c?'

[0044] The frequently used prompts are: color; shape; lesion number, etc. The answer is generated through the OFA model:

[0045]

[0046] Specifically, the general attributes generated by the mask language model in this step The image-specific attributes extracted by visual question answering are fused into a unified template to generate the final prompt:

[0047]

[0048] where is the attribute combination function (such as and c on ”), represents the splicing operation. We usually use mask language model prompts for location attributes, and use visual question answering for the rest of the attributes. This method effectively improves the model's adaptability to new medical image categories by synergistically utilizing structured medical knowledge (MLM) and visual detail information (VQA).

[0049] S3. Input the medical attributes extracted in steps S1 and S2 and the original medical image into the cross-modal target detection large model GLIP to generate the lesion target frame; if the target position cannot be determined, a default target frame covering 80% of the image area is generated;

[0050] Use the attribute description and the image to input into the target detection large model to generate the target frame of the lesion;

[0051] Specifically, based on the medical attribute description extracted in the previous stage and the original image data, the GLIP model is driven using the MMDetection framework to realize open vocabulary lesion positioning and generate the target detection frame, as shown in the upper right corner of Figure 1 ;

[0052] The specific implementation steps are as follows:

[0053] Let the visual encoder be Enc I , the language encoder be Enc L , and the cross-modal fusion module be Fuse. The detection process can be formalized as:

[0054] (1) The visual encoder extracts the candidate region features:

[0055]

[0056] where N is the number of candidate regions and d is the feature dimension.

[0057] (2) Text encoder parses attribute hint features:

[0058]

[0059] M is the number of prompt phrases.

[0060] (3) Cross-modal alignment score calculation

[0061] Generate a cross-modal alignment score by calculating the visual-text feature dot product matrix:

[0062]

[0063] Where S represents the output alignment score. This matrix quantifies the semantic match between the candidate region and the attribute phrase, and is used to generate the initial positioning box. To further enhance detection robustness, the model uses multi-level cross-modal feature interaction to optimize positioning accuracy.

[0064] V',T'=Cross-MHA(V,T)

[0065] The final detection box is generated by fusion of the feature V' and the regression head. Cross-MHA stands for Cross Multi-Head Attention Module.

[0066] Post-processing mechanism:

[0067] The test results must meet two conditions:

[0068] ① The confidence threshold is increased to 0.35 (an increase of 0.05 compared to the conventional setting) to adapt to the high recall requirements of medical images.

[0069] ②The effective detection frame area does not exceed 95% of the image area.

[0070] For cases with insufficient confidence, the system automatically generates a default box covering 80% of the image area. Finally, the top three candidate boxes with the largest areas (taking into account both missed diagnosis prevention and noise suppression) are selected and input into the MedSAM model for segmentation. This design conforms to the clinical characteristics of medical imaging, where a single lesion is dominant and multiple lesions do not exceed three.

[0071] S4. Input the target box generated in step S3 into the medical image segmentation model MedSAM to generate an initial pseudo-label mask image;

[0072] Specifically, this step uses the target box and the medical image segmentation model to create a pseudo-label mask image of the image;

[0073] Specifically, MedSAM integrates target positioning information with a large medical segmentation model and uses an adaptive feature fusion mechanism to achieve accurate reconstruction of lesion masks, such as Figure 1 Shown in the lower right part.

[0074] The specific implementation steps are as follows:

[0075] (1) Image feature encoding: an image encoder is constructed using a visual Transformer architecture to map the input medical image to a deep feature space:

[0076]

[0077] where HxW represents the size of the feature map, and D represents the number of feature channels.

[0078] (2) Prompt encoding: the coordinates (x k ,y min ,x min ,y max ) of the bounding box b max are converted into prompt features by a position encoding module:

[0079]

[0080] where d p is the prompt embedding dimension.

[0081] (3) Cross-modal feature fusion: a mask decoding network with medical semantic perception ability is constructed to realize dynamic fusion of positioning information and visual features through a multi-level feature interaction mechanism:

[0082]

[0083] where CrossAttn is a multi-head cross-attention layer.

[0084] (4) Mask generation: the target mask is output through deconvolution operation and sigmoid activation function:

[0085]

[0086] where H in xW in is the input image resolution, and σ is the sigmoid function.

[0087] If GLIP does not generate an effective bounding box, B default is generated and input into MedSAM to generate a large-scale mask M default . This dual processing strategy sets a dynamic threshold gating mechanism to ensure the robustness and integrity of the pseudo-label generation system, providing reliable data support for subsequent active learning iterations.

[0088] S5. In the segmentation model training process, the pseudo label area screening strategy is used to optimize the quality of the initial pseudo label, the area of the pseudo label binary image is sorted, the pseudo label with larger area is preferentially selected, and the low-quality pseudo label sample is screened by the active learning algorithm and is marked by a professional physician to obtain a real label;

[0089] Specifically, in the process of training the segmentation model, the pseudo label area method is used to replace the low-quality pseudo label with a real label, thereby improving the overall label quality.

[0090] Specifically, the active learning mechanism is used to make an expert label the low-quality pseudo label, and the low-quality pseudo label is replaced with a real label. The segmentation model training process is as follows: Figure 1 The active learning algorithm pseudo code is as follows:

[0091] The specific implementation steps are as follows:

[0092] A candidate pool containing the pseudo label generated by MedSAM is constructed Where M i is the pseudo label mask generated by MedSAM.

[0093] (1) Initial training set construction: K samples are selected from according to the pseudo label area (i.e., the number of white pixels), and are marked by an expert to form an initial training set which is used to establish a basic segmentation model

[0094] (2) Iterative sampling strategy: in the tthiteration, the mask coverage index of the candidate pool sample is calculated:

[0095]

[0096] Where M i (h,w)∈{0,1}, the area a i is the total number of binary mask pixels.

[0097] (3) Sample selection: the first Top-m samples are selected in descending order of a i to generate a high-value candidate set

[0098]

[0099] Where argtop-max represents the m samples with the largest area.

[0100] (4) Label correction and update: for samples in the training set are corrected by the expert annotation system, and the candidate samples are labeled with the real label Y j replace the pseudo label M j , and update the training set and the segmentation model

[0101]

[0102] (5) Hybrid supervised training strategy: when the annotation budget is exhausted, combine high-quality labeled data and remaining pseudo labels for joint training to maximize the utilization efficiency of annotation resources.

[0103] S6. Based on the optimized label data, the DSC loss function is used to train the special segmentation model, and the model parameters are updated by the stochastic gradient descent method to improve the segmentation performance.

[0104] Specifically as follows:

[0105]

[0106]

[0107] Supervised loss Real label set based on expert annotation (contains K+mT samples), through the loss function Constrain the consistency of model prediction f θ (x) and the real label Y;

[0108] Semi-supervised loss Use the remaining pseudo label set (contains N-(K+mT) samples), through Combine pseudo label M to provide weak supervision signal.

[0109] Specifically, in the experiment, three medical image data sets are used in the embodiment: HyperKvasir-SEG (contains 1,000 colonoscopy images, with expert verified pixel level polyp segmentation mask, covering different shapes and sizes, divided by 8:1:1 and adjusted to 224×224 pixels), ISIC 2017 (contains 2,750 skin endoscopy images, divided into 2000 training set, 150 validation set and 600 test set, used for skin lesion segmentation) and ISIC 2018 (provides 2,594 training images, 100 validation and 1,000 test images, covering multiple clinical sources to evaluate the lesion boundary division ability in real scene), all images are uniformly adjusted to 224×224 pixels to ensure model compatibility.

[0110] The implementation parameters of the embodiment are configured as follows:

[0111] 1. Progressive training mechanism

[0112] Initial training set configuration 10 expert annotated samples.

[0113] Perform 5 rounds of iterative annotation optimization, with 10 new expert validation samples added each round (cumulative annotation amount 50)

[0114] Phase 6 starts hybrid supervised training, integrating full-quantity pseudo-label data.

[0115] 2. Model optimization strategy

[0116] Single round training period set to 100 epochs.

[0117] Dynamic early stopping mechanism is adopted: when the Dice coefficient does not improve for 15 consecutive epochs, training is terminated.

[0118] Each round retains the optimal model weight (based on the highest validation Dice value).

[0119] 3. Multi-center experimental verification

[0120] On the HyperKvasir-SEG (50+750), ISIC2017 (50+1950), and ISIC2018 (50+2544) three medical image benchmark datasets, the dynamic label fusion strategy (expert annotation + large model inference pseudo label) is adopted, and the traditional method (real label + model prediction pseudo label) is compared. All experiments are repeated under three independent random seeds, and the performance indicators are presented in the form of "mean ± standard deviation", and the optimal and suboptimal results are presented in bold and underlined Annotation.

[0121] In the polyp segmentation task of the HyperKvasir-SEG dataset, as shown in Table 1 and Figure 2 It is shown that the method has a significant advantage under the condition of using only 50 real annotated samples and 850 pseudo labels. Experimental results show that in this dataset with higher segmentation difficulty, the conventional active learning method is limited by the quality of pseudo labels in the semi-supervised stage due to insufficient training samples, while the proposed method generates pseudo labels through a large model and combines area sampling screening strategy to effectively improve model performance. Taking the Transattunet architecture as an example, the proposed method leads the suboptimal method by 22.46%, 25.39%, and 20.69 percentage points in DSC (79.23%), IOU (68.94%), and HD95 (37.2), respectively, verifying the effectiveness of the method.

[0122] Visual analysis is shown in, for example, Figure 2As shown, under the Unet architecture (1st and 2nd rows), the present method (1h) successfully suppresses false positive regions, and does not misjudge normal tissue as polyps as in the comparative methods (such as 1c, 1f, and 1g). In the SegFormer architecture (5h and 6h), the present method has a slight over-segmentation problem, which is due to the global modeling characteristics of the Transformer encoder needing to rely on sufficient training data, while the HyperKvasir-SEG dataset only contains 800 training samples, and the performance of the model is limited due to the limited data size. The experimental results reveal that the pseudo-label strategy enhanced by the large model prior can effectively alleviate the small sample training problem.

[0123] Table 1

[0124]

[0125]

[0126] In the medical image segmentation task of the ISIC2017 dataset, see Table 2 and Figure 3 As shown, the present method exhibits overall advantages in the semi-supervised training phase. In the final training round, the pseudo-labels generated by the segmentation model of all active learning methods can bring performance gains, but the improvement is still significantly lower than that of the pseudo-labels generated by the large model. Experimental data show that the present method achieves the best results in the Dice coefficient (73.87%), IOU (61.49%), and HD95 (28.37) in the U-net architecture, and exhibits better stability - for example, in the U-net architecture, the standard deviation of the DSC index of the present method is reduced by 4.57 percentage points compared to the suboptimal method BALD, verifying the reliability advantage of the large model pseudo-label under different experimental conditions, while the traditional active learning method is prone to uncertainty due to the dependence on initial sample selection.

[0127] The visualization comparison results Figure 3 further reveal the characteristics of the method. For the case of strong color distribution heterogeneity (1a), the present method (column h) can completely identify the light and dark lesion regions, while other methods (such as 1d and 1e in the U-net architecture) can only detect the dark regions. In the case of hair interference (2a), the comparative methods (4d and 4f) under the CompNet architecture have a significant false negative problem, while the present method (4h) successfully reduces the missed detection area, and exhibits stronger anti-interference ability. Experimental results show that conventional methods generally have a problem of missing light lesion identification in complex samples, and the present method effectively alleviates this defect by fusing the prior knowledge of the large model.

[0128] Table 2

[0129]

[0130] In the medical image segmentation task of ISIC2018 dataset, the proposed method shows significant advantages on four mainstream architectures (Transattunet, U-net, SegFormer, CompNet). As shown in Table 3 data, the proposed method leads all comparison methods in three core indicators of DSC (84.87±0.19), IOU (75.70±0.16) and HD95 (19.70±0.33). Taking the Transattunet architecture as an example, its DSC value is improved by nearly 5 percentage points compared with the suboptimal method BADGE (79.98±2.09), and the standard deviation is reduced by about 10 times, which reflects the outstanding stability advantage. It is worth noting that although the real label budget remains 50 samples, the segmentation model obtains better performance due to the 2594 training samples (the largest among the three datasets) of the ISIC2018 dataset and the generated 2544 high-quality pseudo labels, which verifies the efficiency of the proposed method on large-scale datasets.

[0131] Visualization analysis results Figure 4 further demonstrate the effectiveness of the method. In the case where the lesion-background color difference is small (1a), the proposed method (1h, 3h, 5h, 7h) shows accurate segmentation capability: taking the U-net architecture (first row) as an example, the proposed method accurately outlines the lesion contour, while the comparison methods 1e / 1f show over-segmentation, and 1c / 1g produce large-area false positives. In the SegFormer architecture (fifth row), only the proposed method (5h) avoids misjudging dark normal tissue as lesions. For the case where light-colored lesions dominate (4a), the proposed method (4h) under the CompNet architecture (fourth row) successfully identifies most of the lesion area, while the comparison methods 4d / 4e / 4f can only detect small dark lesions.

[0132] Table 3

[0133]

[0134] To explore the impact of the pseudo-label area sampling active learning method on the entire framework of the proposed method, further ablation studies were conducted on HyperKvasir-SEG. The large model pseudo-label and pseudo-label area sampling method were ablated on the U-net model. And test its corresponding Dice coefficient, IOU, HD95 and other indicators.

[0135] The specific implementation steps are as follows: the large model pseudo-label and pseudo-label area method are used as a separate module, under the same conditions, test three cases:

[0136] 1. Only use the pseudo-label area method.

[0137] 2. Only use the large model pseudo-label.

[0138] 3. A combination of two.

[0139] The specific result indicators are shown in Table 4, where "T" represents the number of true label samples, and "P" represents the number of pseudo-label samples.

[0140] As shown in the experimental data in Table 4, when the pseudo-label area sampling strategy is used alone, the quality of the pseudo-labels generated by the model is insufficient, directly restricting the optimization of the model performance. To solve the sample utilization problem in the low labeling budget scenario, the method innovatively introduces a large model pseudo-label generation framework. Under the condition of limited labeling resources, the strategy generates high-confidence pseudo-labels through a large model, and dynamically replaces low-quality pseudo-labels with high-quality manual labels using the area sampling method, forming a synergistic optimization mechanism: the large model framework improves the overall accuracy of the pseudo-labels, and the area sampling strategy filters the label quality, ultimately enabling the segmentation model to achieve a high performance level under limited labeling budget. This two-stage optimization strategy effectively overcomes the limitations of traditional single methods and verifies the advancement of the heterogeneous label source joint optimization scheme.

[0141] Table 4

[0142]

[0143]

[0144] As shown in the ablation experiment results in Table 5 below, the study systematically evaluates the impact of the area proportion of the default target box generated by GLIP on the algorithm performance. The experiment sets four area proportions: 60%, 70%, 80%, and 90%, aiming to balance the pseudo-label positioning efficiency and quality control: larger target boxes are beneficial to the pseudo-label area sampling strategy for positioning the areas to be labeled, but may introduce low-quality labels; smaller target boxes are prone to pseudo-label confusion, reducing the effectiveness of the sampling strategy. The data shows that when the area proportion is 60%-70%, the performance of the first 30 samples in the training round is similar, reflecting the positioning ambiguity of small target boxes in the feedforward stage; when the proportion is 90%, due to the limitation of labeling budget (only 50 true labels), it may not effectively cover the large-size pseudo-label error area, resulting in a decrease in DSC to 76.62%. The optimal performance appears when the proportion is 80%, with peak values of DSC (78.19%), IOU (67.34%), and HD95 (40.87), confirming that this proportion effectively balances label quality control and sampling efficiency. The experiment also reveals that parameter optimization is data-dependent, and it is recommended to adjust the proportion according to the characteristics of the target data set in practical applications.

[0145] Table 5

[0146]

[0147]

[0148] In summary, the application provides a few-shot medical image segmentation method based on multi-model cooperation and active learning. Through the construction of a multi-model cooperation framework of BiomedBERT+OFA→GLIP→MedSAM, the advantages of different models are utilized to extract rich and accurate medical attribute information from medical images, generate lesion target boxes and initial pseudo-label mask images, and optimize the quality of pseudo-labels through pseudo-label area screening and active learning algorithm. Finally, a special segmentation model is trained based on the optimized label data, effectively improving the accuracy and performance of medical image segmentation, reducing the dependence on large amounts of manually annotated data, and improving the segmentation efficiency and practicality.

[0149] Specifically, the method provided by the application innovatively constructs a four-level cooperation framework of "attribute extraction-target positioning-mask generation-active optimization", realizes multi-dimensional analysis of medical semantics through the attribute complementary mechanism of BiomedBERT and OFA model, breaks through the dependence of traditional detection models on text prompts through the cross-modal alignment ability of GLIP, and couples dynamic pseudo-label optimization strategy and active learning mechanism. Compared with existing traditional full-supervised learning scheme, the method can achieve high segmentation performance with only a small amount of budget, and at the same time, the explainable medical attribute description enhances the clinical decision support capability.

[0150] The above-described embodiments only express several embodiments of the application, which are described in detail and specifically, but should not be understood as limiting the scope of the patent of the application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the application, several modifications and improvements can be made, which belong to the protection scope of the application. Therefore, the protection scope of the patent of the application should be subject to the appended claims.

Claims

1. A medical image segmentation method based on multi-model collaboration and active learning, characterized in that: The steps include: S1. By inputting a medical image and a preset prompt template, the BiomedBERT model, pre-trained on biomedical text, is used to extract the common medical attributes of the image, including the location of the lesion; S2. By inputting a medical image and a customized visual question-answering statement, the large language model One ForAll is used to extract the image's unique medical attributes, including lesion color and shape; S3. Input the medical attributes extracted in steps S1 and S2 and the original medical image into the large cross-modal object detection model GLIP to generate a lesion target box. If the target location cannot be determined, a default target box covering 80% of the image area is generated. S4. Input the target box generated in step S3 into the medical image segmentation model MedSAM to generate an initial pseudo-label mask image; During segmentation model training, a pseudo-label area screening strategy is used to optimize the quality of the initial pseudo-labels. The pseudo-label binary images are sorted by area, with larger pseudo-labels being prioritized. Low-quality pseudo-label samples are screened using an active learning algorithm and assigned to professional physicians for annotation to obtain true labels. S6. Based on the optimized labeled data, the dedicated segmentation model is trained using the DSC loss function, and the model parameters are updated using the stochastic gradient descent method to improve the segmentation performance.

2. The medical image segmentation method based on multi-model collaboration and active learning according to claim 1, characterized in that: The BiomedBERT model achieves breakthrough performance in biomedical NLP tasks by being pre-trained entirely from scratch based on biomedical domain text and using a domain-specific vocabulary.

3. The medical image segmentation method based on multi-model collaboration and active learning according to claim 1, characterized in that: The visual question answering method combines customized question statements with medical images and inputs them into a large language model One For All. The question statements are designed for the specific lesion attributes of a single image.

4. The medical image segmentation method based on multi-model collaboration and active learning according to claim 1, characterized in that: The GLIP model reconstructs object detection into a phrase localization task under text prompts, uses an image encoder to extract regional features, a language encoder to generate text features, and aligns image and text features through cross-modal deep fusion. Finally, a regressor is used to predict the target box position that matches the text semantics.

5. The medical image segmentation method based on multi-model collaboration and active learning according to claim 1, characterized in that: The proposed MedSAM model guides its image encoder, cue encoder, and mask decoder architecture through the target box, and uses a cross-attention mechanism to fuse image features and cue information to generate an accurate segmentation mask of the target area.

6. The medical image segmentation method based on multi-model collaboration and active learning according to claim 1, characterized in that: The pseudo-label area screening strategy calculates the pixel area ratio of the pseudo-label binary image, prioritizes the pseudo-labels with lower area ratios, and actively selects samples with higher area ratios for manual labeling.

7. The medical image segmentation method based on multi-model collaboration and active learning according to claim 1, characterized in that: The dedicated segmentation model includes any one of U-net, CompNet, SegFormer or Transattunet.

8. A medical image segmentation device based on multi-model collaboration and active learning, characterized in that: The method comprises a processor and a memory, wherein the memory stores executable instructions, and when the instructions are executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images