A method for lung nodule classification and segmentation based on Gaussian mixture model

Abstract

The application relates to a lung nodule classification and segmentation method based on a Gaussian mixture model and belongs to the technical field of computer vision image processing.The application comprises the following steps: designing a lesion feature extraction module based on a Gaussian mixture model, exhibiting detailed features of lung nodules in the form of a mathematical model, taking parameters thereof as feature representation in an image processing process, improving understanding ability of the model, and guiding a classification and segmentation process; taking a low-rank matrix as a bypass, injecting the low-rank matrix into a self-attention module in a transformer in an image encoder and a mask decoder, and training the model to learn knowledge of lung nodules.By adding the bypass low-rank matrix and training internal parameters, the application designs a lesion feature extraction module based on a Gaussian mixture model, so that the model is suitable for a lung nodule classification and segmentation task, and the effect of lung nodule classification and segmentation is improved, especially for a lesion initial stage.

Description

Technical Field

[0001] This invention belongs to the field of computer vision image processing technology, specifically relating to a method for classifying and segmenting lung nodules based on a Gaussian mixture model. Background Technology

[0002] In medical imaging, lesion classification and segmentation have always been important topics in clinical research, playing a crucial role in clinical applications such as disease diagnosis and treatment planning. Different types of lesions require different treatment strategies, and accurate segmentation helps improve clinical treatment outcomes. Lung cancer is one of the leading causes of cancer-related deaths worldwide, highlighting the urgent need for effective screening and early diagnosis strategies. Lung nodules are often indicators of malignant or benign lesions and play a key role in the early diagnosis of lung cancer. Therefore, there is a growing interest in developing robust classification and segmentation techniques to accurately identify and analyze lung nodules. Lung nodules refer to small, round or oval lesions appearing in lung imaging and are essential for the early detection of lung cancer. Research on lung nodule classification and segmentation techniques is crucial for the fields of computer vision and medical imaging.

[0003] Advances in medical imaging technologies, such as computed tomography (CT), have revolutionized the detection and characterization of lung nodules. High-resolution imaging modalities provide detailed anatomical information, aiding in the visualization and analysis of lung lesions. Since the advent of deep learning, it has demonstrated powerful data feature learning capabilities. Its application in lesion classification and segmentation tasks in medical images has resulted in significant improvements in classification accuracy, target segmentation precision, and efficiency. Currently, image processing technologies have become powerful tools for the automatic analysis and interpretation of medical images. In the context of lung nodule research, these technologies offer the potential to extract relevant features from lung images and accurately depict nodule boundaries through segmentation. Convolutional neural networks were once the most popular method for using deep learning to handle medical image segmentation problems. U-Net and its variants were specifically developed for medical image segmentation, capturing detailed features through skip connections and convolutional operations, thus improving segmentation accuracy. Transformers, with their attention mechanisms, enhance the model's global attention and understanding capabilities. The VisionTransformer (ViT) model, based on Transformer, combines self-attention layers to learn contextual relationships and spatial information, enabling richer semantic segmentation for application in medical image classification and segmentation tasks. However, the application of these methods in medical imaging tasks remains limited due to factors such as the scarcity of medical image annotations. This is mainly reflected in the fact that the accuracy of lesion classification and segmentation, especially early-stage lesions, still has room for improvement. The clinical application of lung nodule classification and segmentation techniques is not limited to early diagnosis but also involves personalized treatment planning and prognosis assessment. Accurate nodule characterization allows clinicians to tailor treatment strategies to individual patient conditions and optimize treatment outcomes. Therefore, it is necessary to further improve the reliability and clinical applicability of classification and segmentation algorithms.

[0004] With the application and development of large-scale models in downstream tasks, medical image classification and segmentation tasks have seen new breakthroughs. SegmentAnythingModel (SAM), currently the most attention-grabbing large-scale computer vision model, employs the ViT architecture and, after extensive training, can classify and segment any image, exhibiting high classification and segmentation accuracy and wide applicability. SAM introduces prompts to guide the model's classification and segmentation process. However, due to its lack of medical image training, it cannot be directly applied to medical image classification and segmentation tasks. Because retraining is too costly, and the classification and segmentation process requires more semantic information, SAM, despite its powerful image segmentation capabilities, cannot be directly applied to medical image classification and segmentation tasks.

[0005] Research into lung nodule classification and segmentation techniques holds immense potential to improve the early detection and management of lung cancer. By further exploring the clinical applications of medical imaging technologies and deep learning, more accurate and intelligent solutions can be developed for analyzing lung images and identifying nodule abnormalities. Summary of the Invention

[0006] Based on the development history and current challenges of medical image classification and segmentation research, this invention aims to propose a SAM-based method for lung nodule classification and segmentation, improving classification and segmentation performance to meet clinical needs and serve early cancer diagnosis. To achieve this goal, this invention uses SAM as the base network, trains the model by incorporating a low-rank side-channel matrix, designs a feature extraction module based on a Gaussian mixture model, and integrates it into the image classification and segmentation process. This enriches semantic information and uses more accurate and reliable lesion features to guide the model's classification and segmentation process, significantly improving the classification and segmentation performance of lung nodule images.

[0007] The present invention provides a medical image classification and segmentation method based on a Gaussian mixture model, comprising the following steps:

[0008] 1) Obtaining lung nodule datasets: These are obtained through public datasets, academic research papers, medical imaging institutions, research centers, and competitions and challenges.

[0009] 2) Extract and classify lung nodules, including the following:

[0010] 2.1 Extract image annotations from the lung nodule dataset, including information on the location and malignancy of the lung nodules;

[0011] 2.2 According to the degree of malignancy, pulmonary nodules are divided into 5 grades, from 1 to 5, which represent the possibility of cancer from low to high. Grades 1 and 2 are benign, grade 3 is between benign and malignant, and grades 4 and 5 are malignant.

[0012] 3) Construct a feature extraction module based on Gaussian mixture model to extract lesion features, including the following steps:

[0013] 3.1 Establish the joint density function of the multidimensional Gaussian distribution:

[0014]

[0015] Where: d is the dimension of the variable; μ is the mean of each variable; ∑ is the covariance matrix, which for a three-dimensional Gaussian distribution is:

[0016]

[0017] The essence of a Gaussian mixture model is a combination of several Gaussian distributions. Theoretically, by assigning appropriate weights to each Gaussian distribution, any distribution can be fitted, and its probability density function is:

[0018]

[0019] Where: p(x|k)=f(X) is the probability density function of the k-th Gaussian model; p(k) is the weight of the k-th Gaussian model, called the prior probability of the k-th model, and satisfies

[0020] For the Gaussian mixture model with parameters π, u, and ∑, the above equation can be rewritten as:

[0021]

[0022] 3.2 Estimate π, u, and ∑ using the Expectation-Maximization Algorithm

[0023] The first step is to calculate the rough values ​​of the three parameters. The second step is to use the values ​​from the first step to maximize the likelihood function. The specific algorithm steps are as follows:

[0024] 3.2.1 Define the number of components K, and set initial values ​​for π, u, and Σ for each component k. Then, iteratively calculate according to the following formula until the parameters converge, based on the current π. k u k , ∑ k Calculate the posterior probability:

[0025]

[0026] 3.2.2 Calculate the new π k u k , ∑ k

[0027]

[0028] in: N represents the number of points; γ(z) nk ) represents point n(x) n The posterior probability of belonging to cluster k; N k This represents the number of points in the k-th cluster; u k This represents the weighted average of all points;

[0029] Thus, the feature extraction module obtains the feature parameters π, u, ∑ for different types of lesions;

[0030] 4) Embed the extracted lesion features into the cue encoder.

[0031] In the prompt encoder, the lesion features represented by the feature extraction module are fed to the prompt encoder, and the lesion features are embedded as a set of feature vectors as input;

[0032] 5) SAM is selected as the base model, consisting of an image encoder, a cue encoder, and a mask decoder. In the transformers of the image encoder and mask decoder, a self-attention module is configured with a low-rank side-channel matrix for efficient training of the model's understanding of lung nodule images. In the image encoder, a visual transformer (ViT-H / 16) pre-trained with MAE is used, and the image is segmented into many blocks for position encoding. In the image encoder and mask decoder, each transformer block consists of a multi-head attention block and an MLP block. After each block, layer normalization is performed, followed by residual connections. Setting a side-channel matrix decomposes the weights into a low-rank matrix, thereby greatly reducing the number of trainable parameters. Model training includes the following steps:

[0033] 5.1 Encoder Training Hints

[0034] A set of tokens representing lesion features, output by the feature extraction module, is embedded into the cue encoder for cue training, and the model's mask predictions are compared with the ground truth. During training, no cue clicks are required; instead, the tokens representing lesion features guide the model to segment lesions. Different medical lesions vary greatly in shape and texture, and are especially difficult to identify visually in their early stages, significantly reducing human efficiency. The lesion feature extraction module extracts feature parameters from various lesions and embeds them into the cue encoder. These lesion feature parameters are used to supervise the model, significantly improving its performance in medical image classification and segmentation tasks.

[0035] 5.2 Training of Low-Rank Bypass Parameters

[0036] First, the transformer layers are frozen to keep the q-weights fixed. Then, a bypass is added, which includes linear layers A and B, to achieve low-order training and reduce the number of training parameters. After the trainable rank factorization matrix is ​​injected into the Transformer architecture, the low-rank matrix is ​​trained without changing the original weights to obtain the internal parameters of the trainable low-rank matrix of the bypass. This achieves the goal of training fewer parameters while ensuring segmentation performance, thus significantly reducing the computation of the model.

[0037] The process of updating layer W is described as follows:

[0038] W = W0 + △W = W0 + BA;

[0039] The Transformer's self-attention module has four weight matrices: Wq, Wk, Wv, and Wo. The MLP module has two weight matrices: Wi and Wj. Adjusting the attention weights freezes the MLP module. The multi-head attention processing strategy is as follows:

[0040]

[0041] Q = W q +B q A q ;

[0042] K = W k +B K A k ;

[0043] V = W v +B v A v ;

[0044] Among them: W q W k and W v It is the frozen projection layer of SAM; A q B q A k B k A v and B v These are trainable parameters;

[0045] 6) Combine the output of the image encoder and the output of the cue encoder from step 5) and input them into the decoder.

[0046] The mask decoder combines the image embedding and a set of lesion feature cue embeddings and maps them to the output mask. Each transformer module consists of a multi-head attention module and an MLP module. After each module, layer normalization is performed. The multi-head attention module consists of self-attention of tokens, cross-attention from tokens to image embeddings, and cross-attention from image embeddings to tokens, and is connected by residual connections. Lesion features are used as cues to update the image embedding. The next decoder layer obtains the updated tags and image embeddings from the previous layer. Before the decoder runs, the learned output tokens and the lesion-extracted cue tokens embeddings are inserted into the cue embedding set.

[0047] 7) After passing through cross-attention and multi-sensor layers, output the classification and segmentation results.

[0048] After the decoder runs, the updated image is upsampled through a convolutional layer, the tokens are again used in the image embedding and passed to the MLP. Finally, the model predicts a mask with a spatial dot product.

[0049] 8) Calculate the loss function and update the parameters of each layer in the network using gradient descent. After training, the final network model is obtained.

[0050] This invention adapts the SAM model for lung nodule classification and segmentation tasks by implementing bypass training. A low-rank strategy is used to efficiently and cost-effectively adjust the model's suitability for lung nodule segmentation. A lesion feature extraction module is integrated into the model, mathematically guiding the model to accurately learn lesion features to improve lung nodule classification and segmentation performance. This enhances sensitivity to early-stage lesions, improves interpretability, and allows the model to effectively improve lung nodule classification and segmentation under the guidance of lesion features. Attached Figure Description

[0051] Figure 1 A flowchart illustrating methods for classifying and segmenting pulmonary nodules;

[0052] Figure 2 This is a schematic flowchart of the feature extraction module;

[0053] Figure 3 This is a schematic diagram of an image encoder module;

[0054] Figure 4 This is a schematic diagram of a mask decoder module;

[0055] Figure 5 A schematic diagram of a low-rank bypass design in a self-attention module;

[0056] Figure 6 Images of lung nodules randomly selected from the dataset (scale bar: 1:7);

[0057] Figure 7 Example image of lung nodule classification and segmentation results (scale bar: 1:7). Detailed Implementation

[0058] The core content of this invention is as follows: It provides a method for classifying and segmenting lung nodules. This method uses a low-rank matrix fine-tuning strategy and p-tuning to overcome the shortcomings of basic models that are unsuitable for medical image classification and segmentation tasks due to a lack of medical image training data, making it applicable to lung nodule classification and segmentation. A lesion feature extraction module is proposed, using the parameters of a Gaussian mixture model to represent all features of different lesion categories. This module is integrated into the model to improve its lack of training on medical images, making the model more focused on lesion features and improving classification accuracy and segmentation precision. This invention achieves state-of-the-art classification and segmentation results on the LIDC dataset. Notably, it is more sensitive to early-stage lesions, focusing on the classification and segmentation of small lesions, and outperforms current popular methods, which is helpful for the clinical diagnosis and early prevention of cancer.

[0059] The present invention will now be described in conjunction with the accompanying drawings.

[0060] The present invention provides a medical image classification and segmentation method based on a Gaussian mixture model, comprising the following steps:

[0061] 1) Obtaining lung nodule datasets: These are obtained through public datasets, academic research papers, medical imaging institutions, research centers, and competitions and challenges.

[0062] 2) Extract and classify lung nodules, including the following:

[0063] 2.1 Extract image annotations from the lung nodule dataset, including information on the location and malignancy of the lung nodules;

[0064] 2.2 According to the degree of malignancy, pulmonary nodules are divided into 5 grades, from 1 to 5, which represent the possibility of cancer from low to high. Grades 1 and 2 are benign, grade 3 is between benign and malignant, and grades 4 and 5 are malignant.

[0065] 3) A feature extraction module is constructed based on a Gaussian mixture model to extract lesion features, such as... Figure 2 As shown, it includes the following steps:

[0066] 3.1 Establish the joint density function of the multidimensional Gaussian distribution:

[0067]

[0068] Where: d is the dimension of the variable; μ is the mean of each variable; ∑ is the covariance matrix, which for a three-dimensional Gaussian distribution is:

[0069]

[0070] The essence of a Gaussian mixture model is a combination of several Gaussian distributions. Theoretically, by assigning appropriate weights to each Gaussian distribution, any distribution can be fitted, and its probability density function is:

[0071]

[0072] Where: p(x|k)=f(X) is the probability density function of the k-th Gaussian model; p(k) is the weight of the k-th Gaussian model, called the prior probability of the k-th model, and satisfies

[0073] For the Gaussian mixture model with parameters π, u, and ∑, the above equation can be rewritten as:

[0074]

[0075] 3.2 Estimate π, u, and ∑ using the Expectation-Maximization Algorithm

[0076] The first step is to calculate the rough values ​​of the three parameters. The second step is to use the values ​​from the first step to maximize the likelihood function. The specific algorithm steps are as follows:

[0077] 3.2.1 Define the number of components K, and set initial values ​​for π, u, and ∑ for each component k. Then, iteratively calculate according to the following formula until the parameters converge, based on the current π. k u k , ∑ k Calculate the posterior probability:

[0078]

[0079] 3.2.2 Calculate the new π k u k , ∑ k

[0080]

[0081] in: N represents the number of points; γ(z) nk ) represents point n(x) n The posterior probability of belonging to cluster k; N k This represents the number of points in the k-th cluster; u k This represents the weighted average of all points;

[0082] Thus, the feature extraction module obtains the feature parameters π, u, ∑ for different types of lesions;

[0083] 4) Embed the extracted lesion features into the cue encoder.

[0084] In the prompt encoder, the lesion features represented by the feature extraction module are fed to the prompt encoder, and the lesion features are embedded as a set of feature vectors as input;

[0085] 5) SAM is selected as the basic model, consisting of an image encoder, a cue encoder, and a mask decoder, such as... Figure 1 , Figure 3 and Figure 4 In the transformers of the image encoder and mask decoder, the self-attention module sets up a bypass low-rank matrix for efficient training of the model's understanding of lung nodule images. In the image encoder, a visual transformer (ViT-H / 16) pre-trained with MAE is used, and the image is segmented into many blocks for positional encoding. In both the image encoder and mask decoder, each transformer block consists of a multi-head attention block and an MLP block. After each block, layer normalization is performed, followed by residual connections. Setting up bypasses decomposes the weights into a low-rank matrix, thereby greatly reducing the number of trainable parameters. Model training includes the following steps:

[0086] 5.1 Encoder Training Hints

[0087] A set of tokens representing lesion features, output by the feature extraction module, is embedded into the cue encoder for cue training, and the model's mask predictions are compared with the ground truth. During training, no cue clicks are required; instead, the tokens representing lesion features guide the model to segment lesions. Different medical lesions vary greatly in shape and texture, and are especially difficult to identify visually in their early stages, significantly reducing human efficiency. The lesion feature extraction module extracts feature parameters from various lesions and embeds them into the cue encoder. These lesion feature parameters are used to supervise the model, significantly improving its performance in medical image classification and segmentation tasks.

[0088] 5.2 Training of Low-Rank Bypass Parameters

[0089] First, the transformer layers are frozen to keep the q-weights fixed. Then, a bypass is added, containing linear layers A and B, to achieve low-order training and reduce the number of training parameters. After injecting a trainable rank factorization matrix into the Transformer architecture, the low-rank matrix is ​​trained without changing the original weights to obtain the intrinsic parameters of the bypass-trainable low-rank matrix, such as... Figure 5 This achieves the goal of training fewer parameters and significantly reducing model computation while ensuring segmentation performance;

[0090] The process of updating layer W is described as follows:

[0091] W = W0 + △W = W0 + BA;

[0092] The Transformer's self-attention module has four weight matrices: Wq, Wk, Wv, and Wo. The MLP module has two weight matrices: Wi and Wj. Adjusting the attention weights freezes the MLP module. The multi-head attention processing strategy is as follows:

[0093]

[0094] Q = W q +B q A q ;

[0095] K = W k +B K A k ;

[0096] V = W v +B v A v ;

[0097] Among them: W q W k and W v It is the frozen projection layer of SAM; A q B qA k B k A v and B v These are trainable parameters;

[0098] 6) Combine the output of the image encoder and the output of the cue encoder from step 5) and input them into the decoder.

[0099] The mask decoder combines the image embedding and a set of lesion feature cue embeddings and maps them to the output mask. Each transformer module consists of a multi-head attention module and an MLP module. After each module, layer normalization is performed. The multi-head attention module consists of self-attention of tokens, cross-attention from tokens to image embeddings, and cross-attention from image embeddings to tokens, and is connected by residual connections. Lesion features are used as cues to update the image embedding. The next decoder layer obtains the updated tags and image embeddings from the previous layer. Before the decoder runs, the learned output tokens and the lesion-extracted cue tokens embeddings are inserted into the cue embedding set.

[0100] 7) After passing through cross-attention and multi-sensor layers, output the classification and segmentation results.

[0101] After the decoder runs, the updated image is upsampled through a convolutional layer, the tokens are again used in the image embedding and passed to the MLP. Finally, the model predicts a mask with a spatial dot product.

[0102] 8) Calculate the loss function and update the parameters of each layer in the network using gradient descent. After training, the final network model is obtained.

[0103] 9. Working conditions

[0104] This experiment is based on the PyTorch framework and trained using an RTX4090 GPU with two 24GB memory modules. The system is Windows 11, and the chip is an Intel Core i7-12700. The dataset is first classified using a feature extraction module. The dataset is trained for 200 epochs. The environment is CUDA 11.8, the regularizer is SGD with a momentum of 0.9, and the learning rate is 2e-4. During the cue-based fine-tuning training period, the regularizer is AdamW with a learning rate of 2e-4. During the low-rank matrix training period, the regularizer is AdamW with a learning rate of 1e-4.

[0105] 10. Content and Results Analysis

[0106] Table 1 shows the classification results on the LIDC dataset.

[0107]

[0108] Table 1 shows the classification results of some methods on the LIDC dataset, where lung nodules are classified into grades 1 to 5 according to their malignancy. Figure 6 Each degree exhibits different malignant signs. Analysis of the evaluation metrics of the experimental results shows that the precision and recall of this invention are 0.701 and 0.697, respectively, achieving the best performance compared to other methods. Considering both precision and recall, this invention leads other methods by 0.5 to 2.4 percentage points. These results demonstrate that, on the one hand, this invention makes the basic model effective in the classification and segmentation of pulmonary nodules; on the other hand, it shows that the proposed Gaussian mixture model-based lesion feature extraction module plays a guiding role in the classification process, resulting in more accurate results.

[0109] Another phenomenon was discovered during the research: various neural networks are not very good at classifying and segmenting lesions with a small degree of deterioration in lung nodules, but the present invention has a more significant classification ability for lesions with a small degree of deterioration. Therefore, further research and analysis were conducted on lesions with a deterioration degree of less than or equal to 3 in the LIDC dataset. As shown in Table 1, as the degree of deterioration of the experimental subjects decreases, the classification performance advantage of the present invention increases compared with other methods.

[0110] Table 2 shows the segmentation results on the LIDC dataset.

[0111]

[0112] Table 2 shows the segmentation performance evaluation of this invention and other methods on the LIDC dataset. In terms of accuracy, the IoU and Dice of this invention are 0.826 and 0.905 respectively, slightly higher than or roughly equal to other methods, demonstrating a certain advantage in edge segmentation. On SEN, although it did not achieve the best result, the difference from the best result is not significant. Similar to the classification experiment, it shows more significant segmentation performance for lesions with less severe deterioration. The visualization results are as follows: Figure 7 The lung nodules were accurately classified and the segmentation boundaries were clear. These results demonstrate that the feature extraction module based on the Gaussian mixture model can more finely characterize lesion features and reflect local edge information, which is also effective for the segmentation of subtle lesions.

[0113] Based on the comparative experimental results of classification and segmentation, compared with some currently popular deep learning methods, this invention balances the performance of accurate classification and precise segmentation, demonstrating competitiveness. It is more sensitive to early lesions and can classify and segment lesions more accurately in the early stages of deterioration, which will provide important practical ideas for the prevention and early diagnosis of lung cancer. The proposed lesion feature extraction module plays an important role in the accurate classification and segmentation of lung nodules, thereby improving the classification and segmentation effect.

[0114] This invention proposes a novel method that integrates lung nodule classification and segmentation. The proposed model employs a low-rank matrix fine-tuning strategy and p-tuning method for lung nodule classification and segmentation, overcoming the limitation of basic models being unsuitable for medical image classification and segmentation tasks due to a lack of medical image training data. A lesion feature extraction module is proposed, using Gaussian mixture model parameters to represent all features of different lesion categories. This module is integrated into the model to address its lack of training on medical images, allowing the model to focus more on lesion features and improving classification accuracy and segmentation precision. This invention achieves state-of-the-art classification and segmentation results on the LIDC dataset. Notably, it is more sensitive to early-stage lesions, focusing on the classification and segmentation of small lesions, and outperforms some currently popular deep learning methods. This is beneficial for the clinical diagnosis and early prevention of cancer.

Claims

1. A medical image classification and segmentation method based on Gaussian mixture model, characterized in that... Includes the following steps: 1) Obtaining lung nodule datasets: These are obtained through public datasets, academic research papers, medical imaging institutions, research centers, and competitions and challenges. 2) Extract and classify lung nodules, including the following: S2.1 Extract image annotations from the lung nodule dataset, including information on the location and malignancy of the lung nodules; S2.2 Pulmonary nodules are classified into 5 grades according to their malignancy. Grades 1 to 5 represent the probability of cancer from low to high. Grades 1 and 2 are benign, grade 3 is between benign and malignant, and grades 4 and 5 are malignant. 3) Construct a feature extraction module based on Gaussian mixture model to extract lesion features, including the following steps: S3.1 Establish the joint density function of the multidimensional Gaussian distribution: in: It is the dimension of the variable; It is the mean of all variables; It is the covariance matrix. For a three-dimensional Gaussian distribution, we have: The probability density function of the Gaussian mixture model is: in: It is the first The probability density function of a Gaussian model; It is the first The weights of the Gaussian model are called the weights of the nth Gaussian model. The prior probabilities of each model, and satisfying ; The parameters of the Gaussian mixture model are ,μ, The above formula can be rewritten as: S3.2 uses the expectation-maximization algorithm to estimate ,μ, The first step is to calculate the rough values ​​of the three parameters. The second step is to use the values ​​from the first step to maximize the likelihood function. The specific algorithm steps are as follows: S3.2.1 Define the number of components and for each component set up ,μ, The initial value is then calculated iteratively according to the following formula until the parameters converge, based on the current... μ k , Calculate the posterior probability: S3.2.2 Calculate the new μ k , : in: , Indicates the number of points; Point Belongs to clustering The posterior probability; Indicates the first The number of points in each cluster; μ k This represents the weighted average of all points; Thus, the feature extraction module obtains the feature parameters of different types of lesions. ,μ, ; 4) Embed the extracted lesion features into the cue encoder. In the prompt encoder, the lesion features represented by the feature extraction module are fed to the prompt encoder, and the lesion features are embedded as a set of feature vectors as input; 5) SAM is selected as the base model, consisting of an image encoder, a cue encoder, and a mask decoder. In the transformers of the image encoder and mask decoder, a side-channel low-rank matrix is ​​set in the self-attention module. In the image encoder, a visual transformer pre-trained with MAE is used, and the image is segmented into many blocks for positional encoding. In both the image encoder and mask decoder, each transformer block consists of a multi-head attention block and an MLP block. Each block is followed by layer normalization and residual connections. Model training includes the following steps: S5.1 prompt encoder training A set of tokens representing lesion features output by the feature extraction module is embedded into the cue encoder for cue training, and the model's mask predictions are compared with the true values. During training, no cue clicks are required; instead, the tokens representing lesion features guide the model to segment lesions. The lesion feature extraction module extracts feature parameters from various types of lesions and embeds them into the cue encoder. The lesion feature parameters are used to supervise the model, significantly improving the model's performance in medical image classification and segmentation tasks. S5.2 Low-Rank Bypass Parameter Training First, the transformer layers are frozen to keep the q-weights fixed. Then, a bypass is added, which includes linear layers A and B, to achieve low-order training and reduce the number of training parameters. After the trainable rank factorization matrix is ​​injected into the Transformer architecture, the low-rank matrix is ​​trained without changing the original weights to obtain the internal parameters of the trainable low-rank matrix of the bypass. This achieves the goal of training fewer parameters while ensuring segmentation performance, thus significantly reducing the computation of the model. Update layer The processing procedure is described as follows: ； The Transformer's self-attention module has four weight matrices: Wq, Wk, Wv, and Wo. The MLP module has Wi and Wj. Two weight matrices are used to adjust the attention weights, and the MLP module is frozen. The multi-head attention processing strategy is as follows: ； ； ； ； in: , and It is the frozen projection layer of SAM; , , , , and These are trainable parameters; 6) Combine the output of the image encoder and the output of the cue encoder from step 5) and input them into the decoder. The mask decoder combines the image embedding and a set of lesion feature cue embeddings and maps them to the output mask. Each transformer module consists of a multi-head attention module and an MLP module. After each module, layer normalization is performed. The multi-head attention module consists of self-attention of tokens, cross-attention from tokens to image embeddings, and cross-attention from image embeddings to tokens, and is connected by residual connections. Lesion features are used as cues to update the image embedding. The next decoder layer obtains the updated tokens and image embeddings from the previous layer. Before the decoder runs, the learned output tokens and the cue tokens extracted from the lesions are inserted into the cue embedding set. 7) After cross-attention and multi-sensor layers, output classification and segmentation results. After the decoder runs, the updated image is upsampled through a convolutional layer, the tokens are again used in the image embedding and passed to the MLP. Finally, the model predicts a mask with a spatial dot product. 8) Calculate the loss function and update the parameters of each layer in the network using gradient descent. After training, the final network model is obtained.

Images