Fine-grained classification method for medical images guided by large segmentation model

Abstract

The present invention discloses a fine-grained classification method for medical images guided by a large segmentation model, and belongs to the field of medical imaging technology. First, a coarse mask is generated as a prompt for a SAM through a segmentation network, and the coarse mask is refined to obtain a fine mask, and then the fine mask and the maximum connected graph are used to generate a box prompt. SAM can receive various types of input prompts, such as points, boxes, etc., which can help the model to more accurately identify and segment objects in the image. The image is segmented and analyzed using SAM to generate a segmented mask map. The segmented mask is then input into a PaNet classification network to perform fine-grained classification on the image. The present invention is superior to traditional fine-grained classification methods for medical images in terms of object positioning and segmentation, generalization ability, partial discovery performance, and classification accuracy, so that the present invention has broad application prospects in the field of medical image analysis.

Description

Technical Field

[0001] The invention relates to a medical image fine-grained classification method based on a large segmentation model guidance, belonging to the technical field of medical images. Background Art

[0002] Medical images can bring more help to doctors' diagnostic decisions, but interpreting medical images will consume a lot of energy. First, it is necessary to find images that can support diagnostic decisions as accurately as possible within a limited time; second, the difficulty of interpreting medical images is relatively large. For example, the difference between benign lesions and malignant lesions may be in a tiny part, or only a slight change in color. This requires doctors to spend a lot of energy to identify every detail in the image; in addition, the doctor's personal experience and level also have a great impact on the interpretation results.

[0003] Using computer vision to solve image classification problems has been widely used in many scenarios, such as plant and animal species identification, traffic scene analysis, and face recognition, which is common in daily life. However, the difficulty of classification varies depending on the subject. In common natural image classification problems, it is easy to distinguish between cars and airplanes, but it is more difficult to distinguish between different models of the same car. This is a fine-grained classification problem, which aims to distinguish objects belonging to different subcategories under the same general category. Compared with general image classification, the characteristic of fine-grained classification problems is that the granularity of the category to which the image belongs is finer.

[0004] The characteristic of medical images is that they have a single imaging object, which is the target organ, and the slight changes in it reflect different diseases. This characteristic is completely consistent with the current fine-grained image classification problem in the field of computer vision. However, the current research on fine-grained classification problems in the computer field focuses on natural images, which is significantly different from medical images.

[0005] Therefore, how to apply the ideas and methods of fine-grained classification to medical image classification is a current research hotspot. By analyzing and summarizing the existing fine-grained classification methods for medical images, the existing methods have the following shortcomings: (1) Poor saliency localization ability, making it difficult to identify subtle but clinically significant areas in lesions. (2) Poor performance interpretation ability, difficulty in quantifying the basis for model prediction, and lack of interpretability. (3) Limited classification granularity, making it difficult to identify subtle tissue differences. (4) Background noise in medical images can also have a serious impact on fine-grained classification, resulting in low accuracy and precision in medical image classification. Summary of the invention

[0006] The purpose of the present invention is to propose a fine-grained classification method for medical images based on the guidance of a large segmentation model, so as to further improve the accuracy of fine-grained classification of medical images and provide methodological and technical support for pathological diagnosis of medical images.

[0007] In order to achieve the above-mentioned invention object, the specific technical scheme adopted by the present invention is as follows:

[0008] A medical image fine-grained classification method based on segmentation large model guidance includes the following steps:

[0009] S1: Collect a medical image dataset, which first generates a coarse mask through a segmentation network;

[0010] S2: First, the coarse mask is refined to obtain a finer mask, and then the fine mask is used to generate a bounding box through the maximum connected graph method;

[0011] S3: taking the refined mask, the box prompt, and the medical image dataset as input to the segmentation large model SAM for segmentation processing;

[0012] S4: The segmentation processing result obtained in S3 is input into the PaNet classification network as a guide item for fine-grained classification; the PaNet classification network uses image-level class labels and A negative square Euclidean distance function is applied between the two networks, and a partial loss method is proposed, that is, a partial feature vector is discarded at one time to prevent a single part from dominating the classification and improve the robustness of the network; the PaNet classification network also adopts a partial feature vector modulation method;

[0013] S5: training the PaNet classification network, and then using the trained network to perform medical image classification and recognition, and outputting the classification results.

[0014] Furthermore, in S1, a segmentation network (such as U-Net or Mask R-CNN) is used to generate a coarse mask, which is regarded as a binary image, in which the area with a pixel value of 1 represents the location of the object, and the area with a pixel value of 0 represents the background.

[0015] Further, the S2 includes:

[0016] S2-1: The coarse mask generates a fine mask by applying a series of convolution and nonlinear operations on the coarse mask through mask refinement; the objective function of mask refinement is defined as the IoU loss function:

[0017] ;

[0018] in, is the predicted label of pixel i in the fine mask, is the label value corresponding to the rough mask, The probability of predicting the correctness;

[0019] S2-2: The maximum connected graph algorithm is then used to obtain all connected areas in the fine mask, and the area with the largest number of pixels is selected as the maximum connected graph;

[0020] S2-3: Finally, the bounding box is generated by calculating the minimum and maximum x and y coordinates of the maximum connected graph.

[0021] Furthermore, in S3, the segmentation model SAM can receive various types of input prompts, including point prompts and box prompts; for box prompts, a binary image is defined ,in Then, it means that (x, y) is in the prompt box, otherwise, it means that the point is not in the prompt box; then, these prompt images are provided to the SAM model as additional input channels. During the SAM model training process, an objective function is defined , which encourages the SAM model to generate high-confidence masks within the hint region; the objective function is defined as:

[0022] ;

[0023] in, is the predicted probability of the SAM model generating an object mask at position (x, y);

[0024] The refined mask is input into the image encoder of the SAM model to generate high-level semantic features, and then input into the mask decoder of the SAM model to generate a more refined segmentation result.

[0025] Furthermore, the S4 is specifically:

[0026] S4-1: Use the SAM model to segment the image and obtain a binary mask, where the area with a pixel value of 1 represents the object and the area with a pixel value of 0 represents the background;

[0027] S4-2: The binary mask is used to guide the PaNet classification network, and the new image to be classified is input into the PaNet classification network to perform fine-grained classification on the input image:

[0028] ;

[0029] Where x is the input image and y is the output of PaNet. is the forward propagation function of PaNet;

[0030] The objective function of the PaNet classification network is defined as the cross entropy loss function:

[0031] ;

[0032] in, is the true label of pixel i, is the predicted probability of pixel i;

[0033] S4-3: The PaNet classification network uses image-level class labels and prior knowledge: PaNet uses image-level class labels to discover parts of objects;

[0034] By using the feature vector Applying the negative squared Euclidean distance function between them, the attention is calculated as follows:

[0035] ;

[0036] in and are the extracted features and the prototypes of the K parts respectively;

[0037] In addition, the attention map A generated by the segmentation result of the SAM model is used as the regularization term for the PaNet classification network to generate the attention map B. The definition formula is as follows:

[0038] ;

[0039] Use the element Aij in the attention map A to update the corresponding element in the attention map B ; It is a learning rate parameter, which is used to control the step size of the update, thereby constraining the attention mapping;

[0040] Each attention map is then used to compute its corresponding partial vector by calculating the weighted average of the feature vectors in Z using the attention values :

[0041] ;

[0042] S4-4: Partial loss: In order to prevent a single part from dominating the classification, PaNet proposed a partial loss method, that is, discarding part of the feature vector at one time;

[0043] S4-5: Partial feature vector modulation: In addition, PaNet also proposed a method for partial feature vector modulation:

[0044] ;

[0045] In this formula, is the original eigenvector of the kth part, and is the corresponding weight (or linear classifier) ​​and modulation vector, is the modulated eigenvector.

[0046] Furthermore, the loss function of the PaNet classification network is specifically:

[0047] ;

[0048] Among them, the equal variance loss function is:

[0049] ;

[0050] There is a loss function defined as follows:

[0051] ;

[0052] Where avgpool() is a 2D average pooling with a kernel size of 1;

[0053] Attention map The concentration loss function is:

[0054] ;

[0055] in and denote the vertical and horizontal spatial variances, respectively;

[0056] The orthogonality loss function for the partial vectors is calculated by applying the cosine distance between all pairs:

[0057] ;

[0058] in, and is the eigenvector;

[0059] KL divergence function to constrain the function:

[0060] ;

[0061] in, and They are the normalized probability distributions of the feature maps obtained by the classification network and SAM, respectively.

[0062] The present invention is a method for fine-grained classification of medical images guided by a segmentation macro model (SAM). First, a rough mask is generated through a segmentation network as a prompt for SAM, and is used to generate richer prompts. Then, this rough mask and the maximum connected graph are used to generate a bounding box. The mask generated at the same time is optimized through mask refinement, and after passing through an image encoder, high-level semantic features are obtained, which are input into the mask decoder of SAM to guide more accurate segmentation results. SAM can receive various types of input prompts, such as points, boxes, etc., which can help the model more accurately identify and segment objects in the image. Next, SAM is used to segment and analyze the image to generate a segmented mask map.

[0063] Then, the segmented mask is input into the PaNet classification network to perform fine-grained classification of the image. The mask constrains the network to focus on the part of the image, reduces the background noise of the image, and pays more attention to the foreground area, that is, more attention to the discriminative area, so that the network can find the part that should be paid attention to. The PaNet network proposes a new part discovery method that uses image-level class labels to discover parts of objects. At the same time, the concept of part loss is proposed, that is, part of the feature vector is discarded at one time to prevent a single part from dominating the classification. In addition, a method of partial feature vector modulation is proposed so that the information of each part is different from other parts from the perspective of the classifier. Finally, this method provides better part discovery performance than previous methods without any additional hyperparameter adjustment, while not reducing the classification performance.

[0064] Advantages and beneficial effects of the present invention:

[0065] Traditional classification methods usually rely on manual annotation or simple threshold segmentation to locate and segment objects, and these methods may not accurately capture all the details of the object. The present invention can generate a more accurate mask by using a segmentation model (SAM) and precise prompts, thereby more accurately locating and segmenting objects. Traditional methods are usually unable to discover parts of objects, or require additional annotation information to discover parts. The PaNet network in the present invention can use image-level class labels and masks generated by the segmentation model SAM to discover parts of objects, which makes the present invention surpass traditional methods in part discovery performance. By reducing the interference of background noise in the original image, more attention is paid to the foreground area. The final classification accuracy can be improved. Traditional methods are usually interfered by background noise and do not have strong discriminability. This leads to reduced classification accuracy.

[0066] In general, the present invention is superior to traditional medical image fine-grained classification methods in terms of object localization and segmentation, generalization ability, part discovery performance and classification accuracy, which makes the present invention have broad application prospects in the field of medical image analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is the overall flow chart of the present invention.

[0068] Figure 2 It is the overall framework diagram of the present invention.

[0069] Figure 3 This is the processing flow of the SAM model in the present invention Figure 1 .

[0070] Figure 4 This is the processing flow of the SAM model in the present invention Figure 2 .

[0071] Figure 5 It is a flow chart of the PaNet classification network in the present invention.

[0072] The following will be combined with the attached Figure 1-5 The present invention is further described with specific embodiments.

[0073] Embodiment 1:

[0074] A fine-grained classification method for medical images based on segmentation large model (SAM) is proposed. The overall process is as follows Figure 1 As shown, the following steps are included:

[0075] S1: First, a segmentation network (such as U-Net or Mask R-CNN) is used to generate a coarse mask, which is regarded as a binary image, where the area with pixel value 1 represents the location of the object, and the area with pixel value 0 represents the background. The coarse mask description is used as a hint for the subsequent SAM model.

[0076] Another purpose of using a segmentation network to generate a coarse mask is to generate another hint through the coarse mask. This process can be achieved through the mask refinement technique, which generates a fine mask by applying a series of convolutions and nonlinear operations on the coarse mask.

[0077] The objective function of the mask refinement is defined as the IoU loss function:

[0078] ;

[0079] in, is the predicted label of pixel i in the refined mask.

[0080] S2: The fine mask is regarded as a binary image, where the area with pixel value 1 represents the location of the object, and the area with pixel value 0 represents the background. The maximum connected graph algorithm is then used to generate the bounding box from the coarse mask. The basic idea of ​​the maximum connected graph algorithm is to find all connected areas in the mask and select the area with the largest number of pixels as the maximum connected graph. Then, the bounding box is generated by calculating the minimum and maximum x and y coordinates of this maximum connected graph.

[0081] A better segmentation network can generate a better mask, because the goal of the segmentation network is to predict as accurately as possible whether each pixel belongs to an object. Therefore, if the performance of the segmentation network is good, the generated mask will be closer to the actual object shape, and after the mask refinement process, the generated bounding box will be more accurate.

[0082] S3: The fine mask passes through SAM's image encoder to obtain high-level semantic features, which are input into SAM's mask decoder to guide SAM to form a more accurate segmentation. Segment Anything Model (SAM) is a revolutionary computer vision model launched by Meta AI.

[0083] Key features and innovations of SAM include:

[0084] Large-scale training: SAM is trained on a large-scale dataset containing 11 million images and over 1 billion masks.

[0085] Zero-shot generalization capability: SAM is a hintable model that can generate high-quality object masks given hints (e.g., points or boxes). It can perform zero-shot generalization to unfamiliar objects and images without additional training.

[0086] Wide range of application areas: SAM can be used to segment any object in any image. It can cover a wide range of use cases and can be used directly in new image "domains" (such as underwater photos or cell microscopy images) without additional training.

[0087] These features make SAM a powerful tool that can be widely used in various fields of computer vision.

[0088] S4: The Segment Anything Model (SAM) is a powerful computer vision model that can receive various types of input cues, including points, boxes, etc. These cues can be viewed as prior information, providing the SAM model with an initial estimate of the location of the object.

[0089] For this box prompt, define a binary image ,in express In the prompt box, otherwise, it means that the point is not in the prompt box. Then, these prompt images are provided to SAM as additional input channels. During the training process, an objective function is defined , which encourages SAM to generate high-confidence masks within the hint region. This objective function can be defined as:

[0090] ;

[0091] in, is the predicted probability of SAM generating an object mask at position (x, y). In this way, accurate hints can help the SAM model produce more accurate segmentation results. This is because these hints provide prior information about the location of the object, thereby reducing the amount of information the model needs to learn from the data.

[0092] S5: In this process, the image is first segmented using the "Segment Anything" model (SAM). The result is a binary mask where the area with pixel value 1 represents the object and the area with pixel value 0 represents the background.

[0093] Then, this mask is applied to the original image to obtain a mask image containing only the object. This new image only has the corresponding object part, and all the background noise has been removed. The description of SAM segmentation is as follows Figure 4 Flowchart of the cascade post-processing method for obtaining more refined cues.

[0094] Next, this new image is input into the PaNet fine-grained classification network. Figure 5 As shown in Figure 2, PaNet is a deep neural network whose goal is to perform fine-grained classification of input images. This process can be described by the following formula:

[0095] ;

[0096] Where x is the input image and y is the output of PaNet. It is the forward propagation function of PaNet.

[0097] The objective function of PaNet can be defined as the cross entropy loss function:

[0098] ;

[0099] in, is the true label of pixel i, is the predicted probability of pixel i.

[0100] S6: PaNet is a model for fine-grained classification that discovers parts of objects by using image-level class labels and prior knowledge. The key to this approach is that it encourages the model to first detect these discriminative parts and then use them to infer categories, which helps measure whether the model actually focuses on the right details.

[0101] By using the feature vector Applying the negative squared Euclidean distance function between them, the attention is calculated as follows:

[0102] ;

[0103] in and are the extracted features and the prototypes of the K parts respectively.

[0104] In addition, the attention map A generated by the SAM segmentation result is used as the regularization term for the fine-grained network to generate the attention map B. The definition formula is as follows:

[0105] ;

[0106] Use the element Aij in the attention map A to update the corresponding element in the attention map B . It is a learning rate parameter that controls the step size of the update, which constrains the attention mapping.

[0107] Each attention map is then used to compute its corresponding partial vector by calculating the weighted average of the feature vectors in Z using the attention values :

[0108] ;

[0109] An important method of PaNet is partial loss, that is, discarding the complete partial feature vector at one time to prevent a single part from dominating the classification. This can be achieved by introducing a partial loss function, which can be defined as:

[0110] ;

[0111] in, is the activation probability of the ith part, and n is the total number of parts. The goal of this function is to minimize the cross entropy of the activation probability of each part, thereby encouraging the model to use all parts when making decisions rather than relying on a single part.

[0112] S7: Part Feature Vector Modulation is a unique method that ensures that the information of each part is different from other parts from the perspective of the classifier. The key to this method is that it makes the information of each part unique from the perspective of the classifier by applying different linear transformations to each partial feature vector. Specifically, Part Feature Vector Modulation can be implemented by the following mathematical formula:

[0113] ;

[0114] In this formula, is the original eigenvector of the kth part, and is the corresponding weight (or linear classifier) ​​and modulation vector, is the modulated feature vector. The advantage of this method is that it ensures that the information of each part is unique from the perspective of the classifier, thereby improving the classification performance of the model.

[0115] In general, partial eigenvector modulation is a powerful tool that can help the model better understand and utilize partial information, thereby improving the performance of the model.

[0116] S8: Since the focus on the noisy part of the image is reduced and more discriminative regions are introduced, the network can learn the important parts more easily and focus on the regions that are important for the classification results. Without any additional hyperparameter adjustment, it provides better part discovery performance than previous methods without reducing the classification performance. Through these, the network model of the present invention can achieve good results and achieve high fine-grained classification accuracy.

[0117] In addition to the most basic classification cross entropy loss, the loss function of the network model of the present invention is also composed of the following loss functions:

[0118] First is the equivariance loss function, which creates a transformed image by applying a random rigid transformation T to the input image. Then, the original image and the transformed image are passed through the model and the transformation on the attention map is inverted from the transformed image. If is a function that returns the kth attention map of image X, and the equivariance loss is calculated using the cosine distance between the original image and the transformed image:

[0119] ;

[0120] Then there is the existence loss function, since all parts should appear in some images of the dataset. To this end, the absence of parts in the entire batch is penalized during training. The existence loss encourages each part to appear at least once per batch. Given a batch { ,., }, the loss function is defined as follows:

[0121] ;

[0122] Where avgpool() is a 2D average pooling with a kernel size of 1. This operator is used to prevent encouraging single-pixel attention maps.

[0123] Since we want each detected part to consist of a compact and continuous image region, in order to enforce these priors, we use as many loss functions as possible to Concentrated loss:

[0124] ;

[0125] in and denote the vertical and horizontal spatial variances, respectively.

[0126] Additionally, overlap between parts is avoided. This is encouraged by decorrelating the part feature vectors. The orthogonality loss of the part vectors is calculated by applying the cosine distance between all pairs:

[0127] ;

[0128] and is the eigenvector.

[0129] at last, It is the Kullback-Leibler divergence function (KL divergence). KL divergence is used to compare the closeness of two probability distributions. In statistics, a simple probability distribution is often used. To describe, the original distribution is , we need a quantity to measure the approximate distribution and the original distribution Compared to how much information is lost, this is the role of KL divergence. The KL divergence function is used to determine the gap between the attention map obtained after feature extraction and the feature map obtained by SAM segmentation. By using the KL divergence function, the network is constrained to learn more foreground parts:

[0130] ;

[0131] in and They are the normalized probability distributions of the feature maps obtained by the classification network and SAM, respectively, to constrain the classification network to learn more accurate foreground parts.

[0132] The weighted combination of these losses is used as the final loss of the PaNet classification network. The corresponding loss function is:

[0133] .

[0134] Embodiment 2: This embodiment uses embodiment 1 as the basic method to carry out module design.

[0135] A medical image fine-grained classification system based on segmentation large model guidance is composed of a coarse segmentation network module, a hint generation module, a SAM attention mechanism module, a SAM segmentation module, a part discovery module, a part feature vector modulation module and a fine-grained classification module. Figure 2 As shown, each part is described in detail below:

[0136] The coarse segmentation network module: first, the data in the original data set is passed through mask-rcnn or nn-Unet to generate a coarse mask. The original mask-rcnn or nn-Unet is pre-trained to generate a relatively accurate coarse mask, which will be used as the subsequent input.

[0137] The hint generation module performs a maximum connected graph operation on the generated rough mask to find its largest bounding box as another hint. It can also make personalized hints through the mask and a reference image. This mask is also used to guide SAM to form a more accurate segmentation.

[0138] The SAM attention mechanism module: operates the fine mask with the image to obtain a confidence map, combines this map with other prompts, and inputs it into the corresponding attention layer in the SAM decoder to perform attention operations, such as Figure 3 As shown, better results can be achieved through a training-free approach.

[0139] The SAM segmentation module can receive various types of input prompts, including points, boxes, etc. These prompts can be regarded as prior information, which provide the model with an initial estimate of the object location and produce a segmentation result (mask).

[0140] The part discovery module: The model first detects these distinguishable parts and then uses them to infer categories, which helps to better measure whether the model actually pays attention to the correct details, and uses the SAM segmentation results to guide the attention of the discovered parts.

[0141] The partial feature vector modulation module ensures that the information of each part is different from other parts from the perspective of the classifier. The key to this method is that it applies different linear transformations to each partial feature vector so that the information of each part is unique from the perspective of the classifier.

[0142] The fine-grained classification module: combines partial discovery with partial feature vector modulation to achieve a more refined classification result. The description of the last three modules is as follows Figure 5 shown.

[0143] Example 3: This implementation is based on the above method for example verification

[0144] This embodiment is verified based on the above method and system. The overall process is as follows: Figure 1 As shown, in order to verify the accuracy of the model attributes proposed by the present invention, experiments were conducted on the red_4_cells_raw red blood cell dataset, using accuracy as the evaluation index. By using resnet50 as the backbone for the experiment, the test accuracy result was 79.03%. And it was verified on the natural image dataset CUB_200_2011. The CUB-200-2011 dataset was jointly created by the California Institute of Technology and the University of California, San Diego, and is used for fine-grained visual classification tasks. This dataset contains 200 subcategories of birds, with a total of 11,788 pictures, of which 5,994 are used for training and 5,794 are used for testing. When the number of partial attention discoveries K=8 is set, an accuracy of 87.69% is achieved, which is obviously better than other existing methods.

[0145] The above results show that the classification model constructed by the present invention can significantly improve the accuracy of fine-grained classification.

[0146] The above plans are only implementation methods of the present invention, but the protection scope of the present invention is not limited thereto. All replacements or changes that can be understood and thought of by persons familiar with the technology within the technical scope disclosed in the present invention should be included in the protection scope of the present invention.

Claims

1. A fine-grained classification method for medical images based on the guidance of a large segmentation model, characterized in that: The steps include: S1: Collect a medical image dataset, which first generates a coarse mask through a segmentation network; S2: First, the coarse mask is refined to obtain a finer mask, and then the fine mask is used to generate a bounding box through a maximum connected graph method; S3: taking the fine mask, the box prompt, and the medical image dataset as input to the segmentation large model SAM for segmentation processing; S4: The segmentation processing result obtained in S3 is input into the PaNet classification network as a guide item for fine-grained classification; the PaNet classification network uses image-level class labels and performs fine-grained classification by ij The negative square Euclidean distance function is applied between them, and a partial loss method is proposed, that is, a partial feature vector is discarded at one time to prevent a single part from dominating the classification and improve the robustness of the network; the PaNet classification network also adopts a partial feature vector modulation method; the S4 is specifically as follows: S4-1: Use the SAM model to segment the image and obtain a binary mask, where the area with a pixel value of 1 represents the object and the area with a pixel value of 0 represents the background; S4-2: The binary mask is used to guide the PaNet classification network, and the new image to be classified is input into the PaNet classification network to perform fine-grained classification on the input image: y=f PaNet (x) Where x is the input image, y is the output of PaNet, and f PaNet is the forward propagation function of PaNet; The objective function of the PaNet classification network is defined as the cross entropy loss function: Among them, y i is the true label of pixel i, is the predicted probability of pixel i; S4-3: The PaNet classification network uses image-level class labels and prior knowledge: PaNet uses image-level class labels to discover parts of objects; By adding the eigenvector z ij Applying the negative squared Euclidean distance function between them, the attention is calculated as follows: Where z and p are the extracted features and the prototypes of the K parts respectively; In addition, the attention map A generated by the segmentation result of the SAM model is used as the regularization term for the PaNet classification network to generate the attention map B. The definition formula is as follows: a ij =B ij -a(B ij -A ij ) 2 Use the element Aij in the attention map A to update the corresponding element B in the attention map B ij ; α is a learning rate parameter, which is used to control the step size of the update, thereby constraining the attention mapping; Each attention map is then used to compute its corresponding partial vector by calculating the weighted average of the feature vectors in Z using the attention values S4-4: Partial loss: In order to prevent a single part from dominating the classification, PaNet proposed a partial loss method, that is, discarding part of the feature vector at one time; S4-5: Partial feature vector modulation: In addition, PaNet also proposed a method for partial feature vector modulation: s k =W class ·(m k ⊙v k ) In this formula, v k is the original eigenvector of the kth part, W class and m k is the corresponding weight and modulation vector, s k is the modulated eigenvector; S5: training the PaNet classification network, and then using the trained network to perform medical image classification and recognition, and outputting the classification results.

2. The method for fine-grained classification of medical images based on guidance of a large segmentation model as claimed in claim 1, characterized in that: In S1, a segmentation network is used to generate a coarse mask, which is regarded as a binary image, in which the area with a pixel value of 1 represents the location of the object, and the area with a pixel value of 0 represents the background.

3. The medical image fine-grained classification method based on segmentation large model guidance as claimed in claim 1, characterized in that: The S2 includes: S2-1: The coarse mask generates a fine mask by applying a series of convolution and nonlinear operations on the coarse mask through mask refinement; the objective function of mask refinement is defined as the IoU loss function: Among them, y i is the true label of pixel i, is the predicted probability of pixel i, p i The probability of the prediction being correct; S2-2: The maximum connected graph algorithm is then used to obtain all connected areas in the fine mask, and the area with the largest number of pixels is selected as the maximum connected graph; S2-3: Finally, the bounding box is generated by calculating the minimum and maximum x and y coordinates of the maximum connected graph.

4. The method for fine-grained classification of medical images based on guidance of a large segmentation model as claimed in claim 1, characterized in that: In S3, the segmentation model SAM can receive various types of input prompts, including point prompts and box prompts; For the box prompt, define a binary image I box , where I box (x, y) = 1 means that (x, y) is within the prompt box, otherwise, it means that the point is not within the prompt box. Then, these prompt images are provided to the SAM model as additional input channels. During the SAM model training process, an objective function L is defined. SAM , which encourages the SAM model to generate high-confidence masks within the hint region; the objective function is defined as: Among them, p SAM (x,y) is the predicted probability of SAM generating an object mask at position (x,y); The refined mask is input into the image encoder of the SAM model to generate high-level semantic features, and then input into the mask decoder of the SAM model to generate a more refined segmentation result.

5. The method for fine-grained classification of medical images based on guidance of a large segmentation model as claimed in claim 1, characterized in that: The loss function of the PaNet classification network is specifically: Among them, the equal variance loss function is: There is a loss function defined as follows: Where avgpool() is a 2D average pooling with a kernel size of 1; For the attention map A k The concentration loss function is: where σ v and σ h denote the vertical and horizontal spatial variances, respectively; The orthogonality loss function for the partial vectors is calculated by applying the cosine distance between all pairs: Among them, v k and v l is the eigenvector; KL divergence function to constrain the function: Among them, P(i) and Q(i) are the normalized probability distributions of the feature maps obtained by the classification network and SAM model, respectively.

Images