Medical image classification method and system based on prototype enhancement

Abstract

The invention discloses a medical image classification method and system based on prototype enhancement. The method comprises the following steps: acquiring a plurality of medical images with known category labels and image category description text features; determining a class prototype of each image class through a medical image with a known class label; combining the class prototype of each image class with the image class description text features to obtain an enhanced class prototype of each image class; classifying the medical images according to the class label and the enhanced class prototype of each medical image to obtain clean samples and in-distribution noise label samples; training the constructed medical image classification model through the clean sample and the distributed internal noise label sample, and obtaining a trained medical image classification model after the training is completed; and classifying the medical images by using the trained medical image classification model. Accurate recognition of the medical image is realized, and the robustness of the medical image classification model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of model training, and particularly to a medical image classification method and system based on class prototype enhancement. Background Technique

[0002] The statements in this part only provide background technical information related to the present invention and do not necessarily constitute prior art.

[0003] Currently, a medical image classification model is mainly constructed using a deep neural network, and the constructed medical image classification model is trained using labeled medical images. Then, the trained medical image classification model is used for medical image classification. In the wide application of deep neural networks, noisy labels, especially the handling of out-of-distribution noisy label samples, have always been a key challenge. Noisy labels refer to incorrect classification labels existing in the training data, which may cause the model to learn inaccurate features and thus reduce performance. On the other hand, out-of-distribution noisy label samples refer to new category samples that do not belong to the training data distribution, which usually involves data distribution shift, and this may be caused by limitations in data acquisition methods, data sources, or data volumes. Such samples may interfere with the generalization ability of the model and make it perform poorly on unknown categories. The model needs to maintain good generalization ability when facing these distribution differences.

[0004] However, medical image annotation highly depends on expert experience, with high annotation costs and being susceptible to subjective factors, which may introduce noisy labels, such as mislabeling rare lesions as common lesions. In addition, actual clinical data may contain out-of-distribution noisy samples, such as non-lesion areas, unknown lesion types, or heterogeneous data collected by different imaging devices (such as MRI, CT, X-ray, etc.). These samples may be misclassified as known categories, that is, the samples currently used for training the medical image classification model may include not only clean samples, in-distribution noisy samples, but also out-of-distribution noisy samples.

[0005] When training a constructed medical image classification model, existing methods only target in-distribution noisy label samples and enhance the model's robustness by introducing regularization terms or improving the network structure. However, these methods have limited ability to handle out-of-distribution noise samples, unable to effectively identify and process out-of-distribution noise samples that do not belong to the training distribution, resulting in limited robustness of the model. In addition, the class prototype discrimination method adopted by existing medical image classification models mainly identifies and discriminates by using the class prototypes of categories; the class prototypes are constructed based on the features of each class in the training data and can effectively represent the core features of each class; when a sample is input into the model, by calculating the similarity between the sample and each class prototype, it can be determined whether the sample belongs to a known category; although overfitting is alleviated to a certain extent, due to sample noise pollution, the quality of the class prototypes is not high, resulting in unsatisfactory sample purification effects. Moreover, existing methods usually only focus on intra-class features and ignore the differences in inter-class features, which leads to insufficient discrimination of class features, further affecting the performance of the model and ultimately resulting in inaccurate medical image classification. Therefore, current medical image models have the technical problems of inaccurate medical image classification and limited model robustness. Summary of the Invention

[0006] To solve the above problems, the present invention proposes a medical image classification method and system based on class prototype enhancement, achieving accurate recognition of medical images and improving the robustness of the medical image classification model.

[0007] To achieve the above object, the present invention adopts the following technical solutions: In the first aspect, a medical image classification method based on class prototype enhancement is proposed, including: Obtain medical images with multiple known category labels and image category description text features; Determine the class prototype of each image category through the medical images with known category labels; Combine the class prototype of each image category with the image category description text features to obtain an enhanced class prototype for each image category; Classify the medical images according to the category labels and enhanced class prototypes of each medical image to obtain clean samples and in-distribution noisy label samples; Train the constructed medical image classification model through the clean samples and in-distribution noisy label samples, and obtain a trained medical image classification model after training; Use the trained medical image classification model to classify medical images.

[0008] Furthermore, the medical image classification model includes two parallel image classification networks; each image classification network includes a feature extractor and a classifier; the feature extractor extracts the image features of the medical image; the classifier classifies the image features extracted by the feature extractor; the classification results output by the two classifiers are averaged to obtain the final medical image classification result.

[0009] Furthermore, when training the constructed medical image classification model, the large language model is used to extract the key attention regions of the clean samples and the in-distribution noise label samples; the key attention regions extracted by the large language model are used as supervision signals to train the medical image classification model; after the training is completed, a trained medical image classification model is obtained.

[0010] Furthermore, the image features of each medical image are extracted. Based on the image features and labels of each medical image, the class prototype of each image category is calculated and determined.

[0011] Furthermore, for each medical image with a known class label, the predicted class of the medical image is calculated through the two image classification networks respectively; when the two predicted classes are consistent with the class label of the medical image, the medical image sample is used as a clean sample; when only one of the two predicted classes is consistent with the class label of the medical image, the sample is used as an in-distribution noise sample; when the two predicted classes are not consistent with the class label of the medical image, the medical image sample is used as an out-of-distribution noise sample.

[0012] Furthermore, the descriptive text of each medical image category is obtained. Using the text feature extraction model, the features of the descriptive text of each medical image category are extracted to obtain the image category descriptive text features.

[0013] In the second aspect, a medical image classification system based on class prototype enhancement is proposed, including: A data acquisition unit for acquiring multiple medical images with known class labels and image category descriptive text features; A class prototype determination unit for determining the class prototype of each image category through the medical images with known class labels; A class prototype enhancement unit for combining the class prototype of the image category with the image category descriptive text features to obtain the enhanced class prototype of each image category; A sample classification unit for classifying medical images according to the class labels and enhanced class prototypes of each medical image to obtain clean samples and in-distribution noise label samples; A model training unit for training the constructed medical image classification model with clean samples and in-distribution noisy label samples, and obtaining a trained medical image classification model after the training is completed; A medical image classification unit for classifying medical images by using the trained medical image classification model.

[0014] In a third aspect, a computer device is proposed, and the device includes: A processor adapted to execute a computer program; A computer-readable storage medium storing a computer program, and when the computer program is executed by the processor, it implements the medical image classification method based on class prototype enhancement proposed in the first aspect.

[0015] In a fourth aspect, a computer-readable storage medium is proposed, and the computer-readable storage medium stores a computer program, and the computer program is adapted to be loaded and executed by a processor to implement the medical image classification method based on class prototype enhancement proposed in the first aspect.

[0016] In a fifth aspect, a computer program product is proposed, and the computer program product includes a computer program, and when the computer program is executed by a processor, it implements the medical image classification method based on class prototype enhancement proposed in the first aspect.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: For the medical image classification method and system based on class prototype enhancement proposed by the present invention, before training the medical image classification model, the method first enhances the class prototype of each determined image class by using the medical image category description text features, obtains the enhanced class prototype of each image class, and by incorporating rich semantic knowledge into the class prototype, makes the enhanced class prototype more comprehensive and accurate, enhances the stability of the class prototype, and reduces the influence caused by sample feature deviation; then uses the enhanced class prototype to accurately classify the obtained medical images with known class labels, discards the inter-partition noisy label samples, only extracts clean samples and in-distribution noisy label samples, and trains the constructed medical image classification model, thereby reducing the influence of noise on model training, avoiding overconfident prediction of out-of-distribution noisy samples, improving the training effect of the medical image classification model, and finally improving the accuracy of medical image classification.

[0018] The advantages of the additional aspects of the present invention will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the present invention. Description of the Drawings

[0019] The accompanying drawings forming a part of this application are used to provide a further understanding of this application. The schematic embodiments and descriptions thereof of this application are used to explain this application and do not constitute an improper limitation of this application.

[0020] Figure 1 Flowchart of the medical image classification method based on class prototype enhancement disclosed in the embodiment; Figure 2 Overall framework of the existing medical image classification model disclosed in the embodiment; Figure 3 Overall framework of the medical image classification method based on class prototype enhancement disclosed in the embodiment; Figure 4 Detailed flowchart of the overall model training in the medical image classification method based on class prototype enhancement disclosed in the embodiment; Figure 5 Simplified flowchart of the overall model training process in the medical image classification method based on class prototype enhancement disclosed in the embodiment; Figure 6 Flowchart of sample screening disclosed in the embodiment; Figure 7 Flowchart of region-sensitive distillation regularization disclosed in the embodiment. Detailed implementation manners

[0021] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0022] It should be noted that the following detailed descriptions are all illustrative and are intended to provide a further description of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs.

[0023] It should be noted that the terms used herein are only for describing the specific implementation manners and are not intended to limit the exemplary implementation manners according to this application. As used herein, unless the context clearly indicates otherwise, the singular forms are also intended to include the plural forms. In addition, it should be understood that when the terms "include" and / or "comprise" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0024] In the case of no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0025] It should be noted that all data acquisitions are based on compliance with laws, regulations and user consent, and the data is legally applied.

[0026] First, the terms involved in the embodiments of this application are explained, including: Clean sample: The label is correct and belongs to a predefined category; In-distribution noisy-label sample: The label is incorrect, but the true category belongs to a predefined category; Out-of-distribution noisy-label sample: The label is incorrect and the true category does not belong to a predefined category Class prototype: A category representation vector obtained by averaging or weighting the features of samples in the same class, used to describe the core features of the class.

[0027] Large language models (LLMs): General language models pre-trained on massive amounts of data (such as GPT-4, CLIP), with cross-modal semantic understanding and generation capabilities, which can provide rich prior knowledge.

[0028] Multimodal: By processing and integrating information of multiple data types (such as text, images, audio, and video), enabling computers to more comprehensively understand and respond to complex information environments.

[0029] Teacher-student model: A knowledge distillation framework that transfers the knowledge of a pre-trained teacher model (such as a large language model) to a student model (such as a target network) to improve the performance and generalization ability of the student model.

[0030] Region of attention: It refers to the part of the input data that the model dynamically focuses on and extracts through the attention mechanism, which is most relevant to the current task, to improve the pertinence of feature extraction and the performance of the model.

[0031] Upsampling: A technique that increases the number of data points through interpolation or other methods to improve the resolution of signals or images.

[0032] First, the application scenario of the medical image classification method based on class prototype enhancement proposed in the embodiments of this application is described.

[0033] The medical image classification method based on class prototype enhancement proposed in the embodiments of this application is applied to the medical image classification scenario. The medical image can be a dermoscopic image, a pathological section image, an ultrasound image, etc.

[0034] The current method uses a trained medical image classification model to classify medical images. The training samples used in the training of the medical image classification model not only include clean samples and in-distribution noise samples, but may also include out-of-distribution noise samples. Moreover, when the current medical image classification model is trained, only for in-distribution noise-labeled samples, by introducing a regularization term or improving the network structure to enhance the robustness of the model, it is unable to effectively identify and process out-of-distribution noise samples that do not belong to the training distribution, resulting in limited robustness of the model. In addition, the class prototype discrimination method adopted by the existing medical image classification model mainly identifies and discriminates by using the class prototypes of the classes. By calculating the similarity between the sample and each class prototype, it can be determined whether the sample belongs to a known class. The class prototype is constructed based on the training samples of the known classes. Therefore, if the training data itself has noise or is unbalanced (for example, the samples of some classes are too few or of poor quality), the class prototype may not accurately represent the true characteristics of the class. This will lead to inaccurate determination of the model for out-of-distribution noise-labeled samples (OOD samples). The class prototype is usually calculated by the average feature of the samples within the class or a certain aggregation method, and may not be able to fully capture the diversity of the samples within the class. Especially for classes with high variability, the class prototype may not be able to well represent the overall characteristics of the class, resulting in an unsatisfactory detection effect for OOD samples.

[0035] In the embodiment of the present application, the prototype of the medical image category is enhanced through the text features of the image category description, making the enhanced class prototype more comprehensive and accurate, thereby enhancing the stability of the class prototype, reducing the influence brought by sample feature deviation, and using the enhanced class prototype to classify and screen medical images, discarding out-of-distribution noise-labeled samples, and only using clean samples and in-distribution noise-labeled samples to train the constructed medical image classification model, ensuring the training effect of the medical image classification model, improving the accuracy of medical image classification, and enhancing the robustness and adaptability of the medical image classification model in practical applications.

[0036] Such as Figure 2As shown, the method for screening out-of-distribution noise label samples has evolved from traditional statistical methods to deep learning and class prototype methods. Early methods were mainly based on density estimation and probability models, such as Gaussian mixture models (GMMs) and One-Class SVMs. They learned the feature distributions of known classes and detected samples that deviated from these distributions during testing. However, these methods have limited effectiveness on high-dimensional data. With the development of deep learning, confidence detection based on the Softmax distribution has become mainstream, that is, using the maximum Softmax probability of the classifier as the confidence that a sample belongs to a certain known class, and samples with probabilities below a threshold are judged as OOD samples. However, Softmax is prone to overconfidence, leading to misjudgments. To address this, subsequent methods have introduced temperature scaling and perturbation techniques to enhance the model's discrimination ability. In addition, the Mahalanobis distance method calculates the distance between a sample and the class center using the multi-dimensional Gaussian distribution in the feature space to detect OOD samples. Recently, class prototype-based OOD detection methods have become a new trend. These methods construct class prototypes (i.e., the feature centers of classes) for each class and calculate the similarity between test samples and class prototypes during the inference stage. If a sample has a low similarity to all class prototypes, it is judged as OOD. Compared with traditional methods, class prototype methods can more stably describe the commonalities and differences between classes, avoiding misjudgments caused by simply relying on Softmax confidence. At the same time, the latest methods combine multi-modal information (such as class descriptions generated by language models) to enhance the semantic expression ability of class prototypes, making OOD detection more accurate and robust in complex scenarios.

[0037] Next, a detailed description will be given of the medical image classification method based on class prototype enhancement disclosed in the embodiments of the present application. [[ID=?]]

[0038] As Figure 1 and Figures 3 - 7 shown, the medical image classification method based on class prototype enhancement disclosed in the embodiments of the present application includes: Obtain medical images with multiple known class labels and text features of image class descriptions; Determine the class prototype of each image class through the medical images with known class labels; Combine the class prototype of each image class with the text features of the image class description to obtain an enhanced class prototype for each image class; Classify the medical images according to the class labels and enhanced class prototypes of each medical image to obtain clean samples and in-distribution noise label samples; Train the constructed medical image classification model with the clean samples and in-distribution noise label samples, and obtain a trained medical image classification model after training is completed; Use the trained medical image classification model to classify medical images.

[0039] The medical image classification model constructed in the embodiments of this application includes two parallel image classification networks; among them, each image classification network includes a feature extractor and a classifier; the image features of the medical image are extracted through the feature extractor; the image features extracted by the feature extractor are classified through the classifier; the classification results output by the two classifiers are averaged to obtain the final medical image classification result.

[0040] Preferably, calculate the cosine similarity between the image features and the enhanced class prototypes of each image class, and select the class corresponding to the enhanced class prototype with the largest cosine similarity as the image class of the image features and the predicted class of the medical image to which the image features belong.

[0041] Taking the medical image as a dermoscopic image as an example, the corresponding image classes include: normal, melanoma, nevus, etc.

[0042] When training the constructed medical image classification model, use large language models (LLMs) to extract the key attention regions of clean samples and in-distribution noise-labeled samples; use the key attention regions extracted by the large language model as supervision signals to train the medical image classification model; after training is completed, obtain the trained medical image classification model, where the key attention region of a sample refers to the region in the image sample that has the greatest impact on the final classification result.

[0043] The embodiments of this application utilize the powerful feature extraction ability of large language models to enhance the prediction performance of the medical image classification model. To avoid the problem that direct feature supervision causes the model to over-focus on global features and ignore local details, when training the medical image classification model in the embodiments of this application, regional sensitive distillation regularization is introduced. By aligning the deep semantic knowledge of the large language model with the early inaccurate classification network, the network is guided to focus on the key regions of the samples, thereby improving the performance. Specifically, the medical image classification model is used as the student model, the large language model is used as the teacher model, and the attention region of the last layer of the large language model is used as the supervision signal. The size of the attention region is unified through upsampling, and distillation learning is used for knowledge transfer. This method effectively improves the network's learning ability for important regions, thereby optimizing the prediction accuracy of the medical image classification model.

[0044] Preferably, the feature extractor can be constructed using a convolutional neural network.

[0045] The classifier uses a Softmax classifier.

[0046] The large language model selects GPT-4o to generate descriptive text and uses the CLIP model to extract the descriptive text features.

[0047] In the sample screening stage, the embodiments of this application extract the image features of each medical image; Based on the image features and labels of each medical image, calculate and determine the class prototypes of each image category, and use the class prototypes to represent the core image features of each image category.

[0048] The process of classifying medical images includes: for each medical image with a known class label, calculate the predicted class of the medical image through two image classification networks respectively; when the two predicted classes are consistent with the class label of the medical image, regard the medical image sample as a clean sample; when only one of the two predicted classes is consistent with the class label of the medical image, regard the sample as an in-distribution noise sample; when the two predicted classes are both inconsistent with the class label of the medical image, then regard the medical image sample as an out-of-distribution noise sample.

[0049] Among them, the image classification network extracts the image features of the medical image, calculates the cosine similarity between the image features and each enhanced class prototype, and selects the enhanced class prototype with the largest cosine similarity as the predicted class of the medical image.

[0050] In this embodiment, by obtaining the description text of each medical image category; Using a text feature extraction model, extract the features of the description text of each medical image category to obtain image category description text features.

[0051] Preferably, the image category description text features are obtained by extracting through a multi-view description module.

[0052] Through a multi-modal semantic synchronization module, synchronize the image category description text features extracted by the multi-view description module into the class prototypes of the corresponding image categories to strengthen the class prototypes, obtain enhanced class prototypes, and then based on the enhanced class prototypes, screen the medical image samples.

[0053] Through a region-sensitive distillation regularization module, use the selected samples to train the medical image classification model.

[0054] The multi-view description module uses a large language model to generate description texts of each medical image category with intra-class commonality descriptions and inter-class difference descriptions to improve the accuracy of calculating class prototypes in a dataset with noisy labels and out-of-distribution samples. By designing prompt templates, the LLMs generate texts describing category commonalities from multiple perspectives and difference descriptions emphasizing the distinguishing features between categories, and then use a text encoder and an adapter to extract the features of these descriptions as the corresponding image category description text features for optimizing the calculation of class prototypes.

[0055] The descriptive text of the images of tumor categories includes within-class commonality descriptions and between-class difference descriptions. The within-class commonality descriptions include: In CT images, tumors usually appear as abnormal density shadows, presenting as round, oval, or irregular masses. The density can be higher or lower than that of the surrounding normal tissues, such as low-density liver cancer or high-density calcification foci. Benign tumors mostly have clear boundaries and can be seen with a complete capsule surrounding them, while malignant tumors often have blurred edges with a spiculated appearance and signs of infiltration of the surrounding tissues. During enhanced scanning, malignant tumors mostly show inhomogeneous enhancement due to rich blood supply, and the central necrosis area shows no enhancement, while benign tumors show more uniform enhancement or no obvious enhancement. Tumors may compress or invade adjacent structures, such as blood vessel displacement, organ deformation, or lumen obstruction, and some are accompanied by secondary changes such as surrounding edema and effusion. Some tumors show characteristic manifestations, such as the "bull's-eye sign" (metastatic tumor), "fat density" (lipoma), or "popcorn-like calcification" (pulmonary hamartoma), providing clues for differential diagnosis.

[0056] The between-class difference descriptions include: Tumors mostly form localized masses, with a round / irregular mass shape, compressing or infiltrating the surrounding structures (such as blood vessel displacement, organ deformation), and the surrounding edema zone (such as white matter edema around a brain tumor) may gradually worsen; The multi-modal semantic synchronization module uses two parallel feature extractors in the medical image classification model to extract image features from medical images respectively; and based on the image features, determines the class prototypes of each image category; then through two integrators, synchronizes the text features of the image category descriptions into the class prototypes corresponding to the categories, enhances each class prototype using the text features of the image category descriptions, and obtains the corresponding enhanced class prototypes. The enhanced class prototypes contain within-class commonality and between-class difference information. Therefore, when using the enhanced class prototypes for classification, it can reduce the error information brought by noise samples and achieve accurate classification of samples; then, a sample purification strategy is adopted. For each medical image with a known class label, the cosine similarity between the image features of the medical image and each enhanced class prototype is calculated through two image classification networks. According to the calculated cosine similarity, two predicted categories corresponding to the medical image are determined; when the two predicted categories are consistent with the class label of the medical image, the medical image sample is regarded as a clean sample; when only one of the two predicted categories is consistent with the class label of the medical image, the sample is regarded as an in-distribution noise sample; when the two predicted categories are both inconsistent with the class label of the medical image, the medical image sample is regarded as an out-of-distribution noise sample.

[0057] Among them, when determining the predicted category of a medical image according to the cosine similarity, the image category corresponding to the maximum value in the cosine similarity is selected as the predicted category of the medical image.

[0058] After that, only clean samples and in-division noisy-label samples are used to train the medical image classification model, purifying the training data of the medical image classification model and improving the robustness and accuracy of the medical image classification model.

[0059] Preferably, the integrator in this embodiment adopts a classical attention mechanism to fuse the text features of the image category description and the class prototype, integrating the semantic features from the text and the class prototype information together, so as to achieve the purpose of class prototype enhancement and obtain the corresponding enhanced class prototype.

[0060] The region-sensitive distillation regularization module adopts region-sensitive distillation regularization. Using clean samples and in-division noisy-label samples, taking the large language model as the teacher model and the medical image classification model as the student model, the medical image classification model is trained. After the training is completed, a trained medical image classification model is obtained.

[0061] When training the medical image classification model, clean samples and in-distribution noisy-label samples are input into the large language model to extract the key attention regions in the intermediate layer, and they are used as the teacher model to guide the student model, and the region-sensitive distillation regularization loss is calculated therefrom; different strategies are used to calculate the classification loss for clean samples and in-distribution noisy-label samples respectively; all losses are linearly summed to calculate the total loss of the medical image classification model, and the medical image classification model is optimized by backpropagation according to the total loss; when the number of optimization cycles of the medical image classification model reaches the set value, or the optimization effect of the medical image classification model reaches the optimal, it indicates that the training of the medical image classification model is completed, and a trained medical image classification model is obtained.

[0062] The medical image classification method based on class prototype enhancement proposed in the embodiment of this application generates description texts for each medical image category that describe the intra-class commonality and inter-class differences through the large language model, combines multi-modal semantics to synchronously optimize the class prototype, and uses prediction consistency to achieve sample purification to screen out noise samples and remove out-of-distribution noisy-label samples, and further improves the performance of the medical image classification model through region-sensitive knowledge distillation. This series of steps effectively improves the robustness and prediction accuracy of the model in the case of noisy labels and OOD samples.

[0063] Compared with existing methods, the embodiments of the present application add two processes: obtaining text features of image category descriptions and enhancing class prototypes. First, two parallel networks are used to calculate the class prototypes of each image category according to the sample labels respectively. Then, a large language model is used to generate diverse and more discriminative category descriptions for each category, that is, the description text of medical image categories, and a text extractor is used to extract the text features of this part of the description, that is, the text features of image category descriptions. The text features of this part of image category descriptions and the calculated class prototypes are jointly input into the designed semantic compressor. The semantic compressor can fuse multi-modal information from images and texts together to generate class prototypes with more detailed semantics as enhanced class prototypes. Thus, the problems of low quality of class prototypes and lack of information caused by samples with wrong labels and worthless out-of-distribution noises are solved. Then, the enhanced class prototypes are used to classify each sample respectively, removing the worthless out-of-distribution noise label samples, and the clean samples and in-distribution noise label samples are retained as valuable samples.

[0064] For the selected valuable samples, the embodiments of the present application propose a knowledge distillation method based on the teacher-student framework. The large language model contains pre-trained knowledge and thus has a powerful feature extraction ability. To avoid the problem that direct feature supervision causes the network to over-focus on global features and ignore local details, the embodiments of the present application introduce region-sensitive distillation regularization, using the key attention regions of the samples extracted by the large language model as supervision to guide the student network to focus on the key regions of the samples, thereby further improving the performance. Specifically, the attention regions of the last layer of the large language model are selected as the supervision signal. After upsampling to unify the size of the attention regions, knowledge transfer is carried out through distillation. The key learning ability of the network is improved, and the robustness of the model is optimized.

[0065] The framework of the embodiments of the present application demonstrates significant advantages in multi-modal fusion and class prototype optimization. First, by generating multi-perspective descriptions, the model can understand the characteristics of each category more comprehensively and meticulously. These multi-perspective descriptions not only include visual features at the image level but also incorporate rich semantic information, thus providing a more profound and comprehensive category representation. Second, by introducing a multi-modal semantic synchronization mechanism, the text description and image features can be seamlessly fused together, making the prediction results of the model more stable and accurate. This synchronization mechanism ensures the effective integration of information from different modalities, enabling the class prototype to not only have higher representativeness but also enhance the model's adaptability to complex scenarios. In addition, the framework of the embodiments of the present application further improves the model's performance through region-sensitive distillation regularization. This regularization term can guide the model to not only focus on global features but also effectively capture local detailed information in the image during the learning process. In this way, the model can more accurately determine the key regions in the image, thereby improving the accuracy and robustness of the model's prediction. Especially in the face of complex backgrounds or blurred images, the model shows stronger generalization ability. In diverse datasets and complex environments, the model can effectively handle different image features and variations, showing stronger robustness and generalization ability than traditional methods. Overall, the embodiments of the present application effectively improve the quality of class prototype generation in the image classification task and enhance the model's deep semantic learning ability by skillfully combining multi-perspective descriptions generated by LLMs, the multi-modal semantic synchronization mechanism, and region-sensitive distillation regularization. This method provides a new idea and method for solving the image classification problem under out-of-distribution noisy label samples, and has important application value and research significance.

[0066] The embodiments of the present application also provide a medical image classification system based on class prototype enhancement, including: A data acquisition unit for acquiring medical images with multiple known category labels and text features of image category descriptions; A class prototype determination unit for determining the class prototype of each image category through medical images with known category labels; A class prototype enhancement unit for combining the class prototype of the image category with the text features of the image category description to obtain an enhanced class prototype for each image category; A sample classification unit for classifying medical images according to the category labels and enhanced class prototypes of each medical image to obtain clean samples and in-distribution noisy label samples; A model training unit for training the constructed medical image classification model through clean samples and in-distribution noisy label samples, and obtaining a trained medical image classification model after training; A medical image classification unit for classifying medical images by using the trained medical image classification model.

[0067] The present invention also discloses a computer device, which includes: A processor, adapted to execute a computer program; A computer-readable storage medium, in which a computer program is stored. When the computer program is executed by the processor, the medical image classification method based on class prototype enhancement disclosed in the embodiments of the present application is implemented.

[0068] The present invention also discloses a computer-readable storage medium, which stores a computer program. The computer program is adapted to be loaded and executed by a processor to implement the medical image classification method based on class prototype enhancement disclosed in the embodiments of the present application.

[0069] The present invention also discloses a computer program product, which includes a computer program. When the computer program is executed by a processor, the medical image classification method based on class prototype enhancement disclosed in the embodiments of the present application is implemented.

[0070] The method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0071] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with this embodiment can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0072] Although the specific implementation manners of the present invention have been described above in conjunction with the accompanying drawings, it is not a limitation on the protection scope of the present invention. Those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or deformations that can be made by those skilled in the art without creative efforts are still within the protection scope of the present invention.

Claims

1. A medical image classification method based on class prototype enhancement, characterized in that including: Obtaining medical images with multiple known class labels and text features of image class descriptions; Determining the class prototype of each image class through medical images with known class labels; Combining the class prototype of each image class with the text features of the image class description to obtain the enhanced class prototype of each image class; Classifying medical images according to the class label of each medical image and the enhanced class prototype to obtain clean samples and in-distribution noise-labeled samples; Training the constructed medical image classification model with the clean samples and in-distribution noise-labeled samples, and obtaining a trained medical image classification model after training; Classifying medical images using the trained medical image classification model.

2. The medical image classification method based on class prototype enhancement according to claim 1, wherein The medical image classification model includes two parallel image classification networks; among them, each image classification network includes a feature extractor and a classifier; extracting the image features of medical images through the feature extractor; classifying the image features extracted by the feature extractor through the classifier; taking the average of the classification results output by the two classifiers to obtain the final medical image classification result.

3. The medical image classification method based on class prototype enhancement according to claim 2, characterized in that, For each medical image with a known class label, calculating the predicted class of the medical image through the two image classification networks respectively; when the two predicted classes are consistent with the class label of the medical image, taking the medical image sample as a clean sample; when only one of the two predicted classes is consistent with the class label of the medical image, taking the sample as an in-distribution noise sample; when the two predicted classes are not consistent with the class label of the medical image, taking the medical image sample as an out-of-distribution noise sample.

4. The medical image classification method based on class prototype enhancement according to claim 1, characterized in that When training the constructed medical image classification model, using a large language model to extract the key attention areas of clean samples and in-distribution noise-labeled samples; taking the key attention areas extracted by the large language model as supervision signals to train the medical image classification model; After training, obtaining a trained medical image classification model.

5. The medical image classification method based on class prototype enhancement according to claim 1, characterized in that, Extracting the image features of each medical image; Calculating and determining the class prototype of each image class according to the image features and labels of each medical image.

6. The medical image classification method based on class prototype enhancement according to claim 1, characterized in that Obtaining the description text of each medical image class; Using a text feature extraction model to extract the features of the description text of each medical image class to obtain the text features of the image class description.

7. A medical image classification system enhanced based on class prototypes, characterized in that including: A data acquisition unit for obtaining medical images with multiple known class labels and text features of image class descriptions; A class prototype determination unit for determining the class prototype of each image class through medical images with known class labels; A class prototype enhancement unit for combining the class prototype of the image class with the text features of the image class description to obtain the enhanced class prototype of each image class; A sample classification unit for classifying medical images according to the class label of each medical image and the enhanced class prototype to obtain clean samples and in-distribution noise-labeled samples; A model training unit for training the constructed medical image classification model with the clean samples and in-distribution noise-labeled samples, and obtaining a trained medical image classification model after training; A medical image classification unit for classifying medical images using the trained medical image classification model.

8. An electronic device, characterized in that, The device includes: A processor, adapted to execute a computer program; A computer-readable storage medium storing a computer program, which when executed by the processor, implements the method for class prototype-enhanced medical image classification according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is adapted to be loaded and executed by a processor to implement the method for class prototype-enhanced medical image classification according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program, which when executed by a processor, implements the method for class prototype-enhanced medical image classification according to any one of claims 1-6.

Images