Small sample medical image processing method based on two similarity metrics and joint learning
This method for processing small-sample medical images using dual similarity metrics and joint learning addresses the problem of low model generalization ability in small-sample chest X-ray image classification, improving the accuracy and precision of disease classification, especially the identification of rare diseases.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TONGJI UNIV
- Filing Date
- 2023-12-21
- Publication Date
- 2026-06-26
AI Technical Summary
Existing technologies for classifying small-sample chest X-ray images suffer from problems such as large data volume, low model generalization ability, and insufficient classification accuracy. In particular, they are not effective in identifying rare and novel diseases and fail to make effective use of query set sample information.
We employ a few-sample medical image processing method based on dual similarity measurement and joint learning. By constructing a dual similarity measurement module and a target optimizer, we utilize the feature vectors of the support set and query set for bidirectional prediction and loss constraints to improve the classification accuracy of the model.
The dual similarity measurement module reduces similarity bias, improves the model's generalization ability, and utilizes query set sample information to enhance the classification accuracy of chest diseases.
Smart Images
Figure CN117809058B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of medical image processing, specifically relating to a small-sample medical image processing method based on dual similarity measurement and joint learning. Background Technology
[0002] Few-shot learning is a machine learning method that uses a limited number of labeled images to learn from image information and solve problems. Unlike traditional data-driven algorithms, each category in few-shot learning contains only a few instance samples. Therefore, how to improve the performance and classification accuracy of neural networks using a limited number of samples is the focus of its research. Looking at domestic and international research on the classification of diseases (especially chest diseases) with few samples, existing methods can be broadly divided into two categories: data augmentation and metric learning methods. Data augmentation methods can be further divided into data augmentation based on geometric transformations and data augmentation based on deep generative models. The former augments chest X-ray image data through image transformations, while the latter generates new pseudo-labeled chest X-ray image data through a generator. Metric learning methods mainly utilize various distance functions to calculate the similarity between chest diseases.
[0003] In recent years, given the end-to-end training advantages of deep neural networks, researchers have often combined them with simple and effective metric learning, widely applying them to small-sample chest disease classification tasks. The combined deep metric learning directly predicts the query sample label by comparing the similarity between support samples and query samples, thereby improving the model's performance in small-sample scenarios.
[0004] While researchers have made significant progress in classifying diseases using small-sample chest X-ray images, some challenges remain, including the following:
[0005] (1) Existing methods artificially increase the number of positive samples for chest diseases, but they are unable to cope with rare or novel diseases with already scarce sample sizes.
[0006] (2) Current small-sample chest disease classification work based on metric learning only establishes a single similarity distance metric for comparison, which leads to bias in similarity and reduces the generalization ability of the model.
[0007] (3) Current work only learns information from chest X-ray images to support the information of the sample set in order to predict the label of the sample set for the query set, without making use of information from the equally effective sample set for the query set, which limits the performance of the model.
[0008] Chest disease assisted diagnosis is one of the important applications of deep learning technology in the field of medical imaging. Due to the difficulties in obtaining samples, high annotation costs, and long-tailed distribution of chest X-ray image datasets, research on chest X-ray image classification for small samples is of practical significance. Summary of the Invention
[0009] The technical problem to be solved by this invention is to provide a small-sample medical image processing method based on dual similarity measurement and joint learning, which solves the problems of large data volume, low model generalization ability and inaccurate classification in existing X-ray image processing technologies.
[0010] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:
[0011] A small-sample medical image processing method based on dual similarity measurement and joint learning includes the following steps:
[0012] Step 1: First, divide the input sample image dataset into a training set and a test set, and then construct a support set and a query set for the training set and the test set respectively;
[0013] Step 2: During the training phase, feature vectors are first extracted from the support set and query set samples of the training set using a feature extractor. Then, a dual similarity measurement module is constructed to measure and constrain the feature vectors of the support set samples, so that samples of the same class are highly similar and samples of different classes are far apart. Finally, the average feature vector of each class in the support set samples is calculated as a prototype to predict the label of the query set samples.
[0014] Step 3: Perform small-sample joint learning on the model. The predicted labels of the query set and the query set samples form sample-label pairs. Similarly, the class prototype is obtained through the double similarity measurement module. Based on the query set prototype, the support set sample labels are predicted in reverse. The support set sample labels and the true labels of the support set samples obtained by the inverse prediction are used together with the predicted query set sample labels and the true labels of the query set samples to train the objective optimizer.
[0015] Step 4: In the testing phase, feature vectors are extracted from the support set and query set samples of the test set using a feature extractor. The feature vectors of the support set samples are used to construct a prototype based on the dual similarity measurement module. The query set samples are compared with the prototype to obtain preliminary predicted labels. The preliminary predicted labels are then processed by the target optimizer to obtain the final predicted labels.
[0016] In the feature space, the feature vectors are subjected to a loss constraint, the cosine similarity is calculated for samples of the same class, and the Euclidean distance is calculated for the prototypes of each class, so that the intra-class similarity is high and the inter-class differences are large.
[0017] The formula for calculating cosine similarity is as follows:
[0018]
[0019] The loss function is:
[0020]
[0021] Among them, F i ,F j Let i be the feature vectors of sample i and sample j, where i ≠ j, and n be the number of sample pairs within the class.
[0022] The formula for calculating Euclidean distance is as follows:
[0023]
[0024] The loss function is:
[0025]
[0026] Among them, F a ,F b Let a be the feature vectors of prototype a and prototype b, where a ≠ b, m is the number of inter-class sample pairs, and k is the length of the feature vector.
[0027] The target optimizer consists of three fully connected layers, with a ReLU activation function between each pair of layers.
[0028] During the training phase, support set labels are established. The true label y of the sample, and the optimized support set label. Loss constraints between Create query prediction set labels The true label y′ of the query set sample and the optimized query set label Loss constraints between Among them, L C This is the cross-entropy loss function.
[0029] The feature extractor is a ConvNeXt model, and sample I is processed by the feature extractor to obtain a feature vector:
[0030] F = f cnn (I;φ cnn )∈R d
[0031] Where, φ cnn Here, d represents the model parameters, and d represents the length of the feature vector.
[0032] To further address the problem of low classification accuracy in small samples, this invention also provides a method and system for processing small-sample medical images, the specific technical solution of which is as follows:
[0033] A small-sample medical image processing system based on dual similarity measurement and joint learning includes an image storage unit, an image feature extractor, a dual similarity measurement module, and a target optimizer; among which,
[0034] The image storage unit stores several image data for training and testing the model. The training phase uses a training set, and the testing phase uses a test set. The training set and the test set are composed of a meta-task set, and each meta-task contains a set of support set samples and a set of query set samples.
[0035] Image feature extractors are used to capture image features and transform raw image data into feature data with representative information.
[0036] The dual similarity measurement module is used to obtain more representative prototypes;
[0037] The target optimizer is used to learn the difference between the predicted label and the true label of a sample.
[0038] The small-sample medical image processing method first acquires several X-ray images of medical diseases and corresponding normal X-ray images of the body parts, and then filters out the images that meet the requirements. Next, the method described above is applied to process the filtered images to obtain the classification results of all images. Finally, the final classification results are output as the basis or guide for disease judgment.
[0039] A computer storage medium storing computer instructions, which, when invoked, are used to execute all or part of the steps of the method.
[0040] Compared with the prior art, the present invention has the following beneficial effects:
[0041] 1. A dual similarity measurement loss module is proposed, which uses different measurement functions for samples within the same disease class and samples between different disease classes, reducing the similarity bias caused by a single measurement space and improving the generalization ability of the model.
[0042] 2. By using chest X-ray image query set samples to predict support set samples in reverse, the model can fully integrate information from query set samples in the equally effective training set, effectively improving the model's performance.
[0043] 3. A target optimizer was designed to optimize the predicted labels of query set samples during the testing phase by learning the differences between the predicted labels and true labels of support set samples and the predicted labels and true labels of query set samples during the training phase, thereby improving the model's classification accuracy for chest diseases. Attached Figure Description
[0044] Figure 1 This is a structural diagram of the overall model of the small-sample medical image processing method of the present invention. Detailed Implementation
[0045] The structure and working process of the present invention will be further described below with reference to the accompanying drawings.
[0046] To address the shortcomings of existing technologies, this invention conducts in-depth research. First, a meta-learning task is constructed for the dataset, where each task associates and processes a support set S and a query set Q. A sample s from the support set and a sample q from the query set are processed by an image feature extractor f. cnn The eigenvectors F are obtained respectively. s and F q The prototype of the category P s The feature vector F belonging to this category s The mean is taken. In the feature space, for the feature vector F s A loss constraint is implemented, calculating cosine similarity for samples of the same class and Euclidean distance for prototypes of each class. The aim is to achieve high intra-class similarity and large inter-class differences. During the classification phase, the feature vector F... q With prototype P s Similarity calculation is performed to obtain predicted labels. The sample label pair is formed with the query set sample q. Use this as a training sample. Similarly, for the feature vector F q Implement loss constraints for the prototype P q The feature vector F of the support set samples to be predicted s Similarity calculation is performed to obtain the predicted label of s. Will Predicted labels using known true labels y and query set samples A target optimizer is trained together with the true label y′. Make Similarly, this makes This improves the prediction accuracy of the query set sample q. Using the method of this invention, the model provides a general and effective auxiliary diagnostic method for disease classification of small-sample chest X-ray images by jointly learning a dual similarity metric and support set query set samples.
[0047] A small-sample medical image processing method based on dual similarity measurement and joint learning includes the following steps:
[0048] Step 1: First, divide the input sample image dataset into a training set and a test set, and then construct a support set and a query set for the training set and the test set respectively;
[0049] Step 2: During the training phase, feature vectors are first extracted from the support set and query set samples of the training set using a feature extractor. Then, a dual similarity measurement module is constructed to measure and constrain the feature vectors of the support set samples, so that samples of the same class are highly similar and samples of different classes are far apart. Finally, the average feature vector of each class in the support set samples is calculated as a prototype to predict the label of the query set samples.
[0050] Step 3: Perform small-sample joint learning on the model. The predicted labels of the query set and the query set samples form sample-label pairs. Similarly, the class prototype is obtained through the double similarity measurement module. Based on the query set prototype, the support set sample labels are predicted in reverse. The support set sample labels and the true labels of the support set samples obtained by the inverse prediction are used together with the predicted query set sample labels and the true labels of the query set samples to train the objective optimizer.
[0051] Step 4: In the testing phase, feature vectors are extracted from the support set and query set samples of the test set using a feature extractor. The feature vectors of the support set samples are used to construct a prototype based on the dual similarity measurement module. The query set samples are compared with the prototype to obtain preliminary predicted labels. The preliminary predicted labels are then processed by the target optimizer to obtain the final predicted labels.
[0052] Specific embodiments, such as Figure 1 As shown,
[0053] To construct a more representative prototype, cosine similarity was used to constrain intra-class samples, and Euclidean distance was used to constrain inter-class samples. A mechanism for a dual similarity loss measure was also presented. Since the support set and query set play equally important roles, the predicted query set sample q is used to predict the support set sample s, and the predicted support set sample labels are set accordingly. Predicted labels based on the true label y and the query set samples The target optimizer is trained together with the true label y′. Make The loss between y and y is not less than The loss between y and y; similarly, the predicted labels for the query set samples. The loss between the actual label y′ and the true label y′ is no less than The loss between y′ and y′ improves the prediction accuracy of the query set sample q.
[0054] The complete technical solution of this patent is as follows: Figure 1 As shown. During the training phase, firstly, the support set samples s are processed by the feature extractor f. cnn The eigenvector F is then obtained. s F s A category prototype P is constructed using a dual similarity measurement module. s , and the feature vector F of the query set samples qSimilarity calculation is performed to obtain the predicted tags for the query set. Then, the roles of the query set and the support set are reversed. The support set samples are then used as the backpropagation support set samples to predict the labels from the query set samples. Finally, the labels predicted from the support set samples are used to predict the support set samples. The labels predicted from the true label y and the query set samples The true label y′ is input into the target optimizer. In China, learning With y and The difference between y and y′ allows the model to effectively improve sample classification accuracy during the testing phase. Specifically, the complete solution of this patent mainly includes the following three parts:
[0055] (1) Dual Similarity Measurement Module
[0056] For the input chest image I, after passing through feature extractor f cnn The eigenvector F = f is then obtained. cnn (I;φ cnn )∈R d , where: f cnn (·) represents a convolutional neural network model, specifically the ConvNeXt network model, φ cnn Here, d represents the model parameters, specifically the feature vector length, which is 768. After the feature vector F enters the dual similarity measurement module, it considers both intra-class and inter-class constraints. Cosine similarity is used for intra-class constraints. The loss function is used to calculate the similarity between samples within the same category. The goal is to promote similarity between samples of the same class, where n represents the number of sample pairs. Inter-class constraints use Euclidean distance. Calculating the distance between different categories requires creating a representative vector, or category prototype, for each category. In this invention, the average vector of each category is taken as the category prototype, and the loss function is... The goal is to increase the distance between samples from different classes, where m is the number of inter-class sample pairs. After the support set feature vectors pass through the dual similarity measurement module, the average vector for each class is also calculated, resulting in the class prototype P. s .
[0057] (2) Joint learning with few samples
[0058] During the model training phase, the query set and support set have equal learning value. Therefore, the method of this invention uses the labels predicted by the support set from the query set. Sample-label pairs are formed with the query set samples. Similarly, after passing through the dual similarity measurement module, the reverse prediction supports the set of sample labels.
[0059] (3) Objective Optimizer
[0060] Support set labels obtained through back-prediction Compared with the true label y of the sample and the predicted label of the query set sample Together with the true label y′, a target optimizer is trained. The objective optimizer consists of three fully connected layers, with a ReLU activation function between each pair of layers. go through After obtaining the label y, in order to... Effective learning requires establishing loss constraints. Among them, L C This is the cross-entropy loss function. Similarly, during the model training phase, the query set predicts the label. go through Obtain the optimized predicted labels The relationship between it and the true label y′ of the query set sample is: During the model testing phase, go through Output This will be used as the final classification result of the model.
[0061] The specific algorithm is as follows:
[0062] This invention explores the effectiveness of multiple similarity measures and joint learning in disease classification tasks using chest X-ray images in small-sample scenarios. Based on the constructed meta-learning task, the model first obtains image feature vectors, then predicts the query set against the support set. Next, it uses the predicted labels of the query set and the query set samples to form sample-label pairs to predict the labels of the support set samples. Finally, it trains the objective optimizer based on the difference between the predicted and true labels. The specific algorithm is as follows:
[0063] Input: Support set S and its samples s and corresponding labels y, query set Q and its samples q and corresponding labels y′.
[0064] Output: The classification results of the query set sample q.
[0065] During the training phase, the model uses meta-tasks from the training set:
[0066] ① Support set samples s pass through feature extractor f cnn Obtain the eigenvector F s The feature length is 768. After passing through the double similarity measurement module, the mean value of each category feature vector is calculated to obtain the category prototype P. s ;
[0067] ②Feature vector F of query set sample q q With category prototype Ps Calculate similarity Get the predicted labels of the query set Where D(,) is the Euclidean distance formula;
[0068] ③Predict the labels of the query set The query set samples form sample-label pairs, and the feature vector F q Similarly, after passing through the dual similarity measurement module, the category prototype P is calculated. q Inverse prediction of the labels of the support set samples;
[0069] ④ Predict labels based on the support set The relationship between the actual label y and the query set predicted label The relationship between the training objective optimizer and the true label y′
[0070] During the testing phase, the model uses the meta-tasks from the test set:
[0071] ① Support set samples s pass through feature extractor f cnn Obtain the eigenvector F s After passing through the dual similarity measurement module, the mean value of each category feature vector is calculated to obtain the category prototype P. s ;
[0072] ②Feature vector F of query set sample q q With category prototype P s Calculate similarity to obtain the predicted labels for the query set. Then through the target optimizer The final classification result is obtained.
[0073] Application examples are as follows:
[0074] Using the Chest X-ray14 dataset provided by the NIH Research Institute as the experimental subject, this database contains X-ray images of 14 chest diseases and normal X-ray images without any detected diseases. First, the data was filtered, selecting only single-label samples and dividing it into training and test sets. The training set contained 24,706 images across 9 categories, and the test set contained 3,767 images across 6 categories. The experiment employed a scenario-based training method, setting up 20 rounds of training, each round containing 1,000 tasks. Each task included a support set and a query set. Under a 3-way, 5-shot scenario setting, the method of this patent was used in the experiment, achieving a disease classification accuracy of 49.2% for a small sample of chest X-ray images. A prototype network was used as a benchmark model for comparison; under the same experimental settings, the classification accuracy on the prototype network was 44.5%, verifying the innovation and effectiveness of this invention.
[0075] A small-sample medical image processing system based on dual similarity measurement and joint learning includes an image storage unit, an image feature extractor, a dual similarity measurement module, and a target optimizer; among which,
[0076] The image storage unit stores several image data for training and testing the model. The training phase uses a training set, and the testing phase uses a test set. The training set and the test set are composed of a meta-task set, and each meta-task contains a set of support set samples and a set of query set samples.
[0077] Image feature extractors are used to capture image features and transform raw image data into feature data with representative information.
[0078] The dual similarity measurement module is used to obtain more representative prototypes;
[0079] The target optimizer is used to learn the difference between the predicted label and the true label of a sample.
[0080] The small-sample medical image processing method first acquires several X-ray images of medical diseases and corresponding normal X-ray images of the body parts, and then filters out the images that meet the requirements. Next, the method described above is applied to process the filtered images to obtain the classification results of all images. Finally, the final classification results are output as the basis or guide for disease judgment.
[0081] This method can be applied not only to chest image processing, but also, as an example, to illustrate the scheme in detail. This method can be applied to the processing of any X-ray image.
[0082] A computer storage medium storing computer instructions, which, when invoked, are used to execute all or part of the steps of the method.
[0083] Those skilled in the art should understand that variations can be implemented by combining existing technology and the above embodiments. Such variations do not affect the substantive content of this solution and will not be elaborated here.
[0084] It should be understood that this solution is not limited to the specific embodiments described above. Devices and structures not described in detail herein should be understood as being implemented in a manner common to the art. Any person skilled in the art can make many possible variations and modifications to this solution, or modify it into equivalent embodiments, without departing from the scope of this solution, using the methods and techniques disclosed above. This does not affect the substantive content of this solution. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of this solution, without departing from its scope, still fall within the protection scope of this solution.
Claims
1. A small-sample medical image processing method based on dual similarity measurement and joint learning, characterized in that: Includes the following steps: Step 1: First, divide the input sample image dataset into a training set and a test set, and then construct a support set and a query set for the training set and the test set respectively; Step 2: During the training phase, feature vectors are first extracted from the support set and query set samples of the training set using a feature extractor. Then, a dual similarity measurement module is constructed. In the feature space, the feature vectors of the support set samples are subject to a measurement loss constraint. Cosine similarity is calculated for samples of the same class, and Euclidean distance is calculated for the prototypes of each class. This ensures that samples of the same class have high similarity and samples of different classes are far apart from each other. Finally, the average feature vector of each class in the support set samples is calculated as a prototype to predict the label of the query set samples. Step 3: Perform small-sample joint learning on the model. The predicted labels of the query set and the query set samples form sample-label pairs. Similarly, the class prototype is obtained through the double similarity measurement module. Based on the query set prototype, the support set sample labels are predicted in reverse. The support set sample labels and the true labels of the support set samples obtained by the inverse prediction are used together with the predicted query set sample labels and the true labels of the query set samples to train the objective optimizer. The objective optimizer consists of three fully connected layers, with a ReLU activation function between each pair of layers. During the training phase, support set labels are established. Sample True Labels And optimized support for set tags Loss constraints between Establish query prediction set labels Query set sample real labels and optimized query set tags Loss constraints between ,in, The cross-entropy loss function; Step 4: In the testing phase, feature vectors are extracted from the support set and query set samples of the test set using a feature extractor. The feature vectors of the support set samples are used to construct a prototype based on the dual similarity measurement module. The query set samples are compared with the prototype to obtain preliminary predicted labels. The preliminary predicted labels are then processed by the target optimizer to obtain the final predicted labels.
2. The small-sample medical image processing method based on dual similarity measurement and joint learning according to claim 1, characterized in that: The formula for calculating cosine similarity is as follows: ， The loss function is: in, For the sample and samples eigenvectors, and , This represents the number of sample pairs within the class.
3. The small-sample medical image processing method based on dual similarity measurement and joint learning according to claim 1, characterized in that: The formula for calculating Euclidean distance is as follows: The loss function is: in, As prototype and prototype eigenvectors, and , The number of sample pairs between classes. is the length of the feature vector.
4. The small-sample medical image processing method based on dual similarity measurement and joint learning according to claim 1, characterized in that: The feature extractor is the ConvNeXt model, and the samples... The feature vector is obtained after the feature extractor: in, For model parameters, is the length of the feature vector.
5. A system for implementing the small-sample medical image processing method based on dual similarity measurement and joint learning as described in any one of claims 1 to 4, characterized in that: It includes an image storage unit, an image feature extractor, a dual similarity measurement module, and a target optimizer; among which, The image storage unit stores several image data for training and testing the model. The training phase uses a training set, and the testing phase uses a test set. The training set and the test set are composed of a meta-task set, and each meta-task contains a set of support set samples and a set of query set samples. Image feature extractors are used to capture image features and transform raw image data into feature data with representative information. The dual similarity measurement module is used to obtain more representative prototypes; The target optimizer is used to learn the difference between the predicted label and the true label of a sample.
6. A method for processing small-sample medical images, characterized in that: First, acquire several medical disease X-ray images and corresponding normal X-ray images, and select the images that meet the requirements. Then, process the selected images using the method described in any one of claims 1 to 4 to obtain the classification results of all images. Finally, output the final classification results as the basis or guide for disease judgment.
7. A computer storage medium, characterized in that: The computer storage medium stores computer instructions, which, when invoked, are used to perform all or part of the steps of the method according to any one of claims 1 to 4.