Classification method based on retina image and image classification equipment
By combining multimodal information fusion of color fundus images, fundus OCT images, and clinical metadata, the problems of equipment accessibility and accuracy in the diagnosis of hereditary retinal diseases have been solved, achieving high-precision classification prediction and early screening.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-16
- Publication Date
- 2026-04-07
AI Technical Summary
Existing AI-assisted diagnostic technologies for hereditary retinal diseases suffer from problems such as expensive equipment that is difficult to popularize, insufficient accuracy and interpretability of diagnostic results, and lack of specificity in covering pathogenic gene types, making early detection and accurate diagnosis difficult.
A multimodal information fusion method is adopted, which combines color fundus images, fundus OCT images and patient clinical metadata. Deep fusion classification is performed through multilayer perceptron and metadata model to achieve high-precision classification of hereditary retinal diseases.
It significantly improves the classification accuracy and prediction stability of retinal images, reduces prediction errors and verification costs, and provides an efficient and reliable intelligent diagnostic solution suitable for early screening and personalized decision support for hereditary retinal diseases.
Smart Images
Figure CN121811477A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and more specifically to a classification method and image classification device based on retinal images. Background Technology
[0002] Hereditary retinal diseases are a group of gene-related ophthalmic diseases characterized by diverse inheritance patterns, wide age variations in onset, and complex clinical phenotypes. These diseases typically present as progressive visual function decline, and in severe cases, can lead to irreversible blindness. Currently, although molecular genetic testing has become an important diagnostic tool, its high cost, long processing time, and the fact that some pathogenic genes remain undiscovered mean that initial clinical diagnosis still relies primarily on imaging examinations and physician experience. Improving the early detection rate and accurate diagnosis of hereditary retinal diseases is a major challenge in the field of ophthalmology.
[0003] With the development of artificial intelligence technology, especially deep learning and large-scale modeling, medical image analysis has shown great potential in assisted diagnosis. By automatically learning pathological features in images, more objective and stable results can be achieved while reducing the burden on doctors.
[0004] However, existing AI-assisted diagnostic technologies for hereditary retinal diseases still have at least the following problems: First, some methods rely on specific imaging equipment (such as fluorescein autofluorescence imaging), which is expensive and difficult to popularize in primary healthcare institutions, thus limiting the scope of technology promotion and application.
[0005] Second, most existing models rely solely on image features for training, which fails to effectively simulate clinical diagnostic approaches, thus limiting the accuracy and interpretability of diagnostic results.
[0006] Third, while some methods cover a wide range of pathogenic gene types, they lack specificity, have limited clinical value, and their overly broad prediction range can easily introduce errors. Summary of the Invention
[0007] The purpose of this invention is to provide a classification method and image classification device based on retinal images. When classifying patient images, this method integrates the patient's color fundus images, fundus OCT images, and key clinical metadata. The entire classification process is completed based on multimodal data fusion, thereby achieving high-precision classification prediction. Furthermore, it can focus on key aspects for prediction, reducing prediction errors and verification costs. Thus, through deep fusion and integrated optimization of multimodal information, it not only significantly improves the classification accuracy and prediction stability of retinal images but also considers clinical practicality and universality. It provides an efficient, reliable, and scalable intelligent solution that meets the clinical application needs for early screening, typing, and personalized decision support for hereditary retinal diseases.
[0008] To achieve the above objectives, the present invention provides a classification method based on retinal images, comprising: Acquire color fundus images and fundus OCT images of the patient to be classified; The first image classification model is used to classify the color fundus image to obtain the first classification result, and the second image classification model is used to classify the fundus OCT image to obtain the second classification result; The combination of the first classification result and the patient's clinical metadata is input into the multimodal fusion classification module for classification to obtain the third classification result, and the combination of the second classification result and the clinical metadata is input into the multimodal fusion classification module for classification to obtain the fourth classification result; The first predicted category result obtained by weighted fusion of the first classification result and the second classification result is used as the prediction result of the image; the fifth classification result obtained by weighted fusion of the first classification result and the third classification result, the sixth classification result obtained by weighted fusion of the second classification result and the fourth classification result, the second prediction result obtained by weighted fusion of the fifth classification result and the sixth classification result, and the second prediction result is used as the final prediction result.
[0009] The present invention also provides an image classification device, comprising: at least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the above-described classification method based on retinal images.
[0010] The present invention also provides a computer-readable storage medium, which is a non-volatile or non-transient storage medium, on which a computer program is stored, which, when executed by a processor, performs the steps of the retinal image-based classification method described above.
[0011] In one embodiment, the multimodal fusion classification module includes: a metadata model and a multilayer perceptron; The combination of the first classification result and the patient's clinical metadata is input into the multimodal fusion classification module for classification to obtain a third classification result, and the combination of the second classification result and the clinical metadata is input into the multimodal fusion classification module for classification to obtain a fourth classification result, including: The first classification result is fused with the clinical metadata and then input into the metadata model. The metadata model outputs a first feature vector to the multilayer perceptron, and the multilayer perceptron outputs a third classification result. The second classification result is fused with the clinical metadata and then input into the metadata model. The metadata model outputs a second feature vector to the multilayer perceptron, and the multilayer perceptron outputs a fourth classification result.
[0012] In one embodiment, the metadata model includes: an XGBoost classifier, a LightGBM classifier, and a K-nearest neighbor classifier; The first feature vector output by the metadata model includes: feature vectors obtained by the XGBoost classifier, the LightGBM classifier, and the K-nearest neighbor classifier respectively processing the data after fusing the first classification result with the clinical metadata; The second feature vector output by the metadata model includes: feature vectors obtained by the XGBoost classifier, the LightGBM classifier, and the K-nearest neighbor classifier respectively processing the data after fusing the second classification result with the clinical metadata.
[0013] In one embodiment, the method further includes: The classification result obtained by weighted fusion of the first classification result and the third classification result, combined with the clinical metadata, and the classification result obtained by weighted fusion of the second classification result and the fourth classification result, combined with the clinical metadata, are respectively used as inputs to the multimodal fusion classification module to perform at least one round of stacked updates on the metadata model.
[0014] In one embodiment, both the first image classification model and the second image classification model are VisionTransformer models.
[0015] In one embodiment, the clinical metadata includes at least one of the following: Gender, family history, age, course of disease, and age of onset.
[0016] In one instance, the weights of the first classification result and the third classification result (the second classification result and the fourth classification result) are the weights of the first image classification model, the second image classification model, and the multimodal fusion classification module determined by grid search.
[0017] In one embodiment, the weight of the first classification result (second classification result) is 0.974, and the weight of the third classification result (fourth classification result) is 0.026. Attached Figure Description
[0018] Figure 1 This is a flowchart illustrating the classification method based on retinal images in the first embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the classification process implemented by the first image classification model and the second image classification model in the first embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the classification process implemented by the multimodal fusion classification module in the first embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the stacking and updating of the metadata model of the multimodal fusion classification module in the first embodiment of the present invention. Detailed Implementation
[0019] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings to provide a clearer understanding of the purpose, features, and advantages of the present invention. It should be understood that the embodiments shown in the drawings are not intended to limit the scope of the present invention, but are merely illustrative of the essential spirit of the technical solution of the present invention.
[0020] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments in order to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the art will recognize that embodiments may be practiced without one or more of these specific details. In other instances, well-known apparatuses, structures, and techniques associated with this application may not have been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.
[0021] Unless the context requires otherwise, throughout the specification and claims, the word “comprising” and its variations, such as “including” and “having”, shall be understood to have an open, inclusive meaning, that is, to be interpreted as “including, but not limited to”.
[0022] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the appearance of "in an embodiment" or "an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any manner in one or more embodiments.
[0023] The singular forms “a” and “the” used in this specification and the appended claims include plural references unless otherwise expressly stated herein. It should be noted that the term “or” is generally used to include the meaning of “or / and” unless otherwise expressly stated herein.
[0024] In the following description, in order to clearly demonstrate the structure and working method of the present invention, a number of directional terms will be used. However, terms such as "front", "back", "left", "right", "outer", "inner", "outer", "inner", "up", and "down" should be understood as convenient terms and not as limiting terms.
[0025] The first embodiment of the present invention relates to a classification method based on retinal images, which is applied to an image classification device. The image classification device has a built-in image classification model (including a first image classification model and a second image classification model) and a multimodal fusion classification module to determine the probability of a patient's retinal image belonging to various genotype categories. Common genotype categories include, but are not limited to: autosomal dominant inheritance, autosomal recessive inheritance, X-linked inheritance, etc.
[0026] The specific process of the retinal image-based classification method in this embodiment is as follows: Figure 1 As shown.
[0027] Step 101: Obtain the patient's color fundus image and fundus OCT image to be classified.
[0028] Specifically, color fundus images (CFP) are captured using a fundus image acquisition device, and fundus OCT images are acquired using an OCT device (Optical Coherence Tomography). This allows for the acquisition of both color fundus images and fundus OCT images of the same patient for classification.
[0029] Step 102: Classify the color fundus image using the first image classification model to obtain the first classification result, and classify the fundus OCT image using the second image classification model to obtain the second classification result.
[0030] Specifically, the first image classification model and the second image classification model are two models trained independently with the same network structure. The first image classification model is used to extract depth features and predict classification of color fundus images, while the second image classification model is used to extract depth features and predict classification of fundus OCT images.
[0031] For example, both the first image classification model and the second image classification model are VisionTransformer models. Specifically, the VisionTransformer model, serving as the first and second image classification models, includes: an image patch embedding module, an encoder, and a classification head (MLP multilayer perceptron). The image patch embedding module divides the input image into several image patches, for example, dividing the entire image into multiple 16x16 image patches, mapping each image patch to an embedding vector, and adding learnable positional encoding. The embedding vectors of each image patch are input to the encoder, which consists of a multi-head self-attention mechanism and a feed-forward network, used to capture global dependencies between image patches. For example, the 24 Transformer modules included in the VisionTransformer model are used to represent the features of the image patches, and then global average pooling is used to aggregate the features of all image patches, concatenating them to obtain a unified feature vector. This feature vector is input to the classification head, which performs classification prediction and outputs the classification result, including the probability that the image belongs to each genotype category.
[0032] Please refer to Figure 2 It can be seen that color fundus images and fundus OCT images collected from the same patient at the same time period (representing that the patient's eyes are in the same state) are respectively input into the first image classification model and the second image classification model; the first image classification model classifies the color fundus images to obtain the first classification result, and the second image classification model classifies the fundus OCT images to obtain the second classification result.
[0033] Furthermore, the first predicted category result obtained by fusing the first classification result and the second classification result includes the probability that the image belongs to each genotype category. For example, the first classification result and the second classification result can be fused by weighting, which can be an average weighting, manually configuring the weights of the two image classification models, or determining the weights of the first image classification model and the second image classification model through grid search. In other embodiments, the first image classification model and the second image classification model are fused using a model ensemble approach. That is, the first image classification model and the second image classification model are used as two base models, and then these two base models are integrated into a new model for fusing the first classification result and the second classification result.
[0034] Both the first and second image classification models were trained, and their training methods and processes were similar. The main difference lay in the images used for training: the first image classification model used color fundus images, while the second image classification model used fundus OCT images. The training process is explained below using the first image classification model as an example: Multiple samples are obtained from several patients. Each sample includes a color fundus image sample and the genotype category to which the color fundus image sample belongs. From all these samples, the majority (e.g., 80%) is selected as training samples, and the remaining portion (e.g., 20%) is selected as validation samples. A training set is constructed using the selected training samples, and a validation set is constructed using the selected validation samples. Furthermore, a portion of the samples can be reserved as test samples, and a test set is constructed from the selected test samples.
[0035] In one example, data augmentation can be performed on the acquired color fundus image samples to improve the model's robustness and generalization ability. This includes brightness adjustment, contrast enhancement, rotation, scaling, and random erasure. The augmented color fundus image samples are normalized and adjusted to a resolution of 224×224 pixels before being input into the network. In subsequent testing phases, the input images are centered and cropped to 224×224 pixels, using the same normalization parameters as during training to ensure consistency between training and testing.
[0036] Construct a first image classification model that takes a color fundus image as input and the probability of the color fundus image belonging to each genotype category. The first image classification model can be the Vision Transformer model mentioned above.
[0037] The first image classification model is trained using multiple training samples in the training set to obtain a first image classification model that meets the conditions.
[0038] During the training phase, the first image classification model is trained for a total of 50 epochs with a batch size of 32. In the first 10 epochs, a learning rate warm-up strategy is used, linearly increasing the learning rate from 0 to 5×10⁻⁴. Then, in the remaining 40 epochs, cosine annealing is used to gradually decrease the learning rate from 5×10⁻⁴ to 1×10⁻⁶. After each epoch, the performance of the first image classification model on the validation set is evaluated, and the weights with the best performance are saved for subsequent evaluation on the test set.
[0039] As described above, different image classification models with the same structure (i.e., two image classification model encoders with non-shared weights) are used for image data of different modalities (i.e., color fundus images and fundus OCT images), which can ensure the training stability of the image classification models of each modality. By extracting image features in advance, the consistency and reliability of the obtained features can be ensured, providing a stable feature foundation for subsequent multimodal fusion and classification prediction.
[0040] Step 103: Input the combination of the first classification result and the patient's clinical metadata into the multimodal fusion classification module to obtain the third classification result, and input the combination of the second classification result and the clinical metadata into the multimodal fusion classification module to obtain the fourth classification result.
[0041] Specifically, the patient's clinical metadata represents several personal attribute parameters, including at least one of the following: gender, family history characteristics, age, disease duration, and age of onset. Gender is divided into male and female, and can be represented by 1 and 0; family history characteristics can include having a family history (represented by 1) and not having a family history (represented by 0); age, disease duration, and age of onset are represented by specific numbers.
[0042] Please refer to Figure 3 The multimodal fusion classification module classifies the combined data of the first classification result (corresponding to the color fundus image) and the patient's clinical metadata to obtain the third classification result; the multimodal fusion classification module classifies the combined data of the second classification result (corresponding to the fundus OCT image) and the patient's clinical metadata to obtain the fourth classification result.
[0043] In one example, the multimodal fusion classification module includes: a metadata model and a multilayer perceptron; the metadata model processes the combined data of the first classification result and the patient's clinical metadata to extract a first feature vector, and processes the combined data of the second classification result and the patient's clinical metadata to extract a second feature vector; the multilayer perceptron acts as the classification head, performing classification predictions based on the first and second feature vectors respectively. Specifically: The first classification result is combined with the patient's clinical metadata and input into the multimodal fusion classification module for classification to obtain a third classification result. The second classification result is combined with the clinical metadata and input into the multimodal fusion classification module for classification to obtain a fourth classification result, including: The first classification result is fused with the clinical metadata and then input into the metadata model. The metadata model outputs a first feature vector to the multilayer perceptron, and the multilayer perceptron outputs a third classification result. The second classification result is fused with the clinical metadata and then input into the metadata model. The metadata model outputs a second feature vector to the multilayer perceptron, and the multilayer perceptron outputs a fourth classification result.
[0044] In one embodiment, the metadata model includes: an XGBoost classifier, a LightGBM classifier, and a K-nearest neighbor classifier; For the combined data of the first classification result and the patient's clinical metadata, the XGBoost classifier, LightGBM classifier, and K-nearest neighbor classifier will all process the input data and output feature vectors respectively. The first feature vector output by the metadata model includes the feature vectors obtained by the XGBoost classifier, the LightGBM classifier, and the K-nearest neighbor classifier respectively processing the data after the first classification result and the fused clinical metadata. That is, the first feature vector includes the three feature vectors obtained by the three classifiers respectively processing the input data.
[0045] Similarly, for the combined data of the input second classification result and the patient's clinical metadata, the XGBoost classifier, LightGBM classifier, and K-nearest neighbor classifier will all process the input data and output feature vectors respectively. The second feature vector output by the metadata model includes: the feature vectors obtained by the XGBoost classifier, the LightGBM classifier, and the K-nearest neighbor classifier respectively processing the data after the second classification result and the fusion of the clinical metadata. That is, the second feature vector includes: the three feature vectors obtained by the three classifiers respectively processing the input data.
[0046] In the metadata model, the XGBoost classifier uses the Dropouts meet Additive RegressionTrees (DART) algorithm combined with L2 regularization to enhance the model's generalization ability; the LightGBM classifier sets a predefined maximum number of iterations and combines L1 regularization to control model complexity and reduce overfitting; the K nearest neighbor classifier sets the number of neighbors to 9 to improve the stability of local classification.
[0047] Therefore, in the multimodal fusion classification module, a multi-classifier integration and multilayer perceptron feature integration strategy is adopted, and the prediction results are optimized through an iterative weighted fusion mechanism to enhance the model's ability to represent multi-source heterogeneous classification result data and clinical metadata, thereby reducing the dependence on specific imaging equipment.
[0048] Furthermore, the second predicted category result is obtained by fusing the third classification result with the fourth classification result. The second predicted category result includes the probability that the image belongs to each genotype category. For example, the third classification result and the fourth classification result can be fused by weighting, which can be by average weighting or by manually configuring the weights of the two image classification models.
[0049] The multimodal fusion classification module was also trained, and the specific training process is as follows: For example, multiple samples from several patients can be obtained. Each sample includes: a first classification result sample, a second classification result sample, the patient's clinical metadata, and the patient's genotype category. The first classification result sample can be obtained by inputting the patient's color fundus image sample into a first image classification model, and the second classification result sample can be obtained by inputting the patient's fundus OCT image sample into a second image classification model.
[0050] From all these samples, a majority (e.g., 80%) is selected as training samples, and the remaining portion (e.g., 20%) is used as validation samples. A training set is constructed using the selected training samples, and a validation set is constructed using the selected validation samples. Furthermore, a portion of the samples can be reserved as test samples, and a test set is constructed from these selected test samples. The features of each sample are standardized, and the validation and test sets use the same standardized parameters as the training set to ensure a consistent input distribution.
[0051] In one example, to reduce noise from missing values and maintain the original data distribution, different missing value imputation strategies were employed based on the feature types in the patients' clinical metadata. All patients' clinical metadata was arranged, and for gender and family history characteristics, the mode of each column was used for imputation. For age, disease duration, and age of onset, a step-by-step imputation method was used: first, missing values were imputed using the mean of the age column; then, by comparing the mean of the disease duration column with the age of the corresponding sample, if the mean was greater than the individual age, the age value was used for imputation; otherwise, the column mean was used; finally, missing values for age of onset were calculated by subtracting disease duration from age. After imputation, age, disease duration, and age of onset were normalized to ensure the consistency of these variables.
[0052] A multimodal fusion classification module consisting of a metadata model and a multilayer perceptron is constructed. The metadata model includes an XGBoost classifier, a LightGBM classifier, and a K-nearest neighbor classifier. In the metadata model, the `n_estimators` parameter of the XGBoost classifier is set to 21, and L2 regularization is used to control model complexity and reduce the risk of overfitting. The LightGBM classifier is set to increase the number of iterations to 33, with a maximum of 8 leaf nodes per tree, and L1 regularization is used. The KNN classifier uses 9 neighbors to balance the sensitivity to local patterns and the generalization ability of the overall feature space. The multilayer perceptron has a single hidden layer containing 100 neurons, and the initial learning rate is set to 0.0186 to accelerate optimization convergence.
[0053] The constructed multimodal fusion classification module is trained using multiple training samples in the training set to obtain a multimodal fusion classification module that meets the conditions.
[0054] Step 104: The first predicted category result obtained by weighted fusion of the first classification result and the second classification result is used as the predicted result for the image; please refer to... Figure 3 The fifth classification result is obtained by weighted fusion of the first classification result and the third classification result. The sixth classification result is obtained by weighted fusion of the second classification result and the fourth classification result. The second prediction result is obtained by weighted fusion of the fifth classification result and the sixth classification result. The second prediction result is the final prediction result.
[0055] Specifically, the first predicted category result obtained through fusion is used as the prediction result for the multimodal image, the second predicted result is the final prediction result, and the final classification result indicates the probability that the patient image finally belongs to each genotype category.
[0056] For example, the first classification result and the third category result (the second classification result and the fourth classification result) can be fused by weighting. This can be done by average weighting, manually configuring the weights of the two predicted category results, or determining the weights of the first and third classification results through grid search. In other embodiments, the weights of the first and third classification results (the second and fourth classification results) are: the weights of the first image classification model, the metadata classification model, and the multimodal fusion classification module determined by grid search. Specifically, a grid search method is used to assign weights to the first and third classification results (and the second and fourth classification results) of the image classification model (including the first and second image classification models) and the multimodal fusion classification module. For example, for the finally trained image classification model and multimodal fusion classification module, the corresponding test data is used to test the two image classification models and the multimodal fusion classification module respectively. Based on the accuracy of the first and second classification results obtained by the image classification model and the third and fourth classification results obtained by the multimodal fusion classification module, the weights of the first and third classification results and the weights of the second and fourth classification results are determined. The fifth and sixth classification results are calculated, and the fifth and sixth classification results are weighted and fused to determine the second prediction result as the final prediction result.
[0057] For example, the weight of the first classification result is 0.974 and the weight of the third classification result is 0.026.
[0058] In this embodiment, please refer to Figure 4 Furthermore, the classification result obtained by weighted fusion of the first classification result and the third classification result can be combined with the clinical metadata, and the classification result obtained by weighted fusion of the second classification result and the fourth classification result can be combined with the clinical metadata, respectively, as inputs to the multimodal fusion classification module to perform at least one round of stacked updates on the metadata model.
[0059] This step can be performed later in the training process of the image classification model (including the first image classification model and the second image classification model) and the multimodal fusion classification module. Specifically, it involves weighted fusion of the single-modality (color fundus image and fundus OCT image) category results predicted by the image classification model and the multimodal fusion classification module, concatenating this fusion with the original clinical metadata to form a new input, which is then fed into the multimodal fusion classification module for iterative optimization. Through repeated feature extraction and multi-layer weighted fusion, the feature representation capability of the metadata model can be enhanced, thereby gradually improving the final prediction performance and obtaining the optimal parameter set.
[0060] The above adopts a stacking fusion strategy, and determines the optimal fusion weights and stacking layers through grid search. During the fusion process, the prediction results of each image classification model (including the first image classification model and the second image classification model) and the multimodal fusion classification module are weighted and combined according to their weights to generate the final classification output, thereby improving the final prediction accuracy and stability.
[0061] In this embodiment of the retinal image-based classification method, color fundus images and fundus OCT images are classified separately using an image classification model to obtain two classification results. These two classification results are then fused to obtain a first predicted category result, which serves as the image's prediction result. After fusing these two classification results with the patient's key clinical metadata, they are input into a multimodal fusion classification module to obtain two more classification results. These two classification results are then weighted according to modal correspondence with the two original classification results to obtain two new classification results. The two newly obtained classification results are then weighted to obtain a second prediction result, which is the final classification result. In other words, when classifying a patient's images, the patient's color fundus images, fundus OCT images, and key clinical metadata are fused. The entire classification process is completed based on multimodal data fusion, thereby achieving high-precision classification prediction. Furthermore, prediction can be performed only on key aspects, reducing prediction errors and verification costs.
[0062] Thus, through the deep fusion and integrated optimization of multimodal information, not only is the classification accuracy and prediction stability of retinal images significantly improved, but clinical practicality and universality are also taken into account. It also provides an efficient, reliable and scalable intelligent solution to meet the clinical application needs of early screening, typing and personalized auxiliary decision-making for hereditary retinal diseases.
[0063] The second embodiment of the present invention relates to an image classification device, such as a computer, laptop computer, or other electronic device with computing capabilities.
[0064] An image classification device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a retinal image-based classification method as described in the first embodiment.
[0065] The third embodiment of the present invention relates to a computer-readable storage medium, which is a non-volatile or non-transient storage medium, on which a computer program is stored. When the computer program is run by a processor, it performs the steps of the retinal image-based classification method in the first embodiment.
[0066] The preferred embodiments of the present invention have been described in detail above, but it should be understood that, if necessary, aspects of the embodiments can be modified to utilize aspects, features, and concepts from various patents, applications, and publications to provide other embodiments.
[0067] In light of the detailed description above, these and other changes can be made to the embodiments. Generally, the terminology used in the claims should not be considered limited to the specific embodiments disclosed in the specification and claims, but should be understood to include all possible embodiments together with the full scope of equivalents enjoyed by these claims.
Claims
1. A classification method based on retinal images, characterized in that, include: Acquire color fundus images and fundus OCT images of the patient to be classified; The first image classification model is used to classify the color fundus image to obtain the first classification result, and the second image classification model is used to classify the fundus OCT image to obtain the second classification result; The combination of the first classification result and the patient's clinical metadata is input into the multimodal fusion classification module for classification to obtain the third classification result, and the combination of the second classification result and the clinical metadata is input into the multimodal fusion classification module for classification to obtain the fourth classification result; The first predicted category result obtained by weighted fusion of the first classification result and the second classification result is used as the prediction result of the image; The fifth classification result is obtained by weighted fusion of the first classification result and the third classification result. The sixth classification result is obtained by weighted fusion of the second classification result and the fourth classification result. The second prediction result is obtained by weighted fusion of the fifth classification result and the sixth classification result. The second prediction result is used as the final prediction result.
2. The classification method based on retinal images according to claim 1, characterized in that, The multimodal fusion classification module includes: a metadata model and a multilayer perceptron; The combination of the first classification result and the patient's clinical metadata is input into the multimodal fusion classification module for classification to obtain a third classification result, and the combination of the second classification result and the clinical metadata is input into the multimodal fusion classification module for classification to obtain a fourth classification result, including: The first classification result is fused with the clinical metadata and then input into the metadata model. The metadata model outputs a first feature vector to the multilayer perceptron, and the multilayer perceptron outputs a third classification result. The second classification result is fused with the clinical metadata and then input into the metadata model. The metadata model outputs a second feature vector to the multilayer perceptron, and the multilayer perceptron outputs a fourth classification result.
3. The classification method based on retinal images according to claim 2, characterized in that, The metadata model includes: XGBoost classifier, LightGBM classifier, and K nearest neighbor classifier; The first feature vector output by the metadata model includes: feature vectors obtained by the XGBoost classifier, the LightGBM classifier, and the K-nearest neighbor classifier respectively processing the data after fusing the first classification result with the clinical metadata; The second feature vector output by the metadata model includes: feature vectors obtained by the XGBoost classifier, the LightGBM classifier, and the K-nearest neighbor classifier respectively processing the data after fusing the second classification result with the clinical metadata.
4. The classification method based on retinal images according to claim 2, characterized in that, The method further includes: The classification result obtained by weighted fusion of the first classification result and the third classification result, combined with the clinical metadata, and the classification result obtained by weighted fusion of the second classification result and the fourth classification result, combined with the clinical metadata, are respectively used as inputs to the multimodal fusion classification module to perform at least one round of stacked updates on the metadata model.
5. The classification method based on retinal images according to claim 1, characterized in that, Both the first image classification model and the second image classification model are Vision Transformer models.
6. The classification method based on retinal images according to claim 1, characterized in that, The clinical metadata includes at least one of the following: Gender, family history, age, course of disease, and age of onset.
7. The classification method based on retinal images according to claim 1, characterized in that, The weights of the first predicted category result and the second predicted category result are: the weights of the first image classification model, the second image classification model, and the multimodal fusion classification module determined by grid search.
8. The classification method based on retinal images according to claim 1 or 7, characterized in that, The weight of the first classification result is 0.974, the weight of the third classification result is 0.026, the weight of the second classification result is 0.974, and the weight of the fourth classification result is 0.
026.
9. An image classification device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the retinal image-based classification method as described in any one of claims 1 to 8.
10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores processor-executable instructions configured to enable the processor of the autonomous operating device to perform the retinal image-based classification method as described in any one of claims 1 to 8.