A fundus image classification method, device, equipment and medium

Abstract

The application discloses an eye fundus image classification method and device, equipment and medium, and relates to the technical field of medical image processing. The method comprises the following steps: generating a training set based on an eye fundus image, performing feature enhancement and rotation operation on the training set to obtain a rotation enhanced image; performing feature extraction and training operation on the rotation enhanced image to obtain multi-view instance recognition task representation features and rotation prediction task representation features, and calculating instance loss functions and rotation loss functions; adjusting model parameters by using the instance loss functions and the rotation loss functions, generating a model adjustment loss function, performing self-supervised learning of the eye fundus disease diagnosis classification of the model adjustment loss function by using a K nearest neighbor classifier, and obtaining a diagnosis classification result; and classifying the eye fundus image according to the diagnosis classification result. The application can solve the problem that the current eye fundus image classification relies on classification label data, reduce the misdiagnosis and missed diagnosis rate, improve the accuracy and efficiency of eye fundus image classification, and save human resources.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a method, apparatus, device and medium for classifying fundus images. Background Technology

[0002] Currently, fundus image classification typically involves medical experts annotating acquired fundus images to determine the stage of disease progression and propose targeted treatment plans. The accuracy of the classification directly impacts the doctor's assessment and is a crucial step in the diagnostic process. Precise classification and annotation rely on the decision-making of professional physicians. When dealing with a large number of patients, this traditional manual classification method is inefficient, poorly adaptable to complex data, and inevitably introduces classification errors. Furthermore, labeling diseases is a tedious, time-consuming, and labor-intensive task, especially for large-scale datasets. While semi-supervised learning methods can be used for fundus image classification, existing medical imaging resources contain a large amount of data without precise labels, far exceeding the amount of data with precise labels. This may limit model performance. Moreover, semi-supervised learning methods require labeling the fundus dataset, which consumes significant time and effort from professional physicians and can reduce the accuracy of fundus image classification.

[0003] As can be seen from the above, how to solve the current problem of reliance on classification label data in fundus image classification, reduce the rate of misdiagnosis and missed diagnosis, improve the accuracy and efficiency of fundus image classification, and save human resources are problems that need to be solved in this field. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for fundus image classification, which can solve the problem of current fundus image classification relying on classification label data, reduce the rate of misdiagnosis and missed diagnosis, improve the accuracy and efficiency of fundus image classification, and save human resources. The specific solution is as follows:

[0005] In a first aspect, this application discloses a fundus image classification method, which is based on a pre-defined convolutional neural network-graph convolutional network-self-supervised learning network model, including:

[0006] Acquire fundus images, generate a training set based on the fundus images, and perform feature enhancement and rotation operations on the training set to obtain a rotated enhanced image;

[0007] Feature extraction and training operations are performed on the rotation-enhanced image to obtain multi-view instance recognition task representation features and rotation prediction task representation features. Loss functions and cross-entropy loss functions are calculated on the multi-view instance recognition task representation features and the rotation prediction task representation features respectively to obtain instance loss function and rotation loss function.

[0008] The local model parameters are adjusted using the instance loss function and the rotation loss function to generate a model adjustment loss function. A preset K-nearest neighbor classifier is used to perform self-supervised learning of the model adjustment loss function for fundus disease diagnosis and grading to obtain the diagnosis and grading results.

[0009] The fundus images are classified according to the diagnostic grading results.

[0010] Optionally, before acquiring the fundus image, the method further includes:

[0011] Construct the initial network model;

[0012] The initial network model is configured to obtain the convolutional neural network-graph convolutional network-self-supervised learning network model; the convolutional neural network-graph convolutional network-self-supervised learning network model includes an implicit instance graph representation module, an implicit adaptive normalized exponential function embedding module, and a self-attention feature extraction module.

[0013] Optionally, performing feature enhancement and rotation operations on the training set to obtain a rotation-enhanced image includes:

[0014] Random feature enhancement operations are performed on the training set to obtain enhanced images;

[0015] The enhanced image is rotated, and a rotation label is assigned to the rotated enhanced image to obtain the rotated enhanced image.

[0016] Optionally, the feature extraction and training operation on the rotation-enhanced image to obtain representation features for the multi-view instance recognition task and rotation prediction task includes:

[0017] The rotation-enhanced image is subjected to feature extraction using a local convolutional neural network to obtain the extracted features;

[0018] The extracted features are trained using a local self-attention feature extraction module to obtain the representation features for the multi-view instance recognition task and the rotation prediction task.

[0019] Optionally, the local self-attention feature extraction module trains the extracted features to obtain the representation features for the multi-view instance recognition task and the representation features for the rotation prediction task, including:

[0020] The extracted features are trained based on the self-attention feature extraction module to obtain trained features;

[0021] The trained features are decoupled to obtain the multi-view instance recognition task representation features and the rotation prediction task representation features.

[0022] Optionally, the step of calculating the loss function and cross-entropy loss function on the multi-view instance recognition task representation features and the rotation prediction task representation features respectively to obtain the instance loss function and rotation loss function includes:

[0023] The local implicit adaptive normalized exponential function embedding module is used to calculate the loss function for the multi-view instance recognition task representation features and the cross-entropy loss function for the rotation prediction task representation features to obtain the instance loss function and the rotation loss function.

[0024] Optionally, adjusting the local model parameters using the instance loss function and the rotation loss function includes:

[0025] The instance loss function and the rotation loss function are weighted and combined to obtain the overall loss function;

[0026] The local model parameters are adjusted using the overall loss function.

[0027] Secondly, this application discloses a fundus image classification device, the device being based on a preset convolutional neural network-graph convolutional network-self-supervised learning network model, comprising:

[0028] The rotation enhancement module is used to acquire fundus images, generate a training set based on the fundus images, and perform feature enhancement and rotation operations on the training set to obtain a rotation-enhanced image.

[0029] The loss function calculation module is used to perform feature extraction and training operations on the rotation-enhanced image to obtain multi-view instance recognition task representation features and rotation prediction task representation features. The module then performs loss function calculation and cross-entropy loss function calculation on the multi-view instance recognition task representation features and the rotation prediction task representation features to obtain instance loss function and rotation loss function, respectively.

[0030] The diagnostic grading module is used to adjust the local model parameters using the instance loss function and the rotation loss function to generate a model adjustment loss function, and to perform self-supervised learning of the model adjustment loss function using a preset K-nearest neighbor classifier to obtain the diagnostic grading results for fundus diseases.

[0031] An image classification module is used to classify the fundus image according to the diagnostic grading results.

[0032] Thirdly, this application discloses an electronic device, comprising:

[0033] Memory, used to store computer programs;

[0034] A processor is used to execute the computer program to implement the aforementioned fundus image classification method.

[0035] Fourthly, this application discloses a computer storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed fundus image classification method.

[0036] As can be seen, this application provides a fundus image classification method, including acquiring fundus images, generating a training set based on the fundus images, performing feature enhancement and rotation operations on the training set to obtain a rotation-enhanced image; performing feature extraction and training operations on the rotation-enhanced image to obtain multi-view instance recognition task representation features and rotation prediction task representation features, respectively calculating loss functions and cross-entropy loss functions on the multi-view instance recognition task representation features and the rotation prediction task representation features to obtain instance loss functions and rotation loss functions; adjusting local model parameters using the instance loss functions and the rotation loss functions to generate a model adjustment loss function; performing self-supervised learning of fundus disease diagnosis and grading on the model adjustment loss function using a preset K-nearest neighbor classifier to obtain a diagnosis and grading result; and classifying the fundus images according to the diagnosis and grading result. This application's method is based on a pre-defined convolutional neural network-graph convolutional network-self-supervised learning network model. It generates a training set based on fundus images and performs feature enhancement, rotation, feature extraction, and training operations to obtain representation features for multi-view instance recognition and rotation prediction tasks. Instance loss functions and rotation loss functions are calculated to adjust model parameters. A K-nearest neighbor classifier is used for self-supervised learning of fundus disease diagnosis and grading, improving the dependence of fundus image classification on classification label data. Fundus images are classified according to the diagnostic grading results, achieving self-supervised collaborative task learning on the fundus dataset and realizing the goal of fundus image classification. This application has significant implications for computer-aided clinical diagnosis, eliminating the need for time-consuming manual annotation and classification by professional doctors, controlling errors to a certain extent, improving the accuracy and efficiency of fundus image classification, thereby better and more effectively diagnosing conditions, reducing misdiagnosis and missed diagnosis rates, and reducing the consumption of human resources. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0038] Figure 1 This is a flowchart of a fundus image classification method disclosed in this application;

[0039] Figure 2 This is a flowchart of a model construction and configuration process disclosed in this application;

[0040] Figure 3 This is a flowchart illustrating an example of image rotation enhancement and computational representation features disclosed in this application;

[0041] Figure 4 This application discloses a flowchart of a computational representation feature and loss function.

[0042] Figure 5 This is a flowchart of a self-attention feature extraction module disclosed in this application;

[0043] Figure 6 This is a diagnostic grading result diagram disclosed in this application;

[0044] Figure 7 This is a flowchart of a fundus image classification method disclosed in this application;

[0045] Figure 8 This is a step diagram of an implicit structural representation disclosed in this application;

[0046] Figure 9 This is an overall flowchart of a fundus image classification method disclosed in this application;

[0047] Figure 10 This is a schematic diagram of the structure of a fundus image classification device disclosed in this application;

[0048] Figure 11 This application provides a structural diagram of an electronic device. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Currently, fundus image classification typically involves medical experts annotating acquired fundus images to determine the stage of disease progression and propose targeted treatment plans. The accuracy of the classification directly impacts the doctor's assessment and is a crucial step in the diagnostic process. Precise classification and annotation rely on the decision-making of professional physicians. When dealing with a large number of patients, this traditional manual classification method is inefficient, poorly adaptable to complex data, and inevitably introduces classification errors. Furthermore, labeling diseases is a tedious, time-consuming, and labor-intensive task, especially for large-scale datasets. While semi-supervised learning methods can be used for fundus image classification, existing medical imaging resources contain a large amount of data without precise labels, far exceeding the amount of data with precise labels. This may limit model performance. Moreover, semi-supervised learning methods require labeling the fundus dataset, which consumes significant time and effort from professional physicians and can reduce the accuracy of fundus image classification. As can be seen from the above, how to solve the current problem of reliance on classification label data in fundus image classification, reduce the rate of misdiagnosis and missed diagnosis, improve the accuracy and efficiency of fundus image classification, and save human resources are problems that need to be solved in this field.

[0051] See Figure 1 As shown, this invention discloses a fundus image classification method. The method is based on a pre-defined convolutional neural network-graph convolutional network-self-supervised learning network model, and specifically may include:

[0052] Step S11: Obtain fundus images, generate a training set based on the fundus images, and perform feature enhancement and rotation operations on the training set to obtain a rotation-enhanced image.

[0053] In this embodiment, before acquiring the fundus image, the method further includes: constructing an initial network model; configuring the initial network model to obtain the convolutional neural network-graph convolutional network-self-supervised learning network model; the convolutional neural network-graph convolutional network-self-supervised learning network model includes an implicit instance graph representation module, an implicit adaptive normalized exponential function embedding module, and a self-attention feature extraction module.

[0054] The specific model building and configuration process is as follows: Figure 2As shown, the method in this application is based on a CNN (Convolutional Neural Network)-GCN (Graph Convolutional Network)-Hybrid Self-Supervised Learning Network model. The backbone of this model is ResNet18, and it includes a LIGR (Latent Instance Graph Representation) module, a LASoftmax (Latent Adaptable Softmax) module, and a SAFEM (Self-attention Feature Extraction Module). This model is implemented on the PyTorch architecture. The process of adjusting fundus images and configuring the model is as follows: (1) Construct the initial network model; (2) Adjust the fundus image to 320×320, randomly determine the cropping percentage within the range of [20%, 100%], and randomly grayscale the 20% image; (3) Randomly scale and crop the fundus image to 224×224, and then randomly flip it in the horizontal direction. The factors of brightness, contrast, saturation and hue are randomly selected within the range of [0.6, 1.4] to increase the data; (4) Optimize the network using the Adam optimizer; (5) Initialize the learning rate to 0.00001; (6) Set the number of iterations of the model to 2000 and the batch size to 64 to complete the construction and configuration of the model.

[0055] Step S12: Perform feature extraction and training operations on the rotation-enhanced image to obtain multi-view instance recognition task representation features and rotation prediction task representation features. Calculate the loss function and cross-entropy loss function on the multi-view instance recognition task representation features and the rotation prediction task representation features respectively to obtain the instance loss function and rotation loss function.

[0056] In this embodiment, a local convolutional neural network is used to extract features from the rotation enhancement image to obtain extracted features; the extracted features are trained based on a local self-attention feature extraction module to obtain the multi-view instance recognition task representation features and the rotation prediction task representation features; loss functions and cross-entropy loss functions are calculated on the multi-view instance recognition task representation features and the rotation prediction task representation features respectively to obtain the instance loss function and the rotation loss function.

[0057] Specifically, a local convolutional neural network is used to extract features from the rotation-enhanced image to obtain extracted features. The extracted features are then trained based on the self-attention feature extraction module to obtain trained features. The trained features are then decoupled to obtain the multi-view instance recognition task representation features and the rotation prediction task representation features. Then, a local implicit adaptive normalized exponential function embedding module is used to calculate the loss function for the multi-view instance recognition task representation features and the cross-entropy loss function for the rotation prediction task representation features to obtain the instance loss function and the rotation loss function.

[0058] Examples of image rotation enhancement and computational representation features include... Figure 3 As shown, this application employs a strategy of simultaneously training on rotation prediction and multi-view instance recognition tasks. m sample fundus images are randomly selected as the training set. Figure 3 Taking x1 and x2 as examples, each image x i (i = 1, 2) Random features are enhanced twice to generate an enhanced image. Know The enhanced images are then further rotated by 0°, 90°, 180°, and 270° to produce 8m new images as a batch (i.e., rotated enhanced images). Accordingly, each rotated enhanced image is assigned a rotation label 0, 1, 2, and 3, from image x. i (i = 1, 2) generates Know In a batch process, by using the feature embedding module F c (·;θ), will input x i Mapping to high-level feature vector f i This feature vector will be further input into SAFEM and divided into f i (d) and f i (r) , used for learning collaborative tasks. i (d) It represents features for multi-view instance recognition tasks and uses implicit instance graph representations to train models to improve collaborative tasks. It uses L2 normalization layers to calculate the loss function between features and outputs diagnostic results. i (r) It represents the features of the rotation prediction task, and uses a fully connected layer to generate the probability of each rotation type.

[0059] The specific process for calculating the representation features and loss function is as follows: Figure 4As shown: (1) Generate a training set, enhance the features of the training set to obtain an enhanced image; (2) Perform rotation, assign rotation labels to obtain a rotation-enhanced image; (3) Use the self-attention feature extraction module to extract and train features to obtain trained features, and then decouple the trained features to obtain multi-view instance recognition task representation features and rotation prediction task representation features; among them, during the training process, rotation prediction provides rich structural features as an auxiliary task. Multi-view instance recognition is used as the main task to obtain the learned rotation invariant features; (4) Perform loss function calculation and cross-entropy loss function calculation on the multi-view instance recognition task representation features and rotation prediction task representation features respectively to obtain the instance loss function and rotation loss function.

[0060] In this embodiment, the workflow of the self-attention feature extraction module is as follows: Figure 5 As shown: The feature matrix X is calculated by the fully connected layer and transformed through a linear mapping:

[0061] Q = XW Q K = XW K V = XW V .

[0062] Among them W Q W K W V The parameter matrix is ​​used to generate the query, key, and value, respectively, and is updated through backpropagation during model training. Attention coefficients between retinal regions are obtained by calculating the scaled dot product correlation between Q (query) and K (key), and then input into the softmax function. Finally, the self-attention feature matrix is ​​calculated by multiplying the attention weights by V (value).

[0063]

[0064] In the formula, d is the normalization factor, and the dimension N is set to be equal to K. This is a new feature map for the retinal image, where ω is a weight used to adjust the contribution of multi-scale feature extraction, determined by the image size.

[0065] Step S13: Adjust the local model parameters using the instance loss function and the rotation loss function to generate a model adjustment loss function. Use a preset K-nearest neighbor classifier to perform self-supervised learning of fundus disease diagnosis and grading on the model adjustment loss function to obtain the diagnosis and grading results.

[0066] In this embodiment, the diagnostic grading results are divided into the following categories: Figure 6 As shown: (1) Unlabeled data AMD (age-related macular degeneration) lesion grading; (2) Unlabeled data PM (pathological myopia) lesion grading.

[0067] Step S14: Classify the fundus images according to the diagnostic grading results.

[0068] In this embodiment, fundus images are acquired, and a training set is generated based on the fundus images. Feature enhancement and rotation operations are performed on the training set to obtain a rotation-enhanced image. Feature extraction and training operations are performed on the rotation-enhanced image to obtain multi-view instance recognition task representation features and rotation prediction task representation features. Loss functions and cross-entropy loss functions are calculated on the multi-view instance recognition task representation features and the rotation prediction task representation features, respectively, to obtain instance loss functions and rotation loss functions. The local model parameters are adjusted using the instance loss functions and the rotation loss functions to generate a model adjustment loss function. A preset K-nearest neighbor classifier is used to perform self-supervised learning of fundus disease diagnosis and grading on the model adjustment loss function to obtain a diagnosis and grading result. The fundus images are classified according to the diagnosis and grading result. This application's method is based on a pre-defined convolutional neural network-graph convolutional network-self-supervised learning network model. It generates a training set based on fundus images and performs feature enhancement, rotation, feature extraction, and training operations to obtain representation features for multi-view instance recognition and rotation prediction tasks. Instance loss functions and rotation loss functions are calculated to adjust model parameters. A K-nearest neighbor classifier is used for self-supervised learning of fundus disease diagnosis and grading, improving the dependence of fundus image classification on classification label data. Fundus images are classified according to the diagnostic grading results, achieving self-supervised collaborative task learning on the fundus dataset and realizing the goal of fundus image classification. This application has significant implications for computer-aided clinical diagnosis, eliminating the need for time-consuming manual annotation and classification by professional doctors, controlling errors to a certain extent, improving the accuracy and efficiency of fundus image classification, thereby better and more effectively diagnosing conditions, reducing misdiagnosis and missed diagnosis rates, and reducing the consumption of human resources.

[0069] See Figure 7 As shown, this invention discloses a fundus image classification method. The method is based on a pre-defined convolutional neural network-graph convolutional network-self-supervised learning network model, and specifically may include:

[0070] Step S21: Obtain a fundus image, generate a training set based on the fundus image, perform random feature enhancement operation on the training set to obtain an enhanced image, perform a rotation operation on the enhanced image, and assign a rotation label to the rotated enhanced image to obtain the rotated enhanced image.

[0071] In this embodiment, the proposed convolutional neural network-graph convolutional network-self-supervised learning network model includes an implicit instance graph representation module, an implicit adaptive normalized exponential function embedding module, and a self-attention feature extraction module.

[0072] Specifically, the implicit instance graph representation module works as follows:

[0073] Using GCN, instance features (x i As input to the multi-view instance recognition task, an implicit graph representation (z) is obtained. i Specifically, m instances are randomly selected, characterized by: Feature enhancement is performed at each training step. Matrix This represents the set of features enhanced twice. In the SAFEM-based backbone training, the enhanced features are partitioned and input into the multi-view instance recognition task to obtain instance features Z0. Using the relationships between Z0, an undirected graph G = (A, Z1) is created, and the adjacency matrix is ​​initialized using the following formula.

[0074] in Let Z be the identity matrix, representing the connections between each node and itself. An implicit representation Z1 of graph G is obtained using graph convolutional layers, as shown in the following formula:

[0075]

[0076]

[0077] In the formula, Let A be the degree matrix, φ(·) be the Leaky ReLU activation, and W be the degree matrix. (0) and W (1) The input and output weight matrices are implicit graph representations. Context from instance features is merged into... To obtain the implicit representation, a linear layer can also be used instead. The implicit representation feature Z1, updated iteratively by the graph convolutional layer, is further concatenated with the original feature Z0, as shown in the following formula:

[0078] Z = [Z0, Z1];

[0079] The Z features are further trained using LASoftmax embeddings, which helps improve the performance of instance recognition. This method iteratively constructs an undirected graph G after continuous training.

[0080] Step S22: Perform feature extraction and training operations on the rotation-enhanced image to obtain multi-view instance recognition task representation features and rotation prediction task representation features. Calculate the loss function and cross-entropy loss function on the multi-view instance recognition task representation features and the rotation prediction task representation features respectively to obtain the instance loss function and rotation loss function.

[0081] Specifically, the implicit adaptive softmax embedding module works as follows:

[0082] The steps for implicit structural representation using LASoftmax embedding after the LIGR module are explained below. Figure 8 As shown, from each instance x i Enhanced implicit representation Features extracted Features considered as positive samples, while those from other instances Features extracted from implicit representations Features considered as negative samples. Specifically, considering the rotation prediction task, It is instance x i Features The implicit representation features obtained by rotating y times, where y∈{0,1,2,3}. Therefore, Corresponding to Characteristics of positive samples Features corresponding to negative samples. Enhanced samples. Identified as instance Z i The probability is defined as:

[0083]

[0084] in, Know Let |z| represent the loss functions between positive and negative sample pairs, respectively. An adaptive factor η > 1 is used to amplify the similarity difference and increase the negative similarity weights. A temperature parameter τ < 1 is used to smooth the probability distribution. For numerical stability, |z| i |2=1 ensures that all implicit representations are... Normalization.

[0085] Similarly, the enhanced samples Identified as instance z j The probability of (j≠i) is defined as:

[0086]

[0087] The proposed implicit representation LASoftmax minimizes the negative sum of the log probabilities of all instances, as shown in the following formula:

[0088]

[0089] Step S23: Weight the instance loss function and the rotation loss function to obtain the overall loss function. Use the overall loss function to adjust the local model parameters to generate the model adjustment loss function. Use a preset K-nearest neighbor classifier to perform self-supervised learning of fundus disease diagnosis and grading on the model adjustment loss function to obtain the diagnosis and grading results.

[0090] In this model, the loss function L for the rotation prediction task r Defined as:

[0091]

[0092] In the formula, F c (·;θ c () is a fully connected layer. For input X i The rotation feature is obtained by rotating y times by 90°, where y∈{0,1,2,3}, and l represents the cross-entropy loss. The overall loss L is a weighted combination of collaborative tasks, defined as follows:

[0093] L = L d +λ·L r ;

[0094] Among them, L d The loss for the multi-view instance recognition task is expressed in the formula within the LASoftmax module. Specifically, the parameter λ is defined as the weight, representing the importance of the rotation prediction task.

[0095] Step S24: Classify the fundus images according to the diagnostic grading results.

[0096] The overall classification process of this application is as follows: Figure 9As shown, the process is divided into three stages: network model training and configuration, self-supervised learning-based grading of fundus disease diagnosis, and fundus image classification. In the training stage, fundus image data from the training set is used without classification labels. A self-supervised learning approach, employing two collaborative tasks—rotation prediction and multi-view instance recognition—is used to jointly train the fundus images for fundus disease diagnosis and grading. In the fundus image classification stage, the fundus images are classified according to the diagnostic grading results, generating corresponding classification labels. This application uses a CNN-GCN hybrid network and a self-supervised learning approach to improve the dependence of fundus image classification on classification label data. Self-supervised collaborative task learning is performed on the fundus dataset to achieve the goal of fundus image classification. This application has significant implications for computer-aided clinical diagnosis, alleviating the dependence of fundus datasets on labels, eliminating the need for time-consuming manual annotation and classification by professional doctors, controlling errors to a certain extent, ensuring accuracy, and assisting doctors in better and more effective diagnosis, obtaining more reliable disease information, reducing misdiagnosis and missed diagnosis rates, lowering labor costs, and improving diagnostic efficiency.

[0097] In this embodiment, fundus images are acquired, and a training set is generated based on the fundus images. Feature enhancement and rotation operations are performed on the training set to obtain a rotation-enhanced image. Feature extraction and training operations are performed on the rotation-enhanced image to obtain multi-view instance recognition task representation features and rotation prediction task representation features. Loss functions and cross-entropy loss functions are calculated on the multi-view instance recognition task representation features and the rotation prediction task representation features, respectively, to obtain instance loss functions and rotation loss functions. The local model parameters are adjusted using the instance loss functions and the rotation loss functions to generate a model adjustment loss function. A preset K-nearest neighbor classifier is used to perform self-supervised learning of fundus disease diagnosis and grading on the model adjustment loss function to obtain a diagnosis and grading result. The fundus images are classified according to the diagnosis and grading result. This application's method is based on a pre-defined convolutional neural network-graph convolutional network-self-supervised learning network model. It generates a training set based on fundus images and performs feature enhancement, rotation, feature extraction, and training operations to obtain representation features for multi-view instance recognition and rotation prediction tasks. Instance loss functions and rotation loss functions are calculated to adjust model parameters. A K-nearest neighbor classifier is used for self-supervised learning of fundus disease diagnosis and grading, improving the dependence of fundus image classification on classification label data. Fundus images are classified according to the diagnostic grading results, achieving self-supervised collaborative task learning on the fundus dataset and realizing the goal of fundus image classification. This application has significant implications for computer-aided clinical diagnosis, eliminating the need for time-consuming manual annotation and classification by professional doctors, controlling errors to a certain extent, improving the accuracy and efficiency of fundus image classification, thereby better and more effectively diagnosing conditions, reducing misdiagnosis and missed diagnosis rates, and reducing the consumption of human resources.

[0098] See Figure 10 As shown, this invention discloses a fundus image classification device. The device is based on a preset convolutional neural network-graph convolutional network-self-supervised learning network model, and may specifically include:

[0099] The rotation enhancement module 11 is used to acquire fundus images, generate a training set based on the fundus images, and perform feature enhancement and rotation operations on the training set to obtain a rotation-enhanced image.

[0100] The loss function calculation module 12 is used to perform feature extraction and training operations on the rotation enhancement image to obtain multi-view instance recognition task representation features and rotation prediction task representation features, and to calculate the loss function and cross-entropy loss function on the multi-view instance recognition task representation features and the rotation prediction task representation features, respectively, to obtain the instance loss function and the rotation loss function.

[0101] The diagnostic grading module 13 is used to adjust the local model parameters using the instance loss function and the rotation loss function to generate a model adjustment loss function, and to perform self-supervised learning of the model adjustment loss function using a preset K-nearest neighbor classifier to obtain the diagnostic grading result for fundus diseases.

[0102] Image classification module 14 is used to classify the fundus image according to the diagnostic grading results.

[0103] In this embodiment, fundus images are acquired, and a training set is generated based on the fundus images. Feature enhancement and rotation operations are performed on the training set to obtain a rotation-enhanced image. Feature extraction and training operations are performed on the rotation-enhanced image to obtain multi-view instance recognition task representation features and rotation prediction task representation features. Loss functions and cross-entropy loss functions are calculated on the multi-view instance recognition task representation features and the rotation prediction task representation features, respectively, to obtain instance loss functions and rotation loss functions. The local model parameters are adjusted using the instance loss functions and the rotation loss functions to generate a model adjustment loss function. A preset K-nearest neighbor classifier is used to perform self-supervised learning of fundus disease diagnosis and grading on the model adjustment loss function to obtain a diagnosis and grading result. The fundus images are classified according to the diagnosis and grading result. This application's method is based on a pre-defined convolutional neural network-graph convolutional network-self-supervised learning network model. It generates a training set based on fundus images and performs feature enhancement, rotation, feature extraction, and training operations to obtain representation features for multi-view instance recognition and rotation prediction tasks. Instance loss functions and rotation loss functions are calculated to adjust model parameters. A K-nearest neighbor classifier is used for self-supervised learning of fundus disease diagnosis and grading, improving the dependence of fundus image classification on classification label data. Fundus images are classified according to the diagnostic grading results, achieving self-supervised collaborative task learning on the fundus dataset and realizing the goal of fundus image classification. This application has significant implications for computer-aided clinical diagnosis, eliminating the need for time-consuming manual annotation and classification by professional doctors, controlling errors to a certain extent, improving the accuracy and efficiency of fundus image classification, thereby better and more effectively diagnosing conditions, reducing misdiagnosis and missed diagnosis rates, and reducing the consumption of human resources.

[0104] In some specific embodiments, the rotation enhancement module 11 may specifically include:

[0105] The model building module is used to build the initial network model;

[0106] The model configuration module is used to configure the initial network model to obtain the convolutional neural network-graph convolutional network-self-supervised learning network model; the convolutional neural network-graph convolutional network-self-supervised learning network model includes an implicit instance graph representation module, an implicit adaptive normalized exponential function embedding module, and a self-attention feature extraction module.

[0107] In some specific embodiments, the rotation enhancement module 11 may specifically include:

[0108] A random feature enhancement module is used to perform random feature enhancement operations on the training set to obtain an enhanced image;

[0109] A rotation module is used to rotate the enhanced image and assign a rotation label to the rotated enhanced image to obtain the rotated enhanced image.

[0110] In some specific embodiments, the loss function calculation module 12 may specifically include:

[0111] The feature extraction module is used to extract features from the rotation-enhanced image using a local convolutional neural network to obtain extracted features;

[0112] The feature determination module is used to train the extracted features based on the local self-attention feature extraction module to obtain the representation features of the multi-view instance recognition task and the rotation prediction task.

[0113] In some specific embodiments, the loss function calculation module 12 may specifically include:

[0114] The training module is used to perform training operations on the extracted features based on the self-attention feature extraction module to obtain trained features;

[0115] A decoupling module is used to decouple the trained features to obtain the multi-view instance recognition task representation features and the rotation prediction task representation features.

[0116] In some specific embodiments, the loss function calculation module 12 may specifically include:

[0117] The loss function calculation module is used to calculate the loss function for the multi-view instance recognition task representation features using the local implicit adaptive normalized exponential function embedding module, and to calculate the cross-entropy loss function for the rotation prediction task representation features, so as to obtain the instance loss function and the rotation loss function.

[0118] In some specific embodiments, the diagnostic grading module 13 may specifically include:

[0119] The weighted combination module is used to weight and combine the instance loss function and the rotation loss function to obtain the overall loss function;

[0120] The parameter adjustment module is used to adjust the local model parameters using the overall loss function.

[0121] Figure 11This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the fundus image classification method performed by the electronic device disclosed in any of the foregoing embodiments.

[0122] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0123] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored on it include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.

[0124] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device 20 to enable the processor 21 to perform calculations and processing on the data 223 in the memory 22. The operating system 221 can be Windows, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the fundus image classification method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the fundus image classification device from external devices, as well as data collected by its own input / output interface 25.

[0125] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0126] Furthermore, embodiments of this application also disclose a computer-readable storage medium storing a computer program, which, when loaded and executed by a processor, implements the steps of the fundus image classification method disclosed in any of the foregoing embodiments.

[0127] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0128] The foregoing has provided a detailed description of the fundus image classification method, apparatus, device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for classifying fundus images, characterized in that, The method is based on a pre-defined convolutional neural network-graph convolutional network-self-supervised learning network model, including: Acquire fundus images, generate a training set based on the fundus images, and perform feature enhancement and rotation operations on the training set to obtain a rotated enhanced image; Feature extraction and training operations are performed on the rotation-enhanced image to obtain multi-view instance recognition task representation features and rotation prediction task representation features. Loss functions and cross-entropy loss functions are calculated on the multi-view instance recognition task representation features and the rotation prediction task representation features respectively to obtain instance loss function and rotation loss function. The local model parameters are adjusted using the instance loss function and the rotation loss function to generate a model adjustment loss function. A preset K-nearest neighbor classifier is used to perform self-supervised learning of the model adjustment loss function for fundus disease diagnosis and grading to obtain the diagnosis and grading results. The fundus images are classified according to the diagnostic grading results; The loss function and cross-entropy loss function are calculated for the representation features of the multi-view instance recognition task and the rotation prediction task respectively, so as to obtain the instance loss function and the rotation loss function. This includes: using the local implicit adaptive normalized exponential function embedding module to calculate the loss function for the representation features of the multi-view instance recognition task and to calculate the cross-entropy loss function for the representation features of the rotation prediction task, so as to obtain the instance loss function and the rotation loss function. The implicit adaptive softmax embedding module works as follows: LASoftmax embedding is used after the LIGR module, from each instance Enhanced implicit representation Features extracted Features considered as positive samples, while those from other instances Features extracted from implicit representations Features considered as negative samples; taking rotation prediction task into account, It is an example Features The implicit representation features obtained by rotating y times, where ;therefore, Corresponding to Characteristics of positive samples Features corresponding to negative samples; augmented samples Recognized as an instance The probability is defined as: ； in, and Let these represent the loss functions for positive and negative sample pairs, respectively, using adaptive factors. Magnify the similarity difference, increase the negative similarity weight, and use temperature parameters. Smooth the probability distribution; for numerical stability, It ensures that all implicit representations are Normalization.

2. The fundus image classification method according to claim 1, characterized in that, Before acquiring the fundus image, the process also includes: Construct the initial network model; The initial network model is configured to obtain the convolutional neural network-graph convolutional network-self-supervised learning network model; the convolutional neural network-graph convolutional network-self-supervised learning network model includes an implicit instance graph representation module, an implicit adaptive normalized exponential function embedding module, and a self-attention feature extraction module.

3. The fundus image classification method according to claim 1, characterized in that, The step of performing feature enhancement and rotation operations on the training set to obtain a rotation-enhanced image includes: Random feature enhancement operations are performed on the training set to obtain enhanced images; The enhanced image is rotated, and a rotation label is assigned to the rotated enhanced image to obtain the rotated enhanced image.

4. The fundus image classification method according to claim 1, characterized in that, The step of performing feature extraction and training operations on the rotation-enhanced image to obtain representation features for the multi-view instance recognition task and rotation prediction task includes: The rotation-enhanced image is subjected to feature extraction using a local convolutional neural network to obtain the extracted features; The extracted features are trained using a local self-attention feature extraction module to obtain the representation features for the multi-view instance recognition task and the rotation prediction task.

5. The fundus image classification method according to claim 4, characterized in that, The local self-attention feature extraction module trains the extracted features to obtain the representation features for the multi-view instance recognition task and the rotation prediction task, including: The extracted features are trained based on the self-attention feature extraction module to obtain trained features; The trained features are decoupled to obtain the multi-view instance recognition task representation features and the rotation prediction task representation features.

6. The fundus image classification method according to any one of claims 1 to 5, characterized in that, The step of adjusting the local model parameters using the instance loss function and the rotation loss function includes: The instance loss function and the rotation loss function are weighted and combined to obtain the overall loss function; The local model parameters are adjusted using the overall loss function.

7. A fundus image classification device, characterized in that, The device is based on a preset convolutional neural network-graph convolutional network-self-supervised learning network model, including: The rotation enhancement module is used to acquire fundus images, generate a training set based on the fundus images, and perform feature enhancement and rotation operations on the training set to obtain a rotation-enhanced image. The loss function calculation module is used to perform feature extraction and training operations on the rotation-enhanced image to obtain multi-view instance recognition task representation features and rotation prediction task representation features. The module then performs loss function calculation and cross-entropy loss function calculation on the multi-view instance recognition task representation features and the rotation prediction task representation features to obtain instance loss function and rotation loss function, respectively. The diagnostic grading module is used to adjust the local model parameters using the instance loss function and the rotation loss function to generate a model adjustment loss function, and to perform self-supervised learning of the model adjustment loss function using a preset K-nearest neighbor classifier to obtain the diagnostic grading results for fundus diseases. An image classification module is used to classify the fundus image according to the diagnostic grading results; The loss function and cross-entropy loss function are calculated for the representation features of the multi-view instance recognition task and the rotation prediction task respectively, so as to obtain the instance loss function and the rotation loss function. This includes: using the local implicit adaptive normalized exponential function embedding module to calculate the loss function for the representation features of the multi-view instance recognition task and to calculate the cross-entropy loss function for the representation features of the rotation prediction task, so as to obtain the instance loss function and the rotation loss function. The implicit adaptive softmax embedding module works as follows: LASoftmax embedding is used after the LIGR module, from each instance Enhanced implicit representation Features extracted Features considered as positive samples, while those from other instances Features extracted from implicit representations Features considered as negative samples; taking rotation prediction task into account, It is an example Features The implicit representation features obtained by rotating y times, where ;therefore, Corresponding to Characteristics of positive samples Features corresponding to negative samples; augmented samples Recognized as an instance The probability is defined as: ； in, and Let these represent the loss functions for positive and negative sample pairs, respectively, using adaptive factors. Magnify the similarity difference, increase the negative similarity weight, and use temperature parameters. Smooth the probability distribution; for numerical stability, It ensures that all implicit representations are Normalization.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the fundus image classification method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the fundus image classification method as described in any one of claims 1 to 6.

Images