Fundus image classification system based on self-paced learning and adversarial generative network

Abstract

The application discloses an eye fundus image classification system based on self-walking learning and an adversarial generative network, the classification system comprising a down-sampling layer and a full connection layer, the input of the classification system being an eye fundus image, and the output being a class prediction probability of the input image; the training steps of the classification system comprising: S1, constructing a training set; S2, constructing a pre-training subsystem; S3, training the pre-training subsystem; S4, constructing a joint training subsystem; and S5, jointly training the generator, the discriminator and the classification system by using the joint training subsystem. The classification system can train a relatively accurate eye fundus image classification system on small sample data.

Description

Technical Field

[0001] This invention belongs to the field of image classification technology, specifically relating to a fundus image classification system based on auto-learning and adversarial generative networks. Background Technology

[0002] Fundus images are widely used in the computer-aided diagnosis and screening of eye diseases such as diabetic retinopathy (DR). In DR diagnosis, the main manifestations of lesions include aneurysms, hemorrhages, hard exudates, cotton wool spots, and macular edema. Diagnosis is made by determining whether DR fundus images contain early pathological features of DR, such as microaneurysms, hemorrhages, and exudates. While image classification using machine learning or deep learning has achieved relatively accurate results, both require large amounts of training data and perform poorly with small sample sizes. Furthermore, there are not enough samples available for medical images to effectively train machine learning and deep learning methods, thus limiting the application of computer-aided diagnosis.

[0003] Furthermore, in fundus image analysis, features such as lesions and texture are extremely important for DR diagnosis. However, these features are often small areas in the image, hidden in large areas such as shadows, macula, and background. Therefore, a robust learning strategy that learns from high-confidence datasets to avoid misleading the model learning due to a large amount of ambiguous data in a visual environment where lesion representation is weak is crucial for classification accuracy. Summary of the Invention

[0004] Purpose of the invention: To address the problems existing in the prior art, this invention provides a fundus image classification system based on auto-learning and adversarial generative networks. This classification system can be trained on small sample data to obtain a relatively accurate fundus image classification system.

[0005] Technical solution: This invention discloses a fundus image classification system based on auto-learning and generative adversarial networks. The classification system includes a downsampling layer and a fully connected layer. The input of the classification system is a fundus image, and the output is the predicted category probability of the input image.

[0006] The training steps for the classification system include:

[0007] S1. Construct a training set, wherein a sample in the training set consists of a sample fundus image, a lesion type label, and a text description; the resolution of all sample fundus images in the training set is unified to the same dimension, and the pixel values ​​are normalized to the range of [-1, 1]; the text description consists of multiple sentences expressing the same meaning;

[0008] S2. Construct a pre-training subsystem, which includes: an image encoder, an image decoder, a text encoder, and a blood vessel segmentation network;

[0009] The image encoder is used to extract image features from the input image, including two cascaded 2D convolutional units, each of which consists of a cascaded Conv2d layer and a pooling layer.

[0010] The image decoder is used to reconstruct an image based on image features, and includes two cascaded ConvTranspose2d layers;

[0011] The text encoder is used to extract text features from the text description of the image, including a one-hot encoding layer, three cascaded 1D convolutional units and a fully connected layer; each 1D convolutional unit consists of cascaded Conv1d layers and pooling layers; the fully connected layer transforms the output of the convolutional unit into a one-dimensional vector of length 1024.

[0012] The blood vessel segmentation network is used to obtain blood vessel segmentation maps from fundus images;

[0013] S3. Train the pre-training subsystem;

[0014] S4. Construct a joint training subsystem, which includes: a generator, a discriminator, and a sampler;

[0015] The generator is used to generate a virtual fundus image based on the text features of the blood vessel segmentation map and the image text description. The generator includes 8 cascaded downsampling units and 8 cascaded upsampling units. Each downsampling unit includes a downsampling layer, a regularization layer, and an activation function. Each upsampling unit includes an upsampling layer, a regularization layer, and an activation function. The input of the first downsampling unit is the blood vessel segmentation map. The input of the eighth downsampling unit is the concatenation vector of the output of the seventh downsampling unit and the noisy text, where the noisy text is the concatenation vector of text features and randomly sampled Gaussian noise.

[0016] The input to the first upsampling unit is the output of the 8th downsampling unit, and the inputs to the 2nd to 8th upsampling units are:

[0017] Upin l For the input of the l-th upsampling unit, Upout l-1 The output of the (l-1)th upsampling unit, Downout 8-l+1 This is the output of the 8-l+1 group of downsampling units; l = 2, 3, ..., 8;

[0018] The downsampling layer in the downsampling unit and the upsampling layer in the upsampling unit are both convolutional layers with a kernel size of 3*3 and a stride of 1. The number of channels in the convolutional layers in the first to sixth downsampling units is 2. n+1 The number of channels in the convolutional layers of the downsampling units in groups 7 and 8 are 2 respectively. 7 and 2 8 The number of channels in the convolutional layers of the upsampling units from group 1 to group 6 is 2. 9-n n = 1, 2, ..., 6; the number of channels in the convolutional layer of the 7th and 8th downsampling units are 2 respectively. 3 and 2 2 The activation functions in the 8 downsampling units and the 1st to 7th upsampling units are all ReLU activation functions with negative slope L, and the activation function in the 8th upsampling unit is the Tanh function.

[0019] The discriminator takes a fundus image and text features from a text description as input, and determines whether the input fundus image matches the text features of the text description. If they match, it outputs "true"; otherwise, it outputs "false". The discriminator includes a downsampling layer and a fully connected layer. The input fundus image is processed through the downsampling layer to obtain a feature vector. The feature vector is concatenated with the text features of the input text description and then input into the fully connected layer. The output is a binary number indicating whether the image is true or false. The downsampling layer includes six cascaded spectral normalization units. Each spectral normalization unit consists of a cascaded spectral normalization convolutional layer and an activation function. The kernel size of the spectral normalization convolutional layer is 4, and the stride is 2. The activation function in the first to fifth spectral normalization units is LeakyReLU with a negative slope of 0.2, and the activation function in the sixth spectral normalization unit is sigmoid.

[0020] The sampler is used to construct a support set and a query set for classification during training. Specifically, it extracts m samples from each of the c classes, for a total of c*m samples. c*k samples are assigned to the support set, and the remaining c*(mk) samples are assigned to the query set.

[0021] S5. Use the joint training subsystem to jointly train the generator, discriminator and classification system.

[0022] Furthermore, the text description consists of five sentences expressing the same meaning.

[0023] Furthermore, the joint training in step S5 includes multiple rounds, and the specific steps of one round of joint training include:

[0024] S5.1 The sampler extracts m samples from each of the c classes in the training set, for a total of c*m samples. c*k samples are assigned to the support set, and the remaining c*(mk) samples are assigned to the query set.

[0025] S5.2 Generator generates virtual samples: The sample fundus images in the query set are obtained by the blood vessel segmentation network to obtain blood vessel segmentation maps. The blood vessel segmentation maps and corresponding text descriptions are input into the generator, and virtual fundus image samples are generated by forward propagation. The generated virtual fundus image samples are then incorporated into the query set.

[0026] Calculate the loss function value of the generator:

[0027] In the formula, D represents the discriminator. The generated image represents the corresponding text description, which is the text vector encoded by the text encoder, and the weight parameters represent the weight parameters. It is the L2 norm of the generated image and the real image; the objective of updating the model parameters is to minimize the loss function;

[0028] S5.3 Input the sample fundus images and text features of the text descriptions in the query set into the discriminator, and obtain the true and false probabilities through forward propagation;

[0029] The virtual fundus image samples corresponding to the fundus images in the query set and the text features of the text description are input into the discriminator, and the virtual true and false probabilities are obtained by forward propagation.

[0030] Calculate the loss function value of the discriminator:

[0031]

[0032] In the formula, x represents the real image, t represents the matching text description, and ψ(t) w ) represents the text vector encoded by the text encoder, t w A textual description indicating that it does not match x. This represents the generated image;

[0033] S5.4 Input the sample fundus images from the support set into the classification system, generate the embedding space, and calculate the prototype point of each category in the embedding space;

[0034] The sample fundus images in the query set are input into the classification system. The distance from each sample fundus image to its own class prototype point is calculated. Based on this distance, the self-step learning order of real sample fundus images and virtual sample fundus images is determined respectively from small to large.

[0035] The sample fundus images from the query set are input into the classification system in the order of auto-learning to obtain the predicted class probability for each real sample.

[0036] The virtual fundus image samples corresponding to the fundus images in the query set are input into the classification system in the order of auto-learning to obtain the category prediction probability of each virtual sample; if the category prediction probability is less than the preset probability threshold, the corresponding virtual fundus image sample and the corresponding category prediction probability are deleted.

[0037] Calculate the loss function value of the classification system:

[0038] L c (x,y,z)=CE(C(x),y)+λCE(C(G(z)),argmax(C(G(z)))>t)

[0039] In the formula, x represents the real image, y represents the label, λ represents the weight parameter, z represents random Gaussian noise, C(x) represents the classification result of the classification system for x, CE(C(x), y) represents the calculation of the cross-entropy between C(x) and y, G(z) represents the generated image, argmax(C(G(z))) represents the pseudo label, and t represents the preset probability threshold. This loss represents training the classification system with real images and generated images respectively. When training with generated images, the confidence threshold of the pseudo label needs to be higher than t before the sample can be used.

[0040] Based on the loss function values ​​of the generator, discriminator, and classification system, backpropagation and parameter optimization are performed until the changes in the loss function values ​​of the generator, discriminator, and classification system in two consecutive rounds of joint training are all less than a preset loss change threshold.

[0041] Furthermore, in step S5.4, the optimization method for the generator, discriminator, and classification system is Adam, the batch size is 64, the discriminator is optimized once for each batch of images, and then the generator and classification system are optimized once, with the learning rate set to 0.0002.

[0042] Furthermore, the blood vessel segmentation network adopts a U-net network.

[0043] Further, step S3 includes: inputting the sample fundus images from the training set into the image encoder, inputting the extracted image features into the image decoder, and obtaining the reconstructed image of the input image; calculating the BCELoss of the reconstructed image and the input sample fundus images, and then using the Adam optimizer to optimize and train the image encoder and image decoder.

[0044] Further, step S3 includes: inputting the text description of the sample fundus images in the training set into a text encoder to obtain the output text features; inputting the sample fundus images into a trained image encoder to obtain the output image features; and optimizing the parameters of the text encoder by minimizing the structured joint embedding loss of the image features and text features.

[0045] Furthermore, step S3 includes: optimizing the training of the parameters of the blood vessel segmentation network using the DRIVE dataset.

[0046] Beneficial Effects: This invention discloses a fundus image classification system based on auto-step learning and generative adversarial networks. During training, the system utilizes a GAN network combining lesion text descriptions and vascular segmentation maps to generate relatively realistic and effective virtual samples to augment the data, considering the distribution realism and diagnostic reliability of medical images. By constructing a few-shot learning method based on auto-step learning, adaptive learning with a self-determined pace is performed on each lesion category, eliminating the negative impact of scattered data and avoiding falling into local minima. The invention integrates the aforementioned generative adversarial network and few-shot classification method into a unified framework, incorporating real-time generated latest virtual samples to assist classification, thus linking multiple networks and maintaining the multi-task relationship of the entire framework to improve the classification performance of the system. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the composition of the fundus image classification system disclosed in this invention;

[0048] Figure 2 A flowchart of the training steps for a fundus image classification system;

[0049] Figure 3 This is a schematic diagram of a fundus image;

[0050] Figure 4 This is a training diagram of the pre-training subsystem;

[0051] Figure 5 A schematic diagram of the joint training subsystem;

[0052] Figure 6 This is a schematic diagram of the generator's components. Detailed Implementation

[0053] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments.

[0054] This invention discloses a fundus image classification system based on auto-learning and adversarial generative networks, such as... Figure 1 As shown, the classification system includes a downsampling layer and a fully connected layer. The input of the classification system is a fundus image, and the output is the predicted category probability of the input image.

[0055] like Figure 2 As shown, the training steps of the classification system include:

[0056] S1. Construct a training set. A sample in the training set consists of a sample fundus image, a lesion type label, and a text description. The resolution of all sample fundus images in the training set is unified to the same dimension, and the pixel values ​​are normalized to the range [-1, 1]. The text description consists of multiple sentences expressing the same meaning. In this embodiment, the resolution of the sample fundus images is unified to the same dimension of 256*256*3, and the text description consists of 5 sentences. For example... Figure 3 As shown, Figure 3 (a) A fundus image with hard exudate, microaneurysms, and hemorrhage, described in the following 5 sentences:

[0057] 1. This fundus image has hardexudates microaneurysms and hemorrhages.

[0058] 2. There are hardexudates microaneurysms and hemorrhages in this fundus image.

[0059] 3. Afundus image with hardexudates microaneurysms and hemorrhages.

[0060] 4. Adiabetic retinopathy image has hardexudates microaneurysms and hemorrhages.

[0061] 5. Hardexudates microaneurysms and hemorrhages are in a diabeticretinopathy fundus image.

[0062] Figure 3 (b) is a normal fundus image, and its text description is as follows: 5 sentences:

[0063] 1. This is a normal fundus image.

[0064] 2. There is nothing about disease characteristics in this fundusimage.

[0065] 3. A fundus image of healthy person.

[0066] 4. Adiabetic retinopathy image has nothing about diseasecharacteristics.

[0067] 5. A healthy fundus image.

[0068] S2. Construct a pre-training subsystem, which includes: an image encoder, an image decoder, a text encoder, and a blood vessel segmentation network;

[0069] The image encoder is used to extract image features from the input image. It includes two cascaded 2D convolutional units, each of which consists of a cascaded Conv2d layer and a pooling layer. The input is an image with dimensions of 256*256*3, and the output is a 16*16*4 matrix representing the image features.

[0070] The image decoder is used to reconstruct an image based on image features, and includes two cascaded ConvTranspose2d layers;

[0071] The text encoder is used to extract text features from the text description of the image, including a one-hot encoding layer, three cascaded 1D convolutional units and a fully connected layer; each 1D convolutional unit consists of cascaded Conv1d layers and pooling layers; the fully connected layer transforms the output of the convolutional unit into a one-dimensional vector of length 1024.

[0072] The blood vessel segmentation network is used to obtain blood vessel segmentation maps of fundus images. In this embodiment, the blood vessel segmentation network adopts the U-net network provided in the paper "U-Net: Convolutional Networks for Biomedical Image Segmentation".

[0073] S3. Train the pre-training subsystem;

[0074] like Figure 4 As shown in (a), the sample fundus images in the training set are input into the image encoder, the extracted image features are input into the image decoder, and the reconstructed image of the input image is obtained; the BCELoss of the reconstructed image and the input sample fundus images is calculated, and then the Adam optimizer is used to optimize and train the image encoder and image decoder.

[0075] like Figure 4As shown in (b), the text description of the sample fundus images in the training set is input into the text encoder to obtain the output text features; the sample fundus images are input into the trained image encoder to obtain the output image features; the parameters of the text encoder are optimized by minimizing the structured joint embedding loss of image features and text features.

[0076] The parameters of the U-net network, a blood vessel segmentation network, were optimized and trained using the DRIVE dataset, which contains fundus images and their blood vessel segmentation maps.

[0077] S4. Construct a joint training subsystem, which includes: a generator, a discriminator, and a sampler, such as... Figure 5 As shown;

[0078] The generator is used to generate a virtual fundus image based on the text features of the blood vessel segmentation map and the image text description; such as Figure 6 As shown, the generator includes 8 cascaded downsampling units and 8 cascaded upsampling units. Each downsampling unit includes a downsampling layer, a regularization layer, and an activation function; each upsampling unit includes an upsampling layer, a regularization layer, and an activation function. The input of the first downsampling unit is a blood vessel segmentation map; the input of the 8th downsampling unit is a concatenation vector of the output of the 7th downsampling unit and noisy text, wherein the noisy text is a concatenation vector of text features and randomly sampled Gaussian noise.

[0079] The input to the first upsampling unit is the output of the 8th downsampling unit, and the inputs to the 2nd to 8th upsampling units are:

[0080] Upin l For the input of the l-th upsampling unit, Upout l-1 The output of the (l-1)th upsampling unit, Downout 8-l+1 This is the output of the 8-l+1 group of downsampling units; l = 2, 3, ..., 8;

[0081] The downsampling layer in the downsampling unit and the upsampling layer in the upsampling unit are both convolutional layers with a kernel size of 3*3 and a stride of 1. The number of channels in the convolutional layers in the first to sixth downsampling units is 2. n+1 The number of channels in the convolutional layers of the downsampling units in groups 7 and 8 are 2 respectively. 7 and 2 8 The number of channels in the convolutional layers of the upsampling units from group 1 to group 6 is 2. 9-n n = 1, 2, ..., 6; the number of channels in the convolutional layer of the 7th and 8th downsampling units are 2 respectively. 3 and 2 2The activation functions in the 8 downsampling units and the 1st to 7th upsampling units are all ReLU activation functions with negative slope L, and the activation function in the 8th upsampling unit is the Tanh function.

[0082] The discriminator takes a fundus image and text features from a text description as input, and determines whether the input fundus image matches the text features of the text description. If they match, it outputs "true"; otherwise, it outputs "false". The discriminator includes a downsampling layer and a fully connected layer. The input fundus image is processed through the downsampling layer to obtain a feature vector. The feature vector is concatenated with the text features of the input text description and then input into the fully connected layer. The output is a binary number indicating whether the image is true or false. The downsampling layer includes six cascaded spectral normalization units. Each spectral normalization unit consists of a cascaded spectral normalization convolutional layer and an activation function. The kernel size of the spectral normalization convolutional layer is 4, and the stride is 2. The activation function in the first to fifth spectral normalization units is LeakyReLU with a negative slope of 0.2, and the activation function in the sixth spectral normalization unit is sigmoid.

[0083] The sampler is used to construct a support set and a query set for classification during training. Specifically, it extracts m samples from each of the c classes, for a total of c*m samples. c*k samples are assigned to the support set, and the remaining c*(mk) samples are assigned to the query set.

[0084] S5. Utilize the joint training subsystem to jointly train the generator, discriminator, and classification system. Joint training includes multiple rounds, and the specific steps of one round of joint training include:

[0085] S5.1 The sampler extracts m samples from each of the c classes in the training set, for a total of c*m samples. c*k samples are assigned to the support set, and the remaining c*(mk) samples are assigned to the query set.

[0086] S5.2 Generator generates virtual samples: The sample fundus images in the query set are obtained by the blood vessel segmentation network to obtain blood vessel segmentation maps. The blood vessel segmentation maps and corresponding text descriptions are input into the generator, and virtual fundus image samples are generated by forward propagation. The generated virtual fundus image samples are then incorporated into the query set.

[0087] Calculate the loss function value of the generator:

[0088] In the formula, D represents the discriminator. The generated image represents the corresponding text description, which is the text vector encoded by the text encoder, and the weight parameters represent the weight parameters. The L2 norm is the difference between the generated image and the real image; the goal of updating the model parameters is to minimize the loss function; the first term in the loss function is used to control the image generated by the generative network to match the text description as closely as possible; the L2 norm loss controls the generated image to be consistent with the original image; the two work together to update the model parameters to generate an image similar to the original distribution.

[0089] S5.3 Input the sample fundus images and text features of the text descriptions in the query set into the discriminator, and obtain the true and false probabilities through forward propagation;

[0090] The virtual fundus image samples corresponding to the fundus images in the query set and the text features of the text description are input into the discriminator, and the virtual true and false probabilities are obtained by forward propagation.

[0091] Calculate the loss function value of the discriminator:

[0092]

[0093] In the formula, represents the real image, represents the matching text description, represents the text vector encoded by the text encoder, and represents the non-matching text description. The generated image is represented by the loss function, which controls the discrimination network to output a larger discrimination result (first term) when the input image and the corresponding text description are real images and corresponding text descriptions, and to output a smaller discrimination probability (second term) when the input image and the text description do not match, and to output a smaller discrimination probability (third term) when the input image and the corresponding discrimination probability are generated images.

[0094] S5.4 Input the sample fundus images from the support set into the classification system, generate the embedding space, and calculate the prototype point of each category in the embedding space;

[0095] The sample fundus images in the query set are input into the classification system. The distance from each sample fundus image to its own class prototype point is calculated. Based on this distance, the self-step learning order of real sample fundus images and virtual sample fundus images is determined respectively from small to large.

[0096] The sample fundus images from the query set are input into the classification system in the order of auto-learning to obtain the predicted class probability for each real sample.

[0097] The virtual fundus image samples corresponding to the fundus images in the query set are input into the classification system in the order of auto-learning to obtain the category prediction probability of each virtual sample; if the category prediction probability is less than the preset probability threshold, the corresponding virtual fundus image sample and the corresponding category prediction probability are deleted.

[0098] Calculate the loss function value of the classification system:

[0099] L c(x,y,z)=CE(C(x),y)+λCE(C(G(z)),argmax(C(G(z)))>t)

[0100] In the formula, x represents the real image, y represents the label, λ represents the weight parameter, z represents random Gaussian noise, C(x) represents the classification result of the classification system for x, CE(C(x), y) represents the calculation of the cross-entropy between C(x) and y, G(z) represents the generated image, argmax(C(G(z))) represents the pseudo label, and t represents the preset probability threshold. This loss represents training the classification system with real images and generated images respectively. When training with generated images, the confidence threshold of the pseudo label needs to be higher than t before the sample can be used.

[0101] Based on the loss function values ​​of the generator, discriminator, and classification system, backpropagation and parameter optimization are performed. The Adam optimization method is used, with a batch size of 64. The discriminator is optimized once for each batch of images, followed by optimization of the generator and classification system. The learning rate is set to 0.0002, until the changes in the loss function values ​​of the generator, discriminator, and classification system in two consecutive rounds of joint training are all less than a preset loss change threshold.

[0102] The fundus image to be classified is input into a trained classification system to obtain the predicted probability of the corresponding category, which is used to assist in diagnosis.

Claims

1. A fundus image classification system based on autosynchronous learning and generative adversarial networks, characterized in that, The classification system includes a downsampling layer and a fully connected layer. The input of the classification system is a fundus image, and the output is the predicted category probability of the input image. The training steps for the classification system include: S1. Construct a training set, wherein a sample in the training set consists of a sample fundus image, a lesion type label, and a text description; the resolution of all sample fundus images in the training set is unified to the same dimension, and the pixel values ​​are normalized to the range of [-1, 1]; the text description consists of multiple sentences expressing the same meaning; S2. Construct a pre-training subsystem, which includes: an image encoder, an image decoder, a text encoder, and a blood vessel segmentation network; The image encoder is used to extract image features from the input image, including two cascaded 2D convolutional units, each of which consists of a cascaded Conv2d layer and a pooling layer. The image decoder is used to reconstruct an image based on image features, and includes two cascaded ConvTranspose2d layers; The text encoder is used to extract text features from the text description of the image, including a one-hot encoding layer, three cascaded 1D convolutional units and a fully connected layer; each 1D convolutional unit consists of cascaded Conv1d layers and pooling layers; the fully connected layer transforms the output of the convolutional unit into a one-dimensional vector of length 1024. The blood vessel segmentation network is used to obtain blood vessel segmentation maps from fundus images; S3. Train the pre-training subsystem; S4. Construct a joint training subsystem, which includes: a generator, a discriminator, and a sampler; The generator is used to generate a virtual fundus image based on the text features of the blood vessel segmentation map and the image text description. The generator includes 8 cascaded downsampling units and 8 cascaded upsampling units. Each downsampling unit includes a downsampling layer, a regularization layer, and an activation function. Each upsampling unit includes an upsampling layer, a regularization layer, and an activation function. The input of the first downsampling unit is the blood vessel segmentation map. The input of the eighth downsampling unit is the concatenation vector of the output of the seventh downsampling unit and the noisy text, where the noisy text is the concatenation vector of text features and randomly sampled Gaussian noise. The input to the first upsampling unit is the output of the 8th downsampling unit, and the inputs to the 2nd to 8th upsampling units are: ; in For the first The input of the group upsampling unit, For the first The output of the group upsampling unit, For the first Output of the group downsampling unit; ; The downsampling layer in the downsampling unit and the upsampling layer in the upsampling unit are both convolutional layers with a kernel size of 3*3 and a stride of 1. The number of channels in the convolutional layers of the first to sixth downsampling units is... The number of channels in the convolutional layers of the 7th and 8th downsampling units are respectively and The number of channels in the convolutional layers of the upsampling units from group 1 to group 6 is... , The number of channels in the convolutional layers of the 7th and 8th downsampling units are respectively and The activation functions in the 8 downsampling units and the 1st to 7th upsampling units are all ReLU activation functions with negative slope L, and the activation function in the 8th upsampling unit is the Tanh function. The discriminator takes a fundus image and text features from a text description as input, and determines whether the input fundus image matches the text features of the text description. If they match, it outputs "true"; otherwise, it outputs "false". The discriminator includes a downsampling layer and a fully connected layer. The input fundus image is processed through the downsampling layer to obtain a feature vector. The feature vector is concatenated with the text features of the input text description and then input into the fully connected layer. The output is a binary number indicating whether the image is true or false. The downsampling layer includes six cascaded spectral normalization units. Each spectral normalization unit consists of a cascaded spectral normalization convolutional layer and an activation function. The kernel size of the spectral normalization convolutional layer is 4, and the stride is 2. The activation function in the first to fifth spectral normalization units is LeakyReLU with a negative slope of 0.2, and the activation function in the sixth spectral normalization unit is sigmoid. The sampler is used to construct a support set and a query set for classification during training. Specifically, it extracts m samples from each of the c classes, for a total of c*m samples. c*k samples are assigned to the support set, and the remaining c*(mk) samples are assigned to the query set. S5. The generator, discriminator, and classification system are jointly trained using the joint training subsystem; the joint training includes multiple rounds, and the specific steps of one round of joint training include: S5.1 The sampler extracts m samples from each of the c classes in the training set, for a total of c*m samples. c*k samples are assigned to the support set, and the remaining c*(mk) samples are assigned to the query set. S5.2 Generator generates virtual samples: The sample fundus images in the query set are obtained by the blood vessel segmentation network to obtain blood vessel segmentation maps. The blood vessel segmentation maps and corresponding text descriptions are input into the generator, and virtual fundus image samples are generated by forward propagation. The generated virtual fundus image samples are then incorporated into the query set. Calculate the loss function value of the generator: ; In the formula, D represents the discriminator. This represents the generated image. This indicates the corresponding text description. The text vector encoded by the text encoder. Represents the weight parameters. It is the L2 norm of the generated image and the real image; the objective of updating the model parameters is to minimize the loss function; S5.3 Input the sample fundus images and text features of the text descriptions in the query set into the discriminator, and obtain the true and false probabilities through forward propagation; The virtual fundus image samples corresponding to the fundus images in the query set and the text features of the text description are input into the discriminator, and the virtual true and false probabilities are obtained by forward propagation. Calculate the loss function value of the discriminator: ； In the formula, Represents a real image. This indicates a matching text description. This represents the text vector after being encoded by the text encoder. Indicates and Mismatched text descriptions This represents the generated image; S5.4 Input the sample fundus images from the support set into the classification system, generate the embedding space, and calculate the prototype point of each category in the embedding space; The sample fundus images in the query set are input into the classification system. The distance from each sample fundus image to its own class prototype point is calculated. Based on this distance, the self-step learning order of real sample fundus images and virtual sample fundus images is determined respectively from small to large. The sample fundus images from the query set are input into the classification system in the order of auto-learning to obtain the predicted class probability for each real sample. The virtual fundus image samples corresponding to the fundus images in the query set are input into the classification system in the order of auto-learning to obtain the category prediction probability of each virtual sample; if the category prediction probability is less than the preset probability threshold, the corresponding virtual fundus image sample and the corresponding category prediction probability are deleted. Calculate the loss function value of the classification system: ； In the formula, Represents a real image. The corresponding tags Represents the weight parameters. Represents random Gaussian noise. The classification system represents the pair The classification results Indicates calculation and cross-entropy, The generated image, Indicates a pseudo tag. This represents the preset probability threshold; the loss represents the training of the classification system using real images and generated images, respectively. When training with generated images, the confidence threshold of the pseudo-labels needs to be higher than [a certain threshold]. Only then can the sample be used; Based on the loss function values ​​of the generator, discriminator, and classification system, backpropagation and parameter optimization are performed until the changes in the loss function values ​​of the generator, discriminator, and classification system in two consecutive rounds of joint training are all less than the preset loss change threshold.

2. The fundus image classification system according to claim 1, characterized in that, The text description consists of five sentences expressing the same meaning.

3. The fundus image classification system according to claim 1, characterized in that, In step S5.4, the optimization method for the generator, discriminator, and classification system is Adam, the batch size is 64, the discriminator is optimized once for each batch of images, and then the generator and classification system are optimized once, with the learning rate set to 0.0002.

4. The fundus image classification system according to claim 1, characterized in that, The blood vessel segmentation network uses a U-net network.

5. The fundus image classification system according to claim 1, characterized in that, Step S3 includes: inputting sample fundus images from the training set into the image encoder, inputting the extracted image features into the image decoder, and obtaining the reconstructed image of the input image; calculating the BCELoss of the reconstructed image and the input sample fundus images, and then using the Adam optimizer to optimize and train the image encoder and image decoder.

6. The fundus image classification system according to claim 1, characterized in that, Step S3 includes: inputting the text description of the sample fundus images in the training set into a text encoder to obtain the output text features; inputting the sample fundus images into a trained image encoder to obtain the output image features; and optimizing the parameters of the text encoder by minimizing the structured joint embedding loss of image features and text features.

7. The fundus image classification system according to claim 1, characterized in that, Step S3 includes: optimizing the parameters of the blood vessel segmentation network using the DRIVE dataset.

Images