Multi-modal information network-based glaucoma different-stage discrimination system

Abstract

The invention discloses a glaucoma different-stage discrimination system based on a multi-modal information network, relates to a glaucoma different-stage discrimination system, and belongs to the computer vision and machine learning technology. The invention aims to solve the problem of low judgment accuracy of different stages of glaucoma in the prior art. The system comprises a data acquisition module, a data preprocessing module, a super-resolution reconstruction network model, an image-text multi-modal convolutional network model and a to-be-tested module. The data acquisition module is used for acquiring an OCT image, an OCTA image, an RNFL image and medical record information; the data preprocessing module is used for obtaining a preprocessed OCT image training set, an OCTA image training set, an RNFL image training set and medical record information; the super-resolution reconstruction network model is used for obtaining a trained super-resolution reconstruction network model; the image-text multi-modal convolutional network model is used for obtaining a trained image-text multi-modal convolutional network model; and the to-be-tested module is used for outputting the probabilities of glaucoma in different periods of the to-be-tested data.

Description

Technical Field

[0001] The invention relates to a system for distinguishing different stages of glaucoma, and belongs to computer vision and machine learning technology. Background Art

[0002] Glaucoma is characterized by its progressive nature and can cause vision loss and irreversible blindness if not detected and treated promptly. Early identification and prevention of glaucoma are critical to addressing this serious eye disease.

[0003] However, traditional glaucoma diagnostic methods are not only time-consuming, but also easily affected by human factors during the operation, resulting in errors in the diagnostic results and low overall diagnostic efficiency. These problems limit the application of traditional diagnostic methods in glaucoma screening and management.

[0004] In order to improve the accuracy of diagnosis and simplify the diagnostic process, the introduction of automated diagnostic technology has become particularly important and necessary.

[0005] There is already an automated glaucoma diagnosis system based on GoogleNet. The system uses a sliding window approach combined with a network trained on manually extracted OCTA structures and region of interest (ROI) sub-images. After training, the algorithm showed good accuracy even with images from a poor quality database. Implementing the preprocessing step of the database images adds computational overhead that can slow down the overall workflow and increase resource requirements.

[0006] The accuracy rate (AR) of DenseNet169 reached 85.19%, indicating that it is effective in distinguishing mild from severe glaucoma. However, DenseNet169 is a deep neural network with a large number of parameters, making its training computationally intensive and requiring a lot of computing resources.

[0007] The CAD framework for assisting social professionals in glaucoma screening achieved a high accuracy of 96% in glaucoma classification, showing its potential utility in glaucoma detection for examining various eye conditions. However, the CAD framework with DLN requires a large and diverse dataset for effective training, which can be challenging, especially for rare diseases or specific patient populations. Summary of the invention

[0008] The purpose of the present invention is to solve the problem of low accuracy in distinguishing different stages of glaucoma, and to propose a system for distinguishing different stages of glaucoma based on a multimodal information network.

[0009] The glaucoma different stage discrimination system based on multimodal information network includes: data acquisition module, data preprocessing module, super-resolution reconstruction network model, image and text multimodal convolution network model and test module;

[0010] The data acquisition module is used to collect fundus optical coherence tomography (OCT) image training sets, fundus optical coherence tomography (OCTA) image training sets, RNFL image training sets, and medical record information; the medical record information includes the patient's gender and age information;

[0011] The data preprocessing module is used to preprocess the fundus optical coherence tomography (OCT) image training set, the fundus optical coherence tomography (OCTA) image training set, the RNFL image training set and the medical record information collected by the data acquisition module, respectively, to obtain the preprocessed fundus optical coherence tomography (OCT) image training set, the fundus optical coherence tomography (OCTA) image training set, the RNFL image training set and the medical record information; the medical record information includes the patient's gender and age information;

[0012] The super-resolution reconstruction network model is used to use the preprocessed fundus optical coherence tomography (OCT) image training set, fundus optical coherence tomography (OCTA) image training set and RNFL image training set as inputs of the super-resolution reconstruction network model, and the high-resolution OCT image training set, OCTA image training set and RNFL image training set as outputs of the super-resolution reconstruction network model, to obtain a trained super-resolution reconstruction network model;

[0013] The image-text multimodal convolutional network model is used to use the high-resolution OCT images, OCTA images, RNFL images, age, gender, and joint features output by the trained super-resolution reconstruction network model as inputs of the image-text multimodal convolutional network model, and the probability of glaucoma at different stages as outputs of the image-text multimodal convolutional network model, to obtain a trained image-text multimodal convolutional network model;

[0014] The module to be tested is used to obtain the OCT images, OCTA images, RNFL images, age and gender information in the cases to be tested, preprocess the OCT images, OCTA images, RNFL images, age and gender information in the cases to be tested, and obtain the preprocessed OCT images, OCTA images, RNFL images, gender, age and joint indicators; the preprocessed OCT images, OCTA images, and RNFL images are input into the trained super-resolution reconstruction network model, and the generator in the trained super-resolution reconstruction network model outputs high-resolution OCT images, OCTA images, and RNFL images; the high-resolution OCT images, OCTA images, and RNFL images output by the generator and the preprocessed gender, age, and joint indicators are input into the trained graphic multimodal convolutional network model, and the trained graphic multimodal convolutional network model outputs the probabilities of different stages of glaucoma, including early, middle and late stages of glaucoma.

[0015] Preferably, the data acquisition module is used to collect fundus optical coherence tomography (OCT) image training set, fundus optical coherence tomography (OCTA) image training set, RNFL image training set, and medical record information; the specific process is:

[0016] The fundus optical coherence tomography (OCT) image training set includes labeled normal OCT images, early glaucoma OCT images, mid-stage glaucoma OCT images, and late-stage glaucoma OCT images;

[0017] The fundus optical coherence tomography (OCTA) image training set includes labeled normal OCTA images, early glaucoma OCTA images, mid-stage glaucoma OCTA images, and late-stage glaucoma OCTA images;

[0018] The RNFL image training set includes labeled normal RNFL images, early glaucoma RNFL images, mid-stage glaucoma RNFL images, and late-stage glaucoma RNFL images;

[0019] The OCT is optical coherence tomography technology;

[0020] The OCTA is optical coherence tomography angiography;

[0021] The RNFL is the retinal nerve fiber layer.

[0022] Preferably, the data preprocessing module is used to preprocess the fundus optical coherence tomography OCT image training set, the fundus optical coherence tomography OCTA image training set, the RNFL image training set and the medical record information collected by the data acquisition module, respectively, to obtain the preprocessed fundus optical coherence tomography OCT image training set, the fundus optical coherence tomography OCTA image training set, the RNFL image training set and the medical record information; the specific process is:

[0023] 1) Preprocessing the fundus optical coherence tomography (OCT) image training set collected by the data acquisition module to obtain a preprocessed fundus optical coherence tomography (OCT) image training set; the specific process is as follows:

[0024] 11) Use median filtering technology to denoise the OCT image to obtain a denoised OCT image;

[0025] 12) Use the Sobel edge detection technology to locate the boundary of the denoised OCT image to obtain an OCT image with clear boundaries;

[0026] 13) Adaptive thresholding technology was used to segment the OCT images with clear boundaries into vitreous, retinal nerve fiber layer, ganglion cell layer, inner nuclear layer, outer plexiform layer, outer nuclear layer and retinal pigment epithelium;

[0027] 14) Adjust the layered OCT images to 224×224 pixels, normalize the OCT images adjusted to 224×224 pixels, and obtain a preprocessed OCT image training set;

[0028] 2) Preprocessing the fundus optical coherence tomography (OCTA) image training set collected by the data acquisition module to obtain a preprocessed fundus optical coherence tomography (OCTA) image training set; the specific process is as follows:

[0029] 21) Using the diffusion model to denoise the OCTA image to obtain a denoised OCTA image;

[0030] 22) Calculate the Hessian matrix of each pixel in the denoised OCTA image;

[0031] 23) Calculate the eigenvalue of the Hessian matrix of each pixel and identify the potential vascular structure in the OCTA image based on the eigenvalue;

[0032] 24) Design a matched filter to enhance the identified potential vascular structure and suppress background noise to obtain OCTA images;

[0033] 25) Processing the OCTA image obtained in 24) using adaptive threshold technology to obtain a screened OCTA image;

[0034] 26) adjusting the screened OCTA images obtained in 25) to 224×224 pixels, normalizing the OCTA images adjusted to 224×224 pixels, and obtaining a preprocessed OCTA image training set;

[0035] 3) Preprocess the RNFL image training set collected by the data acquisition module to obtain the preprocessed RNFL image training set; the specific process is:

[0036] The RNFL image is adjusted to 224×224 pixels, and the RNFL image adjusted to 224×224 pixels is normalized to obtain a preprocessed RNFL image training set;

[0037] 4) Preprocess the medical record information collected by the data collection module to obtain preprocessed medical record information; the preprocessed medical record information includes preprocessed age, processed gender, and processed joint features.

[0038] Preferably, in said 4), the medical record information collected by the data collection module is preprocessed to obtain preprocessed medical record information; the preprocessed medical record information includes preprocessed age, processed gender, and processed joint features;

[0039] The specific process is:

[0040] 41) Preprocess the age data to obtain preprocessed age data; the specific process is:

[0041] The age groups are divided into four stages: 0-17, 18-39, 40-79, and 80 and above;

[0042] The value corresponding to 0-17 years old is 1; the value corresponding to 18-39 years old is 2; the value corresponding to 40-79 years old is 3; the value corresponding to 80 years old and above is 4;

[0043] 42) Preprocess the gender data to obtain preprocessed gender data; the specific process is:

[0044] Use the get_dummies function in the pandas library to convert the gender variable into a dummy variable;

[0045] 43) Based on the preprocessed age data and the preprocessed gender data, calculate the joint feature; expressed as:

[0046] Joint feature = Beta1×age+Beta2×gender+Beta3

[0047] Among them, Beta1, Beta2, and Beta3 are coefficients;

[0048] The process of obtaining the coefficients Beta1, Beta2, and Beta3 is as follows:

[0049] Based on the joint features, the Logistic regression model is trained to obtain the predicted probability of suffering from glaucoma. The roc_auc_score function in the scikit-learn library is called to calculate the area under the ROC curve AUC corresponding to the Logistic regression model. At this point, one cycle ends.

[0050] Modify the values ​​of coefficients Beta1, Beta2, and Beta3 in each cycle, and take the set of Beta1, Beta2, and Beta3 coefficients that makes the AUC value closest to 1 as the optimal Beta1, Beta2, and Beta3 coefficients;

[0051] Calculate the joint features based on the optimal Beta1, Beta2, and Beta3 coefficients.

[0052] Preferably, the super-resolution reconstruction network model is used to use the preprocessed fundus optical coherence tomography (OCT) image training set, fundus optical coherence tomography (OCTA) image training set and RNFL image training set as inputs of the super-resolution reconstruction network model, and the high-resolution OCT image training set, OCTA image training set and RNFL image training set as outputs of the super-resolution reconstruction network model to obtain a trained super-resolution reconstruction network model; the specific process is:

[0053] 1) The super-resolution reconstruction network model includes a data conversion module, a generator network and a discriminator network in sequence;

[0054] The data conversion module includes: a high-resolution conversion module and a low-resolution conversion module;

[0055] The generator network includes: input layer, initial convolution layer, residual block, upsampling block, final convolution layer, and output layer;

[0056] The discriminator network includes: input layer, convolution layer, batch normalization layer, LeakyReLU activation function layer, adaptive average pooling layer, fully connected layer, and output layer;

[0057] 2) Input the preprocessed OCT image into the data conversion module, and the data conversion module outputs a pair of low-resolution and high-resolution images corresponding to the OCT image;

[0058] The preprocessed OCTA image is input into a data conversion module, and the data conversion module outputs a pair of low-resolution and high-resolution images corresponding to the OCTA image;

[0059] The preprocessed RNFL images are respectively input into a data conversion module, and the data conversion module outputs a pair of low-resolution and high-resolution images corresponding to the RNFL images;

[0060] 3) The low-resolution images corresponding to the OCT images, OCTA images, and RNFL images output by the data conversion module are respectively input into the generator network, and the generator network outputs high-resolution images respectively;

[0061] 4) Inputting the high-resolution image corresponding to the OCT image output by the data conversion module or the high-resolution image corresponding to the OCT image output by the generator network into the discriminator network, the discriminator network determines the probability P that the output image is the high-resolution image corresponding to the OCT image output by the data conversion module;

[0062] The high-resolution image corresponding to the OCTA image output by the data conversion module or the high-resolution image corresponding to the OCTA image output by the generator network is input into the discriminator network, and the discriminator network determines the probability P that the output image is the high-resolution image corresponding to the OCTA image output by the data conversion module;

[0063] The high-resolution image corresponding to the RNFL image output by the data conversion module or the high-resolution image corresponding to the RNFL image output by the generator network is input into the discriminator network, and the discriminator network determines the probability P that the output image is the high-resolution image corresponding to the RNFL image output by the data conversion module;

[0064] 5) Repeat 1) to 5) until the probability P that the discriminator network determines that the output image is a high-resolution image corresponding to the OCT image output by the data conversion module, the probability P that the discriminator network determines that the output image is a high-resolution image corresponding to the OCTA image output by the data conversion module, and the probability P that the discriminator network determines that the output image is a high-resolution image corresponding to the RNFL image output by the data conversion module all reach the maximum, and a trained deep learning module is obtained.

[0065] Preferably, in 2), the preprocessed OCT image is input into a data conversion module, and the data conversion module outputs a pair of low-resolution and high-resolution images corresponding to the OCT image;

[0066] The preprocessed OCTA image is input into a data conversion module, and the data conversion module outputs a pair of low-resolution and high-resolution images corresponding to the OCTA image;

[0067] The preprocessed RNFL images are respectively input into a data conversion module, and the data conversion module outputs a pair of low-resolution and high-resolution images corresponding to the RNFL images;

[0068] The specific process is:

[0069] The data conversion module includes: a high-resolution conversion module and a low-resolution conversion module;

[0070] The pre-processed OCT images, OCTA images and RNFL images are respectively input into the high-resolution conversion module, and the high-resolution conversion module outputs high-resolution images respectively;

[0071] The pre-processed OCT images, OCTA images and RNFL images are respectively input into low-resolution conversion modules, and the low-resolution conversion modules output low-resolution images respectively;

[0072] The pre-processed OCT image, OCTA image and RNFL image are respectively input into the high-resolution conversion module, and the high-resolution conversion module outputs high-resolution images respectively; the process is:

[0073] The preprocessed OCT images, OCTA images and RNFL images are randomly cropped, and the randomly cropped OCT images, OCTA images and RNFL images are converted into tensors respectively;

[0074] The pre-processed OCT images, OCTA images and RNFL images are respectively input into the low-resolution conversion module, and the low-resolution conversion module outputs low-resolution images respectively; the process is:

[0075] The preprocessed OCT images, OCTA images and RNFL images are converted into PIL images respectively, and the PIL images are resized to be equal to the size of the A image by bicubic interpolation, and the resized PIL images are converted into tensors;

[0076] The A image size is the result of dividing the image size after random cropping in the high-resolution conversion module by the upsampling factor.

[0077] Preferably, in said 3), the low-resolution images corresponding to the OCT images, OCTA images, and RNFL images output by the data conversion module are respectively input into the generator network, and the generator network outputs high-resolution images respectively;

[0078] The specific process is:

[0079] The generator network includes: input layer, initial convolution layer, residual block, upsampling block, final convolution layer, and output layer;

[0080] Among them, the initial convolution layer includes a convolution layer and a pre-activation function layer in sequence;

[0081] The residual block includes convolution layer, batch normalization layer, pre-activation function layer, convolution layer, batch normalization layer in sequence;

[0082] The upsampling block includes convolution layer, PixelShuffle upsampling, and pre-activation function layer in sequence;

[0083] The final convolutional layer includes convolutional layer and tanh activation function layer in sequence;

[0084] The low-resolution images corresponding to the OCT images, OCTA images, and RNFL images output by the data conversion module are respectively input into the initial convolution layer, the residual block, the upsampling block, the final convolution layer, and the output layer through the input layer, and the output layer outputs high-resolution images.

[0085] Preferably, in said 4), the high-resolution image corresponding to the OCT image output by the data conversion module or the high-resolution image corresponding to the OCT image output by the generator network is input into the discriminator network, and the discriminator network determines the probability P that the output image is the high-resolution image corresponding to the OCT image output by the data conversion module;

[0086] The high-resolution image corresponding to the OCTA image output by the data conversion module or the high-resolution image corresponding to the OCTA image output by the generator network is input into the discriminator network, and the discriminator network determines the probability P that the output image is the high-resolution image corresponding to the OCTA image output by the data conversion module;

[0087] The high-resolution image corresponding to the RNFL image output by the data conversion module or the high-resolution image corresponding to the RNFL image output by the generator network is input into the discriminator network, and the discriminator network determines the probability P that the output image is the high-resolution image corresponding to the RNFL image output by the data conversion module;

[0088] The specific process is:

[0089] The discriminator network includes: input layer, convolution layer, batch normalization layer, LeakyReLU activation function layer, adaptive average pooling layer, fully connected layer, and output layer;

[0090] The high-resolution image corresponding to the OCT image output by the data conversion module or the high-resolution image corresponding to the OCT image output by the generator network is sequentially input into the convolution layer, batch normalization layer, LeakyReLU activation function layer, adaptive average pooling layer, fully connected layer, and output layer through the input layer. The output layer determines the probability P that the output image is the high-resolution image corresponding to the OCT image output by the data conversion module;

[0091] The high-resolution image corresponding to the OCTA image output by the data conversion module or the high-resolution image corresponding to the OCTA image output by the generator network is sequentially input into the convolution layer, batch normalization layer, LeakyReLU activation function layer, adaptive average pooling layer, fully connected layer, and output layer through the input layer. The output layer determines the probability P that the output image is the high-resolution image corresponding to the OCTA image output by the data conversion module;

[0092] The high-resolution image corresponding to the RNFL image output by the data conversion module or the high-resolution image corresponding to the RNFL image output by the generator network is input into the convolution layer, batch normalization layer, LeakyReLU activation function layer, adaptive average pooling layer, fully connected layer, and output layer in sequence through the input layer. The output layer determines the probability P that the output image is the high-resolution image corresponding to the RNFL image output by the data conversion module.

[0093] Preferably, 1) to 5) are repeatedly performed in 5) until the probability P of the discriminator network determining that the output image is a high-resolution image corresponding to the OCT image output by the data conversion module, the probability P of the discriminator network determining that the output image is a high-resolution image corresponding to the OCTA image output by the data conversion module, and the probability P of the discriminator network determining that the output image is a high-resolution image corresponding to the RNFL image output by the data conversion module all reach a maximum, thereby obtaining a trained deep learning module;

[0094] The specific process is:

[0095] Set the total discriminator loss Loss_D = 1-Loss_real+Loss_fake

[0096] Loss_real = -log(D(x))

[0097] D(x) is the probability mean of the high-resolution image output by the discriminator network as the high-resolution image x converted by the data conversion module;

[0098] Loss_fake = -log(1-D(G(z)))

[0099] G(z) is the probability mean that the high-resolution image output by the discriminator network is the high-resolution image generated by the generator network;

[0100] Use the Adam optimizer to update the discriminator parameter weights so that the high-resolution image output by the discriminator network is closer to the high-resolution image converted by the data conversion module;

[0101] Set the total generator loss Loss_G:

[0102] Loss_G=Adversarial_Loss+λ p Perception_Loss+λ i Image_Loss+λ tv TV_Loss

[0103] Adversarial_Loss is the adversarial loss;

[0104] Perception_Loss is the perception loss;

[0105] Image_Loss is the image loss;

[0106] TV_Loss is the total variation loss;

[0107] where λ p , i , tv is the weight;

[0108] Use the Adam optimizer to update the generator parameter weights so that the high-resolution image generated by the generator is closer to the high-resolution image converted by the data conversion module;

[0109] Repeat 1) to 5), alternately updating the generator and discriminator until the total discriminator loss and the total generator loss converge to obtain a trained super-resolution reconstruction network model.

[0110] Preferably, the image-text multimodal convolutional network model is used to use the high-resolution OCT images, OCTA images, RNFL images and age, gender, and joint features output by the trained super-resolution reconstruction network model as the input of the image-text multimodal convolutional network model, and the probability of glaucoma at different stages as the output of the image-text multimodal convolutional network model to obtain a trained image-text multimodal convolutional network model; the specific process is:

[0111] The image-text multimodal convolutional network model includes a feature extraction network and a feature fusion network;

[0112] The feature extraction network includes an image feature extraction network and a text information feature extraction network;

[0113] The image feature extraction network includes: input layer, convolution layer, batch normalization layer, ReLU activation function layer, 50 residual blocks, global average pooling layer, and output layer;

[0114] Each residual block contains a convolutional layer and an identity connection;

[0115] The text information feature extraction network includes: input layer, embedding layer, convolution layer, maximum pooling layer, and output layer;

[0116] The feature fusion network includes: input layer, concatenation layer, fully connected layer, softmax layer and output layer;

[0117] The high-resolution OCT image, OCTA image, and RNFL image are respectively input into the image feature extraction network, and the image feature extraction network outputs the OCT image feature vector, the OCTA image feature vector, and the RNFL image feature vector respectively;

[0118] Age, gender, and joint features are input into the text information feature extraction network respectively, and the text information feature extraction network outputs feature vectors of age, gender, and joint features respectively;

[0119] The OCT image feature vector, OCTA image feature vector, RNFL image feature vector and the feature vectors of age, gender and joint features are used as inputs of the feature fusion network, and the disease probability features of glaucoma at different stages are output by the fusion network;

[0120] Define the cross entropy loss function;

[0121] Use the Adam optimization algorithm to update the parameters of the feature extraction network and the feature fusion network until the cross entropy loss function converges to obtain a trained image-text multimodal convolutional network model.

[0122] The beneficial effects of the present invention are:

[0123] The present invention first inputs fundus optical coherence tomography (OCT) images, labeled fundus optical coherence tomography (OCTA) images, and retinal nerve fiber layer (RNFL) images into a super-resolution reconstruction module to improve the resolution of the images and facilitate subsequent diagnosis and prediction. Then, when the image data is insufficient, consider adding the text information in the medical record to the diagnosis and prediction of glaucoma, which helps to reduce the amount of data required for the image. The text information in the medical record is jointly analyzed to find the joint indicators that have the greatest impact on the diagnosis of glaucoma. Finally, the image, text information, and joint indicators are all input into a graphic multimodal convolutional neural network to output the probability of different stages of glaucoma, including the early, middle, and late stages of glaucoma. BRIEF DESCRIPTION OF THE DRAWINGS

[0124] Figure 1 The figure is a flow chart of the system of the present invention. DETAILED DESCRIPTION

[0125] Specific implementation method 1: In this implementation method, the system for distinguishing different stages of glaucoma based on a multimodal information network includes: a data acquisition module, a data preprocessing module, a super-resolution reconstruction network model, a graphic multimodal convolutional network model and a module to be tested;

[0126] The data acquisition module is used to collect fundus optical coherence tomography (OCT) image training sets, fundus optical coherence tomography (OCTA) image training sets, RNFL image training sets, and medical record information; the medical record information includes the patient's gender and age information;

[0127] The data preprocessing module is used to preprocess the fundus optical coherence tomography (OCT) image training set, the fundus optical coherence tomography (OCTA) image training set, the RNFL image training set and the medical record information collected by the data acquisition module, respectively, to obtain the preprocessed fundus optical coherence tomography (OCT) image training set, the fundus optical coherence tomography (OCTA) image training set, the RNFL image training set and the medical record information; the medical record information includes the patient's gender and age information;

[0128] The super-resolution reconstruction network model is used to use the preprocessed fundus optical coherence tomography (OCT) image training set, fundus optical coherence tomography (OCTA) image training set and RNFL image training set as inputs of the super-resolution reconstruction network model, and the high-resolution OCT image training set, OCTA image training set and RNFL image training set as outputs of the super-resolution reconstruction network model, to obtain a trained super-resolution reconstruction network model;

[0129] The image-text multimodal convolutional network model is used to use the high-resolution OCT images, OCTA images, RNFL images, age, gender, and joint features output by the trained super-resolution reconstruction network model as inputs of the image-text multimodal convolutional network model, and the probability of glaucoma at different stages as outputs of the image-text multimodal convolutional network model, to obtain a trained image-text multimodal convolutional network model;

[0130] The module to be tested is used to obtain the OCT images, OCTA images, RNFL images, age and gender information in the cases to be tested, preprocess the OCT images, OCTA images, RNFL images, age and gender information in the cases to be tested, and obtain the preprocessed OCT images, OCTA images, RNFL images, gender, age and joint indicators; the preprocessed OCT images, OCTA images, and RNFL images are input into the trained super-resolution reconstruction network model, and the generator in the trained super-resolution reconstruction network model outputs high-resolution OCT images, OCTA images, and RNFL images; the high-resolution OCT images, OCTA images, and RNFL images output by the generator and the preprocessed gender, age, and joint indicators are input into the trained graphic multimodal convolutional network model, and the trained graphic multimodal convolutional network model outputs the probabilities of different stages of glaucoma, including early, middle and late stages of glaucoma.

[0131] Multimodal information refers to OCT images, OCTA images, RNFL images, gender, age, and combined indicators;

[0132] Multimodal information network refers to the super-resolution reconstruction network model and the image-text multimodal convolutional network model.

[0133] Specific implementation method 2: This implementation method is different from the specific implementation method 1 in that: the data acquisition module is used to collect fundus optical coherence tomography (OCT) image training set, fundus optical coherence tomography (OCTA) image training set, RNFL image training set, and medical record information; the specific process is:

[0134] The fundus optical coherence tomography (OCT) image training set includes labeled normal OCT images, early glaucoma OCT images, mid-stage glaucoma OCT images, and late-stage glaucoma OCT images;

[0135] The fundus optical coherence tomography (OCTA) image training set includes labeled normal OCTA images, early glaucoma OCTA images, mid-stage glaucoma OCTA images, and late-stage glaucoma OCTA images;

[0136] The RNFL image training set includes labeled normal RNFL images, early glaucoma RNFL images, mid-stage glaucoma RNFL images, and late-stage glaucoma RNFL images;

[0137] The OCT is optical coherence tomography technology;

[0138] The OCTA is optical coherence tomography angiography;

[0139] The RNFL is the retinal nerve fiber layer.

[0140] The other steps and parameters are the same as those in the first embodiment.

[0141] Specific implementation method three: This implementation method is different from specific implementation method one or two in that: the data preprocessing module is used to preprocess the fundus optical coherence tomography OCT image training set, fundus optical coherence tomography OCTA image training set, RNFL image training set and medical record information collected by the data acquisition module, respectively, to obtain the preprocessed fundus optical coherence tomography OCT image training set, fundus optical coherence tomography OCTA image training set, RNFL image training set and medical record information; the specific process is:

[0142] 1) Preprocessing the fundus optical coherence tomography (OCT) image training set collected by the data acquisition module to obtain a preprocessed fundus optical coherence tomography (OCT) image training set; the specific process is as follows:

[0143] 11) Use median filtering technology to denoise the OCT image to obtain a denoised OCT image;

[0144] 12) Use the Sobel edge detection technology to locate the boundary of the denoised OCT image to obtain an OCT image with clear boundaries;

[0145] Since the boundaries between retinal layers in OCT images often show obvious gradient changes, the Sobel edge detection technique is used to locate the boundaries of the denoised OCT images;

[0146] 13) Adaptive thresholding technology was used to segment the OCT images with clear boundaries into vitreous, retinal nerve fiber layer, ganglion cell layer, inner nuclear layer, outer plexiform layer, outer nuclear layer and retinal pigment epithelium;

[0147] 14) Adjust the layered OCT images to 224×224 pixels (there are 7 layers inside the OCT image), perform minimum-maximum normalization on the OCT images adjusted to 224×224 pixels, and obtain the preprocessed OCT image training set;

[0148] Taking advantage of the different reflection intensities of different retinal layers, adaptive thresholding technology is used to segment the boundaries. The retina is segmented into seven main layers;

[0149] 2) Preprocessing the fundus optical coherence tomography (OCTA) image training set collected by the data acquisition module to obtain a preprocessed fundus optical coherence tomography (OCTA) image training set; the specific process is as follows:

[0150] 21) Using the diffusion model to denoise the OCTA image to obtain a denoised OCTA image;

[0151] 22) Calculate the Hessian matrix of each pixel in the denoised OCTA image;

[0152] 23) Calculate the eigenvalue of the Hessian matrix of each pixel and identify the potential vascular structure in the OCTA image based on the eigenvalue;

[0153] 24) Design a matched filter to enhance the identified potential vascular structure and suppress background noise to obtain OCTA images;

[0154] 25) Processing the OCTA image obtained in 24) using adaptive threshold technology to obtain a screened OCTA image;

[0155] 26) adjusting the screened OCTA images obtained in 25) to 224×224 pixels, performing minimum-maximum normalization on the OCTA images adjusted to 224×224 pixels, and obtaining a preprocessed OCTA image training set;

[0156] 3) Preprocess the RNFL image training set collected by the data acquisition module to obtain the preprocessed RNFL image training set; the specific process is:

[0157] The RNFL image is adjusted to 224×224 pixels, and the RNFL image adjusted to 224×224 pixels is subjected to minimum-maximum normalization to obtain a preprocessed RNFL image training set;

[0158] 4) Preprocess the medical record information collected by the data collection module to obtain preprocessed medical record information; the preprocessed medical record information includes preprocessed age, processed gender, and processed joint features.

[0159] The other steps and parameters are the same as those in the first or second embodiment.

[0160] Specific implementation method 4: This implementation method is different from specific implementation methods 1 to 3 in that: in the above 4), the medical record information collected by the data collection module is preprocessed to obtain preprocessed medical record information; the preprocessed medical record information includes the preprocessed age, processed gender, and processed joint features;

[0161] The specific process is:

[0162] 41) Preprocess the age data to obtain preprocessed age data; the specific process is:

[0163] The age groups are divided into four stages: 0-17, 18-39, 40-79, and 80 and above;

[0164] The value corresponding to 0-17 years old is 1; the value corresponding to 18-39 years old is 2; the value corresponding to 40-79 years old is 3; the value corresponding to 80 years old and above is 4;

[0165] Since age is continuous data, converting it into discrete data can simplify the analysis process, so the age data is binned. According to research, one in every 200 people over 40 years old has glaucoma, and one in every eight people over 80 years old has glaucoma. Therefore, the age is divided into 0-17 years old, 18-39 years old, 40-79 years old, 80 years old and above.

[0166] 42) Preprocess the gender data to obtain preprocessed gender data; the specific process is:

[0167] Use the get_dummies function in the pandas library to convert the gender variable into a dummy variable;

[0168] Since gender is non-numeric data, the get_dummies function in the pandas library is used to convert the gender variable into a dummy variable, that is, one-hot encoding, that is, creating a new binary column (0 or 1) for each gender category, 1 means belonging to the category, and 0 means not belonging to the category.

[0169] 43) Based on the preprocessed age data and the preprocessed gender data, calculate the joint feature; expressed as:

[0170] Joint feature = Beta1×age+Beta2×gender+Beta3

[0171] Among them, Beta1, Beta2, and Beta3 are all coefficients.

[0172] The process of obtaining the coefficients Beta1, Beta2, and Beta3 is as follows:

[0173] Based on the joint features, the Logistic regression model is trained to obtain the predicted probability of suffering from glaucoma. The roc_auc_score function in the scikit-learn library is called to calculate the area under the ROC curve AUC corresponding to the Logistic regression model. At this point, one cycle ends.

[0174] Modify the values ​​of coefficients Beta1, Beta2, and Beta3 in each cycle, and take the set of Beta1, Beta2, and Beta3 coefficients that makes the AUC value closest to 1 as the optimal Beta1, Beta2, and Beta3 coefficients;

[0175] Calculate the joint features based on the optimal Beta1, Beta2, and Beta3 coefficients.

[0176] Medical records contain information about the patient's gender and age. Glaucoma can occur in people of any age, but the probability of developing the disease increases greatly in people over 40 years old. Therefore, age is positively correlated with the probability of developing glaucoma. The older you are, the higher the probability of developing glaucoma. For women, the risk of developing glaucoma increases accordingly before and after menopause due to changes in hormone levels. Therefore, gender factors also have a certain impact on the probability of developing glaucoma.

[0177] For continuous diagnostic indicators, a linear combination of multiple diagnostic indicators is usually used as a new indicator, and this new indicator is used as the basis for diagnosis. Since the diagnostic outcome variable is a binary variable, that is, whether or not the patient has glaucoma, the Logistic regression model is used to predict the probability of glaucoma based on the combined indicators, and the Beta coefficient (OR = exp(Beta), Beta = log(OR)) in the Logistic regression model is used to explain the influence of the independent variable (feature) on the dependent variable (outcome variable) in the model.

[0178] Use a for loop structure. In each loop, input the Beta coefficients Beta1, Beta2, and Beta3 of age, gender, and intercept term to obtain the joint features of this loop. Use the joint features to train a Logistic regression model to obtain the predicted probability of having glaucoma. Call the roc_auc_score function in the scikit-learn library to calculate the area under the ROC curve AUC. At this point, a loop ends. Modify the Beta coefficient in each loop so that the calculated AUC value is close to 1. The set of Beta coefficients that makes the AUC value closest to 1 is the desired Beta coefficient.

[0179] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0180] Specific embodiment 5: This embodiment is different from any one of specific embodiments 1 to 4 in that: the super-resolution reconstruction network model is used to use the preprocessed fundus optical coherence tomography (OCT) image training set, fundus optical coherence tomography (OCTA) image training set and RNFL image training set as the input of the super-resolution reconstruction network model, and the high-resolution OCT image training set, OCTA image training set and RNFL image training set as the output of the super-resolution reconstruction network model to obtain a trained super-resolution reconstruction network model; the specific process is:

[0181] 1) The super-resolution reconstruction network model includes a data conversion module, a generator network and a discriminator network in sequence;

[0182] The data conversion module includes: a high-resolution conversion module and a low-resolution conversion module;

[0183] The generator network includes: input layer, initial convolution layer, residual block, upsampling block, final convolution layer, and output layer;

[0184] The discriminator network includes: input layer, convolution layer (Conv), batch normalization layer (BatchNorm2d), LeakyReLU activation function layer, adaptive average pooling layer (AdaptiveAvgPool2d), fully connected layer, and output layer;

[0185] 2) Input the preprocessed OCT image into the data conversion module, and the data conversion module outputs a pair of low-resolution and high-resolution images corresponding to the OCT image;

[0186] The preprocessed OCTA image is input into a data conversion module, and the data conversion module outputs a pair of low-resolution and high-resolution images corresponding to the OCTA image;

[0187] The preprocessed RNFL images are respectively input into a data conversion module, and the data conversion module outputs a pair of low-resolution and high-resolution images corresponding to the RNFL images;

[0188] 3) The low-resolution images corresponding to the OCT images, OCTA images, and RNFL images output by the data conversion module are respectively input into the generator network, and the generator network outputs high-resolution images respectively;

[0189] 4) Inputting the high-resolution image corresponding to the OCT image output by the data conversion module or the high-resolution image corresponding to the OCT image output by the generator network into the discriminator network, the discriminator network determines the probability P that the output image is the high-resolution image corresponding to the OCT image output by the data conversion module;

[0190] The high-resolution image corresponding to the OCTA image output by the data conversion module or the high-resolution image corresponding to the OCTA image output by the generator network is input into the discriminator network, and the discriminator network determines the probability P that the output image is the high-resolution image corresponding to the OCTA image output by the data conversion module;

[0191] The high-resolution image corresponding to the RNFL image output by the data conversion module or the high-resolution image corresponding to the RNFL image output by the generator network is input into the discriminator network, and the discriminator network determines the probability P that the output image is the high-resolution image corresponding to the RNFL image output by the data conversion module;

[0192] 6) Repeat 1) to 5) until the probability P that the discriminator network determines that the output image is a high-resolution image corresponding to the OCT image output by the data conversion module, the probability P that the discriminator network determines that the output image is a high-resolution image corresponding to the OCTA image output by the data conversion module, and the probability P that the discriminator network determines that the output image is a high-resolution image corresponding to the RNFL image output by the data conversion module all reach the maximum, and a trained deep learning module is obtained.

[0193] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0194] Specific implementation example 6: This implementation example is different from any one of specific implementation examples 1 to 5 in that: in the above 2), the pre-processed OCT image is input into a data conversion module, and the data conversion module outputs a pair of low-resolution and high-resolution images corresponding to the OCT image;

[0195] The preprocessed OCTA image is input into a data conversion module, and the data conversion module outputs a pair of low-resolution and high-resolution images corresponding to the OCTA image;

[0196] The preprocessed RNFL images are respectively input into a data conversion module, and the data conversion module outputs a pair of low-resolution and high-resolution images corresponding to the RNFL images;

[0197] The specific process is:

[0198] The data conversion module includes: a high-resolution conversion module and a low-resolution conversion module;

[0199] The pre-processed OCT images, OCTA images and RNFL images are respectively input into the high-resolution conversion module, and the high-resolution conversion module outputs high-resolution images respectively;

[0200] The pre-processed OCT images, OCTA images and RNFL images are respectively input into low-resolution conversion modules, and the low-resolution conversion modules output low-resolution images respectively;

[0201] A pair of low-resolution and high-resolution images refers to images obtained by inputting the same pre-processed fundus optical coherence tomography (OCT) image into a high-resolution conversion module and a low-resolution conversion module respectively.

[0202] The pre-processed OCT image, OCTA image and RNFL image are respectively input into the high-resolution conversion module, and the high-resolution conversion module outputs high-resolution images respectively; the process is:

[0203] The preprocessed OCT images, OCTA images and RNFL images are randomly cropped, and the randomly cropped OCT images, OCTA images and RNFL images are converted into tensors respectively;

[0204] The pre-processed OCT images, OCTA images and RNFL images are respectively input into the low-resolution conversion module, and the low-resolution conversion module outputs low-resolution images respectively; the process is:

[0205] The preprocessed OCT images, OCTA images and RNFL images are converted into PIL images respectively, and the PIL images are resized to be equal to the size of the A image by bicubic interpolation, and the resized PIL images are converted into tensors;

[0206] The A image size is the result of dividing the image size after random cropping in the high-resolution conversion module by the upsampling factor.

[0207] The upsampling factor is set.

[0208] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0209] Specific embodiment 7: This embodiment differs from any one of specific embodiments 1 to 6 in that: in said 3), the low-resolution images corresponding to the OCT images, OCTA images, and RNFL images output by the data conversion module are respectively input into the generator network, and the generator network outputs high-resolution images respectively;

[0210] The specific process is:

[0211] The generator network includes: input layer, initial convolution layer, residual block, upsampling block, final convolution layer, and output layer;

[0212] Among them, the initial convolution layer includes a convolution layer (Conv) and a pre-activation function layer (PReLU) in sequence;

[0213] The residual block includes a convolutional layer (Conv), a batch normalization layer (BatchNorm2d), a pre-activation function layer (PReLU), a convolutional layer (Conv), and a batch normalization layer (BatchNorm2d);

[0214] The upsampling block includes convolution layer (Conv), PixelShuffle upsampling, and pre-activation function layer (PReLU) in sequence;

[0215] The final convolutional layer includes the convolutional layer (Conv) and the tanh activation function layer in sequence;

[0216] The low-resolution images corresponding to the OCT images, OCTA images, and RNFL images output by the data conversion module are respectively input into the initial convolution layer, the residual block, the upsampling block, the final convolution layer, and the output layer through the input layer, and the output layer outputs high-resolution images.

[0217] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0218] Specific embodiment eight: This embodiment differs from any one of specific embodiments one to seven in that: in said 4), the high-resolution image corresponding to the OCT image output by the data conversion module or the high-resolution image corresponding to the OCT image output by the generator network is input into the discriminator network, and the discriminator network determines the probability P that the output image is the high-resolution image corresponding to the OCT image output by the data conversion module;

[0219] The high-resolution image corresponding to the OCTA image output by the data conversion module or the high-resolution image corresponding to the OCTA image output by the generator network is input into the discriminator network, and the discriminator network determines the probability P that the output image is the high-resolution image corresponding to the OCTA image output by the data conversion module;

[0220] The high-resolution image corresponding to the RNFL image output by the data conversion module or the high-resolution image corresponding to the RNFL image output by the generator network is input into the discriminator network, and the discriminator network determines the probability P that the output image is the high-resolution image corresponding to the RNFL image output by the data conversion module;

[0221] The specific process is:

[0222] The discriminator network includes: input layer, convolution layer (Conv), batch normalization layer (BatchNorm2d), LeakyReLU activation function layer, adaptive average pooling layer (AdaptiveAvgPool2d), fully connected layer, and output layer;

[0223] The high-resolution image corresponding to the OCT image output by the data conversion module or the high-resolution image corresponding to the OCT image output by the generator network is sequentially input into the convolution layer (Conv), the batch normalization layer (BatchNorm2d), the LeakyReLU activation function layer, the adaptive average pooling layer (AdaptiveAvgPool2d), the fully connected layer, and the output layer through the input layer. The output layer determines the probability P that the output image is the high-resolution image corresponding to the OCT image output by the data conversion module;

[0224] The high-resolution image corresponding to the OCTA image output by the data conversion module or the high-resolution image corresponding to the OCTA image output by the generator network is sequentially input into the convolution layer (Conv), the batch normalization layer (BatchNorm2d), the LeakyReLU activation function layer, the adaptive average pooling layer (AdaptiveAvgPool2d), the fully connected layer, and the output layer through the input layer. The output layer determines the probability P that the output image is the high-resolution image corresponding to the OCTA image output by the data conversion module;

[0225] The high-resolution image corresponding to the RNFL image output by the data conversion module or the high-resolution image corresponding to the RNFL image output by the generator network is input into the convolution layer (Conv), batch normalization layer (BatchNorm2d), LeakyReLU activation function layer, adaptive average pooling layer (AdaptiveAvgPool2d), fully connected layer, and output layer in sequence through the input layer. The output layer determines the probability P that the output image is the high-resolution image corresponding to the RNFL image output by the data conversion module.

[0226] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0227] Specific implementation nine: This implementation is different from any one of specific implementations one to eight in that: 1) to 5) are repeatedly performed in the above 5) until the probability P of the discriminator network judging that the output image is a high-resolution image corresponding to the OCT image output by the data conversion module, the probability P of the discriminator network judging that the output image is a high-resolution image corresponding to the OCTA image output by the data conversion module, and the probability P of the discriminator network judging that the output image is a high-resolution image corresponding to the RNFL image output by the data conversion module are all maximized, thereby obtaining a trained deep learning module;

[0228] The specific process is:

[0229] Define loss function: loss functions include GAN loss, perceptual loss PerceptionLoss (usually measured by the difference between pre-trained CNN feature maps), image loss ImageLoss (such as L1 or L2 loss), and total variation loss TVLoss (used to smooth images); the specific process of each training cycle is:

[0230] Set the total discriminator loss Loss_D = 1-Loss_real+Loss_fake

[0231] Loss_real = -log(D(x))

[0232] D(x) is the probability mean of the high-resolution image output by the discriminator network as the high-resolution image x converted by the data conversion module;

[0233] Loss_fake = -log(1-D(G(z)))

[0234] G(z) is the probability mean that the high-resolution image output by the discriminator network is the high-resolution image generated by the generator network;

[0235] Use the Adam optimizer to update the discriminator parameter weights so that the high-resolution image output by the discriminator network is closer to the high-resolution image converted by the data conversion module (with a probability close to 1);

[0236] Set the total generator loss Loss_G:

[0237] Loss_G=Adversarial_Loss+λ p Perception_Loss+λ i Image_Loss+λ tv TV_Loss

[0238] Adversarial_Loss is the adversarial loss, which is the average probability that the discriminator network determines that the generated high-resolution image is not the high-resolution image converted by the data conversion module;

[0239] Perception_Loss is the perceptual loss. The high-resolution image generated by the generator and the high-resolution image converted by the data conversion module are respectively input into the VGG16 network. The 31st layer of the VGG16 network outputs the feature map of the high-resolution image generated by the generator and the feature map of the high-resolution image converted by the data conversion module, and then the mean square error loss function MSELoss() is used to calculate the difference between the two feature maps. The difference between the two feature maps is used as the perceptual loss;

[0240] Image_Loss is the image loss, which is the difference between the square of the high-resolution image pixels generated by the generator and the square of the high-resolution image pixels converted by the data conversion module using the mean square error loss function MSELoss();

[0241] TV_Loss is the total variation loss, that is, first calculate the difference between adjacent pixels in each row of the high-resolution image generated by the generator and sum the squares of these differences, then calculate the difference between adjacent pixels in each column of the high-resolution image generated by the generator and sum the squares of these differences, then add the sum of the squares of the row and column differences to get the total variation, then divide the total variation by the total number of pixels of the high-resolution image generated by the generator to get a loss value that is independent of the image size, and finally multiply the loss value by a set weight coefficient to get the final total variation loss;

[0242] where λ p , i , tv is the weight to balance different loss terms;

[0243] Use the Adam optimizer to update the generator parameter weights so that the high-resolution image generated by the generator is closer to the high-resolution image converted by the data conversion module;

[0244] Repeat 1) to 5), alternately updating the generator and discriminator until the total discriminator loss and the total generator loss converge to obtain a trained super-resolution reconstruction network model.

[0245] “Alternating” means first fixing the discriminator and updating the generator; then fixing the generator and updating the discriminator. “Converging” means that the total discriminator loss and the total generator loss gradually stabilize and no longer change significantly.

[0246] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0247] Specific embodiment ten: This embodiment differs from any one of specific embodiments one to nine in that: the image-text multimodal convolutional network model is used to use the high-resolution OCT images, OCTA images, RNFL images and age, gender, and joint features output by the trained super-resolution reconstruction network model as inputs of the image-text multimodal convolutional network model, and the probability of glaucoma at different stages (including early, middle and late stages of glaucoma) as outputs of the image-text multimodal convolutional network model to obtain a trained image-text multimodal convolutional network model; the specific process is:

[0248] The image-text multimodal convolutional network model includes a feature extraction network and a feature fusion network;

[0249] The feature extraction network includes an image feature extraction network and a text information feature extraction network;

[0250] The image feature extraction network includes: input layer, convolution layer (Conv), batch normalization layer (BatchNorm2d), ReLU activation function layer, 50 residual blocks (Residual Block), global average pooling layer (GlobalAveragePooling Layer), and output layer;

[0251] Each residual block contains a convolution layer and an identity connection (the working process of each residual block is: the input of the convolution layer and the output of the convolution layer are added, and the summed features are used as the input of the next residual block); the convolution group is composed of residual blocks, and between the convolution groups, the maximum pooling layer is used for downsampling to reduce the size of the feature map and increase the receptive field.

[0252] The text information feature extraction network includes: input layer, embedding layer, convolution layer, maximum pooling layer (MaxPooling Layer), and output layer;

[0253] The feature fusion network includes: input layer, concatenation layer, fully connected layer, softmax layer and output layer;

[0254] The high-resolution OCT image, OCTA image, and RNFL image are respectively input into the image feature extraction network, and the image feature extraction network outputs the OCT image feature vector, the OCTA image feature vector, and the RNFL image feature vector respectively;

[0255] Age, gender, and joint features are input into the text information feature extraction network respectively, and the text information feature extraction network outputs feature vectors of age, gender, and joint features respectively;

[0256] The OCT image feature vector, OCTA image feature vector, RNFL image feature vector, and the feature vectors of age, gender, and joint features are used as inputs to the feature fusion network, and the probability of glaucoma at different stages (including early, middle, and late stages of glaucoma) is output from the feature fusion network;

[0257] Define the cross entropy loss function;

[0258] Use the Adam optimization algorithm to update the parameters of the feature extraction network and the feature fusion network until the cross entropy loss function converges to obtain a trained image-text multimodal convolutional network model.

[0259] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0260] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A glaucoma stage discrimination system based on a multimodal information network, characterized by: The system comprises: a data acquisition module, a data preprocessing module, a super-resolution reconstruction network model, a graphic and text multimodal convolutional network model and a module to be tested; The data acquisition module is used to collect fundus optical coherence tomography (OCT) image training sets, fundus optical coherence tomography (OCTA) image training sets, RNFL image training sets, and medical record information; the medical record information includes the patient's gender and age information; The data preprocessing module is used to preprocess the fundus optical coherence tomography (OCT) image training set, the fundus optical coherence tomography (OCTA) image training set, the RNFL image training set and the medical record information collected by the data acquisition module, respectively, to obtain the preprocessed fundus optical coherence tomography (OCT) image training set, the fundus optical coherence tomography (OCTA) image training set, the RNFL image training set and the medical record information; the medical record information includes the patient's gender and age information; The super-resolution reconstruction network model is used to use the preprocessed fundus optical coherence tomography (OCT) image training set, fundus optical coherence tomography (OCTA) image training set and RNFL image training set as inputs of the super-resolution reconstruction network model, and the high-resolution OCT image training set, OCTA image training set and RNFL image training set as outputs of the super-resolution reconstruction network model, to obtain a trained super-resolution reconstruction network model; The image-text multimodal convolutional network model is used to use the high-resolution OCT images, OCTA images, RNFL images, age, gender, and joint features output by the trained super-resolution reconstruction network model as inputs of the image-text multimodal convolutional network model, and the probability of glaucoma at different stages as outputs of the image-text multimodal convolutional network model, to obtain a trained image-text multimodal convolutional network model; The module to be tested is used to obtain the OCT images, OCTA images, RNFL images, age and gender information in the cases to be tested, preprocess the OCT images, OCTA images, RNFL images, age and gender information in the cases to be tested, and obtain the preprocessed OCT images, OCTA images, RNFL images, gender, age and joint indicators; the preprocessed OCT images, OCTA images, and RNFL images are input into the trained super-resolution reconstruction network model, and the generator in the trained super-resolution reconstruction network model outputs high-resolution OCT images, OCTA images, and RNFL images; the high-resolution OCT images, OCTA images, and RNFL images output by the generator and the preprocessed gender, age, and joint indicators are input into the trained graphic multimodal convolutional network model, and the trained graphic multimodal convolutional network model outputs the probabilities of different stages of glaucoma, including early, middle and late stages of glaucoma.

2. The system for distinguishing different stages of glaucoma based on a multimodal information network according to claim 1, characterized in that: The data acquisition module is used to collect fundus optical coherence tomography (OCT) image training set, fundus optical coherence tomography (OCTA) image training set, RNFL image training set, and medical record information; the specific process is: The fundus optical coherence tomography (OCT) image training set includes labeled normal OCT images, early glaucoma OCT images, mid-stage glaucoma OCT images, and late-stage glaucoma OCT images; The fundus optical coherence tomography (OCTA) image training set includes labeled normal OCTA images, early glaucoma OCTA images, mid-stage glaucoma OCTA images, and late-stage glaucoma OCTA images; The RNFL image training set includes labeled normal RNFL images, early glaucoma RNFL images, mid-stage glaucoma RNFL images, and late-stage glaucoma RNFL images; The OCT is optical coherence tomography technology; The OCTA is optical coherence tomography angiography; The RNFL is the retinal nerve fiber layer.

3. The system for distinguishing different stages of glaucoma based on a multimodal information network according to claim 2, characterized in that: The data preprocessing module is used to preprocess the fundus optical coherence tomography (OCT) image training set, the fundus optical coherence tomography (OCTA) image training set, the RNFL image training set and the medical record information collected by the data acquisition module, respectively, to obtain the preprocessed fundus optical coherence tomography (OCT) image training set, the fundus optical coherence tomography (OCTA) image training set, the RNFL image training set and the medical record information; the specific process is: 1) Preprocessing the fundus optical coherence tomography (OCT) image training set collected by the data acquisition module to obtain a preprocessed fundus optical coherence tomography (OCT) image training set; the specific process is as follows: 11) Use median filtering technology to denoise the OCT image to obtain a denoised OCT image; 12) Use the Sobel edge detection technology to locate the boundary of the denoised OCT image to obtain an OCT image with clear boundaries; 13) Adaptive thresholding technology was used to segment the OCT images with clear boundaries into vitreous, retinal nerve fiber layer, ganglion cell layer, inner nuclear layer, outer plexiform layer, outer nuclear layer and retinal pigment epithelium; 14) Adjust the layered OCT images to 224×224 pixels, normalize the OCT images adjusted to 224×224 pixels, and obtain a preprocessed OCT image training set; 2) Preprocessing the fundus optical coherence tomography (OCTA) image training set collected by the data acquisition module to obtain a preprocessed fundus optical coherence tomography (OCTA) image training set; the specific process is as follows: 21) Using the diffusion model to denoise the OCTA image to obtain a denoised OCTA image; 22) Calculate the Hessian matrix of each pixel in the denoised OCTA image; 23) Calculate the eigenvalue of the Hessian matrix of each pixel and identify the potential vascular structure in the OCTA image based on the eigenvalue; 24) Design a matched filter to enhance the identified potential vascular structure and suppress background noise to obtain OCTA images; 25) Processing the OCTA image obtained in 24) using adaptive threshold technology to obtain a screened OCTA image; 26) adjusting the screened OCTA images obtained in 25) to 224×224 pixels, normalizing the OCTA images adjusted to 224×224 pixels, and obtaining a preprocessed OCTA image training set; 3) Preprocess the RNFL image training set collected by the data acquisition module to obtain the preprocessed RNFL image training set; the specific process is: The RNFL image is adjusted to 224×224 pixels, and the RNFL image adjusted to 224×224 pixels is normalized to obtain a preprocessed RNFL image training set; 4) Preprocessing the medical record information collected by the data collection module to obtain preprocessed medical record information; The preprocessed medical record information includes preprocessed age, preprocessed gender, and preprocessed joint characteristics.

4. The system for distinguishing different stages of glaucoma based on a multimodal information network according to claim 3, characterized in that: In the step 4), the medical record information collected by the data collection module is preprocessed to obtain preprocessed medical record information; the preprocessed medical record information includes preprocessed age, processed gender, and processed joint features; The specific process is: 41) Preprocess the age data to obtain preprocessed age data; the specific process is: The age groups are divided into four stages: 0-17, 18-39, 40-79, and 80 and above; The value corresponding to 0-17 years old is 1; the value corresponding to 18-39 years old is 2; the value corresponding to 40-79 years old is 3; the value corresponding to 80 years old and above is 4; 42) Preprocess the gender data to obtain preprocessed gender data; the specific process is: Use the get_dummies function in the pandas library to convert the gender variable into a dummy variable; 43) Calculate the joint features based on the preprocessed age data and the preprocessed gender data; It is expressed as: Joint feature = Beta1×age+Beta2×gender+Beta3 Among them, Beta1, Beta2, and Beta3 are coefficients; The process of obtaining the coefficients Beta1, Beta2, and Beta3 is as follows: Based on the joint features, the Logistic regression model is trained to obtain the predicted probability of suffering from glaucoma. The roc_auc_score function in the scikit-learn library is called to calculate the area under the ROC curve AUC corresponding to the Logistic regression model. At this point, one cycle ends. Modify the values ​​of coefficients Beta1, Beta2, and Beta3 in each cycle, and take the set of Beta1, Beta2, and Beta3 coefficients that makes the AUC value closest to 1 as the optimal Beta1, Beta2, and Beta3 coefficients; Calculate the joint features based on the optimal Beta1, Beta2, and Beta3 coefficients.

5. The system for distinguishing different stages of glaucoma based on a multimodal information network according to claim 4, characterized in that: The super-resolution reconstruction network model is used to use the preprocessed fundus optical coherence tomography (OCT) image training set, fundus optical coherence tomography (OCTA) image training set and RNFL image training set as inputs of the super-resolution reconstruction network model, and the high-resolution OCT image training set, OCTA image training set and RNFL image training set as outputs of the super-resolution reconstruction network model to obtain a trained super-resolution reconstruction network model; the specific process is: 1) The super-resolution reconstruction network model includes a data conversion module, a generator network and a discriminator network in sequence; The data conversion module includes: a high-resolution conversion module and a low-resolution conversion module; The generator network includes: input layer, initial convolution layer, residual block, upsampling block, final convolution layer, and output layer; The discriminator network includes: input layer, convolution layer, batch normalization layer, LeakyReLU activation function layer, adaptive average pooling layer, fully connected layer, and output layer; 2) Input the preprocessed OCT image into the data conversion module, and the data conversion module outputs a pair of low-resolution and high-resolution images corresponding to the OCT image; The preprocessed OCTA image is input into a data conversion module, and the data conversion module outputs a pair of low-resolution and high-resolution images corresponding to the OCTA image; The preprocessed RNFL images are respectively input into a data conversion module, and the data conversion module outputs a pair of low-resolution and high-resolution images corresponding to the RNFL images; 3) The low-resolution images corresponding to the OCT images, OCTA images, and RNFL images output by the data conversion module are respectively input into the generator network, and the generator network outputs high-resolution images respectively; 4) Inputting the high-resolution image corresponding to the OCT image output by the data conversion module or the high-resolution image corresponding to the OCT image output by the generator network into the discriminator network, the discriminator network determines the probability P that the output image is the high-resolution image corresponding to the OCT image output by the data conversion module; The high-resolution image corresponding to the OCTA image output by the data conversion module or the high-resolution image corresponding to the OCTA image output by the generator network is input into the discriminator network, and the discriminator network determines the probability P that the output image is the high-resolution image corresponding to the OCTA image output by the data conversion module; The high-resolution image corresponding to the RNFL image output by the data conversion module or the high-resolution image corresponding to the RNFL image output by the generator network is input into the discriminator network, and the discriminator network determines the probability P that the output image is the high-resolution image corresponding to the RNFL image output by the data conversion module; 5) Repeat 1) to 5) until the probability P that the discriminator network determines that the output image is a high-resolution image corresponding to the OCT image output by the data conversion module, the probability P that the discriminator network determines that the output image is a high-resolution image corresponding to the OCTA image output by the data conversion module, and the probability P that the discriminator network determines that the output image is a high-resolution image corresponding to the RNFL image output by the data conversion module all reach the maximum, and a trained deep learning module is obtained.

6. The system for distinguishing different stages of glaucoma based on a multimodal information network according to claim 5, characterized in that: In the step 2), the preprocessed OCT image is input into a data conversion module, and the data conversion module outputs a pair of low-resolution and high-resolution images corresponding to the OCT image; The preprocessed OCTA image is input into a data conversion module, and the data conversion module outputs a pair of low-resolution and high-resolution images corresponding to the OCTA image; The preprocessed RNFL images are respectively input into a data conversion module, and the data conversion module outputs a pair of low-resolution and high-resolution images corresponding to the RNFL images; The specific process is: The data conversion module includes: a high-resolution conversion module and a low-resolution conversion module; The pre-processed OCT images, OCTA images and RNFL images are respectively input into the high-resolution conversion module, and the high-resolution conversion module outputs high-resolution images respectively; The pre-processed OCT images, OCTA images and RNFL images are respectively input into low-resolution conversion modules, and the low-resolution conversion modules output low-resolution images respectively; The pre-processed OCT image, OCTA image and RNFL image are respectively input into the high-resolution conversion module, and the high-resolution conversion module outputs high-resolution images respectively; the process is: The preprocessed OCT images, OCTA images and RNFL images are randomly cropped, and the randomly cropped OCT images, OCTA images and RNFL images are converted into tensors respectively; The pre-processed OCT images, OCTA images and RNFL images are respectively input into the low-resolution conversion module, and the low-resolution conversion module outputs low-resolution images respectively; the process is: The preprocessed OCT images, OCTA images and RNFL images are converted into PIL images respectively, and the PIL images are resized to be equal to the size of the A image by bicubic interpolation, and the resized PIL images are converted into tensors; The A image size is the result of dividing the image size after random cropping in the high-resolution conversion module by the upsampling factor.

7. The system for distinguishing different stages of glaucoma based on a multimodal information network according to claim 6, characterized in that: In the above 3), the low-resolution images corresponding to the OCT images, OCTA images, and RNFL images output by the data conversion module are respectively input into the generator network, and the generator network outputs high-resolution images respectively; The specific process is: The generator network includes: input layer, initial convolution layer, residual block, upsampling block, final convolution layer, and output layer; Among them, the initial convolution layer includes a convolution layer and a pre-activation function layer in sequence; The residual block includes convolution layer, batch normalization layer, pre-activation function layer, convolution layer, batch normalization layer in sequence; The upsampling block includes convolution layer, PixelShuffle upsampling, and pre-activation function layer in sequence; The final convolutional layer includes convolutional layer and tanh activation function layer in sequence; The low-resolution images corresponding to the OCT images, OCTA images, and RNFL images output by the data conversion module are respectively input into the initial convolution layer, the residual block, the upsampling block, the final convolution layer, and the output layer through the input layer, and the output layer outputs high-resolution images.

8. The system for distinguishing different stages of glaucoma based on a multimodal information network according to claim 7, characterized in that: In the above 4), the high-resolution image corresponding to the OCT image output by the data conversion module or the high-resolution image corresponding to the OCT image output by the generator network is input into the discriminator network, and the discriminator network determines the probability P that the output image is the high-resolution image corresponding to the OCT image output by the data conversion module; The high-resolution image corresponding to the OCTA image output by the data conversion module or the high-resolution image corresponding to the OCTA image output by the generator network is input into the discriminator network, and the discriminator network determines the probability P that the output image is the high-resolution image corresponding to the OCTA image output by the data conversion module; The high-resolution image corresponding to the RNFL image output by the data conversion module or the high-resolution image corresponding to the RNFL image output by the generator network is input into the discriminator network, and the discriminator network determines the probability P that the output image is the high-resolution image corresponding to the RNFL image output by the data conversion module; The specific process is: The discriminator network includes: input layer, convolution layer, batch normalization layer, LeakyReLU activation function layer, adaptive average pooling layer, fully connected layer, and output layer; The high-resolution image corresponding to the OCT image output by the data conversion module or the high-resolution image corresponding to the OCT image output by the generator network is sequentially input into the convolution layer, batch normalization layer, LeakyReLU activation function layer, adaptive average pooling layer, fully connected layer, and output layer through the input layer. The output layer determines the probability P that the output image is the high-resolution image corresponding to the OCT image output by the data conversion module; The high-resolution image corresponding to the OCTA image output by the data conversion module or the high-resolution image corresponding to the OCTA image output by the generator network is sequentially input into the convolution layer, batch normalization layer, LeakyReLU activation function layer, adaptive average pooling layer, fully connected layer, and output layer through the input layer. The output layer determines the probability P that the output image is the high-resolution image corresponding to the OCTA image output by the data conversion module; The high-resolution image corresponding to the RNFL image output by the data conversion module or the high-resolution image corresponding to the RNFL image output by the generator network is input into the convolution layer, batch normalization layer, LeakyReLU activation function layer, adaptive average pooling layer, fully connected layer, and output layer in sequence through the input layer. The output layer determines the probability P that the output image is the high-resolution image corresponding to the RNFL image output by the data conversion module.

9. The system for distinguishing different stages of glaucoma based on a multimodal information network according to claim 8, characterized in that: In the above 5), 1) to 5) are repeatedly performed until the probability P of the discriminator network determining that the output image is a high-resolution image corresponding to the OCT image output by the data conversion module, the probability P of the discriminator network determining that the output image is a high-resolution image corresponding to the OCTA image output by the data conversion module, and the probability P of the discriminator network determining that the output image is a high-resolution image corresponding to the RNFL image output by the data conversion module are all maximized, thereby obtaining a trained deep learning module; The specific process is: Set the total discriminator loss Loss_D = 1-Loss_real+Loss_fake Loss_real = -log(D(x)) D(x) is the probability mean of the high-resolution image output by the discriminator network as the high-resolution image x converted by the data conversion module; Loss_fake = -log(1-D(G(z))) G(z) is the probability mean that the high-resolution image output by the discriminator network is the high-resolution image generated by the generator network; Use the Adam optimizer to update the discriminator parameter weights so that the high-resolution image output by the discriminator network is closer to the high-resolution image converted by the data conversion module; Set the total generator loss Loss_G: Loss_G=Adversarial_Loss+λ p Perception_Loss+λ i Image_Loss+λ tv TV_Loss Adversarial_Loss is the adversarial loss; Perception_Loss is the perception loss; Image_Loss is the image loss; TV_Loss is the total variation loss; where λ p , i , tv is the weight; Use the Adam optimizer to update the generator parameter weights so that the high-resolution image generated by the generator is closer to the high-resolution image converted by the data conversion module; Repeat 1) to 5), alternately updating the generator and discriminator until the total discriminator loss and the total generator loss converge to obtain a trained super-resolution reconstruction network model.

10. The system for distinguishing different stages of glaucoma based on a multimodal information network according to claim 9, characterized in that: The image-text multimodal convolutional network model is used to use the high-resolution OCT images, OCTA images, RNFL images, age, gender, and joint features output by the trained super-resolution reconstruction network model as inputs of the image-text multimodal convolutional network model, and the probability of glaucoma at different stages as outputs of the image-text multimodal convolutional network model to obtain a trained image-text multimodal convolutional network model; the specific process is: The image-text multimodal convolutional network model includes a feature extraction network and a feature fusion network; The feature extraction network includes an image feature extraction network and a text information feature extraction network; The image feature extraction network includes: input layer, convolution layer, batch normalization layer, ReLU activation function layer, 50 residual blocks, global average pooling layer, and output layer; Each residual block contains a convolutional layer and an identity connection; The text information feature extraction network includes: input layer, embedding layer, convolution layer, maximum pooling layer, and output layer; The feature fusion network includes: input layer, concatenation layer, fully connected layer, softmax layer and output layer; The high-resolution OCT image, OCTA image, and RNFL image are respectively input into the image feature extraction network, and the image feature extraction network outputs the OCT image feature vector, the OCTA image feature vector, and the RNFL image feature vector respectively; Age, gender, and joint features are input into the text information feature extraction network respectively, and the text information feature extraction network outputs feature vectors of age, gender, and joint features respectively; The OCT image feature vector, OCTA image feature vector, RNFL image feature vector and the feature vectors of age, gender and joint features are used as inputs of the feature fusion network, and the disease probability features of glaucoma at different stages are output by the fusion network; Define the cross entropy loss function; Use the Adam optimization algorithm to update the parameters of the feature extraction network and the feature fusion network until the cross entropy loss function converges to obtain a trained image-text multimodal convolutional network model.

Images