A multimodal intelligent recognition system for glaucoma based on transfer learning

Abstract

The present invention discloses a glaucoma multimodal intelligent recognition system based on transfer learning. The system uses a transfer learning method to construct a deep learning model of glaucoma subtype cases for multi-label and multimodal data sets, including a multimodal data preprocessing module and a multimodal recognition module. By pre-training network parameters, the amount of data used for training the network is greatly reduced, and the text information and image information of the data are fully utilized. Multi-feature comprehensive diagnosis greatly improves the accuracy of glaucoma diagnosis. The present invention realizes in-depth diagnosis of glaucoma subtypes, breaks through the traditional technical means that can only diagnose whether it is glaucoma, and realizes automatic learning of multimodal features for diagnosing glaucoma from a training database through a glaucoma multimodal intelligent diagnosis network based on transfer learning.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal neural network classification, and in particular to a glaucoma multimodal intelligent recognition system based on transfer learning. Background Art

[0002] The latest advances in deep learning have been applied to different medical fields for early detection or prediction of certain abnormalities. In the field of ophthalmology, medical image analysis using deep learning methods has begun to receive widespread attention. Among the major ophthalmic abnormalities, glaucoma is the most serious one, which may cause irreversible damage to human vision because it affects the optic nerve fibers responsible for transmitting sensory information related to vision. As an asymptomatic disease, most patients do not notice obvious signs, and it is often too late when symptoms are discovered. Therefore, early detection and treatment are extremely important for preventing vision loss. Among the existing intelligent diagnostic methods for glaucoma, deep learning research is generally based on fundus photos. However, the deep learning method based on fundus photos has the following defects: First, the extracted features are single and limited, and the effective information of glaucoma in fundus photos cannot be fully utilized, and the accuracy is low. Second, the distribution of data sets is uneven. Due to the influence of external conditions such as light and angle when collecting data, the accuracy of the trained neural network is often limited by the data set used.

[0003] How to diagnose glaucoma as early as possible and improve the accuracy of diagnosis has become an urgent problem to be solved in the diagnosis of glaucoma. The main feature of glaucoma is visual field loss, which is mainly caused by increased intraocular pressure (IOP) and / or loss of optic nerve blood flow. However, since visual impairment still exists when IOP is normal, the specificity and sensitivity of IOP are not enough to be an effective glaucoma indicator. Glaucoma can also be diagnosed by measuring features such as glaucoma angle and optic nerve fiber layer thickness. However, the symptoms of glaucoma are complex and the diagnostic accuracy is low. Previous studies have shown that the accuracy of neural networks based on color fundus photographs and Heidelberg optical coherence tomography images for early diagnosis of glaucoma is close to or better than that of ophthalmologists. At the same time, this method can also predict future disease changes such as the degree of visual field loss and nerve fiber layer thickness based on baseline examination results. Another method for diagnosing glaucoma is to identify vascular distribution characteristics through optical coherence tomography angiography (OCTA). OCTA is a new non-invasive fundus imaging technology that can identify retinal choroidal blood flow information with high resolution and image the retinal choroidal microvascular circulation in vivo. It has unique advantages in normal retinal choroidal vascular changes, disease management follow-up, and treatment effect detection. Based on high-quality multimodal image data and combined with neural networks, it is expected to greatly improve the diagnostic accuracy of glaucoma patients. The combination of artificial intelligence and medical imaging has brought breakthrough innovations to the clinical diagnosis and treatment of glaucoma.

[0004] Transfer learning is a learning method that uses a large amount of prior knowledge from other related fields to imitate the human visual system in the process of performing new tasks in a specific field. Using transfer learning, ideal results can be quickly trained even with a small number of medical image datasets. Therefore, it is of great practical significance to realize a multimodal glaucoma recognition network system based on transfer learning. Summary of the invention

[0005] In order to solve the above technical problems, the purpose of the present invention is to provide a glaucoma multimodal intelligent recognition system based on transfer learning, which extracts the patient's age, gender, chief complaint, current medical history, past medical history, vision, intraocular pressure, various specialist examinations and key vocabulary information of diagnosis by using a text extraction neural network; uses a U-Net network to segment the blood vessels in the fundus photo image, outputs the glaucoma blood vessel distribution characteristics, segments the glaucoma optic disc and optic cup in the fundus photo, and outputs the area ratio of the optic disc and optic cup; uses a deep learning classification network to construct a neural network, captures the global and local characteristics of glaucoma, trains model parameters, and then uses the gradient weighted class activation mapping (Grad-CAM) method to calibrate the key focus areas of the neural network in the glaucoma recognition process; and comprehensively identifies the glaucoma disease type through the above three branches.

[0006] To achieve the above technical effects, the present invention proposes a glaucoma multimodal intelligent recognition system based on transfer learning, including a multimodal data preprocessing module and a multimodal recognition module; the multimodal data preprocessing module is used to extract text information and preprocess image data to obtain a glaucoma data set; the multimodal recognition module is used to construct a glaucoma subtype case recognition model based on the transfer learning method for multi-label and multimodal data sets.

[0007] The data preprocessing module includes a keyword extraction module, a Gaussian filtering module, an image enhancement module, and an image cropping module; the keyword extraction module uses a deep learning (DL) method to extract key words of glaucoma symptoms in the electronic medical records of the database; the filtering module is used to perform Gaussian filtering on the image data; the image enhancement module uses an image enhancement method to expand the training samples to avoid overfitting; the cropping module performs 1.5 times square grid pixel cropping around the optic cup disc on the original image based on a convolutional neural network (CNN).

[0008] The multimodal recognition module includes three branches. The first branch is to input the text information of the patient's electronic medical record processed by the preprocessing module, and obtain the feature matrix of the glaucoma keyword after processing by the convolution layer, the recurrent layer and the long short-term memory neural network (LSTM), and the output is connected to the Catboost classifier; the second branch is built based on the U-Net network, which is used to segment the blood vessels in the fundus photo image, output the distribution characteristics of the glaucoma blood vessels, segment the glaucoma optic disc and optic cup in the fundus photo, output the area ratio of the optic disc and the optic cup, and connect the output to the Catboost classifier; the third branch is based on A deep learning classification network is constructed, and the preprocessed data set is input. The global and local features of Heidelberg optical coherence tomography images and optic nerve fiber layer thickness images are captured through the deep learning network, and the model parameters are trained. Then the gradient weighted class activation mapping (Grad-CAM) method is used to calibrate the key focus areas of the neural network in the glaucoma identification process; the output of the first branch is fused with the output of the second branch, and after the fusion, the Catboost classifier is connected, and the output of the Catboost classifier is fused with the output of the third branch for multimodal information, and the interpretable visual analysis and glaucoma type are output.

[0009] The deep learning networks in the third branch include ResNet, GoogLeNet, DenseNet, ShuffleNet, EfficientNet and Vision Transformer. The preprocessed image data are respectively input into the deep learning networks for model parameter training. According to the accuracy of the training samples, the deep learning network with the highest accuracy is selected as the deep learning network to be fused with the output end of the Catboost classifier.

[0010] The system execution process includes:

[0011] Step 1: Collect the original electronic medical records and medical imaging data of glaucoma, annotate the data labels as normal (N), primary open-angle glaucoma (POAG), primary angle-closure glaucoma (PACG), and normal-tension glaucoma (NG), and construct the original glaucoma multimodal dataset;

[0012] Step 2: Input the original glaucoma multimodal dataset into the data preprocessing module for preprocessing to generate a glaucoma dataset;

[0013] Step 3: Input the preprocessed data set to perform parameter training on the multimodal recognition module to obtain a multimodal recognition module based on glaucoma classification;

[0014] Step 4: Input the patient data that needs to be classified into glaucoma subtypes into the trained multimodal recognition module for glaucoma recognition.

[0015] The step 2 comprises:

[0016] Step 2.1: Based on the convolutional neural network, the text information of glaucoma subtypes in the patient's electronic medical records is extracted. The extracted information includes the patient's age, gender, chief complaint, current medical history, past medical history, visual acuity, intraocular pressure, various specialist examinations, and diagnostic key words, which are used as the input of the first branch of the multimodal recognition module;

[0017] Step 2.2: crop the fundus photograph into a 572×572 image centered on the optic disc, and crop the Heidelberg optical correlation tomography image and optic nerve fiber layer thickness image into a 224×224 image centered on the optic disc;

[0018] Step 2.3: Perform data augmentation on the data, including vertical and horizontal mirroring, 90°, 180° and 270° rotation, and sliding window segmentation at the original resolution using a fixed-size frame to obtain an augmented data set;

[0019] Step 2.4: Perform Gaussian filtering on the data to balance the suppression of image noise and the blurring of the image. The input is the amplified data set, and the output is the mean value of the pixels in the filter window.

[0020] Step 2.5: Use the maximum and minimum normalization method to standardize the image. After data standardization, all indicators are at the same order of magnitude.

[0021] The step 3 comprises:

[0022] The preprocessed text information is input into the first branch of the multimodal recognition module, and the words in the electronic medical records of glaucoma patients are clustered using the K-Means algorithm;

[0023] The second branch uses the deep learning model U-Net, which inputs fundus photos to segment the optic nerve vessels and outputs the vascular distribution characteristics of glaucoma. It also inputs three-dimensional fundus photos to segment the optic cup and optic disc and outputs the optic nerve vascular distribution characteristics and cup-to-disc ratio.

[0024] The outputs of the first and second branches are fused together as metadata and processed using the Catboost classifier;

[0025] The third branch of the multimodal neural network simultaneously inputs the Heidelberg OCT, optic nerve fiber layer thickness and fundus photographs in the training set into the deep learning network model, uses the transfer learning method to pre-train the parameters of the constructed neural network, and uses the degree-weighted class activation mapping (Grad-CAM) method to calibrate the key focus areas of the neural network during glaucoma identification, which are then fused with the output of the Catboost classifier to obtain a comprehensive diagnostic result for glaucoma subtypes.

[0026] The clustering of words in the electronic medical records of glaucoma patients by using the K-Means algorithm includes:

[0027] The Word2vec model was trained using the glaucoma keyword corpus, with the input being the corpus data and the output being the glaucoma disease type;

[0028] Perform data preprocessing operations such as word segmentation, part-of-speech tagging, deduplication, and removal of stop words on the text in the patient's electronic medical records, and finally obtain n candidate keywords;

[0029] Traverse the candidate keywords and extract the word vector representation of the candidate keywords from the word vector file;

[0030] Perform K-Means clustering on the candidate keywords to obtain the cluster centers of each category;

[0031] Calculate the Euclidean distance between the words in each group and the cluster center, and sort them in ascending order according to the size of the cluster;

[0032] The candidate keywords are calculated and the top N words are obtained as text keywords for identifying glaucoma.

[0033] The beneficial effects of the present invention are:

[0034] The present invention proposes a glaucoma multimodal intelligent recognition system based on transfer learning. The system uses a transfer learning method to construct a deep learning model for glaucoma subtype cases for multi-label and multimodal data sets, including a multimodal data preprocessing module and a multimodal recognition module. By pre-training network parameters, the amount of data used for training the network is greatly reduced, and the text information and image information of the data are fully utilized to construct a multimodal neural network model for multi-feature comprehensive recognition, which effectively improves the accuracy of glaucoma recognition. The system of the present invention realizes the recognition of subtypes of glaucoma, breaking through the traditional technical means that can only identify whether it is glaucoma. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a block diagram of the multimodal intelligent recognition system for glaucoma based on transfer learning in the present invention;

[0036] Figure 2It is the network flow chart of Resnet in the present invention;

[0037] Figure 3 It is the network flow chart of GoogLeNet in the present invention;

[0038] Figure 4 It is the network flow chart of DenseNet in the present invention;

[0039] Figure 5 It is the network flow chart of ShuffleNet in the present invention;

[0040] Figure 6 It is the network flow chart of ShuffleNet in the present invention;

[0041] Figure 7 This is the network flow chart of the Vision Transformer in the present invention. DETAILED DESCRIPTION

[0042] The invention will be further described below with reference to the accompanying drawings and specific implementation examples.

[0043] like Figure 1 As shown, a glaucoma multimodal intelligent recognition system based on transfer learning uses a transfer learning method to construct a deep learning model for glaucoma subtype cases for multi-label and multimodal data sets, including a multimodal data preprocessing module and a multimodal recognition module; the data preprocessing module is used to preprocess the original images that have been collected and annotated in the database to obtain a data set, including a keyword extraction module, a Gaussian filtering module, an image enhancement module, and an image cropping module; the keyword extraction module uses a deep learning (DL) method to extract key words of glaucoma diseases in the electronic medical records of the database; the filtering module is used to perform Gaussian filtering on the image data; the image enhancement module uses an image enhancement method to expand the training sample to avoid overfitting; the cropping module performs 1.5 times square grid pixel cropping around the optic cup and disc on the original image based on a convolutional neural network (CNN).

[0044] The multimodal recognition module includes three branches. The first branch inputs the text information of the patient's electronic medical record processed by the preprocessing module, uses a text extraction neural network, and obtains the feature matrix of the glaucoma keyword after processing by a convolutional layer, a recurrent layer and a long short-term memory neural network (LSTM), and connects the output to the Catboost classifier; the second branch is constructed based on the U-Net network, which is used to segment the blood vessels in the fundus photo image, output the distribution characteristics of the glaucoma blood vessels, segment the glaucoma optic disc and optic cup in the fundus photo, output the area ratio of the optic disc and the optic cup, and connect the output to the Catboost classifier; the third branch is used to segment the blood vessels in the fundus photo, output the area ratio of the optic disc and the optic cup, and connect the output to the Catboost classifier; The three branches are constructed based on a deep learning classification network. The preprocessed data set is input, and the global and local features of the Heidelberg optical coherence tomography images and the optic nerve fiber layer thickness images are captured through the deep learning network to train the model parameters. Then the gradient weighted class activation mapping (Grad-CAM) method is used to calibrate the key focus areas of the neural network in the glaucoma identification process; the output of the first branch is fused with the output of the second branch, and after fusion, it is connected to the Catboost classifier. The output of the Catboost classifier is fused with the output of the third branch for multimodal information, and the interpretable visual analysis and glaucoma type are output.

[0045] The deep learning networks in the third branch include ResNet, GoogLeNet, DenseNet, ShuffleNet, EfficientNet and Vision Transformer. The preprocessed image data are respectively input into the deep learning networks for model parameter training. According to the accuracy of the training samples, the deep learning network with the highest accuracy is selected as the deep learning network to be fused with the output end of the Catboost classifier.

[0046] In ResNet, the Heidelberg OCT and fundus photographs are fused after passing through three convolution (conv) modules, and then after passing through two more conv modules, they are fused with the optic nerve fiber layer image that has passed through five conv modules alone. Other neural network models also perform similar processing. The Heidelberg OCT and fundus photographs are fused after passing through a certain number of convolution modules, and then after passing through the remaining convolution modules together, they are fused with the optic nerve fiber layer image that has been convolved alone. This is because both the Heidelberg OCT and fundus photographs sample the area around the optic disc, and after multiple convolutions, the feature extraction is roughly the same, while the optic nerve fiber layer thickness image is an extended image of the first two layers of the optic nerve fiber layer, and the extracted feature is the optic nerve fiber layer thickness.

[0047] Catboost is a popular ensemble boosting method that introduces an ordered target statistical strategy to effectively process categorical features. It replaces the categorical features that are ranked in front of the sample with the expected feature value and adds priorities and weights. Therefore, it can transform categorical features into numerical features, effectively reduce the noise of low-frequency categorical features, and enhance the robustness of the algorithm.

[0048] For the multi-module recognition module, all models used Adam optimization, with an initial learning rate of 0.0001 as the benchmark, and were fine-tuned for 300 epochs with a batch size of 32. The network was implemented based on Pytorch and Tensorflow, and all experiments were conducted on NVIDIA Tesla A100.

[0049] Introducing image features into metadata, the present invention adopts a popular deep learning model U-Net when segmenting the optic disc and optical cup areas. This is a well-known FCN-based U-shaped structure model, which consists of a downsampling and upsampling structure similar to the English capital letter U, and uses skip connections in the same stage, thereby ensuring that more high-level and low-level features are integrated into the final functional map.

[0050] The present invention uses feature representation level knowledge transfer learning, which uses a set of carefully crafted features to map the target domain to the source domain. Through this feature representation level knowledge transfer, the data divergence between the target domain and the source domain can be significantly reduced, thereby improving the performance of the task in the target domain.

[0051] The present invention selects focal loss as the loss function, which is a variant of categorical cross entropy loss and is used to process imbalanced data. Imbalanced data is common in the medical field, which makes the classifier pay more attention to the main class and ignore the secondary class, resulting in lower sensitivity to the secondary class and lower specificity to the main class, which can be solved to a certain extent by modifying the loss function. Categorical cross entropy loss is a commonly used loss function in multi-class classification learning, which assigns the same weight to each class, which results in little attention to the secondary class and lower sensitivity to underrepresented classes. In order to overcome the impact of data imbalance, the present invention introduces focal loss as the loss function. Focal loss is a variant of categorical cross entropy loss, which has been proposed to process imbalanced data.

[0052] The execution process of the glaucoma multimodal intelligent recognition system based on transfer learning is as follows:

[0053] Step 1: Collect the original electronic medical records and medical imaging data of glaucoma, annotate the data labels as normal (N), primary open-angle glaucoma (POAG), primary angle-closure glaucoma (PACG), and normal-tension glaucoma (NG), and construct a glaucoma multimodal dataset;

[0054] The dataset is based on the medical images of 317 individuals collected based on the Declaration of Helsinki. Each individual has an electronic medical record, which clearly marks the person's age, gender, chief complaint, current medical history, past medical history, visual acuity, intraocular pressure, various specialist examinations and diagnosis text information; at the same time, all individuals have collected Heidelberg OCT, retinal nerve fiber layer images and fundus photographs, and most individuals have UBM, SLO, OCTA and intraocular pressure data.

[0055] Step 2: Preprocess the original image input data preprocessing module that has been collected and annotated in the database to generate a training set; including:

[0056] Step 2.1: Based on the convolutional neural network, the text information of glaucoma subtypes in the patient's electronic medical records is extracted. The extracted information includes the patient's age, gender, chief complaint, current medical history, past medical history, visual acuity, intraocular pressure, various specialist examinations and key words for recognition, which are used as the input of the first branch of the multimodal recognition module;

[0057] Step 2.2: Crop the original fundus photograph into an image of 572×572 pixels centered on the optic disc, and crop the original Heidelberg OCT and optic nerve fiber layer thickness images into an image of 224×224 pixels centered on the optic disc. The cropping method is to find the most likely pixel in the optic disc area through the basic CNN, and use the threshold to screen the candidate pixels to complete the image cropping. The basic CNN consists of 5 convolution modules, each of which consists of 3 convolution layers and 1 maximum pooling layer. The convolution kernel size of the convolution layer is 3, the padding is 2, the kernel size of the pooling layer is 2, and the stride is 2. The final output of the convolution will be convolved and spliced ​​by convolution modules 3, 4, and 5 with convolution kernels of different sizes;

[0058] The fundus images of the fine-tuning process are automatically cropped around a bounding box of 1.5 times the radius of the optic disc. In addition, it also includes: Gaussian filtering, image enhancement, image histogram equalization, normalization, etc. The purpose of preprocessing images and metadata is to reduce the impact of noise and imbalanced classes in the dataset, thereby improving the model's ability to learn important features hidden in metadata and images. In deep learning (image field), data augmentation is a very common means of increasing the number of training samples, especially for medical data with a small amount. The number of normal eyes and other glaucoma subtypes in the medical images that constitute the training examples is close to 1:1.

[0059] Step 2.3: Data enhancement is performed on the data by using vertical and horizontal mirroring, 90°, 180° and 270° rotation methods, and a sliding window segmentation method using a fixed-size frame at the original resolution. The above methods are used to obtain an augmented data set. The ratio of normal eyes to glaucoma subtypes in the data set should be as close as possible to 1:1.

[0060] Step 2.4: Perform Gaussian filtering on the data to balance the suppression of image noise and the blurring of the image. The input is the amplified data set, and the output is the mean of the pixels in the filter window. The coefficient of the window template decreases as the distance from the template center increases.

[0061] Step 2.5: Use the maximum and minimum standardization method to standardize the image. After data standardization, all indicators are at the same order of magnitude, which is suitable for comprehensive comparative evaluation;

[0062] Step 3: Use the transfer learning method to assign the text recognition network parameters to the convolutional neural network in branch one, assign the segmentation network parameters to the U-net network in branch two, assign the classification network model parameters to the neural network in branch three in the multimodal glaucoma recognition module, and fix the parameters of the neural network except the output layer, input the preprocessed training set to train the model parameters of the multimodal recognition module, and obtain the trained multimodal recognition module; including:

[0063] The preprocessed text information is input into the first branch of the multimodal recognition module. The words in the electronic medical records of glaucoma patients are clustered using the K-Means algorithm. The cluster center is selected as a main keyword. The distance between other words and the cluster center, i.e., the similarity, is calculated. The first N words closest to the cluster center are selected as text keywords.

[0064] The Word2vec model was trained using the glaucoma keyword corpus, with the input being the corpus data and the output being the glaucoma disease type;

[0065] Perform data preprocessing operations such as word segmentation, part-of-speech tagging, deduplication, and removal of stop words on the text in the patient's electronic medical records, and finally obtain n candidate keywords;

[0066] Traverse the candidate keywords and extract the word vector representation of the candidate keywords from the word vector file;

[0067] Perform K-Means clustering on the candidate keywords to obtain the cluster centers of each category;

[0068] Calculate the distance (Euclidean distance) between the words in each group and the cluster center under each category, and sort them in ascending order according to the size of the cluster;

[0069] The candidate keywords are calculated to obtain the top N words as the text keywords for diagnosing glaucoma;

[0070] The second branch uses the deep learning model U-Net, which inputs fundus photos for optic nerve vascular segmentation, inputs three-dimensional fundus photos for optic cup and optic disc segmentation, and outputs optic nerve vascular distribution characteristics and cup-disc ratio. U-Net uses a fully convolutional neural network to perform convolution and pooling operations on the image to reduce the data dimension, then uses upsampling operations to expand the image dimension, and concatenates the generated feature map with the feature map obtained by convolution and pooling operations, and finally generates a feature map through two more convolution operations.

[0071] The outputs of the first and second branches are fused together as metadata and input into the Catboost classifier for processing;

[0072] The third branch of the multimodal neural network simultaneously inputs the Heidelberg OCT and optic nerve fiber layer thickness in the training set into the deep learning network model, uses the transfer learning method to pre-train the parameters of the constructed neural network, and uses the degree-weighted class activation mapping (Grad-CAM) method to calibrate the key focus area of ​​the neural network during glaucoma identification. It is then fused with the output of the Catboost classifier to obtain a comprehensive diagnosis result of glaucoma subtypes.

[0073] For the second branch of the multimodal recognition module, the image characteristics need to be introduced into the metadata. Therefore, the present invention adopts a popular deep learning model U-Net when segmenting the optic disc and the optical cup area. This is a well-known U-shaped structure model based on FCN. It consists of a downsampling and upsampling structure similar to the English capital letter U, and uses jump connections at the same stage to ensure that more high-level and low-level functions are integrated into the final function map. UNet uses a fully convolutional neural network. It consists of a feature extraction network using conv and pooling and a feature fusion network that uses upsampling to generate a feature map and concatenate the feature map on the left. Finally, after two convolution operations, a feature map is generated, and then two convolution kernels with a size of 1*1 are used for classification to obtain the last two heatmaps, which are then used as the input of the softmax function to calculate the softmax with a relatively large probability, and then loss and back propagation calculations are performed. The first branch and the second branch are then mixed together as metadata and processed using a Catboost classifier.

[0074] The third branch of the multimodal recognition module simultaneously inputs Heidelberg OCT, optic nerve fiber layer thickness and color fundus photographs into a neural network based on transfer learning. The backbone of the deep learning model includes ResNet, GoogLeNet, DenseNet, ShuffleNet, EfficientNet and Vision Transformer.

[0075] like Figure 2 As shown in the figure, the deep learning model backbone Resnet uses residual modules and batch normalization (BN) layers to standardize the data, effectively solving the problems of gradient vanishing and gradient exploding. The residual module uses three convolutional layers. The first is a 1x1 convolutional layer to compress the channel dimension, the second is a 3x3 convolutional layer, and the third is a 1x1 convolutional layer to restore the channel dimension. Both Heidelberg OCT and fundus photos sample the area around the optic disc. After multiple convolutions, the feature extraction is roughly the same, so when building the network, these two types of information are fused at the output of convolution module 3. The optic nerve fiber layer thickness image is an extended image of the optic nerve fiber layer, and the extracted feature is the optic nerve fiber layer thickness, so the image is convolved with 5 modules separately. Finally, the three types of data are fused at the output of convolution module 5, enter the average pool together, and output the glaucoma classification.

[0076] like Figure 3 As shown in the figure, the backbone of the deep learning model GoogLeNet is composed of an Inception module and an auxiliary classifier. The Inception structure has a total of 4 branches. Branch 1 is a convolution layer with a convolution kernel size of 1x1 and stride=1; branch 2 first passes through a convolution layer with a convolution kernel size of 1x1, and then a convolution layer with a convolution kernel size of 3x3, stride=1, padding=1 (to ensure that the height and width of the output feature matrix are equal to the input feature matrix); branch 3 first passes through a convolution layer with a convolution kernel size of 1x1, and then a convolution layer with a convolution kernel size of 5x5, stride=1, padding=2 (to ensure that the height and width of the output feature matrix are equal to the input feature matrix), branch 4 first passes through a maximum pooling downsampling with a pooling kernel size of 3x3, stride=1, padding=1 (to ensure that the height and width of the output feature matrix are equal to the input feature matrix), and then passes through a convolution layer with a convolution kernel size of 1x1. The data of Heidelberg OCT and fundus photographs are fused at the output of the Inception module (4d). The optic nerve fiber layer is processed separately and fused with the first two data at the output of Inception (5b). They enter the softmax together for glaucoma classification.

[0077] like Figure 4As shown in the figure, one advantage of the deep learning model backbone DenseNet is that the network is narrower and has fewer parameters. Each layer can directly obtain the gradient of the loss function and the original input signal, thereby obtaining implicit deep supervision. After the 7×7 convolution layer and the pooling layer, it passes through the cross structure of the Dense Block and the Transition layer. The main idea of ​​the Dense Block is: for each layer, the feature maps of all previous layers are used as the input of the current layer, and its own feature maps are used as the input of the subsequent layers to form a fully interconnected link. The feature maps extracted from each layer can be used by subsequent layers. The Heidelberg OCT and fundus photograph images are fused at the output of the Transition layer2, merging the three branch networks into two branches. After a Dense Block and a Transition layer, the optic nerve fiber layer thickness information is fused with the previous two information, and enters the global average pooling layer together for glaucoma classification.

[0078] like Figure 5 As shown in the figure, the deep learning model backbone ShuffleNet uses pointwise group convolution and channel shuffle to greatly reduce the amount of calculation at similar accuracy compared with existing advanced models. The Heidelberg OCT and fundus photo images are fused at the output of module 3. After module 4, the optic nerve fiber layer thickness information is fused with the previous two information and enters the convolution layer and global average pooling layer together for glaucoma classification.

[0079] like Figure 6 As shown in the figure, the Efficient network, the backbone of the deep learning model, is divided into 9 stages in total. The first stage is a common convolution layer with a convolution kernel size of 3x3 and a stride of 2 (including BN and activation function Swish). Stage2 to Stage8 are all repeated stacking MBConv structures. Stage9 consists of a common 1x1 convolution layer (including BN and activation function Swish), an average pooling layer, and a fully connected layer. The MBConv structure mainly consists of a 1x1 common convolution (dimensionality increase, including BN and Swish), a kxk Depthwise Conv convolution (including BN and Swish), a SE module, a 1x1 common convolution (dimensionality reduction, including BN), and a Dropout layer.

[0080] like Figure 7As shown in the figure, the model framework of Vision Transformer (ViT), the backbone of the deep learning model. In simple terms, the model consists of three modules: Embedding layer, Transformer Encoder and MLP Head. Transformer Encoder is actually a repeated stacking of Encoder Block L times, mainly composed of the following parts: Layer Norm, Multi-Head Attention, Dropout / DropPath and MLP Block.

[0081] Step 4: Input the patient data that needs to be classified into the trained multimodal recognition module for glaucoma recognition. The text information in the patient's electronic medical record is output to the first branch for feature text extraction, and the patient's medical images are pre-processed and input into the second and third branches respectively. The second branch is used to extract the cup-to-disc ratio feature; the third branch is used to obtain the glaucoma type diagnosed by multiple medical images.

[0082] The present invention can use the method of transfer learning, and by pre-training the network parameters, the amount of data used for training the network is greatly reduced. The present invention proposes a multimodal neural network for glaucoma diagnosis for the first time, which makes full use of the text information and image information of the data, and multi-feature comprehensive diagnosis, which greatly improves the accuracy of glaucoma diagnosis. At the same time, the present invention realizes the diagnosis of the subtypes of glaucoma, breaking through the traditional technical means that can only diagnose whether it is glaucoma. Through the glaucoma multimodal intelligent diagnosis network based on transfer learning, it is possible to automatically learn the multimodal features for diagnosing glaucoma from the training database. In the test, the method used the data of 317 human bodies for training. Compared with the manual diagnosis of doctors and traditional technologies, the overall accuracy of the method was 94.7% on average. Our data set is superior to most data sets in glaucoma diagnosis, with an accuracy rate of 87.8%. The present invention can be used for clinical diagnosis in hospitals, large-scale physical examinations, examinations in remote mountainous areas, and home examinations.

Claims

1. A multimodal intelligent recognition system for glaucoma based on transfer learning, characterized in that: It includes a multimodal data preprocessing module and a multimodal recognition module; the multimodal data preprocessing module is used to extract text information and preprocess image data to obtain a glaucoma data set; the multimodal recognition module is used to construct a glaucoma subtype case recognition model based on a transfer learning method for multi-label and multimodal data sets; The multimodal recognition module includes three branches: the first branch is to input the text information of the patient's electronic medical record processed by the multimodal data preprocessing module, obtain the feature matrix of the glaucoma keyword after processing by the convolution layer, the recurrent layer and the long short-term memory neural network, and connect the output with the Catboost classifier; the second branch is built based on the U-Net network, which is used to segment the blood vessels in the fundus photo image, output the distribution characteristics of the glaucoma blood vessels, segment the glaucoma optic disc and optic cup in the fundus photo, output the area ratio of the optic disc and optic cup, and connect the output with the Catboost classifier. The first branch is connected to the Catboost classifier; the third branch captures the global and local features of the Heidelberg optical coherence tomography images and the optic nerve fiber layer thickness images through a deep learning network based on the input preprocessed data set, trains the model parameters, and then uses the gradient weighted class activation mapping method to calibrate the neural network focus area in the glaucoma recognition process; the output of the first branch is fused with the output of the second branch, and after fusion, it is connected to the Catboost classifier. The output of the Catboost classifier is fused with the output of the third branch for multimodal information, and outputs interpretable visual analysis and glaucoma types.

2. According to claim 1, a glaucoma multimodal intelligent recognition system based on transfer learning is characterized in that: The multimodal data preprocessing module includes a keyword extraction module, a Gaussian filtering module, an image enhancement module, and an image cropping module; the keyword extraction module uses a deep learning method to extract key words of glaucoma symptoms in the electronic medical records of the database; the Gaussian filtering module is used to perform Gaussian filtering on the image data; the image enhancement module uses an image enhancement method to expand the training samples to avoid overfitting; the image cropping module crops the original image with 1.5 times the square pixels around the optic cup based on a convolutional neural network.

3. The multimodal intelligent recognition system for glaucoma based on transfer learning according to claim 1, characterized in that: The deep learning networks in the third branch include ResNet, GoogLeNet, DenseNet, ShuffleNet, EfficientNet and Vision Transformer. The preprocessed image data are respectively input into the deep learning networks for model parameter training. According to the accuracy of the training samples, the deep learning network with the highest accuracy is selected as the deep learning network to be fused with the output end of the Catboost classifier.

4. The multimodal intelligent recognition system for glaucoma based on transfer learning according to any one of claims 1 to 3, characterized in that: The system execution process includes: Step 1: Collect the original electronic medical records and medical imaging data of glaucoma, annotate the data labels as normal, primary open-angle glaucoma, primary angle-closure glaucoma, and normal-tension glaucoma, and construct the original glaucoma multimodal dataset; Step 2: input the original glaucoma multimodal data set into the multimodal data preprocessing module for preprocessing to generate a glaucoma data set for neural network training; Step 3: Input the preprocessed data set to perform parameter training on the multimodal recognition module to obtain a multimodal recognition module based on glaucoma classification; Step 4: Input the patient data that needs to be classified into glaucoma subtypes into the trained multimodal recognition module based on glaucoma classification to perform glaucoma recognition.

5. The multimodal intelligent glaucoma recognition system based on transfer learning according to claim 4 is characterized in that: The step 2 comprises: Step 2.1: Based on the convolutional neural network, the text information of glaucoma subtypes in the patient's electronic medical records is extracted. The extracted information includes the patient's age, gender, chief complaint, current medical history, past medical history, visual acuity, intraocular pressure, various specialist examinations, and diagnostic key words, which are used as the input of the first branch of the multimodal recognition module; Step 2.2: crop the fundus photograph into a 572×572 image centered on the optic disc, and crop the Heidelberg optical coherence tomography image and optic nerve fiber layer thickness image into a 224×224 image centered on the optic disc; Step 2.3: Perform data augmentation on the data, including vertical and horizontal mirroring, 90°, 180° and 270° rotation, and sliding window segmentation at the original resolution using a fixed-size frame to obtain an augmented data set; Step 2.4: Perform Gaussian filtering on the data to balance the suppression of image noise and the blurring of the image. The input is the amplified data set, and the output is the mean value of the pixels in the filter window. Step 2.5: Use the maximum and minimum normalization method to standardize the image. After data standardization, all indicators are at the same order of magnitude.

6. The multimodal intelligent glaucoma recognition system based on transfer learning according to claim 4 is characterized in that: Step 3 includes: The preprocessed text information is input into the first branch of the multimodal recognition module, and the words in the electronic medical records of glaucoma patients are clustered using the K-Means algorithm; The second branch uses the deep learning model U-Net, which inputs fundus photos to segment the optic nerve vessels and outputs the vascular distribution characteristics of glaucoma. It also inputs three-dimensional fundus photos to segment the optic cup and optic disc and outputs the optic nerve vascular distribution characteristics and cup-to-disc ratio. The outputs of the first and second branches are fused together as metadata and processed using the Catboost classifier; The third branch simultaneously inputs the Heidelberg optical coherence tomography images and optic nerve fiber layer thickness images in the training set into the deep learning network model, uses the transfer learning method to pre-train the parameters of the constructed neural network, and uses the gradient weighted class activation mapping method to calibrate the key focus areas of the neural network in the glaucoma identification process, which are then fused with the output of the Catboost classifier to obtain a comprehensive diagnostic result of glaucoma subtypes.

7. The multimodal intelligent glaucoma recognition system based on transfer learning according to claim 6 is characterized in that: The clustering of words in the electronic medical records of glaucoma patients by using the K-Means algorithm includes: The Word2vec model was trained using the glaucoma keyword corpus, with the input being the corpus data and the output being the glaucoma disease type; Perform data preprocessing operations such as word segmentation, part-of-speech tagging, deduplication, and stop word removal on the text in the patient's electronic medical record, and finally obtain n candidate keywords; Traverse the candidate keywords and extract the word vector representation of the candidate keywords from the word vector file; Perform K-Means clustering on the candidate keywords to obtain the cluster centers of each category; Calculate the Euclidean distance between the words in each group and the cluster center, and sort them in ascending order according to the size of the cluster; The candidate keywords are calculated and the top N words are obtained as text keywords for diagnosing glaucoma.

Images