Glaucoma multi-mode auxiliary diagnosis device and electronic equipment

Abstract

According to the glaucoma multi-mode auxiliary diagnosis device and the electronic equipment, firstly, first feature extraction and second feature extraction are carried out on an eye fundus image and an OCT image respectively, then semantic spaces of an eye fundus image mode and an OCT image mode are aligned by utilizing comparison loss, and the first feature and the second feature are fused by utilizing a cross attention mechanism, so that an eye fundus image is obtained. According to the method, the two modal data are subjected to fusion to obtain fusion features, then the fusion features are subjected to feature extraction to obtain third features, and glaucoma classification judgment is performed based on the third features, so that global modeling of the two modal data can be realized, effective features are extracted to perform glaucoma classification judgment, and the accuracy and timeliness of diagnosis are ensured.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a multimodal auxiliary diagnostic device and electronic device for glaucoma. Background Technology

[0002] Glaucoma is a chronic eye disease that permanently impairs vision and is a leading cause of irreversible blindness worldwide. The pathogenesis of glaucoma is related to optic nerve damage and visual field loss caused by the degeneration of retinal ganglion cells. The characteristics of glaucoma include optic disc atrophy and cupping, visual field loss, elevated intraocular pressure, and decreased vision. If glaucoma is not treated promptly, patients will suffer permanent vision loss due to damage to the optic nerve head and the retinal nerve fiber layer. Therefore, early diagnosis of glaucoma is crucial. In the diagnosis of glaucoma, relying solely on patient feedback and visual observation by doctors is insufficient to meet the diagnostic needs of complex pathologies and a large patient population.

[0003] Fundus imaging is a technique used to capture detailed images of the retina and its outer diameter. Fundus images help magnify the biological features and details of the patient's fundus, assisting doctors in assessing the appearance of the optic nerve and examining vascular changes to identify abnormalities. Fundus images are an effective diagnostic tool for various eye diseases, including glaucoma. Fundus images primarily include the macula, optic disc (OD), optic cup (OC), and optic nerve vessels. In clinical settings, doctors mainly rely on the analysis of the characteristics of the OD and OC areas to determine the progression of glaucoma. The main observation indicators are the ratio of the OC diameter to the OD diameter and the ratio of the OC area to the OD area; generally, these two indicators are directly proportional to the severity of the disease. Optical coherence tomography (OCT) is a non-invasive imaging technique that generates detailed cross-sectional images of the retina, optic nerve, and other eye structures. Doctors can use it to observe the retinal layer structure and measure the thickness of the retinal nerve fiber layer, thereby diagnosing glaucoma. OCT can effectively complement fundus imaging, helping doctors better diagnose glaucoma and reducing missed or misdiagnosed cases.

[0004] Clinically, the diagnosis of glaucoma relies on a variety of examination methods, such as fundus images and optical coherence tomography (OCT). However, these methods not only require a large amount of clinical data, but also the ophthalmologist's comprehensive interpretation of complex images and clinical information is easily affected by clinical experience and equipment conditions, which may lead to missed diagnosis or misdiagnosis. Summary of the Invention

[0005] This application proposes a multimodal auxiliary diagnostic device and electronic device for glaucoma, which can solve one of the problems existing in the background art.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] In a first aspect, a multimodal auxiliary diagnostic device for glaucoma is provided, comprising:

[0008] The feature extraction module is used to perform first feature extraction on fundus images and second feature extraction on optical coherence tomography (OCT) images.

[0009] A feature fusion module is used to align the semantic spaces of fundus image modalities and OCT image modalities using contrastive loss, and to fuse the first feature and the second feature using a cross-attention mechanism to obtain fused features; and,

[0010] The classification module is used to extract features from the fused features to obtain a third feature, and to classify and determine glaucoma based on the third feature.

[0011] Based on the above technical solution, firstly, the first and second features are extracted from the fundus image and OCT image respectively. Then, the semantic spaces of the fundus image modality and the OCT image modality are aligned using contrast loss. The first and second features are then fused using a cross-attention mechanism to obtain the fused feature. Next, the fused feature is used to extract the third feature, and glaucoma classification is performed based on the third feature. In this way, global modeling of the two modalities of data can be achieved, and effective features can be extracted for glaucoma classification, ensuring the accuracy and timeliness of diagnosis.

[0012] In one possible design of the first aspect, the feature extraction module includes: a first initial convolutional layer, a first max pooling layer, a first residual multi-head self-attention layer, a second initial convolutional layer, a second max pooling layer, and a second residual multi-head self-attention layer. The first initial convolutional layer, the first max pooling layer, and the first residual multi-head self-attention layer are sequentially connected to achieve the extraction of the first feature, and the second initial convolutional layer, the second max pooling layer, and the second residual multi-head self-attention layer are sequentially connected to achieve the extraction of the second feature.

[0013] In one possible design of the first aspect, the initial convolutional layer comprises, in sequence: a two-dimensional convolutional layer, a first batch of normalization layers, and a first activation function layer; the residual multi-head self-attention layer comprises, in sequence: a residual module, a multi-head self-attention module, and a third max pooling layer; the output of the first activation function layer is connected to the input of the residual module; and the output of the third max pooling layer is connected to the input of the feature fusion module.

[0014] In one possible design of the first aspect, the residual module comprises, in sequence, a first convolutional layer, a second batch normalization layer, a second activation function layer, a second convolutional layer, a third batch normalization layer, and a third activation function layer. The input of the first convolutional layer and the output of the third batch normalization layer are connected via a residual connection, and the output of the residual connection is connected to the input of the third activation function layer.

[0015] The multi-head self-attention module includes a third convolutional layer, a fourth batch normalization layer, and a multi-head self-attention layer connected in sequence, wherein the multi-head self-attention layer adopts local attention.

[0016] In one possible design of the first aspect, the feature fusion module includes: a first projection head, a second projection head, and a cross-attention layer. The inputs of the first projection head and the second projection head are connected to the output of the feature extraction module to project the first feature and the second feature into the same semantic space. The outputs of the first projection head and the second projection head are connected to the input of the cross-attention layer, which is used to fuse the features to obtain the fused features.

[0017] In one possible design of the first aspect, the projection head includes, in sequence: a fully connected layer, a layer normalization layer, and a fourth activation function layer.

[0018] The cross-attention layer is used to: perform a first attention-based fusion using the first projected feature as the query and the second projected feature as the key; and perform a second attention-based fusion using the second projected feature as the query and the first projected feature as the key.

[0019] In one possible design of the first aspect, the classification module includes, in sequence, a third residual multi-head self-attention layer, a fourth max pooling layer, a fourth residual multi-head self-attention layer, and a global average pooling layer.

[0020] In one possible design of the first aspect, the glaucoma multimodal auxiliary diagnostic device employs a joint loss function, which includes a contrast loss term and a cross-entropy loss term. The contrast loss term is used to calculate the similarity between two modalities of samples in the same batch in the low-level projection space, and the cross-entropy loss term is used to measure the difference between the model's predicted label and the actual label.

[0021] In one possible design of the first aspect, the glaucoma multimodal auxiliary diagnostic device further includes:

[0022] The preprocessing module is used to perform contrast enhancement, threshold segmentation, normalization, and / or size cropping on the original image to obtain the fundus image and the OCT image.

[0023] In a second aspect, an electronic device is provided, comprising: a processor and a memory coupled to the processor, the memory for storing a computer program; the processor for executing the computer program stored in the memory, such that the electronic device performs the following steps: extracting a first feature from a fundus image, extracting a second feature from an optical coherence tomography (OCT) image; aligning the semantic spaces of the fundus image modality and the OCT image modality using a contrast loss, and fusing the first feature and the second feature using a cross-attention mechanism to obtain a fused feature; and extracting a third feature from the fused feature, and classifying glaucoma based on the third feature. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a glaucoma multimodal assisted diagnosis method based on multimodal imaging provided in an embodiment of this application;

[0026] Figure 2 This is a structural diagram of the glaucoma multimodal auxiliary diagnostic device provided in the embodiments of this application;

[0027] Figure 3 This is a structural diagram of the residual multi-head self-attention layer provided in the embodiments of this application.

[0028] Figure 4 This is a structural diagram of the cross-attention layer provided in an embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0030] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0032] Before introducing the embodiments of this application, a brief description of the current stage of technical research of this application will be given:

[0033] In recent years, research on the use of computer vision and deep learning technologies for the auxiliary diagnosis of glaucoma has been increasing. Computer vision methods can automatically extract and analyze minute pathological changes in ocular images, thereby enabling rapid and accurate early screening and risk assessment, giving patients valuable intervention time and reducing the risk of irreversible blindness due to disease progression. Meanwhile, the application of multimodal data fusion technology further enhances the comprehensive judgment capabilities of multimodal assisted diagnostic systems for glaucoma.

[0034] Deep learning, a subfield of machine learning, differs from traditional machine learning in that it simulates the hierarchical information processing mechanism of the human brain by constructing multi-layered neural network models, thereby achieving efficient analysis and understanding of complex data. Traditional machine learning methods rely on manually designed filters and image processing techniques to extract features, requiring rich prior knowledge. However, deep learning adopts an end-to-end learning approach, which can directly and automatically extract abstract features from complex raw data. This characteristic gives deep learning a significant advantage over traditional machine learning methods in the field of glaucoma assisted diagnosis. This advantage stems from the fact that traditional machine learning methods, when processing complex data such as fundus images, rely on prior knowledge and cannot efficiently utilize the effective information within the data. Although deep learning networks such as convolutional neural networks (CNNs) perform well in many classic image classification tasks, they are less effective in tasks requiring the processing of global information, such as considering the proportional changes of the optic disc and optic cup in multimodal glaucoma assisted diagnosis. Transformer, as a sequence-to-sequence deep learning architecture, possesses powerful global modeling capabilities and was initially applied to natural language processing with significant success. Since its inception, ViT has enabled Transformers to achieve competitive performance across a range of computer vision tasks. The Transformer architecture effectively captures long-range dependencies in images through its self-attention mechanism, giving it a significant advantage in capturing scattered lesion regions. Glaucoma diagnosis relies not only on single-modality images but also, more commonly, on combining data from multiple modalities for accurate diagnosis. This necessitates that diagnostic models possess strong extraction capabilities across different modalities and can effectively fuse these data to form better feature sets. Traditional CNN networks require complex branching structures to extract and fuse features from different modalities. However, the Transformer's attention mechanism has a natural advantage in multimodal fusion, enabling dynamic information fusion between different modalities. By effectively integrating features from different sources through cross-modal attention, it improves diagnostic accuracy.

[0035] To achieve feature extraction from both fundus images and OCT data, this embodiment designs a lightweight network that effectively combines residual convolution and attention modules. After extracting high-frequency information from the images using residual convolution, low-frequency information is extracted through multi-head attention operations within local windows. This enables global modeling and effective feature extraction for both modalities. To fuse the two modalities and achieve better diagnostic results, the model in this embodiment uses projection and contrast loss to bridge the semantic gap between the feature data of the two modalities, and then uses cross-attention operations to complete the fusion of the two modalities. The method of this invention achieved better diagnostic results in experiments.

[0036] This embodiment constructs an end-to-end deep learning network for the multimodal glaucoma auxiliary diagnostic model based on multimodal images. To combine local details and global structure in the images, this invention uses a lightweight feature extraction module to extract features from fundus images and OCT data. The feature extraction module designed in this embodiment combines the global expressive power of a multi-head self-attention mechanism with the optimized stability of a residual network. The multi-head self-attention mechanism can learn long-distance dependencies between features from different subspaces, enabling it to perform global modeling and extract effective features from both fundus images and OCT data. Simultaneously, to maintain the module's lightweight nature, local attention is used instead of global attention. Unlike common local attention operations, the local attention operation in this embodiment does not segment the feature map; instead, considering that discriminative features in glaucoma diagnosis are concentrated in a small region, independent attention operations are performed only within a local window. The introduction of a residual network helps solve the gradient vanishing or exploding problem during training, contributing to faster model convergence and improved generalization ability. Since the diagnosis of glaucoma requires combining local features such as blood vessel orientation and global features such as cup-to-disc ratio, feature extraction using a combination of residual networks and Transformers can also combine local and global feature information, which can better identify relevant features of glaucoma and thus achieve more accurate multimodal assisted diagnosis of glaucoma.

[0037] The feature fusion module designed in this embodiment effectively fuses features extracted from fundus images and OCT data. By using projection and contrast loss to bridge the semantic gap between the feature data of the two modalities, and then employing cross-attention operations to allow the feature data from fundus images and OCT to complement each other, the fusion of the two modalities is completed. This enables the model to use the fused, effective features to diagnose glaucoma, improving diagnostic accuracy.

[0038] like Figure 1As shown, the glaucoma multimodal assisted diagnosis method based on multimodal imaging proposed in this embodiment mainly includes the following steps:

[0039] Step 1: Preprocess the images in the dataset and randomly divide them into training and test sets;

[0040] Preprocessing of fundus images mainly includes contrast enhancement, threshold segmentation, and standardization. Contrast enhancement employs an adaptive histogram equalization method with limited contrast to improve image detail contrast. Threshold segmentation uses histogram analysis to adaptively select thresholds and extract the foreground region of the fundus. Based on this, the target region is cropped according to the maximum connected component, and the results are uniformly scaled to a fixed size to achieve image standardization and ensure consistency between different images in subsequent processing. For the OCT modality, since OCT is greatly affected by the data acquisition equipment, different devices may produce datasets with significantly different data distributions. Therefore, this embodiment only performs simple preprocessing on the OCT data. The cropping size is used to highlight image portions related to the thickness and morphology of the retinal nerve fiber layer, thereby preserving structural information closely related to glaucoma diagnosis. The preprocessed data from both modalities are divided into training and test sets.

[0041] Step 2: Train the proposed glaucoma multimodal auxiliary diagnostic network on the training set defined in Step 1;

[0042] The aforementioned glaucoma multimodal assisted diagnostic network mainly consists of a feature extraction module, a feature fusion module, and a classification module, such as... Figure 2 As shown, the feature extraction module efficiently extracts effective features from fundus images and OCT by combining a residual network with a Transformer; the feature fusion module aligns the semantic spaces of the two modalities using contrastive loss, and then fuses the features of the two modalities using a cross-attention mechanism; the classification module uses the same structure as the feature extraction module, combining a residual network and a Transformer, to extract the fused features, helping the model extract glaucoma-related features.

[0043] The feature extraction module consists of two branches, responsible for extracting features from fundus images and OCT images respectively. Each branch comprises an initial convolutional layer and a residual multi-head self-attention layer (ResMSA), connected by a 2×2 max-pooling layer. The residual multi-head self-attention layer combines the advantages of convolution and attention mechanisms to achieve efficient feature extraction.

[0044] The initial convolutional layer consists of a two-dimensional convolution plus batch normalization and the activation function ReLU. The kernel size in the two-dimensional convolution is 3, the stride is 1, and the padding is 1.

[0045] like Figure 3 As shown, the Residual Multi-Head Self-Attention Layer (ResMSA) consists of a Residual module (ResNet), a Multi-Head Self-Attention module (Local MSA), and a 2×2 max-pooling layer connected sequentially. The Residual module and the Multi-Head Self-Attention module are complementary. The Multi-Head Self-Attention module tends to suppress high-frequency components in the image, while the Residual module tends to amplify them. Low-frequency signals correspond to shapes in the image, while high-frequency signals correspond to texture information; this complementarity is beneficial for efficient glaucoma feature extraction.

[0046] The ResNet consists of two consecutive convolutional layers (Conv), with a kernel size of 3×3, a stride of 1, and padding of 1. The two convolutional layers are connected by batch normalization (BN) and the ReLU activation function. Simultaneously, the input is added to the output of the two consecutive convolutional networks via residual skip connections. The input channel length is adjusted using a non-activated 1×1 convolutional layer with batch normalization to match the output size of the two consecutive convolutional layers. Finally, the residual module output is processed by the ReLU activation function. The formula for calculating BN is as follows:

[0047] in Let represent the input feature of the i-th sample in a batch of samples, and d represent the dimension of the input feature. These are the features after normalization. For learnable parameters, Let these represent the mean and variance within a batch, respectively. Here are the error coefficients; the formula for calculating the ReLU activation function is as follows:

[0048] in As input to the activation function, This is the value after the activation function; the calculation formula of the ResNet module is as follows:

[0049] in This represents the input to the residual module. This indicates the output after passing through the residual module. This represents a two-dimensional convolution, where both convolutions have a kernel size of 3, a stride, and padding of 1. It refers to a two-dimensional convolution with a kernel size of 1, used to adjust the number of input channels. Refers to the activation function ReLU. Refer to batch normalization operation.

[0050] The Local MSA consists of a 1×1 convolutional layer and a multi-head attention layer, connected by a batch normalization operation. The probability of the Dropout operation in the multi-head attention layer is 0.2. To improve the computational efficiency of the Transformer, this embodiment uses local attention instead of global attention in its multi-head attention layer, and its approach differs from common local attention operations that segment the feature map. Considering that the discriminative features for glaucoma diagnosis are concentrated in small regions, this embodiment employs local attention, which involves performing attention operations on each pixel within a 7×7 local window, with each window operating independently.

[0051] The original formula for calculating the self-attention module is as follows:

[0052] in:

[0053] As input features, It is the number of columns in matrices Q and K, i.e., the vector dimension. This is the parameter matrix that linearly transforms the input. This refers to the result calculated by the attention module. It is a function that transforms a vector into a probability distribution, and its calculation formula is as follows:

[0054] in The i-th element in the input vector. and Representing the i-th and j-th dimensions in the input vector, Let be the dimension of the input vector. The essence of multi-head attention mechanisms is to split the query, key, and value parameters multiple times while keeping the total number of parameters constant. Each set of split parameters is mapped to different subspaces in a high-dimensional space to calculate attention weights, thereby focusing on different parts of the input. The calculation formula is as follows:

[0055] in , , , The parameter matrix for linear transformation, This represents the i-th attention head. This refers to the result calculated after multi-head attention operations. The result calculated after performing self-attention operations on the i-th attention head. This refers to splicing along the feature dimension.

[0056] The formula for calculating the Local MSA is as follows:

[0057] in: , As input features, This indicates the output after passing through the Local MSA module. Multi-head attention operations are performed only within a 7×7 window. Indicate batch normalization operation, This is the parameter matrix that linearly transforms the input.

[0058] The feature fusion module consists of two projection heads (Proj) and a cross-attention layer (CrossAttn). The two projection heads project the features of the fundus image and the OCT features into the same semantic space, facilitating the subsequent fusion of the two modalities by the cross-attention layer.

[0059] The proposed project uses essentially the same design for both modalities, consisting of fully connected layers plus layer normalization and activation functions. It first projects the features extracted by the feature extraction module from both modalities into a semantic space of the same dimension using fully connected layers. Then, it uses layer normalization (LN) to reduce the differences in data distribution between the two modalities. Finally, it uses the GELU activation function to enhance the non-linear modeling capability of the projection, allowing the features to be better represented. A dropout operation is added to the projection head of the OCT modality to reduce noise; the probability of dropout is 0.5. The formula for calculating the GELU activation function is as follows:

[0060] in As input to the activation function, The formula for calculating the error function is as follows:

[0061] in The input to the error function is used; the calculation formula for the Dropout operation is as follows:

[0062] in For the input vector, This is the output after the Dropout operation. It is a random mask vector. To preserve probability, This indicates element-wise multiplication; the formula for calculating layer-normalized LN is as follows:

[0063] in The input features represent a single sample. These are the features after normalization. For learnable parameters, The mean and variance of all feature dimensions of the sample are calculated respectively. The error coefficient is used; the calculation formula for the projector head of Proj is as follows:

[0064] in This represents the calculation results of the fully connected layer. Represents the input vector. This represents the weight matrix used to perform a linear transformation on the input. This represents the bias matrix. Layer normalization operation, This refers to the activation function GELU.

[0065] like Figure 4 As shown, the core idea of ​​CrossAttn is to utilize the attention mechanism in Transformer to interact and fuse data from two different modalities. CrossAttn first uses the features of the fundus image projected by Proj as the query (Q) and the features of the OCT modality as the keys (K and V) for an attention operation to fuse the features. Then, it performs a second fusion using the features of the OCT modality as the query and the features of the fundus image as the keys. First, the input features of both modalities are processed by layer normalization (LN). Then, the inputs of both modalities are fed into a multi-head attention module to calculate the attention weights between different regions of the two modalities. The attention weights dynamically select the most relevant features for the current task from the modalities, and combine them with the original input of the query modality to form a residual connection to preserve global information. Next, a Dropout operation with a probability of 0.5 is performed. The fused features are then processed by layer normalization (LN) again and fed into a two-layer fully connected multilayer perceptron (MLP) for transformation. Finally, the fused features are combined with the transformed features again through a residual connection. The calculation formula for the MLP is as follows:

[0066] in For the input feature vector, and This represents the weight matrix used to perform a linear transformation on the input. and Represents the bias matrix. Refers to the activation function ReLU. CrossAttn refers to the output after passing through a multilayer perceptron; the formula for calculating CrossAttn is shown below:

[0067] in: ,in This represents a feature vector of a mode. It represents a feature vector of another modality. This is the parameter matrix that linearly transforms the input. This refers to the computation in a multilayer perceptron. Layer normalization operation, This refers to multi-head attention operations.

[0068] The classification module consists of two ResMSA layers (mentioned in the feature extraction module) plus a global average pooling layer. ResMSA's ability to extract high-frequency and low-frequency data allows it to accurately extract features relevant to glaucoma diagnosis from the fused data, ensuring high accuracy in assisted diagnosis. The two ResMSA layers are connected by a 2×2 max pooling layer. Global average pooling is an operation that compresses the spatial features of each channel into a single value. It does not require the introduction of additional parameters, effectively reducing model complexity and the risk of overfitting, while preserving the global semantic information of each channel.

[0069] During model training, the feature data of the two modalities after projection by the projection head Proj in the feature fusion module, the results obtained after the classification module output passes through a fully connected layer, and the true diagnostic labels are input into the loss function, and the model parameters are optimized based on the loss results. The loss function is a joint loss function of contrastive loss and cross-entropy loss. Contrastive loss calculates the similarity between the two modalities of samples in the same batch in the low-dimensional projection space, and its calculation formula is as follows:

[0070] Where B represents the number of samples in a batch. The temperature coefficient is set to 0.1. These represent the feature vectors of two modalities after passing through the projection head Proj and undergoing regularization. and Let represent the i-th and j-th samples in the batch. The cross-entropy loss is a simple binary cross-entropy loss, used to measure the difference between the model's prediction and the actual sample label. Its calculation formula is as follows:

[0071] Where B represents the number of samples in a batch. This represents the true label of the sample. This represents the i-th sample in the batch. This represents the predicted value output by the model. This represents the output value after the predicted value is input into the sigmoid function. The formula for calculating the sigmoid function is as follows:

[0072] in Let the input vector be denoted as , and the overall loss function be defined as follows:

[0073] in The weights, which dynamically change with each training epoch, are used to balance the contrastive and classification losses during training. This represents classification loss. This represents the loss in comparison. The calculation formula is as follows:

[0074] in This means the initial value of the weight is set to 0.1. This indicates that the final weight value is set to 0.9. t represents the current training epoch, and T represents the total number of training epochs.

[0075] Data augmentation was incorporated during training to prevent overfitting, including random horizontal and vertical flips (p=0.5) and random rotations (−30° to 30°). The Adam optimizer was used to optimize the model's parameters. The initial learning rate was set to 0.001; the training epoch was set to 150, and the batch size of the model was 10.

[0076] Step 3: Test the trained network model on the test set to obtain the model's auxiliary diagnostic performance on the test set.

[0077] Experimental verification:

[0078] To evaluate the effectiveness of the auxiliary diagnostic model proposed in this embodiment, it was compared with three other methods: two convolutional neural network-based methods, ResNet18 and EfficientNetB0, and one Transformer-based method, DeiT-S (a smaller version of DeiT). In reproducing these methods, adaptive modifications were made to accommodate different modal inputs. For a fair comparison, the same data preprocessing and data partitioning were used for the input data of each method during training. The comparison methods and the method in this embodiment were compared using evaluation metrics on the GAMMA dataset.

[0079] The GAMMA dataset was meticulously constructed by Zhongshan Ophthalmic Center of Sun Yat-sen University and includes multimodal paired samples from 300 subjects. Each sample contains a 2D fundus color image with the central optic disc and macula clearly visible, along with the corresponding 3D OCT scan data (composed of 256 scan screenshots from different locations).

[0080] ResNet18: It proposes a method to alleviate the vanishing gradient problem during the training of deep networks through residual structures. It consists of multiple residual blocks, each containing two convolutional layers. An identity mapping is used to directly add the input to the output, allowing the network to better train deeper structures. In the classification stage, ResNet18 compresses high-dimensional feature maps into vectors using global average pooling, and finally connects them to fully connected layers to complete the classification task.

[0081] EfficientNetB0 proposes an efficient convolutional neural network architecture based on a composite scaling strategy. EfficientNetB0 uses an improved MBConv as its base module, with each module combining depthwise separable convolutions and SE attention mechanisms. Between adjacent MBConv modules, a composite scaling approach is used to adjust the kernel size, number of channels, and resolution, achieving more efficient multi-scale feature representation. In the classification stage, EfficientNetB0 uses global average pooling to compress feature maps into vectors, which are then connected to fully connected layers to complete the classification task.

[0082] DeiT-S proposes a lightweight Vision Transformer architecture suitable for training scenarios with limited data. It employs a standard Transformer Block in the encoder, consisting of a multi-head self-attention network and a feedforward network, for extracting global features. Different Blocks maintain the same sequence length and achieve progressive semantic modeling through stacking. To improve performance on small datasets, DeiT-S introduces an additional distillation label on top of the classification label, with a teacher model providing supervision. In the classification stage, the model inputs the features of both labels into a fully connected layer to obtain the final prediction result.

[0083] In the experiments, AUROC (Area Under the Curve) is used to measure the model's ability to distinguish between positive and negative samples; a value closer to 1 indicates better performance. Accuracy represents the proportion of correct predictions made by the model overall, suitable for class-balanced data. The F1 score considers both precision and recall, making it more suitable for imbalanced scenarios. Precision refers to the proportion of actually positive samples out of those predicted as positive, while recall is the proportion of correctly predicted samples out of those actually positive; both reflect the model's ability to identify the positive class. Specificity measures the model's ability to correctly identify the negative class, reflecting its performance in distinguishing negative samples.

[0084] The formula for calculating AUROC is as follows:

[0085] The formulas for calculating TPR and FPR are as follows:

[0086] Where TP, FN, and FP represent true positive, false negative, and false positive, respectively; the formula for calculating the F1-score is as follows:

[0087] The formulas for calculating Precision and Recall are as follows:

[0088] The formula for calculating Specificity is as follows:

[0089] The formula for calculating accuracy is as follows:

[0090] TP, TN, FP, and FN represent true positive, true negative, false positive, and false negative, respectively.

[0091] The data used in this experiment consisted of a dataset containing fundus images and corresponding OCT data. The method in this embodiment and the comparative method were evaluated on the same test set, and the experimental results are shown in Tables 1 and 2.

[0092] Table 1: Comparison of the auxiliary diagnostic performance of the method in this embodiment with the comparative method trained on fundus images. The bolded part is the best indicator.

[0093] As shown in Table 1, the model proposed in this embodiment exhibits significant advantages in the multimodal assisted diagnosis task of glaucoma compared to the comparative model trained on fundus images: In terms of key indicators, the model achieves an accuracy of 98% (higher than ResNet18's 81% and EfficientNetB0's 77%), while the AUC-ROC value reaches 0.9896 (close to perfect classification performance), indicating that its ability to distinguish between glaucoma cases and non-cases far exceeds that of the baseline model; its recall rate is as high as 98% (12%-64% higher than other models), significantly reducing the risk of missed diagnoses, while the 98% specificity (better than the best baseline model ResNet18's 94%) takes into account the control of false positives; although the model has only 1.07M parameters (only 9.6% of ResNet18), it achieves the best balance between precision (98%) and recall with an F1 score of 0.98. This lightweight and highly sensitive characteristic makes it particularly suitable for early glaucoma screening in clinically resource-constrained scenarios. It can accurately capture lesion features and can be efficiently deployed on mobile terminals or portable devices. Since the comparison models are all trained on fundus images, while the model in this embodiment is trained on two modalities, it is possible that the effective information in the OCT data improved the model's performance on the dataset.

[0094] Table 2 shows that the model in this embodiment exhibits significant comprehensive advantages compared to the comparative model trained on OCT in the multimodal assisted diagnosis task of glaucoma. Firstly, in terms of model efficiency, its parameter count is only 1.07M, less than 9% of ResNet18, 5% of DeiT-S, and even 73.8% less than the lightweight EfficientNetB0. Regarding core performance metrics, AUROC reaches 0.9896, an improvement of 6.6 percentage points compared to the second-best model, ResNet18, demonstrating superior disease discrimination ability. Diagnostic accuracy (98%), F1 score (0.9800), and recall (98%) all significantly outperform the comparative model, especially in recall, where it is 2 percentage points higher than the second-best EfficientNetB0, which has important clinical value in avoiding missed glaucoma diagnoses. Simultaneously, the model's simultaneous improvement in specificity (98%) and precision (98%) indicates excellent performance in reducing false positives.

[0095] Table 2: Comparison of the auxiliary diagnostic performance of the method in this embodiment with the comparative method trained on OCT. The bolded part is the best indicator.

[0096] In summary, this embodiment proposes a multimodal assisted diagnosis method for glaucoma based on multimodal imaging. Experimental results show that this method can effectively extract features related to glaucoma diagnosis from both fundus images and OCT data while ensuring the model's lightweight nature, and can fuse the features of the two modalities for efficient assisted diagnosis.

[0097] This embodiment also provides a corresponding multimodal assisted diagnostic device for glaucoma, including:

[0098] The feature extraction module is used to perform first feature extraction on fundus images and second feature extraction on optical coherence tomography (OCT) images.

[0099] A feature fusion module is used to align the semantic spaces of fundus image modalities and OCT image modalities using contrastive loss, and to fuse the first feature and the second feature using a cross-attention mechanism to obtain fused features; and,

[0100] The classification module is used to extract features from the fused features to obtain a third feature, and to classify and determine glaucoma based on the third feature.

[0101] In one possible design, the feature extraction module includes: a first initial convolutional layer, a first max pooling layer, a first residual multi-head self-attention layer, a second initial convolutional layer, a second max pooling layer, and a second residual multi-head self-attention layer. The first initial convolutional layer, the first max pooling layer, and the first residual multi-head self-attention layer are sequentially connected to achieve the extraction of the first feature, and the second initial convolutional layer, the second max pooling layer, and the second residual multi-head self-attention layer are sequentially connected to achieve the extraction of the second feature.

[0102] In a first possible design, the initial convolutional layer comprises, in sequence: a two-dimensional convolutional layer, a first batch of normalization layers, and a first activation function layer; the residual multi-head self-attention layer comprises, in sequence: a residual module, a multi-head self-attention module, and a third max pooling layer; the output of the first activation function layer is connected to the input of the residual module; and the output of the third max pooling layer is connected to the input of the feature fusion module.

[0103] In one possible design, the residual module comprises, in sequence, a first convolutional layer, a second batch normalization layer, a second activation function layer, a second convolutional layer, a third batch normalization layer, and a third activation function layer. The input of the first convolutional layer and the output of the third batch normalization layer are connected via a residual connection, and the output of the residual connection is connected to the input of the third activation function layer.

[0104] The multi-head self-attention module includes a third convolutional layer, a fourth batch normalization layer, and a multi-head self-attention layer connected in sequence, wherein the multi-head self-attention layer adopts local attention.

[0105] In one possible design, the feature fusion module includes: a first projection head, a second projection head, and a cross-attention layer. The inputs of the first projection head and the second projection head are connected to the output of the feature extraction module to project the first feature and the second feature into the same semantic space. The outputs of the first projection head and the second projection head are connected to the input of the cross-attention layer, which is used to fuse the features to obtain the fused features.

[0106] In one possible design, the projection head comprises, in sequence: a fully connected layer, a layer normalization layer, and a fourth activation function layer.

[0107] The cross-attention layer is used to: perform a first attention-based fusion using the first projected feature as the query and the second projected feature as the key; and perform a second attention-based fusion using the second projected feature as the query and the first projected feature as the key.

[0108] In one possible design, the classification module comprises, in sequence: a third residual multi-head self-attention layer, a fourth max pooling layer, a fourth residual multi-head self-attention layer, and a global average pooling layer.

[0109] In one possible design, the glaucoma multimodal auxiliary diagnostic device employs a joint loss function, which includes a contrast loss term and a cross-entropy loss term. The contrast loss term is used to calculate the similarity between two modalities of samples in the same batch in the low-level projection space, and the cross-entropy loss term is used to measure the difference between the model's predicted label and the actual label.

[0110] In one possible design, the glaucoma multimodal auxiliary diagnostic device further includes:

[0111] The preprocessing module is used to perform contrast enhancement, threshold segmentation, normalization, and / or size cropping on the original image to obtain the fundus image and the OCT image.

[0112] This application also provides an electronic device, including: a processor, and a memory coupled to the processor, the memory being used to store a computer program; the processor being used to execute the computer program stored in the memory, so that the electronic device performs the method as described in any of the above embodiments.

[0113] Electronic devices can be computing devices such as desktop computers, laptops, handheld computers, and cloud servers. These electronic devices may include, but are not limited to, processors and memory.

[0114] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting various parts of the device via various interfaces and lines.

[0115] The memory can be used to store the computer program, and the processor implements various functions of the electronic device by running or executing the computer program stored in the memory and calling the data stored in the memory.

[0116] The memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc.; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0117] This application also provides a storage medium, which is a computer-readable storage medium. The computer program is stored in the computer-readable storage medium, and when executed by a processor, the computer program can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0118] This application also provides a computer program product, including: a computer program or instructions that, when the computer program or instructions are run on a computer, cause the computer to perform any of the above possible implementation methods.

[0119] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A multimodal auxiliary diagnostic device for glaucoma, characterized in that, include: The feature extraction module is used to perform first feature extraction on fundus images and second feature extraction on optical coherence tomography (OCT) images. A feature fusion module is used to align the semantic spaces of fundus image modalities and OCT image modalities using contrastive loss, and to fuse the first feature and the second feature using a cross-attention mechanism to obtain fused features; and, The classification module is used to extract features from the fused features to obtain a third feature, and to classify and determine glaucoma based on the third feature.

2. The glaucoma multimodal auxiliary diagnostic device as described in claim 1, characterized in that, The feature extraction module includes: a first initial convolutional layer, a first max pooling layer, a first residual multi-head self-attention layer, a second initial convolutional layer, a second max pooling layer, and a second residual multi-head self-attention layer. The first initial convolutional layer, the first max pooling layer, and the first residual multi-head self-attention layer are sequentially connected to achieve the extraction of the first feature, and the second initial convolutional layer, the second max pooling layer, and the second residual multi-head self-attention layer are sequentially connected to achieve the extraction of the second feature.

3. The glaucoma multimodal auxiliary diagnostic device as described in claim 2, characterized in that, The initial convolutional layer comprises, in sequence: a two-dimensional convolutional layer, a first batch of normalization layers, and a first activation function layer. The residual multi-head self-attention layer comprises, in sequence: a residual module, a multi-head self-attention module, and a third max pooling layer. The output of the first activation function layer is connected to the input of the residual module, and the output of the third max pooling layer is connected to the input of the feature fusion module.

4. The glaucoma multimodal auxiliary diagnostic device as described in claim 3, characterized in that, The residual module comprises, in sequence, a first convolutional layer, a second batch normalization layer, a second activation function layer, a second convolutional layer, a third batch normalization layer, and a third activation function layer. The input of the first convolutional layer and the output of the third batch normalization layer are connected via a residual connection, and the output of the residual connection is connected to the input of the third activation function layer. The multi-head self-attention module includes a third convolutional layer, a fourth batch normalization layer, and a multi-head self-attention layer connected in sequence, wherein the multi-head self-attention layer adopts local attention.

5. The glaucoma multimodal auxiliary diagnostic device as described in claim 1, characterized in that, The feature fusion module includes a first projection head, a second projection head, and a cross-attention layer. The inputs of the first projection head and the second projection head are connected to the output of the feature extraction module to project the first feature and the second feature into the same semantic space. The outputs of the first projection head and the second projection head are connected to the input of the cross-attention layer, which is used to fuse the features to obtain the fused features.

6. The glaucoma multimodal auxiliary diagnostic device as described in claim 5, characterized in that, The projection head consists of, in sequence: a fully connected layer, a layer normalization layer, and a fourth activation function layer. The cross-attention layer is used to: perform a first attention-based fusion using the first projected feature as the query and the second projected feature as the key; and perform a second attention-based fusion using the second projected feature as the query and the first projected feature as the key.

7. The glaucoma multimodal auxiliary diagnostic device as described in claim 1, characterized in that, The classification module comprises, in sequence: a third residual multi-head self-attention layer, a fourth maximum pooling layer, a fourth residual multi-head self-attention layer, and a global average pooling layer.

8. The glaucoma multimodal auxiliary diagnostic device as described in claim 1, characterized in that, The glaucoma multimodal auxiliary diagnostic device employs a joint loss function, which includes a contrast loss term and a cross-entropy loss term. The contrast loss term is used to calculate the similarity between two modalities of samples in the same batch in the low-level projection space, and the cross-entropy loss term is used to measure the difference between the model's predicted label and the actual label.

9. The glaucoma multimodal auxiliary diagnostic device as described in claim 1, characterized in that, The glaucoma multimodal auxiliary diagnostic device also includes: The preprocessing module is used to perform contrast enhancement, threshold segmentation, normalization, and / or size cropping on the original image to obtain the fundus image and the OCT image.

10. An electronic device, characterized in that, The electronic device includes: a processor, and a memory coupled to the processor. The memory is used to store computer programs; and The processor is configured to execute the computer program stored in the memory, causing the electronic device to perform the following steps: extracting a first feature from a fundus image and extracting a second feature from an optical coherence tomography (OCT) image; aligning the semantic spaces of the fundus image modality and the OCT image modality using contrast loss, and fusing the first feature and the second feature using a cross-attention mechanism to obtain a fused feature; and extracting a third feature from the fused feature, and classifying glaucoma based on the third feature.

Images