Glaucoma diagnosis and progression prediction method and device based on deep multi-modal fusion and collaborative attention mechanism

Abstract

This invention discloses a method and device for glaucoma diagnosis and progression prediction based on deep multimodal fusion and collaborative attention mechanisms, belonging to the field of medical artificial intelligence and computer-aided diagnostic technology. This invention acquires multimodal data including clinical text, fundus images, and optical coherence tomography (OCT) scans; for low-quality fundus images, it employs an algorithm that fuses nonlocal denoising and contrast-constrained histogram equalization to enhance image quality; it constructs independent feature extractors to obtain clinical text features, enhanced fundus image features, and OCT features; it effectively fuses multimodal features through a dual-gate residual attention module and a dual-spectrum cross-attention module; finally, based on a gated attention module, it uses the fused features to diagnose glaucoma and predict disease progression. This method fully utilizes the complementarity between multimodal information and effectively processes low-quality images, significantly improving the accuracy of glaucoma diagnosis and the performance of progression prediction.

Description

Technical Field

[0001] This invention belongs to the field of medical artificial intelligence and computer-aided diagnostic technology, and particularly relates to a method and device for glaucoma diagnosis and progression prediction based on deep multimodal fusion and collaborative attention mechanism. Background Technology

[0002] Glaucoma is a leading cause of irreversible blindness worldwide. In developed countries, approximately 50% of glaucoma patients remain undiagnosed, while in developing countries, this figure exceeds 90%. However, the etiology of this multifactorial disease remains unclear, making the development of new diagnostic tools crucial for early detection and intervention. Early symptoms of glaucoma are often subtle and difficult to diagnose. Traditional diagnostic methods rely on subjective judgment through visual field testing. While visual field testing remains a diagnostic standard, the process is time-consuming and requires close patient cooperation. Imaging techniques such as optical coherence tomography (OCT) can provide objective data such as retinal thickness measurements, but they cannot directly quantify disease severity. Therefore, more efficient and accurate auxiliary diagnostic tools are urgently needed in clinical practice.

[0003] In recent years, deep learning methods based on convolutional neural networks have significantly improved the accuracy of glaucoma diagnosis by analyzing multimodal data such as fundus photography and optical coherence tomography (OCT). For example, some studies have used the TabNet architecture to fuse electronic medical records and OCT data to predict the longitudinal progression of the disease; others have used the Transformer's cross-attention mechanism to combine fundus images with clinical text data to improve diagnostic accuracy; and still others have constructed a CNN-LSTM framework to generate synthetic visual field maps using OCT time series for rapid identification of patients with disease progression. However, these existing methods still have significant shortcomings. First, they fail to fully and effectively fuse data from different modalities, limiting the model's ability to leverage the complementary advantages between information. For example, they can only combine some modalities and cannot simultaneously integrate the deep correlations between clinical text, fundus images, and OCT. Second, the accuracy of these methods is highly sensitive to the image quality of fundus photography, and images acquired in clinical settings often suffer from various quality problems such as noise, blurring, and insufficient contrast, severely impacting the robustness and reliability of the models in practical applications and making it difficult for them to stably play an auxiliary diagnostic role in real clinical settings. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a method and apparatus for glaucoma diagnosis and progression prediction based on deep multimodal fusion and collaborative attention mechanisms, thereby resolving the issues present in the prior art.

[0005] In a first aspect, to achieve the above objectives, the present invention provides a method for glaucoma diagnosis and progression prediction based on deep multimodal fusion and collaborative attention mechanisms, comprising the following steps: Acquire multimodal glaucoma data including clinical texts, fundus images, and optical coherence tomography (OCT) scans; An algorithm that combines nonlocal denoising and contrast-constrained histogram equalization is used to enhance the image quality of fundus image data. The corresponding modal features were extracted from clinical text, enhanced fundus images, and optical coherence tomography images using independent feature extraction models. The extracted multimodal features are fused using a dual-gate residual attention module and a dual-spectral cross-attention module. Glaucoma diagnosis and disease progression prediction are based on the gated attention module and the fused multimodal features.

[0006] Optionally, the dual-gate residual attention module fuses the extracted multimodal features by: After stitching together optical coherence tomography (OCT) features and fundus image features, the data is input into a dynamic gating network consisting of a compression layer, a dilation layer, and a gating layer to generate an attention weight matrix. The original stitched features are then combined with nonlinearly transformed features through a residual connection structure. This weighted combination is dynamically modulated by the attention weight matrix, and the intensity of the residual signal is adjusted by a learnable scaling parameter.

[0007] Optionally, the dual-spectral cross-attention module fuses the extracted multimodal features by: Using optical coherence tomography (OCT) features as the query matrix and fundus image features as the key and value matrices, the input features are converted into query, key, and value vectors through a learnable projection matrix. The dot product of the query matrix and the transpose of the key matrix is ​​calculated and divided by the dimension of the key vector for scaling. After processing by a normalized exponential function, the product is multiplied by the value matrix to output the fused features.

[0008] Optionally, image quality enhancement of fundus image data includes: The multi-scale enhancement fusion module employs parallel processing of dilated convolutions with different dilation rates to capture vascular structural features at different scales; the convolutional block attention module applies channel attention and spatial attention sequentially for dynamic fusion; and the edge-guided fusion module adaptively weights and fuses the decoder output features with the original image based on the edge detection probability map.

[0009] Optionally, the edge-guided fusion process includes: extracting a probability map of vascular edges from the original fundus image using an edge detection convolution kernel; using the probability map as weights to perform element-wise weighted fusion of the upsampled features finally output by the decoder with the original fundus image, so that the enhanced image retains the vascular edge structure while improving contrast.

[0010] Optionally, the method employs a composite loss function for end-to-end optimization. The composite loss function consists of a weighted sum of image enhancement loss, feature alignment loss, and classification loss. The image enhancement loss includes structural similarity loss and edge preservation loss. The feature alignment loss maps the optical coherence tomography feature space to the fundus image feature space using a learnable projection matrix and calculates the difference between the two. The classification loss employs a cross-entropy loss function.

[0011] Optionally, glaucoma diagnosis and disease progression prediction based on the gated attention module using fused multimodal features include: The output features of the dual-gate residual attention module, the output features of the dual-spectrum cross-attention module, and the clinical text features are concatenated after linear transformation. Attention scores for the three sources are calculated using a gating network. The attention scores are used as weights to perform a weighted summation of the features from the three sources to generate a unified fusion feature. The unified fusion feature is then input into a classifier consisting of a fully connected layer and a normalized exponential function to output the probability distribution of diagnosis and prediction.

[0012] Secondly, the present invention also provides a glaucoma diagnosis and progression prediction device based on deep multimodal fusion and collaborative attention mechanism, used to implement a glaucoma diagnosis and progression prediction method based on deep multimodal fusion and collaborative attention mechanism, the device comprising: The multimodal data acquisition module is used to acquire multimodal glaucoma data, including clinical texts, fundus images, and optical coherence tomography scans. The fundus image quality enhancement module is used to enhance the image quality of the fundus image data by using an algorithm that combines nonlocal denoising and contrast-constrained histogram equalization. The modal feature extraction module is used to extract corresponding modal features from clinical text, enhanced fundus images, and optical coherence tomography images using independent feature extraction models. The multimodal feature fusion module is used to fuse the extracted multimodal features through a dual-gate residual attention module and a dual-spectral cross-attention module; The diagnosis and prediction module is used to diagnose glaucoma and predict disease progression based on the fused multimodal features of the gated attention module.

[0013] Thirdly, the present invention also provides a computer terminal device, comprising: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the glaucoma diagnosis and progression prediction method based on deep multimodal fusion and collaborative attention mechanism in the first aspect described above.

[0014] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the glaucoma diagnosis and progression prediction method based on deep multimodal fusion and collaborative attention mechanism described in the first aspect above.

[0015] Compared with the prior art, the present invention has the following advantages and technical effects: This invention provides a glaucoma diagnosis and progression prediction method based on deep multimodal fusion and collaborative attention mechanisms. By fusing nonlocal denoising and contrast-constrained histogram equalization algorithms, it enhances low-quality fundus images, effectively addressing low-quality imaging issues in clinical settings and improving the model's robustness and reliability in practical applications. Deep fusion of multimodal features via a dual-gate residual attention module and a bispectral cross-attention module enables adaptive selection and bidirectional enhancement among clinical text, fundus images, and optical coherence tomography (OCT) information, fully leveraging the complementarity of multimodal information. Feature alignment loss constrains the alignment of different image modalities in the semantic space, narrowing the semantic gap between modalities. A gated attention module dynamically aggregates features from each modality, ensuring that the unified feature representation after fusion simultaneously contains complementary information from all modalities. Finally, glaucoma diagnosis and progression prediction based on the fused features are correlated with key clinical features, significantly improving diagnostic accuracy and progression prediction performance, and enhancing the reliability of the results. Attached Figure Description

[0016] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart illustrating the glaucoma diagnosis and progression prediction method based on deep multimodal fusion and collaborative attention mechanism according to an embodiment of the present invention. Figure 2 This is a performance comparison chart of MUGLDEM in glaucoma diagnosis, based on an embodiment of the present invention. Figure 3 This is a performance comparison chart of MUGLDEM in predicting glaucoma progression, based on an embodiment of the present invention. Figure 4 This is a performance comparison chart of MUGLDEM in glaucoma image enhancement, as evaluated according to an embodiment of the present invention. Figure 5This is a weighted graph of different text data on the prediction results of MUGLDEM in the interpretability analysis of this embodiment of the invention; Figure 6 This is a comparison chart of the performance of MUGLDEM and removal of each module in the ablation experiment analysis of this invention; Figure 7 This is a performance comparison chart of MUGLDEM and data using different modalities in the ablation experiment analysis of this invention. Detailed Implementation

[0017] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0019] This embodiment provides a method for glaucoma diagnosis and progression prediction based on deep multimodal fusion and collaborative attention mechanisms, including: Acquire multimodal glaucoma data including clinical texts, fundus images, and optical coherence tomography (OCT) scans; An algorithm that combines nonlocal denoising and contrast-constrained histogram equalization is used to enhance the image quality of fundus image data. The corresponding modal features were extracted from clinical text, enhanced fundus images, and optical coherence tomography images using independent feature extraction models. The extracted multimodal features are fused using a dual-gate residual attention module and a dual-spectral cross-attention module. Glaucoma diagnosis and disease progression prediction are based on the gated attention module and the fused multimodal features.

[0020] Furthermore, the dual-gate residual attention module fuses the extracted multimodal features, including: After stitching together optical coherence tomography (OCT) features and fundus image features, the data is input into a dynamic gating network consisting of a compression layer, a dilation layer, and a gating layer to generate an attention weight matrix. The original stitched features are then combined with nonlinearly transformed features through a residual connection structure. This weighted combination is dynamically modulated by the attention weight matrix, and the intensity of the residual signal is adjusted by a learnable scaling parameter.

[0021] Furthermore, the dual-spectral cross-attention module fuses the extracted multimodal features, including: Using optical coherence tomography (OCT) features as the query matrix and fundus image features as the key and value matrices, the input features are converted into query, key, and value vectors through a learnable projection matrix. The dot product of the query matrix and the transpose of the key matrix is ​​calculated and divided by the dimension of the key vector for scaling. After processing by a normalized exponential function, the product is multiplied by the value matrix to output the fused features.

[0022] Further image quality enhancement of fundus image data includes: The multi-scale enhancement fusion module employs parallel processing of dilated convolutions with different dilation rates to capture vascular structural features at different scales; the convolutional block attention module applies channel attention and spatial attention sequentially for dynamic fusion; and the edge-guided fusion module adaptively weights and fuses the decoder output features with the original image based on the edge detection probability map.

[0023] Furthermore, the edge-guided fusion process includes: extracting a probability map of vascular edges from the original fundus image using an edge detection convolution kernel; using the probability map as weights to perform element-wise weighted fusion of the upsampled features finally output by the decoder with the original fundus image, so that the enhanced image retains the vascular edge structure while improving contrast.

[0024] Furthermore, the method employs a composite loss function for end-to-end optimization. The composite loss function consists of a weighted sum of image enhancement loss, feature alignment loss, and classification loss. The image enhancement loss includes structural similarity loss and edge preservation loss. The feature alignment loss maps the optical coherence tomography feature space to the fundus image feature space using a learnable projection matrix and calculates the difference between the two. The classification loss employs a cross-entropy loss function.

[0025] Furthermore, the use of fused multimodal features based on the gated attention module for glaucoma diagnosis and disease progression prediction includes: The output features of the dual-gate residual attention module, the output features of the dual-spectrum cross-attention module, and the clinical text features are concatenated after linear transformation. Attention scores for the three sources are calculated using a gating network. The attention scores are used as weights to perform a weighted summation of the features from the three sources to generate a unified fusion feature. The unified fusion feature is then input into a classifier consisting of a fully connected layer and a normalized exponential function to output the probability distribution of diagnosis and prediction.

[0026] Specifically, the implementation process of this embodiment includes: like Figure 1As shown, this invention provides a method for glaucoma diagnosis and progression prediction based on deep multimodal fusion and collaborative attention mechanisms. Taking the following application scenario as an example, in this scenario, multimodal data is defined as M={T,F,O}, where T represents clinical text data, F represents fundus photography image data, and O represents optical coherence tomography (OCT) image data. Glaucoma diagnosis and progression prediction are considered the primary tasks, and image quality enhancement of fundus photography and multimodal feature fusion are considered the core processing steps. The input is the multimodal data M, and the output is the disease diagnosis classification and progression risk assessment results.

[0027] To address the aforementioned application scenarios, this invention proposes a glaucoma diagnosis and progression prediction method based on deep multimodal fusion and collaborative attention mechanisms, which can also be referred to as the MUGLDEM model in this embodiment. In summary, MUGLDEM first acquires and preprocesses multimodal glaucoma data including clinical text, fundus photography, and optical coherence tomography (OCT). For low-quality fundus photography data, an algorithm combining nonlocal denoising and contrast-constrained histogram equalization is used to enhance image quality. Independent deep networks are used to extract features from each modality to obtain high-quality modality-specific representations. An innovative dual-gate residual attention module and a dual-spectrum cross-attention module are employed to perform deep adaptive fusion of the extracted multimodal features to fully exploit complementary information between modalities. Finally, based on the unified feature representation after fusion, a classifier is used to achieve glaucoma diagnosis and disease progression prediction. The image quality enhancement algorithm integrates multi-scale enhancement fusion, convolutional block attention mechanism, and edge-guided fusion to improve overall image quality while precisely preserving vascular structures. The multimodal feature fusion mechanism effectively addresses feature redundancy and semantic gap issues through dynamic gating and bidirectional semantic cross-attention. Finally, the optimized fused feature representation is used to complete disease identification.

[0028] like Figure 1As shown, MUGLDEM can be divided into several stages. The first stage, multimodal data preprocessing and enhancement, enhances low-quality fundus images based on clinical text, original fundus photographs, and optical coherence tomography (OCT) data, and extracts initial features for each modality. The second stage, multimodal feature deep fusion, utilizes gated residual attention and cross-attention mechanisms for generative adversarial complementary learning between image and text features, as well as between features from different image modalities, to explore and fuse detailed differences and consistency information among multimodal data. The third stage, graph node feature learning, uses "nodes" analogous to information units from different modalities or feature channels. Through the adjacency aggregation concept in graph neural networks, attention mechanisms are used to interact and aggregate features from different sources, further exploring the consistency and complementarity among multimodal information. The fourth stage, feature optimization and prediction, performs end-to-end optimization of the overall model based on a composite loss function encompassing image reconstruction, feature alignment, and classification tasks, and uses the final optimized fused features for disease diagnosis and prediction.

[0029] This invention relates to a glaucoma diagnosis and progression prediction method based on deep multimodal fusion and collaborative attention mechanisms, specifically including: Step 1. Multimodal data acquisition and fundus image quality enhancement: 1.1 Definition and Acquisition of Multimodal Data: The multimodal data This data comprises three categories: clinical text, fundus images, and optical coherence tomography (OCT). Clinical text data refers to structured information extracted from the electronic medical record system, including demographic characteristics, medical history, intraocular pressure, corneal thickness, and other multivariate indicators. Fundus images refer to two-dimensional images of the retina captured by a fundus camera, showcasing key anatomical structures such as the optic disc, vascular network, and macula. Optical coherence tomography (OCT) refers to high-resolution cross-sectional images of the retina obtained through low-coherence interferometry, capable of quantifying the thickness of different retinal layers. These data collectively constitute a complementary multimodal representation. ,in These are the feature representations extracted from the corresponding modes.

[0030] 1.2 Fundus Image Quality Enhancement Algorithm: For raw fundus images acquired clinically, which often have quality problems such as noise, blurriness, and insufficient contrast. (i.e., the original fundus image) This method employs an enhanced algorithm integrating nonlocal denoising and contrast-constrained histogram equalization to improve the robustness of subsequent feature extraction. The algorithm is implemented through an encoder-decoder network, whose core comprises three sequentially executed sub-modules: (1) Multi-scale enhanced fusion: The third layer of the encoder network outputs a feature map. Above, three dilated convolutions with different dilation rates are applied in parallel to capture contextual information of vascular structures at different scales. This process is defined as follows: ; in, Indicates having the first Each feature map represents an expansion rate. Subsequently, these three feature maps at different scales are concatenated along the channel dimension to form a multi-scale fused feature map. : ; Here This indicates a splicing operation.

[0031] (2) Convolutional block attention mechanism: This mechanism addresses multi-scale fusion features. By applying channel attention and spatial attention sequentially, the network can dynamically focus on information-rich feature channels and spatial regions.

[0032] First, two one-dimensional channel descriptors are generated through global average pooling and global max pooling: ; ; Next, the channel attention weights are calculated using the following formula: ; in, for function, It is a multilayer perceptron. This indicates element-wise multiplication.

[0033] Then, based on channel attention output Calculate the spatial attention weights: ; in, This represents the convolution operation. This represents concatenation along the channel dimension. The final output is the feature modulated by channel and spatial attention. .

[0034] (3) Edge-guided fusion: This module aims to adaptively fuse enhanced global features with the detailed structure of the original image.

[0035] First, the decoder uses transposed convolution pairs Perform upsampling and combine with shallow features from the encoder. Resolution is gradually restored by combining skip connections: ; ; ; in, This represents transposed convolution. This indicates a cropping operation. This indicates the addition of features.

[0036] Meanwhile, through an edge detection convolution kernel From the original image Probability map for extracting blood vessel edges : ; in, This represents the convolution operation.

[0037] Finally, the enhanced fundus image From marginal probability graph As weights, the final output features of the decoder are weighted and fused with the original image to generate the following: ; in, This represents element-wise multiplication. This is the convolution weight matrix of the decoder output layer. This is the activation function. This process ensures that while improving overall image contrast and clarity, fine structures such as the edges of key blood vessels are accurately preserved.

[0038] Step 2. Multimodal feature extraction: In this step, the present invention constructs three independent deep feature extraction networks for three heterogeneous data sources: clinical text, enhanced fundus images, and optical coherence tomography (OCT). The core of this design is to maximize the preservation of the specificity and integrity of each modality of data, avoiding information loss or confusion caused by early simple fusion. Each feature extractor is specifically optimized to capture the most discriminative high-level semantic features from its corresponding data source. Through parallel feature extraction processes, high-quality, modality-specific feature representations are obtained, providing a solid foundation for subsequent deep cross-modal fusion.

[0039] (1) Clinical text feature extraction: Preprocessed structured clinical data (including d-dimensional features such as age, intraocular pressure, and corneal thickness) are input into a fully connected network, which nonlinearly maps them into low-dimensional, dense feature vectors. This process is defined as follows: ; in, To input clinical feature vectors, and These represent the weights and biases of the two fully connected layers, respectively, and GELU is the Gaussian error linear unit activation function. Output This refers to the coded clinical text features.

[0040] (2) Feature extraction from fundus images: The enhanced image obtained in step 1 As input, using This network serves as the backbone, and is adaptively modified. The original network's global average pooling layer and classification layer are removed, retaining only its densely connected feature extraction portion. The input image is passed through this network, and feature maps are extracted from the last dense block. These feature maps are then transformed into fixed-dimensional feature vectors through an adaptive global average pooling layer. ; in, The feature vector is used to characterize the appearance and structural information of the fundus.

[0041] (3) OCT image feature extraction: OCT data consists of volumetric data comprising a series of cross-sectional slices. This invention employs a method based on... Feature extraction is performed using an improved network architecture. To integrate prior knowledge of the retinal layer structure, a lightweight attention module is introduced. This module dynamically assigns weights to different spatial regions based on the anatomical importance of each retinal layer (such as the nerve fiber layer and ganglion cell layer). Specifically, for the input OCT slice sequence, the feature extraction is first performed through... Basic convolutional layers extract initial features Then, spatial attention weights are calculated. : ; in, This represents global average pooling along the slice dimension. These are learnable parameters. The weighted feature calculation is as follows: ; in, It is the final extracted high-level feature representation that can reflect changes in retinal thickness and structural abnormalities.

[0042] Step 3. Deep fusion of multimodal features: After obtaining features from each modality, deep fusion is performed through a series of innovative attention mechanism modules, aiming to fully mine and integrate heterogeneous information from clinical text, fundus images, and optical coherence tomography (OCT). This process is the core of the model, designed to address issues such as feature redundancy and semantic gaps in multimodal fusion, achieving adaptive selection and enhancement of complementary information. Figure 1 As shown, this step sequentially passes through a dual-gate residual attention module, a dual-spectrum cross-attention module, and a gated attention module to ultimately generate a unified and robust multimodal fusion feature.

[0043] (1) Dual-door residual attention module: This module is designed to adaptively filter image features from fundus images and OCT scans, suppressing redundant information while enhancing complementary information. Its working principle is as follows: First, OCT features Features of fundus images Concatenate along the feature dimension: ; in, This indicates a splicing operation. .

[0044] Subsequently, a dynamic gating network consisting of a compression layer, an expansion layer, and a gating layer is constructed to generate a network that is compatible with... Same-dimensional attention weight matrix This weight matrix is ​​used to evaluate the importance of each feature element. The specific calculation process is defined as follows: ; ; ; in, and These are the trainable weight matrices for the compression layer, dilation layer, and gating layer, respectively, serving to reduce dimensionality, increase dimensionality, and generate gating weights. BN represents batch normalization, and GELU is the activation function. for The function constrains the weight values ​​to the interval (0,1).

[0045] Ultimately, the output characteristics of this module It is calculated through a residual connection structure that combines the original features with the features after nonlinear transformation, and is dynamically modulated by gated weights: ; in, Represents the identity matrix. This represents element-wise multiplication. It is a learnable scaling parameter used to adjust the strength of the residual signal. This embodiment enables the module to retain necessary basic information while flexibly enhancing useful cross-modal features.

[0046] (2) Bispectral Cross-Attention Module: To establish a deep semantic association between fundus images and OCT features, and to bridge the semantic gap between their appearance and structural information, this embodiment introduces a bispectral cross-attention module. This module uses features from one modality as "queries" and features from another modality as "keys" and "values" to achieve bidirectional semantic alignment and information retrieval.

[0047] Specifically, using OCT features As a query for fundus image features Cross-attention is computed using the key and value pairs. First, the input features are transformed into a query, key, and value vector using a learnable projection matrix: ; in, These are the corresponding projection matrices.

[0048] Then, the attention weights are calculated and the value vectors are aggregated to output the fused features. ; Equivalent to: ; in, This is the dimension of the key vector, used to scale the dot product result to prevent gradient vanishing. This operation enables the model to actively query and aggregate the most relevant regional appearance information in the fundus image based on the structural information reflected by OCT, achieving deep semantic complementarity between the two image modalities.

[0049] (3) Gated attention module: After obtaining the output characteristics of the first two modules and Then, this module is responsible for combining it with clinical text features. The final aggregation is performed to generate a unified feature representation for prediction.

[0050] First, combine the three components: ; in, It is a trainable linear transformation matrix used to map different features to the same dimension for fusion.

[0051] Subsequently, a lightweight gating network was developed based on... Calculate the attention weights for these three sources: ; here, and It is the weight of the gating network. It is a three-dimensional vector whose elements correspond to the pairs of... The normalized attention score.

[0052] Ultimately, the unified integration characteristics Generated by weighted sum: ; This feature It integrates complementary information from all modalities and dynamically adjusts the contribution of each information source according to task requirements. This information is then directly input into the downstream classifier for glaucoma diagnosis and progression prediction.

[0053] Step 4. Diagnosis, Prediction, and Model Optimization: In this step, this embodiment uses the unified multimodal fusion features obtained in step 3. For the final prediction task, a carefully designed composite loss function is used to perform end-to-end joint optimization of the model. This process aims to ensure that the three core sub-tasks—image quality enhancement, multimodal feature fusion, and disease diagnosis / prediction—work synergistically, promote each other, and jointly drive the model performance to the global optimum.

[0054] 4.1 Diagnosis and Progression Prediction: Fusion features Input a fully connected layer and A classifier composed of functions. This classifier maps high-dimensional features to a final predicted probability distribution. For the glaucoma diagnosis task, this is a binary classification problem, outputting probabilities for "healthy" and "glaucoma". The prediction process is formally defined as: ; in, and For the weights and biases of the classification layer, This is the predicted class probability vector.

[0055] 4.2 Composite Loss Function and Cooperative Optimization: To achieve end-to-end training and balance the subtasks, this embodiment defines a composite loss function. The function consists of a weighted sum of four loss terms, corresponding to the three optimization objectives: image enhancement quality, feature semantic alignment, and classification accuracy.

[0056] (1) Image enhancement quality loss: This loss is used to supervise the fundus image enhancement network in step 1, ensuring that its output image approaches the ideal high-quality image in terms of perceptual quality and structural detail. It consists of two sub-items: Structural similarity loss: a measure of enhanced images With high-quality reference images Similarities in brightness, contrast, and structure; ; in, The calculation involves the mean, variance, and covariance of the local window.

[0057] Edge Preservation Loss: The augmented image is forced to maintain consistency with the reference image on key structures such as blood vessel edges, and the gradient difference is measured using the L1 norm; ; in, Represents the gradient operator. This represents the total number of pixels.

[0058] (2) Feature alignment loss: This loss is used to constrain the multimodal fusion process in step 3, forcing features from different image modalities to align in the semantic space to promote effective cross-modal interaction.

[0059] ; in, and These are the basic image features obtained from the feature extraction network. It is a learnable linear projection matrix used to map the OCT feature space to the fundus image feature space. Minimizing this loss means making the two image features as close as possible in the projected space, thereby reducing the semantic gap between modalities.

[0060] (3) Loss from classifying tasks: This is the model's main task loss, using the standard cross-entropy loss function to optimize the final diagnostic or prediction accuracy.

[0061] ; in, This represents the number of categories (C=2 in the diagnostic task). It is the one-hot encoding of the real label. It is the category predicted by the model. The probability of.

[0062] (4) Total loss and balancing weight: Ultimately, the model's total loss is the weighted sum of the four losses mentioned above: ; in, These are hyperparameters used to balance the magnitude and importance of different loss terms, ensuring that each subtask is optimized in a balanced manner during training. and These are structural similarity loss and edge preservation loss for image enhancement, respectively. For feature alignment loss, Let λ be the classification cross-entropy loss used for diagnosis and progression prediction, and λ be the weight coefficient of each loss term. The gradient is calculated from the classification loss using the backpropagation algorithm. The model progresses through the fusion module, feature extraction module, and back to the image enhancement module, enabling the entire MUGLDEM model to perform global and collaborative parameter updates in a goal-oriented manner. This end-to-end optimization mechanism is key to the high performance of this invention. It avoids suboptimal solutions that may arise from staged training and ensures a smooth and consistent flow of information from the original input to the final prediction.

[0063] The following embodiments of the present invention will be combined with specific experimental verification to demonstrate the feasibility and progressiveness of the present invention.

[0064] During experimental verification, the MUGLDEM model will be evaluated from the following aspects.

[0065] •RQ1: Does MUGLDEM outperform other benchmark multimodal glaucoma models in glaucoma diagnosis? • Does RQ2: Does MUGLDEM outperform other benchmark multimodal glaucoma models in predicting glaucoma progression? •RQ3: Does the design of different modules help improve the performance of MUGLDEM? •RQ4: Does inputting different modal data help improve the performance of MUGLDEM? • Does RQ5:MUGLDEM outperform other benchmark image augmentation models? •RQ6: Which text data types have the greatest impact on glaucoma diagnosis in MUGLDEM? Dataset Description: To validate the performance of the MUGLDEM model of this invention, experiments were conducted using a multimodal glaucoma dataset from a large biomedical database. This dataset includes clinical text data, fundus photographic images, and optical coherence tomography (OCT) images, along with glaucoma diagnostic labels and disease progression annotations for some samples. The data underwent rigorous screening to ensure the completeness and quality of the multimodal data. Due to its large scale, comprehensive modalities, and widespread use in related research, this dataset was chosen as a reliable dataset for evaluating the performance of glaucoma-assisted diagnosis and progression prediction.

[0066] Parameter configuration: Consistent training parameter settings were used across all experiments, including a learning rate of 0.001, 100 training epochs, and the Adam optimizer. An early stopping mechanism was employed during training: training was automatically terminated when the validation set loss no longer decreased within 10 consecutive epochs, and the best-performing model weights were saved. Hyperparameters related to the attention mechanism in the image augmentation network and multimodal fusion network (number of attention heads, loss balancing weights λ) were determined on the validation set using grid search.

[0067] Evaluation Metrics: To evaluate model performance, MUGLDEM selected several representative metrics. For diagnosis and prediction tasks, the area under the receiver operating characteristic (AUC) curve, accuracy, and F1 score were used. A higher AUC value indicates stronger discriminative ability. For image enhancement tasks, the structural similarity index and peak signal-to-noise ratio (PSNR) were used to quantitatively measure the consistency of visual quality and structural fidelity between the enhanced image and the high-quality reference image. Higher values ​​for these evaluation metrics indicate better model performance.

[0068] Environment Setup: The MUGLDEM method is implemented based on Python 3.12 and the PyTorch 2.2 deep learning framework, and integrates... Image preprocessing and analysis were performed using libraries. Model training and evaluation were completed on a workstation platform equipped with two NVIDIA RTX 4090 graphics processors (24GB VRAM).

[0069] 2. Results Analysis: The model of this invention was compared with nine advanced benchmark models.

[0070] Traditional machine learning methods are as follows: The K-Nearest Neighbors (KNN) algorithm is an instance-based lazy learning algorithm that makes predictions by measuring the distance between the sample to be classified and its nearest neighbor in the training set.

[0071] Stochastic gradient descent (SGD) is a classic algorithm that minimizes the loss function through iterative optimization. Each update is based on the gradient of only one or a small batch of samples, making it computationally efficient.

[0072] Logistic Regression (LR) is a generalized linear model commonly used for binary classification tasks. It uses the Sigmoid function to map linear combinations into probability outputs.

[0073] Naive Bayes (NB) is a probabilistic classifier based on Bayes' theorem and the assumption of conditional independence of features. It has a simple model and fast training speed.

[0074] Deep learning methods are as follows: Dense neural networks (DNNs) are a type of feedforward neural network consisting of multiple hidden layers that can learn high-level abstract features of data through multi-layer nonlinear transformations.

[0075] • ResNet50V2 is an important improvement to the ResNet residual network. It adopts a "pre-activation" structure to optimize gradient flow and improve the training stability and performance of deep networks.

[0076] The glaucoma diagnostic model is as follows: MDLGO is an innovative deep learning model that integrates OCT, fundus imaging, and visual field data, leveraging the advantages of multimodal fusion to provide comprehensive analysis for glaucoma diagnosis.

[0077] GMNNnet is a novel multimodal neural network for glaucoma diagnosis and classification. It extracts features from glaucoma segmentation based on domain knowledge and uses ResForformer to directly process glaucoma medical images.

[0078] The TabNet fusion method integrates the TabNet deep learning architecture with the baseline XGBoost model. The fusion model combines RNFL and electronic health record data to predict the progression of glaucoma.

[0079] The experimental results are analyzed in detail below: Performance Evaluation (RQ1): This embodiment evaluates the glaucoma diagnostic performance of MUGLDEM. To ensure comprehensive comparison, nine representative benchmark methods were selected on the UK Biobank dataset for comparison, including K-Nearest Neighbors (KNN), Stochastic Gradient Descent (SGD), Logistic Regression (LR), Naive Bayes (NB), Dense Neural Networks (DNN), ResNet50V2, MDLGO, GMNNnet, and the TabNet fusion model. A comprehensive evaluation was performed using three metrics: AUC, accuracy, and F1 score.

[0080] like Figure 2As shown, in the glaucoma diagnosis task, MUGLDEM achieved an AUC of 0.968, significantly outperforming all nine baseline methods. Specifically, compared to traditional machine learning methods (KNN, SGD, LR, NB), MUGLDEM's AUC improvement ranged from 19% to 25%. Compared to deep learning methods (DNN, ResNet50V2), the AUC improvements were 13% and 6%, respectively. Compared to existing state-of-the-art glaucoma diagnosis models (MDLGO, GMNNnet, TabNet fusion), MUGLDEM also achieved AUC improvements of 5%, 4%, and 12%, respectively. This demonstrates MUGLDEM's superiority in integrating multimodal data from clinical text, fundus photography (FP), and optical coherence tomography (OCT) for efficient feature fusion.

[0081] Progression Prediction Assessment (RQ2): MUGLDEM also performed excellently in the glaucoma progression prediction task. For example... Figure 3 As shown, MUGLDEM achieved an AUC of 0.853, surpassing all comparable methods. Its AUC improvement is significant compared to traditional machine learning methods. Compared to specialized methods such as MDLGO, GMNNnet, and TabNet fusion, MUGLDEM achieved AUC improvements of 6%, 8%, and 2%, respectively. This advantage is attributed to MUGLDEM's ability to effectively integrate multi-timepoint FP and OCT features from the same patient, thereby capturing dynamic information about disease progression, whereas traditional methods often rely on static features.

[0082] Image Augmentation Performance (RQ5): To evaluate the performance of the FP image augmentation module in MUGLDEM, we compare it with state-of-the-art image augmentation models such as CycleGAN, CutGAN, StillGAN, PCE-Net, ArcNet, and SCR-Net. Figure 4 As shown, MUGLDEM achieves the best results in both Structural Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PSNR), at 0.914 and 24.78, respectively. Compared to traditional GAN ​​methods, MUGLDEM improves SSIM by 4% to 10%, thanks to its Multi-Scale Enhancement Fusion (MEF) and Convolutional Block Attention (CBA) modules, which effectively alleviate the information loss problem of GAN models. Furthermore, its performance surpasses other network-based enhancement models (PCE-Net, ArcNet, SCR-Net), attributed to its edge-guided fusion strategy, which better preserves the edge details of key structures such as blood vessels while enhancing the image.

[0083] 3. MUGLDEM Model Analysis: A) Ablation experiment.

[0084] 1) Module Effectiveness Analysis (RQ3): The contributions of different key modules in MUGLDEM were evaluated through systematic ablation experiments. Experimental Results Figure 6 The results show that removing the bispectral cross-attention module leads to an 8% decrease in diagnostic AUC, confirming its crucial role in establishing cross-modal (OCT and FP) semantic mapping. Removing the dual-gated residual module reduces AUC by 9%, demonstrating its indispensability in dynamically fusing and selecting multimodal features. Furthermore, removing the feature alignment module and the FP image enhancement module also results in a 7% decrease in AUC, highlighting the importance of feature alignment in addressing intermodal differences and image enhancement in improving input data quality. These results collectively demonstrate that each core design module of MUGLDEM substantially contributes to the final performance.

[0085] 2) Multimodal Contribution Analysis (RQ4): The contributions of different modal data inputs were evaluated through ablation experiments. For example... Figure 7 Integrating clinical text, FP, and OCT modalities achieved the best predictive performance (AUC 0.968). When using clinical text alone, the AUC was 0.929, outperforming FP or OCT alone, indicating that clinical text contains crucial diagnostic information. In the bimodal combination, clinical text + OCT (AUC 0.954) outperformed clinical text + FP (AUC 0.912) and FP + OCT (AUC 0.884), demonstrating that MUGLDEM effectively captures the highly complementary potential interactions between clinical text and OCT images, thus significantly improving the model's diagnostic capabilities.

[0086] B) Interpretability Analysis (RQ6): To clarify the basis of the model's decisions and enhance its clinical credibility, this embodiment uses the SHAP (SHapley Additive exPlanations) method to perform attribution analysis on the clinical text features on which the model relies. For example... Figure 5 As shown, the SHAP value heatmap visualizes the contribution and direction of each clinical feature to the model's prediction of glaucoma risk. The analysis results show that age, EPRS (POAG) (enhanced polygenic risk score for primary open-angle glaucoma), and IOP-C (corneal compensatory intraocular pressure) are the features with the most significant impact on the model's decision-making, exhibiting the highest SHAP values. This finding is consistent with current clinical understanding and literature reports; EPRS (POAG) has been proven to be significantly associated with glaucoma risk. This analysis demonstrates that MUGLDEM's decision-making mechanism focuses on clinically recognized strong risk factors, rather than unreliable noisy associations, thus proving the rationality and interpretability of its decision-making process and providing a basis for physicians to understand and trust the model's output. Based on the same general inventive concept, this invention also provides a glaucoma diagnosis and progression prediction device based on deep multimodal fusion and collaborative attention mechanism. The glaucoma diagnosis and progression prediction device based on deep multimodal fusion and collaborative attention mechanism provided by this invention is described below. The glaucoma diagnosis and progression prediction device described below can be referred to in correspondence with the glaucoma diagnosis and progression prediction method based on deep multimodal fusion and collaborative attention mechanism described above. The device includes: The multimodal data acquisition module is used to acquire multimodal glaucoma data, including clinical texts, fundus images, and optical coherence tomography scans. The fundus image quality enhancement module is used to enhance the image quality of the fundus image data by using an algorithm that combines nonlocal denoising and contrast-constrained histogram equalization. The modal feature extraction module is used to extract corresponding modal features from clinical text, enhanced fundus images, and optical coherence tomography images using independent feature extraction models. The multimodal feature fusion module is used to fuse the extracted multimodal features through a dual-gate residual attention module and a dual-spectral cross-attention module; The diagnosis and prediction module is used to diagnose glaucoma and predict disease progression based on the fused multimodal features of the gated attention module.

[0087] In this embodiment, a computer terminal device is provided, including: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the above-described method for glaucoma diagnosis and progression prediction based on deep multimodal fusion and collaborative attention mechanism.

[0088] In this embodiment, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the above-described method for glaucoma diagnosis and progression prediction based on deep multimodal fusion and collaborative attention mechanism.

[0089] This invention provides a glaucoma diagnosis and progression prediction method based on deep multimodal fusion and collaborative attention mechanisms. By fusing nonlocal denoising and contrast-constrained histogram equalization algorithms, it enhances low-quality fundus images, effectively addressing low-quality imaging issues in clinical settings and improving the model's robustness and reliability in practical applications. Deep fusion of multimodal features via a dual-gate residual attention module and a bispectral cross-attention module enables adaptive selection and bidirectional enhancement among clinical text, fundus images, and optical coherence tomography (OCT) information, fully leveraging the complementarity of multimodal information. Feature alignment loss constrains the alignment of different image modalities in the semantic space, narrowing the semantic gap between modalities. A gated attention module dynamically aggregates features from each modality, ensuring that the unified feature representation after fusion simultaneously contains complementary information from all modalities. Finally, glaucoma diagnosis and progression prediction based on the fused features are correlated with key clinical features, significantly improving diagnostic accuracy and progression prediction performance, and enhancing the reliability of the results.

[0090] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for glaucoma diagnosis and progression prediction based on deep multimodal fusion and collaborative attention mechanism, characterized in that, Includes the following steps: Acquire multimodal glaucoma data including clinical texts, fundus images, and optical coherence tomography (OCT) scans; An algorithm that combines nonlocal denoising and contrast-constrained histogram equalization is used to enhance the image quality of fundus image data. The corresponding modal features were extracted from clinical text, enhanced fundus images, and optical coherence tomography images using independent feature extraction models. The extracted multimodal features are fused using a dual-gate residual attention module and a dual-spectral cross-attention module. Glaucoma diagnosis and disease progression prediction are based on the gated attention module and the fused multimodal features.

2. The method according to claim 1, characterized in that, The dual-gate residual attention module fuses the extracted multimodal features, including: After stitching together optical coherence tomography (OCT) features and fundus image features, the data is input into a dynamic gating network consisting of a compression layer, a dilation layer, and a gating layer to generate an attention weight matrix. The original stitched features are then combined with nonlinearly transformed features through a residual connection structure. This weighted combination is dynamically modulated by the attention weight matrix, and the intensity of the residual signal is adjusted by a learnable scaling parameter.

3. The method according to claim 1, characterized in that, The bispectral cross-attention module fuses the extracted multimodal features, including: Using optical coherence tomography (OCT) features as the query matrix and fundus image features as the key and value matrices, the input features are converted into query, key, and value vectors through a learnable projection matrix. The dot product of the query matrix and the transpose of the key matrix is ​​calculated and divided by the dimension of the key vector for scaling. After processing by a normalized exponential function, the product is multiplied by the value matrix to output the fused features.

4. The method according to claim 1, characterized in that, Image quality enhancement for fundus image data includes: The multi-scale enhancement fusion module employs parallel processing of dilated convolutions with different dilation rates to capture vascular structural features at different scales; the convolutional block attention module applies channel attention and spatial attention sequentially for dynamic fusion; and the edge-guided fusion module adaptively weights and fuses the decoder output features with the original image based on the edge detection probability map.

5. The method according to claim 4, characterized in that, The edge-guided fusion process includes: extracting a probability map of vascular edges from the original fundus image using an edge detection convolution kernel; using the probability map as weights to perform element-wise weighted fusion of the upsampled features finally output by the decoder with the original fundus image, so that the enhanced image retains the vascular edge structure while improving contrast.

6. The method according to claim 1, characterized in that, The method employs a composite loss function for end-to-end optimization. The composite loss function consists of a weighted sum of image enhancement loss, feature alignment loss, and classification loss. The image enhancement loss includes structural similarity loss and edge preservation loss. The feature alignment loss maps the optical coherence tomography feature space to the fundus image feature space using a learnable projection matrix and calculates the difference between the two. The classification loss employs a cross-entropy loss function.

7. The method according to claim 1, characterized in that, Glaucoma diagnosis and disease progression prediction based on gated attention modules utilizing fused multimodal features include: The output features of the dual-gate residual attention module, the output features of the dual-spectrum cross-attention module, and the clinical text features are concatenated after linear transformation. Attention scores for the three sources are calculated using a gating network. The attention scores are used as weights to perform a weighted summation of the features from the three sources to generate a unified fusion feature. The unified fusion feature is then input into a classifier consisting of a fully connected layer and a normalized exponential function to output the probability distribution of diagnosis and prediction.

8. A glaucoma diagnosis and progression prediction device based on deep multimodal fusion and collaborative attention mechanism, characterized in that, For implementing the method according to any one of claims 1-7, the apparatus comprises: The multimodal data acquisition module is used to acquire multimodal glaucoma data, including clinical texts, fundus images, and optical coherence tomography scans. The fundus image quality enhancement module is used to enhance the image quality of the fundus image data by using an algorithm that combines nonlocal denoising and contrast-constrained histogram equalization. The modal feature extraction module is used to extract corresponding modal features from clinical text, enhanced fundus images, and optical coherence tomography images using independent feature extraction models. The multimodal feature fusion module is used to fuse the extracted multimodal features through a dual-gate residual attention module and a dual-spectral cross-attention module; The diagnosis and prediction module is used to diagnose glaucoma and predict disease progression based on the fused multimodal features of the gated attention module.

9. A computer terminal device, characterized in that, include: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the steps of the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-7.

Images