Multi-modal ophthalmologic image analysis method, model and system based on optimal transmission image diffusion and medium of multi-modal ophthalmologic image analysis method, model and system

Abstract

The invention relates to the technical field of multi-modal ophthalmology image analysis, and particularly discloses a multi-modal ophthalmology image analysis method, model and system based on optimal transmission image diffusion and a medium thereof.The method at least comprises the steps that S100, the distance between fundus color photo and OCT image modals is explicitly calculated through an optimal transmission algorithm, and the distance between fundus color photo and OCT image modals is calculated; carrying out collaborative evolution on the features in a continuous time domain in combination with a graph neural network and a Sheng differential equation model; and step S200, applying self-attention and cross attention in parallel, reserving modal specificity with low-layer fine granularity, and progressively fusing the color photo texture and the OCT depth structure in a high layer to realize cross-layer long-range dependent depth semantic fusion. The method not only promotes the leading-edge development of multi-modal medical image analysis, but also provides important scientific basis and technical support for constructing a high-precision and high-robustness intelligent ophthalmology diagnosis system.

Description

Technical Field

[0001] This invention relates to the field of multimodal ophthalmic image analysis technology, specifically to a multimodal ophthalmic image analysis method based on optimal transmission map diffusion, a multimodal ophthalmic image analysis model based on optimal transmission map diffusion, a multimodal ophthalmic image analysis system based on optimal transmission map diffusion, and a computer-readable storage medium related to multimodal ophthalmic image analysis based on optimal transmission map diffusion. Background Technology

[0002] Eye diseases, as a leading cause of blindness and vision impairment worldwide, constitute a significant public health challenge. Early and accurate diagnosis plays a crucial role in preventing irreversible visual loss. Color fundus photography (CFP) captures two-dimensional color images of the retinal surface, visually revealing key pathological features such as vascular morphology changes, optic disc margin abnormalities, and macular lesions, providing clinicians with important morphological diagnostic evidence. Meanwhile, optical coherence tomography (OCT), with its superior three-dimensional imaging capabilities, can quantify the structural parameters and thickness distribution of each retinal layer with micron-level precision, providing objective and reliable biomarkers for quantitative assessment and dynamic monitoring of diseases.

[0003] In recent years, deep learning-based methods have been applied to the automated diagnosis of ophthalmic diseases. Previous work mainly focused on the use of single modalities, whether CFP or OCT, neglecting the potential synergistic effects that can be achieved by combining the two methods. Typically, multimodal learning provides more supplementary information than single-modal learning. This organic combination of multi-dimensional information not only enriches the data foundation for clinical diagnosis but also lays a solid foundation for building intelligent and precise computer-aided diagnostic systems. The revolutionary breakthroughs of deep learning technology in computer vision have further catalyzed the rapid development of multimodal medical image analysis. Advanced architectures such as convolutional neural networks, visual Transformers, and graph neural networks have demonstrated superior performance in single-modal medical image processing, providing a strong technical reserve for solving complex multimodal fusion problems.

[0004] Traditional multimodal fusion methods reveal deep theoretical and technical limitations when dealing with highly heterogeneous data such as ophthalmic images. While feature-level fusion strategies are intuitive, their reliance on simple concatenation or weighted averaging mechanisms is essentially a linear transformation, failing to effectively capture the complex nonlinear interactions between different modalities. This results in a significant dilution of valuable complementary information during the fusion process. Modern attention-based fusion methods, while introducing adaptive weight allocation and demonstrating some flexibility, still rely on a shallow feature combination framework and lack the ability to understand and model the deep structural relationships of multimodal data within a high-dimensional semantic space.

[0005] More importantly, existing methods generally lack rigorous theoretical guidance, mostly adopting experience-driven fusion strategies, which makes it difficult to guarantee the optimality and robustness of cross-modal feature alignment. Fundus color images and OCT data exhibit significant heterogeneity in feature distribution: the former shows continuous color texture distribution features, while the latter shows a discrete hierarchical structure distribution pattern. This distributional difference renders traditional Euclidean geometric measurement methods ineffective, and a more refined geometric theory is urgently needed to establish effective cross-modal semantic mapping relationships.

[0006] Another significant drawback of existing fusion frameworks lies in their static network architecture design, which lacks an adaptive adjustment mechanism for different sample feature complexities and fusion requirements. This often results in performance degradation and insufficient generalization ability when dealing with complex cases. Summary of the Invention

[0007] In view of this, the present invention provides a multimodal ophthalmic image analysis method based on optimal transmission map diffusion to overcome the technical problem that the simple stitching or weighted averaging mechanism relied upon by the prior art cannot effectively capture the complex nonlinear interaction relationship between different modalities, resulting in the effective complementary information being diluted in large quantities during the fusion process.

[0008] To achieve the above objectives, the basic solution of this invention provides a multimodal ophthalmic image analysis method based on optimal transport map diffusion, comprising at least:

[0009] Step S100: The distance between fundus color images and OCT image modalities is explicitly calculated using the optimal transmission algorithm. Combined with graph neural networks and neural network frequent differential equation models, features are co-evolved in the continuous time domain. Step S200: Self-attention and cross-attention are applied in parallel to preserve modal specificity at the low-level fine-grained level and progressively fuse the texture of the fundus color images and the deep structure of the OCT at the high-level level, achieving deep semantic fusion with long-range dependencies across layers. Step S300: Symmetric InfoNCE contrast constraints are applied simultaneously in the projection space of fundus color images, OCT, and fused features to explicitly promote geometric consistency between modalities. Step S400: Based on steps S100-S300, a multimodal ophthalmic image analysis model based on optimal transmission graph diffusion is constructed to analyze multimodal ophthalmic images.

[0010] In one possible design, in step S100, the two modalities first need to be projected into a unified semantic space, given a 2D fundus image and a 3D fundus OCT image, respectively represented as: ,in For batch size, , Given the number of tokens in both modalities, a symmetric encoder with shared parameters projects both into a unified latent space. : ; The encoder uses a standard linear transform-normalization-activation structure: ; Parameter sharing ensures that the two modalities are represented in the same semantic space, providing a consistent metric basis for subsequent optimal transmission computation. (Decoder Dec:) Symmetrically, this ensures approximate invertibility and facilitates returning to the original dimension. Optimal transport theory establishes precise cross-modal correspondences by solving for the "minimum transport cost." Its physical meaning is: how to transform one distribution into another with minimal cost. In a unified space, the Euclidean distance cost of all cross-modal feature pairs is first calculated. ; Each element of cost matrix C Let represent the geometric distance between the i-th fundus feature and the j-th OCT feature. To avoid instability caused by hard matching, an optimal transmission problem with entropy regularization is introduced. Assume that the edge distributions of the two modalities are uniform vectors. The equilibrium OT problem with entropy regularization ε>0 is: ; in For the optimal transfer matrix, It is the total transmission cost, and the entropy regularization term. To ensure the smoothness and numerical stability of the solution, the Sinkhorn iterative algorithm is used for efficient solution in the logarithmic field. (Definition...) Then the iterative formula is: ; Converged transfer matrix The soft correspondence between fundus features and OCT features is provided, and the corresponding Wasserstein distance is: This distance can be used as a measure of cross-modal alignment, based on The alignment is reconstructed as , representing the transmission alignment result of 2D fundus images to OCT and its reverse alignment, respectively, to obtain the optimal transmission matrix. Next, the co-evolution of features needs to be achieved on a structured graph. The core of this step is to construct a dynamic graph structure that simultaneously encodes semantic similarity and geometric correspondence, and to achieve continuous-time feature diffusion through neural ordinary differential equations; a graph is then built on the set of nodes in the modality concatenation: Let For each attention head : The semantic adjacency matrix is ​​obtained after softmax normalization: Simultaneously, introduce OT-induced geometric edges (bidirectional symmetry) and add self-loops: ;coefficient By balancing semantic and geometric information sources to ensure that there is no over-reliance on any one connection pattern, a row-normalized adjacency matrix is ​​obtained that combines semantic similarity and geometric alignment information. On the constructed graph structure, define the continuous-time feature evolution process; let the initial node features be... ,exist Define the constant differential equation of God above: The derivative function can be obtained by combining the current node state, graph structure, and time information. : ;in For the neighborhood aggregation information of node i, the MLP consists of two fully connected subnetworks: message passing and node update. For temporal embedding, for adaptive control of evolution duration, the termination time is predicted based on the complexity of the input features: Generate discrete time point sequences Iterate using the Dormand-Prince solver to... This allows for the smooth fusion of the two modal features within a continuous domain. Finally, through: The weighted aggregated features from the two evolution paths are then processed by a symmetric decoder. Mapping back to the original dimension, the final cross-modal fusion representation is output. The entire process utilizes the optimal transport explicit alignment distribution and uses graph neural ODE to achieve semantic collaborative diffusion, taking into account both geometric matching and deep fusion.

[0011] In one possible design, in step S100, for each batch Estimate an entropy-regularized optimal transfer matrix to provide fundus color image features. With OCT features The soft coupling between them, specifically, firstly, constructing the cost matrix Its elements are given by Euclidean distance. ,right Perform maximum value normalization To improve the numerical condition number and ensure the stability of GPU computation, the source and target marginal distributions are then initialized with equal weights, and the entropy regularization term is applied. Under the constraints, the solution is obtained through Sinkhorn–Knopp iteration: ;in, It is a double random coupled polyhedron. For Shannon entropy, this iteration is in It achieves convergence within a time complexity while maintaining gradient analytical differentiability, allowing the process to be seamlessly embedded into an end-to-end backpropagation chain; when all batches obtain their respective... Then, by superimposing them along the batch dimension, a four-dimensional tensor is obtained. The elements in this tensor It measures both geometric distance and implicitly contains uncertainty in cross-modal pairing, by... The elements are fused with the self-attention similarity matrix to generate an adjacency matrix that simultaneously embeds geometric consistency and contextual dependency.

[0012] In one possible design, in step S100, after obtaining the batch node representation... Next, it is first linearly mapped to the multi-head query-key space, resulting in... and Calculate the scaled dot product for each attention head. Then, the mean value is calculated along the first dimension to form the basic similarity matrix. Softmax operation will Normalization to a probability distribution provides an unbiased, numerically stable set of adjacency weights, and the preliminary stage provides the optimal transport tensor. Further construct sparse matrices Then press Integrating geometric priors and contextual similarity, then... Add an identity matrix to avoid isolated nodes, and arrange them according to row degree vectors. Right normalization yields the normalized adjacency matrix in random walk form. This allows the global statistical relationships captured by self-attention to be embedded in the graph topology along with the provided cross-modal first-order geometric consistency, thereby improving the discriminativeness of adjacency weights.

[0013] In one possible design, in step S100, the node state is set. Satisfying first-order ordinary differential equations Given the solver at time t and the adjacency matrix First, a learnable temporal embedder is used. After feeding the scalar t into a higher dimension, it is broadcast to each node, resulting in... Then, the neighbor features of the i-th node are aggregated row by row: and cascade with its own characteristics to form This tensor is processed by message MLP. After mapping, the data is aggregated into a global message tensor. ,Will With time embedding Element-wise addition, followed by concatenation with the original feature, yields: Where h is the current node feature M, which is the message obtained by aggregating from neighbors and passing through the MLP; T is the time-embedded broadcast, which is first added element-wise by M+T, then concatenated with h in the channel dimension to form Z, and then sent to the node-level MLP. Output derivative The neural ODE constructed in this way outputs within each adaptive step size It inherits the local dependency modeling capability of graph networks and endows the diffusion process with reversible and time-continuous properties, providing a stable evolution for subsequent solvers.

[0014] In one possible design, in step S200, the color image vector... With OCT vector Pull into a unified metric domain, and set the number of layers as . , No. Layer input is denoted as The two vectors are first processed through a linear embedding mapping. Upgrade to Dimension Then it enters the parallel intramodal self-attention and cross-attention pathways: ;in, and It is the representation after self-attention is performed on each of the two modalities; and Cross-attention allows OCT←fundus and fundus←OCT ​​to retrieve supplementary information from each other's representations. Self-attention maintains the discrimination patterns within each modality, while cross-attention exchanges context, enabling the color image path to "question" the deeper structure of the OCT, and vice versa. These two outputs, after residual connections and layer normalization, enter the parallel feedforward network. This ultimately generates the input for the next layer: Among them, the first Representation after cross-modal attention , The feedforward network of this layer is fed in place by place. (MLP) performs a nonlinear mapping, and the output is the two-mode representation of the next layer. , The progression between layers allows the network to retain modality-specific fine-grained features at lower levels and gradually inject cross-modal complementary information at higher levels. At the same time, the parallel splitting of multi-head attention enables the model to capture multi-scale correspondences in different subspaces, fusing from global contours to local lesions step by step. The final output of two vectors is concatenated at the channel level and subjected to lightweight mapping to obtain a fused representation. This vector retains both the exclusive criteria of the two modalities and cross-modal consistent semantics.

[0015] In one possible design, in step S300, the color photograph features... OCT features With fusion features After sharing two projection heads Later normalized to : In this projection space, intramodal InfoNCE is applied to each modality to bring similar samples closer together and push dissimilar samples further apart: Where m∈{f,o,z} represents the modality (fundus / OCT / fusion). For the set of positive sample pairs within this modality, For its quantity, (i,j) is one of the pairs. It is a projected representation of the sample. The similarity is calculated as (dot product / cosine), where τ is the temperature coefficient. The denominator is summed exponentially over all k (negative samples) of the same mode except itself. The overall similarity is averaged over all positive pairs across the three modes. Simultaneously, in the three sets of mode pairs... Apply cross-modal InfoNCE: ; where (m,n) enumerates the pairings of the three modes {f,o,z} (e.g., (f,o), (f,z), (o,z)). , For the projection representation of the corresponding mode, This represents a directed InfoNCE with modality m as the query and n as the key value. The sum of the two losses yields the total comparison target. And linearly weighted with the cross-entropy classification loss to form the final training criterion: .

[0016] This invention also provides a multimodal ophthalmic image analysis model based on optimal transport graph diffusion. Based on the aforementioned multimodal ophthalmic image analysis method based on optimal transport graph diffusion, it includes at least: an OT-GND module, which explicitly calculates the distance between fundus color images and OCT image modalities using an optimal transport algorithm, and combines graph neural networks and neural network frequent differential equation models to perform collaborative evolution of features in the continuous time domain; a hierarchical multi-head collaborative attention fusion module, which applies self-attention and cross-attention in parallel, preserving modal specificity at a low-level fine-grained level and progressively fusing color image texture and OCT deep structure at a high level, achieving deep semantic fusion with long-range dependencies across layers; and a contrastive learning module, which simultaneously applies symmetric InfoNCE contrast constraints in the projection space of fundus color images, OCT, and fused features, explicitly promoting geometric consistency between modalities.

[0017] The present invention also provides a system including a memory, a control processor, and a computer program stored in the memory and executable on the control processor, the control processor executing the program to implement the multimodal ophthalmic image analysis method based on optimal transmission map diffusion as described above.

[0018] The present invention also provides a computer-readable storage medium storing computer-executable instructions for implementing the multimodal ophthalmic image analysis method based on optimal transmission graph diffusion as described above.

[0019] Compared with existing technologies, the advantages of this invention are as follows: This invention proposes an innovative multimodal fusion framework based on Optimal Transport Graph Neural Diffusion Network (OT-GND). This framework deeply integrates the geometric insights of optimal transport theory with the structural modeling capabilities of graph neural networks, and realizes feature evolution in the continuous time domain through neural ordinary differential equations, thus opening up a completely new theoretical path for multimodal medical image alignment.

[0020] This invention transforms the cross-modal feature alignment problem into a process of finding the optimal mapping between probability distributions. By constructing a dynamic graph topology structure based on transmission cost, it adaptively reflects the semantic association strength and optimal correspondence between different modalities.

[0021] The OT-GND framework proposed in this invention uses a specially designed encoder to map heterogeneous modal features to a unified geometric manifold space. Then, an efficient algorithm is used to solve for the optimal transmission plan, and based on this, a dynamically evolving graph neural network structure is constructed. This invention introduces graph god frequent differential equations to model the diffusion and propagation process of features in continuous time and space. This framework achieves innovation from discrete feature matching to continuous semantic fusion, not only ensuring the smoothness and stability of feature evolution but also providing refined fusion control capabilities. To further enhance the model's representation learning ability, this invention designs a hierarchical collaborative attention mechanism to deeply mine complex dependencies within and between modalities, and introduces a multi-scale contrastive learning strategy to optimize the geometric structure of the feature space, ensuring the tight clustering of similar samples and the effective separation of dissimilar samples.

[0022] In summary, this invention establishes a mathematical framework for multimodal feature alignment based on optimal transmission at the theoretical level, realizes semantic evolution modeling in the continuous time domain at the methodological level, constructs a complete end-to-end learning system at the technical level, and verifies the significant advantages of the framework in large-scale ophthalmic disease diagnosis tasks at the application level. Through this systematic theoretical innovation and technological breakthrough, this invention not only advances the cutting-edge development of multimodal medical image analysis, but also provides an important scientific foundation and technical support for building a high-precision and robust intelligent ophthalmic diagnostic system. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A schematic diagram of the OHC-Net network structure in an embodiment of this application is shown.

[0025] Figure 2 A schematic diagram of the OT-GND network structure in an embodiment of this application is shown.

[0026] Figure 3 A schematic diagram of the HCAM network structure in an embodiment of this application is shown.

[0027] Figure 4 A schematic diagram of comparative learning in an embodiment of this application is shown.

[0028] Figure 5 A scatter plot illustrating the comparison of the method of this application embodiment with other prior art methods on the GAMMA dataset is shown.

[0029] Figure 6 This illustration shows a schematic diagram of adding Gaussian noise to Fundus on the GAMMA dataset in an embodiment of this application.

[0030] Figure 7 This illustration shows a schematic diagram of adding Gaussian noise to OCT on the GAMMA dataset in an embodiment of this application.

[0031] Figure 8 The diagram illustrates the changes in ACC and Kappa after adding Gaussian noise to Fundus on the GAMMA dataset in an embodiment of this application.

[0032] Figure 9 The diagram illustrates the changes in ACC and Kappa after adding Gaussian noise to OCT on the GAMMA dataset in an embodiment of this application.

[0033] Figure 10 This illustration shows a diagram of adding Gaussian noise to Fundus on the OLIVES dataset in an embodiment of this application.

[0034] Figure 11 This illustration shows a schematic diagram of adding Gaussian noise to OCT on the OLIVES dataset in an embodiment of this application.

[0035] Figure 12 The diagram illustrates the changes in ACC and Kappa after adding Gaussian noise to Fundus on the OLIVES dataset in an embodiment of this application.

[0036] Figure 13 The diagram illustrates the changes in ACC and Kappa after adding Gaussian noise to OCT on the OLIVES dataset in an embodiment of this application. Detailed Implementation

[0037] To further illustrate the various embodiments, the present invention provides accompanying drawings, which are part of the disclosure of the present invention. These drawings are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementation methods and the advantages of the present invention. The components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0038] This invention provides a multimodal ophthalmic image analysis method based on optimal transport graph diffusion, comprising the following steps: The basic scheme of this invention provides a multimodal ophthalmic image analysis method based on optimal transport graph diffusion, comprising at least the following steps: Step S100, explicitly calculating the distance between fundus color images and OCT image modalities using an optimal transport algorithm, and combining graph neural networks and neural network frequent differential equation models to perform co-evolution of features in the continuous time domain; Step S200, applying self-attention and cross-attention in parallel to preserve modal specificity at a low-level fine-grained level and progressively fuse color image texture and OCT depth structure at a high level to achieve deep semantic fusion with long-range dependencies across layers; Step S300, simultaneously applying symmetric InfoNCE contrast constraints in the projection space of fundus color images, OCT, and fused features to explicitly promote geometric consistency between modalities; Step S400, based on steps S100-S300, constructing a multimodal ophthalmic image analysis model based on optimal transport graph diffusion, and then analyzing multimodal ophthalmic images.

[0039] In step S100, the two modalities first need to be projected into a unified semantic space. Given a 2D fundus image and a 3D fundus OCT image, they are represented as follows: ,in For batch size, , Let be the number of tokens in both modalities. A symmetric encoder with shared parameters projects both into a unified latent space. : The encoder employs a standard linear transform-normalization-activation structure. Parameter sharing ensures that both modalities are represented in the same semantic space, providing a consistent metric basis for subsequent optimal transmission computation. The corresponding decoder, Decoder Dec: Symmetric to it (Linear–LN–GELU–Linear), it guarantees approximate invertibility, making it easy to return to the original dimension.

[0040] Optimal transport theory establishes precise cross-modal correspondences by solving for the "minimum transport cost," which physically means: how to transform one distribution into another with minimal cost. In a unified space, we first calculate the Euclidean distance cost for all cross-modal feature pairs: Each element of cost matrix C This represents the geometric distance between the i-th fundus feature and the j-th OCT feature.

[0041] To avoid instability caused by hard matching, we introduce an entropy-regularized optimal transport problem. Let the marginal distributions of the two modes be uniform vectors. The equilibrium OT problem with entropy regularization ε>0 is: ;in For the optimal transfer matrix, It is the total transmission cost, and the entropy regularization term. To ensure the smoothness and numerical stability of the solution, we employ the Sinkhorn iterative algorithm for efficient solution in the logarithmic field, defining... Then the iterative formula is: The converged transfer matrix The soft correspondence between fundus features and OCT features is provided, and the corresponding Wasserstein distance is: This distance can be used as a measure of cross-modal distribution alignment. Based on The alignment is reconstructed as , representing the transmission alignment result of the 2D fundus image to OCT and its reverse alignment, respectively.

[0042] Obtain the optimal transfer matrix Next, we need to achieve the co-evolution of features on a structured graph. The core of this step is to construct a dynamic graph structure that simultaneously encodes semantic similarity and geometric correspondence, and to achieve feature diffusion in the continuous time domain through neuromorphic differential equations.

[0043] Construct a graph on the node set of modal splicing: Let For each attention head : The semantic adjacency matrix is ​​obtained after softmax normalization: Simultaneously, introduce OT-induced geometric edges (bidirectional symmetry) and add self-loops: ;coefficient By balancing semantic and geometric information sources, we ensure that we do not overly rely on any one connection pattern. This yields a row-normalized adjacency matrix that incorporates both semantic similarity and geometric alignment information. .

[0044] On the constructed graph structure, we define a continuous-time feature evolution process. Let the initial node features be... ,exist Define the constant differential equation of God above: Derivative function Combining the current node state, graph structure, and time information: ;in For the neighborhood aggregation information of node i, the MLP consists of two fully connected subnetworks: message passing and node update. For temporal embedding. To adaptively control the evolution duration, we predict the termination time based on the complexity of the input features: Generate discrete time point sequences Iterate using the Dormand-Prince solver to... This allows for the smooth fusion of features from the two modalities within a continuous domain. Finally, through: The weighted aggregated features from the two evolution paths are then processed by a symmetric decoder. Mapping back to the original dimension, the final cross-modal fusion representation is output. The entire process utilizes the optimal transport explicit alignment distribution and uses graph neural ODE to achieve semantic collaborative diffusion, taking into account both geometric matching and deep fusion.

[0045] In step S100, for each batch Estimate an entropy-regularized optimal transfer matrix to provide fundus color image features. With OCT features The soft coupling between them, specifically, firstly, constructing the cost matrix Its elements are given by Euclidean distance. ,right Perform maximum value normalization To improve the numerical condition number and ensure the stability of GPU computation, the source and target marginal distributions are then initialized with equal weights, and the entropy regularization term is applied. Under the constraints, the solution is obtained through Sinkhorn–Knopp iteration: ;in, It is a double random coupled polyhedron. For Shannon entropy, this iteration is in It achieves convergence within a time complexity while maintaining gradient analytical differentiability, allowing the process to be seamlessly embedded into an end-to-end backpropagation chain; when all batches obtain their respective... Then, by superimposing them along the batch dimension, a four-dimensional tensor is obtained. The elements in this tensor It measures both geometric distance and implicitly contains uncertainty in cross-modal pairing, by... The elements are fused with the self-attention similarity matrix to generate an adjacency matrix that simultaneously embeds geometric consistency and contextual dependency.

[0046] In step S100, after obtaining the batch node representation... Next, it is first linearly mapped to the multi-head query-key space, resulting in... and Calculate the scaled dot product for each attention head. Then, the mean value is calculated along the first dimension to form the basic similarity matrix. Softmax operation will Normalization to a probability distribution provides an unbiased, numerically stable set of adjacency weights, and the preliminary stage provides the optimal transport tensor. Further construct sparse matrices Then press Integrating geometric priors and contextual similarity, then... Add an identity matrix to avoid isolated nodes, and arrange them according to row degree vectors. Right normalization yields the normalized adjacency matrix in random walk form. This allows the global statistical relationships captured by self-attention to be embedded in the graph topology along with the provided cross-modal first-order geometric consistency, thereby improving the discriminativeness of adjacency weights.

[0047] In step S100, the node state is set. Satisfying first-order ordinary differential equations Given the solver at time t and the adjacency matrix First, a learnable temporal embedder is used. After feeding the scalar t into a higher dimension, it is broadcast to each node, resulting in... Then, the neighbor features of the i-th node are aggregated row by row: and cascade with its own characteristics to form This tensor is processed by message MLP. After mapping, the data is aggregated into a global message tensor. ,Will With time embedding Element-wise addition, followed by concatenation with the original feature, yields: Where h is the current node feature M, which is the message obtained by aggregating from neighbors and passing through the MLP; T is the time-embedded broadcast, which is first added element-wise by M+T, then concatenated with h in the channel dimension to form Z, and then sent to the node-level MLP. Output derivative The neural ODE constructed in this way outputs within each adaptive step size It inherits the local dependency modeling capability of graph networks and endows the diffusion process with reversible and time-continuous properties, providing a stable evolution for subsequent solvers.

[0048] And sent to the node-level MLP Output derivative The neural ODE constructed in this way outputs within each adaptive step size It inherits the local dependency modeling capability of graph networks and endows the diffusion process with reversible and time-continuous properties, providing a stable evolution for subsequent solvers.

[0049] In step S200, the color image vector With OCT vector Pull into a unified metric domain, and set the number of layers as . , No. Layer input is denoted as The two vectors are first processed through a linear embedding mapping. Upgrade to Dimension Then it enters the parallel intramodal self-attention and cross-attention pathways: ;in, and It is the representation after self-attention is performed on each of the two modalities; and This is a representation obtained by using cross-attention to allow OCT←fundus and fundus←OCT ​​to retrieve supplementary information from each other. Self-attention maintains the discrimination pattern within each modality, while cross-attention exchanges context, allowing the color image path to "question" the deeper structure of OCT, and vice versa. After residual connections and layer normalization, these two outputs enter the parallel feedforward network. This ultimately generates the input for the next layer: Among them, the first Representation after cross-modal attention , The feedforward network of this layer is fed in place by place. (MLP) performs a nonlinear mapping, and the output is the two-mode representation of the next layer. , .

[0050] The progression between layers allows the network to retain modality-specific fine-grained features at lower levels and gradually inject cross-modal complementary information at higher levels. At the same time, the parallel splitting of multi-head attention enables the model to capture multi-scale correspondences in different subspaces, fusing from global contours to local lesions step by step. The final output vectors are concatenated at the channel level and subjected to lightweight mapping to obtain a fused representation. This vector retains both the exclusive criteria of the two modalities and cross-modal consistent semantics.

[0051] In step S300, color photo features OCT features With fusion features After sharing two projection heads Later normalized to : In this projection space, intramodal InfoNCE is applied to each modality to bring similar samples closer together and push dissimilar samples further apart: Where m∈{f,o,z} represents the modality (fundus / OCT / fusion). For the set of positive sample pairs within this modality, For its quantity, (i,j) is one of the pairs. It is a projected representation of the sample. Let τ be the similarity (dot product / cosine), and τ be the temperature coefficient. The denominator is calculated by summing the exponential similarity over all k (negative samples) of the same mode except itself. The overall similarity is then averaged over all positive pairs across the three modes. Simultaneously, in the three modal pairs... Apply cross-modal InfoNCE: ; where (m,n) enumerates the pairings of the three modes {f,o,z} (e.g., (f,o), (f,z), (o,z)). , For the projection representation of the corresponding mode, This represents a directed InfoNCE with modality m as the query and n as the key. The sum of the two losses yields the total comparison target. And linearly weighted with the cross-entropy classification loss to form the final training criterion: .

[0052] This invention also provides a multimodal ophthalmic image analysis model based on optimal transport graph diffusion. Based on the aforementioned multimodal ophthalmic image analysis method based on optimal transport graph diffusion, it includes at least: an OT-GND module, which explicitly calculates the distance between fundus color images and OCT image modalities using an optimal transport algorithm, and combines graph neural networks and neural network frequent differential equation models to perform collaborative evolution of features in the continuous time domain; a hierarchical multi-head collaborative attention fusion module, which applies self-attention and cross-attention in parallel, preserving modal specificity at a low-level fine-grained level and progressively fusing color image texture and OCT deep structure at a high level, achieving deep semantic fusion with long-range dependencies across layers; and a contrastive learning module, which simultaneously applies symmetric InfoNCE contrast constraints in the projection space of fundus color images, OCT, and fused features, explicitly promoting geometric consistency between modalities.

[0053] The present invention also provides a system including a memory, a control processor, and a computer program stored in the memory and executable on the control processor, the control processor executing the program to implement the multimodal ophthalmic image analysis method based on optimal transmission map diffusion as described above.

[0054] The present invention also provides a computer-readable storage medium storing computer-executable instructions for implementing the multimodal ophthalmic image analysis method based on optimal transmission graph diffusion as described above.

[0055] Specifically, this invention proposes a novel multimodal ophthalmic image analysis framework, such as... Figure 1 As shown, this invention aims to address a core challenge in current multimodal learning. In the joint analysis of fundus color images and OCT volumetric data, significant statistical distribution differences exist between different modalities. Traditional methods often rely on simple feature concatenation to achieve multimodal fusion; however, this concatenation method fails to effectively eliminate distributional shifts between modalities, resulting in incomplete or distorted information fusion. To solve this problem, this invention designs the Optimal-TransportGraphNeuralDiffusion (OT-GND) module.

[0056] OT-GND explicitly calculates the distance between modalities using an optimal transmission algorithm and combines graph neural networks and neural ordinary differential equation (ODE) models to co-evolve features in the continuous time domain. This method not only accurately aligns the distribution of fundus and OCT features but also achieves efficient integration of information from two modalities through the diffusion mechanism of graph neural networks, thus effectively solving the problem of distribution offset between modalities and ensuring the alignment of cross-modal features.

[0057] After feature alignment, a further challenge is how to preserve the unique information of each modality during multimodal feature fusion while fully exploiting their complementarity. To address this issue, this invention proposes a hierarchical multi-head co-attention module. This module, through a hierarchical multi-head attention mechanism, preserves the local information of each modality and progressively extracts and fuses complementary features of two modalities at multiple levels through a cross-modal attention mechanism. Improving the model's discriminative ability remains a significant issue, especially when the amount of data is small or the differences between modalities are large. To effectively address this challenge, this invention introduces intra-modal and cross-modal contrastive learning losses. By performing contrastive learning on fundus images, OCT images, and fused features, the model can explicitly promote geometric consistency between modalities.

[0058] In multimodal learning, especially in the joint analysis of fundus color images and OCT images, the distributional bias between modalities is a key factor affecting the fusion effect. Traditional fusion methods, especially simple stitching operations, often fail to effectively handle such cross-modal statistical differences, leading to incomplete information alignment and fusion distortion. To address this challenge, this invention proposes the Optimal-Transport Graph Neural Diffusion (OT-GND) module, which combines the methods of Optimal Transport (OT) and Graph Neural Networks (GNN), and performs time-continuous feature evolution within the framework of neural ordinary differential equations (ODE). The structure is as follows: Figure 2 As shown.

[0059] First, the two modalities need to be projected into a unified semantic space. Given a 2D fundus image and a 3D fundus OCT image, they are represented as follows: ,in For batch size, , Let be the number of tokens in both modalities. A symmetric encoder with shared parameters projects both into a unified latent space. : The encoder employs a standard linear transform-normalization-activation structure. Parameter sharing ensures that both modalities are represented in the same semantic space, providing a consistent metric basis for subsequent optimal transmission computation. The corresponding decoder, Decoder Dec: Symmetric to it (Linear–LN–GELU–Linear), it guarantees approximate invertibility, making it easy to return to the original dimension.

[0060] Optimal transport theory establishes precise cross-modal correspondences by solving for the "minimum transport cost," which physically means: how to transform one distribution into another with minimal cost. In a unified space, we first calculate the Euclidean distance cost for all cross-modal feature pairs: Each element of cost matrix C This represents the geometric distance between the i-th fundus feature and the j-th OCT feature.

[0061] To avoid instability caused by hard matching, we introduce an entropy-regularized optimal transport problem. Let the marginal distributions of the two modes be uniform vectors. The equilibrium OT problem with entropy regularization ε>0 is: ;in For the optimal transfer matrix, It is the total transmission cost, and the entropy regularization term. To ensure the smoothness and numerical stability of the solution, we employ the Sinkhorn iterative algorithm for efficient solution in the logarithmic field, defining... Then the iterative formula is: The converged transfer matrix The soft correspondence between fundus features and OCT features is provided, and the corresponding Wasserstein distance is: This distance can be used as a measure of cross-modal distribution alignment. Based on The alignment is reconstructed as , representing the transmission alignment result of the 2D fundus image to OCT and its reverse alignment, respectively.

[0062] Obtain the optimal transfer matrix Next, we need to achieve the co-evolution of features on a structured graph. The core of this step is to construct a dynamic graph structure that simultaneously encodes semantic similarity and geometric correspondence, and to achieve feature diffusion in the continuous time domain through neuromorphic differential equations.

[0063] Construct a graph on the node set of modal splicing: Let For each attention head : The semantic adjacency matrix is ​​obtained after softmax normalization: Simultaneously, introduce OT-induced geometric edges (bidirectional symmetry) and add self-loops: ;coefficient By balancing semantic and geometric information sources, we ensure that we do not overly rely on any one connection pattern. This yields a row-normalized adjacency matrix that incorporates both semantic similarity and geometric alignment information. .

[0064] On the constructed graph structure, we define a continuous-time feature evolution process. Let the initial node features be... ,exist Define the constant differential equation of God above: Derivative function Combining the current node state, graph structure, and time information: ;in For the neighborhood aggregation information of node i, the MLP consists of two fully connected subnetworks: message passing and node update. For temporal embedding. To adaptively control the evolution duration, we predict the termination time based on the complexity of the input features: Generate discrete time point sequences Iterate using the Dormand-Prince solver to... This allows for the smooth fusion of features from the two modalities within a continuous domain. Finally, through: The weighted aggregated features from the two evolution paths are then processed by a symmetric decoder. Mapping back to the original dimension, the final cross-modal fusion representation is output. The entire process utilizes the optimal transport explicit alignment distribution and uses graph neural ODE to achieve semantic collaborative diffusion, taking into account both geometric matching and deep fusion.

[0065] This method enables the OT-GND module to achieve precise numerical alignment of modal features and capture more complex dependencies between modalities through the nonlinear characteristics of graph neural networks. Furthermore, the continuity of the ODE ensures the smoothness and differentiability of the entire process. Compared to traditional concatenation or mapping methods, the OT-GND module significantly reduces information loss due to differences in modal distribution, thereby improving the effectiveness of subsequent deep fusion. This innovation achieves precise alignment of multimodal information, laying a more solid foundation for subsequent semantic fusion and task-oriented learning.

[0066] In the primary stage of cross-modal fusion, this invention addresses each batch Estimate an entropy-regularized optimal transfer matrix to provide fundus color image features. With OCT features The soft coupling between them. First, construct the cost matrix. Its elements are given by Euclidean distance. Because different batches may have significant differences in scale, this invention addresses... Perform maximum value normalization This is done to improve the numerical condition number and ensure the stability of GPU computation. Subsequently, the source and target marginal distributions are initialized with equal weights, and the entropy regularization term is then applied. Under the constraints, the solution is obtained through Sinkhorn–Knopp iteration: ;in, It is a double random coupled polyhedron. The entropy is Shannon's. This iteration occurs at... It achieves convergence within a time complexity while maintaining gradient analytical differentiability, allowing the process to be seamlessly embedded into an end-to-end backpropagation chain.

[0067] When all batches obtain their respective Subsequently, the present invention superimposes these along batch dimensions to obtain a four-dimensional tensor. The elements in this tensor Considered a "transportation probability," it measures both geometric distance and implicitly contains uncertainty regarding cross-modal pairings. The graph construction stage involves... The element-wise fusion with the self-attention similarity matrix generates an adjacency matrix that simultaneously embeds geometric consistency and contextual dependence, providing a continuous, smooth, and unbiased structural prior for subsequent graph neural diffusion.

[0068] Obtaining batch node representations Next, it is first linearly mapped to the multi-head query-key space, resulting in... and Calculate the scaled dot product for each attention head. Then, the mean value is calculated along the first dimension to form the basic similarity matrix. Softmax operation will Normalization to a probability distribution provides an unbiased, numerically stable set of adjacency weights.

[0069] The preliminary stage provides the optimal transport tensor. Further construct sparse matrices Then press Integrate geometric priors with contextual similarity. Then in Add an identity matrix to avoid isolated nodes. And arrange by row degree vector. Right normalization yields the normalized adjacency matrix in random walk form. .

[0070] This construction strategy embeds both the global statistical relationships captured by self-attention and the provided cross-modal first-order geometric consistency into the graph topology, improving the discriminative power of adjacency weights. The algorithm's overhead primarily comes from scaling the dot product and softmax, with a time complexity of O(log n). The space complexity is This provides a stable structural basis with cross-modal priors for subsequent graph neural diffusion.

[0071] In the graph diffusion phase, this invention elevates the discrete message passing process to the continuous time domain, allowing node states to... Satisfying first-order ordinary differential equations Given the solver at time t and the adjacency matrix... First, a learnable temporal embedder is used. After feeding the scalar t into a higher dimension, it is broadcast to each node, resulting in... Then, the neighbor features of the i-th node are aggregated row by row: and cascade with its own characteristics to form This tensor is processed by message MLP. After mapping, the data is aggregated into a global message tensor. .

[0072] In order to simultaneously consider node ontology, neighborhood information, and time modulation in continuous-time evolution, this invention will With time embedding Element-wise addition, followed by concatenation with the original feature, yields: And send it to the node-level MLP. Output derivative The computational cost of aggregation operations is... (Dense adjacency), the overhead of a two-level MLP is The overall structure remains differentiable, satisfying the solver continuity requirement. The neural ODE constructed thus outputs within each adaptive step. It inherits the local dependency modeling capability of graph networks and endows the diffusion process with reversible and time-continuous properties, providing a stable evolution for subsequent solvers.

[0073] After multimodal features are aligned in geometric space, a dilemma remains: preserving the inherent discriminative semantics of each modality while maximizing the extraction of complementary information among them. HCAM resolves this contradiction through an attention paradigm of "dual-stream preservation + progressive collaboration + multi-head decomposition." (Color image vector) With OCT vector They have been brought into a unified metric domain, but they still carry two complementary cues: texture and layer thickness, respectively. For example... Figure 3As shown, HCAM employs a hierarchical and progressive collaborative attention mechanism, extracting cross-modal complementary semantics layer by layer without diluting the discriminative information of each component.

[0074] Let the number of floors be , No. Layer input is denoted as The two vectors are first processed through a linear embedding mapping. Upgrade to Dimension Then it enters the parallel intramodal self-attention and cross-attention pathways: Self-attention maintains the discrimination patterns within each modality (fundus blood vessel orientation, OCT layer texture, etc.); cross-attention exchanges context, allowing the color image path to "interrogate" the depth structure of the OCT, and vice versa. These two outputs, after residual connections and layer normalization, enter a parallel feedforward network. This ultimately generates the input for the next layer: The progressive layering allows the network to retain modality-specific fine-grained features at lower levels and gradually inject cross-modal complementary information at higher levels. Simultaneously, the parallel splitting of multi-head attention enables the model to capture multi-scale correspondences across different subspaces, fusing from global contours to local lesions step-by-step. The final output vectors are concatenated at the channel level and lightweightly mapped to obtain a fused representation. This vector retains both modality-specific criteria and cross-modal consistent semantics, providing an information-dense and highly complementary feature foundation for downstream classification and contrastive learning tasks.

[0075] When the sample size is limited and noise and scale differences are still present between the two modalities, the inter-class boundaries tend to become blurred, and samples of the same class may be misaligned in geometric space. To enhance discriminative ability in this "few samples + strong domain difference" scenario, this invention introduces a contrastive learning objective that couples intra-modal and inter-modal learning, in addition to the classification loss. Specifically, color photo features... OCT features With fusion features After sharing two projection heads Later normalized to : This unified projection avoids inconsistencies in the dimensions of different modes and provides a good measurement basis for contrast loss.

[0076] like Figure 4 As shown, in this projection space, the present invention applies intramodal InfoNCE to each modality, bringing similar samples closer together and pushing dissimilar samples further apart: Meanwhile, to reduce residual domain differences and ensure geometric consistency, this invention addresses the issue in three sets of modal pairs. Apply cross-modal InfoNCE: The sum of the two losses yields the total comparison target. The final training criterion is formed by linearly weighting the cross-entropy classification loss: .

[0077] This two-layer contrast mechanism geometrically creates a tension field of "bringing closer similar samples, aligning modalities, and pushing away dissimilar samples": intramodal terms directly broaden the category boundaries, thus maintaining clear discrimination under small sample conditions; cross-modal terms force the orientation of similar samples in different modalities to align, suppressing representation collapse and reducing the risk of domain misalignment; the fusion vector participates in both class constraints simultaneously, thus avoiding being dominated by a single modality, and providing a stable and interpretable highly discriminative representation for the classification head and subsequent retrieval tasks.

[0078] Experiment and Results Analysis: 1. Dataset and Training Information (1) Experimental Dataset: This paper comprehensively evaluates the performance of the OT-GDNet model on two public datasets: GAMMA dataset and OLIVES dataset.

[0079] The different datasets for different diseases are detailed below: GAMMA dataset for glaucoma identification: To evaluate the effectiveness of the proposed method in glaucoma identification, its performance was assessed on the GAMMA dataset. This data includes 100 case pairs, each assigned a three-stage glaucoma classification. The original image sizes of the OCT and near-infrared fundus images are 256×512×992 and 1956×1934, respectively, where 256 is the total number of OCT slices. More details about the original dataset can be found in the GAMMA dataset. These cases were carefully split into training and test sets in an 8:2 ratio, and a rigorous five-fold cross-validation method was used to eliminate the influence of random factors on the evaluation results.

[0080] The OLIVES dataset for DR and DME screening: The effectiveness of the proposed algorithm in identifying DR and DME was subsequently validated using the OLIVES dataset. This dataset contains paired CFP and OCT images from 96 patients (56 with DME and 40 with DR), collected over multiple cycles, resulting in 3128 paired cases. The original image size was 768×768 for CFP and 48×504×496 for OCT images. To ensure the reliability of the results, a five-fold cross-validation method was also applied to this data.

[0081] (2) Training Information: The method proposed in this invention is implemented based on the PyTorch framework and trained on an NVIDIA GeForce RTX 3090 graphics card. This invention uses the Adam optimizer to train the overall parameters, with an initial learning rate of 5e-5 and a maximum epoch value of 100. For the GAMMA dataset, the sizes of all inputs are adjusted to 512×512 and 128×256×128 for CFP and OCT images. For the OLIVES dataset, all inputs for CFP and OCT images are uniformly adjusted to 512×512 and 48×248×248.

[0082] 2. Evaluation Metrics To comprehensively measure the model's discriminative ability and clinical consistency in ophthalmic multimodal grading tasks, this invention reports two metrics simultaneously: classification accuracy (ACC) and quadratic weighted kappa (QWK). ACC directly reflects the model's overall accuracy across all categories, while QWK further considers the order relationship between categories and the severity of prediction errors, thus more closely reflecting actual clinical assessment needs.

[0083] Accuracy (ACC) is the most intuitive overall performance metric for classification models, measuring the proportion of times the model makes completely correct predictions on all test samples. When the sample sizes of each class tend to be balanced, ACC can directly reflect the model's overall discriminative ability. In scenarios with imbalanced class distribution, although it may be dominated by high-frequency classes, it still has reference value as a basic metric. The calculation formula is as follows: ;in This represents the number of samples whose model predictions perfectly match the true labels. This represents the total number of samples in the test set.

[0084] Secondary weighted Kappa QWK is the official metric in Kaggle's diabetic retinopathy grading competition. It comprehensively evaluates model performance. Unlike simple consistency rate, QWK not only deducts for "accidental consistency," but also applies a secondary weighting to the error distance: the greater the grade difference, the heavier the penalty, thus better reflecting clinical sensitivity to severe misclassifications. The formula for calculating the secondary weighted Kappa is as follows: Where C represents the total number of classes, It is a quadratic weight matrix, with subscripts... `j` and `i` represent the row and column indices of the matrix, respectively. Weights Defined as . The range is from -1 to 1, where -1 and 1 represent completely inconsistent and completely consistent, respectively.

[0085] 3. Comparison with other advanced methods in the prior art (Table 1) Figure 5 The performance of nine methods on the GAMMA dataset is listed. It can be seen that the baseline only achieves 0.750 / 0.709 in ACC and QWK. With progressively stronger fusion strategies, COROLLA, TMC, and EyeMost improve ACC to 0.820, 0.800, and 0.850 respectively, but their QWK improvements are not entirely synchronized, remaining at a maximum of 0.765. MM-RAF improves QWK to 0.847, indicating that there is still room for improvement under the same classification error penalty. The publicly available method LMSF, with the strongest overall performance, stabilizes both metrics at 0.860 / 0.865, while the QWK of the single-modal upper bound O-fundus and O-OCT are 0.833 and 0.814 respectively, further confirming the important role of multimodal fusion for classification consistency. Compared to all comparable methods, the method of this invention achieved the highest ACC (0.860) tied with LMSF, and a new high of 0.893 in QWK, a further improvement of 0.028 over the previous best. Figure 5 It can also be seen that the method of the present invention has achieved the optimal method on both ACC and kappa, located in the top right corner.

[0086] Table 1. Comparison results of the method of the present invention with the latest method on the GAMMA dataset.

[0087] Methods ACC Kappa Baseline 0.750 0.709 COROLLA 0.820 0.765 TMC 0.800 0.660 EyeMost 0.850 0.754 MM-RAF - 0.847 Zhaoetal. <![CDATA[ 0.860 ]]> 0.865 O-fundus 0.780 0.833 O-OCT 0.800 0.814 Our <![CDATA[ 0.860 ]]> <![CDATA[ 0.893 ]]>

[0088] Although the overall accuracy is on par with LMSF, the model of this invention significantly outperforms in quadratic weighted Kappa. The key lies in the model's systemic approach of explicitly constraining the probability of large-scale rank errors, thereby reducing the risk of cross-rank misclassification. First, OT-GND uses entropy-regularized optimal transmission to accurately measure and align cross-modal distributions, making it more difficult for higher-order errors to cross critical ranks. Second, hierarchical multi-head collaborative attention, while preserving modality criteria through self-attention, utilizes multi-level cross-attention to refine complementary information, reducing the distance between samples of the same rank. Finally, intra-modal and cross-modal symmetric InfoNCE loss constructs a global force field in the projection space that "brings similar classes closer and pushes away dissimilar classes," compressing class edges at the gradient level and simultaneously correcting domain drift. These three mechanisms work synergistically to significantly reduce the probability of "mild misclassification as severe" or reverse misclassification while maintaining overall accuracy, thus achieving a higher consistency score on the QWK metric, which penalizes rank errors twice.

[0089] Table 2 lists the results of nine methods on the OLIVES dataset. The baseline method only achieved 0.949ACC / 0.957QWK; Huaetal. improved the metrics to 0.986 / 0.968 by refining feature extraction and fusion strategies, but still suffered from considerable misclassification. Zoueetal.'s results dropped to 0.932 / 0.838 under the same settings, indicating insufficient adaptability of their method to this data domain. The single-modal upper limits for color images and OCT, O-fundus and O-OCT, reached 0.965 / 0.971 and 0.988 / 0.985 respectively, showing that depth morphology information is particularly crucial for classification. Wooeetal., Kendall & Gal, Lakshminara et al., and EyeMost all reported 1.000ACC / 1.000QWK, representing the current publicly available limits on the OLIVES dataset. The model of this invention also achieves full marks on both metrics, matching the best results; compared to Huaetal.'s latest improved scheme, the accuracy is further improved by 0.014, the double-weighted Kappa is further improved by 0.032, and compared to the upper bound of the single mode, it is improved by 0.035–0.012 (ACC) and 0.029–0.015 (QWK), respectively, further verifying the advantages of multimodal coupling in terms of level consistency.

[0090] Table 2 shows the comparison results of the method of the present invention with the latest method on the OLIVES dataset.

[0091] Methods ACC Kappa Baseline 0.949 0.957 Huaetal 0.986 0.968 Wooetal 1.000 1.000 KendallandGal 1.000 1.000 Lakshminarayananetal 1.000 1.000 Zouetal 0.932 0.838 EyeMost 1.000 1.000 O-fundus 0.965 0.971 O-OCT 0.988 0.985 Our <![CDATA[ 1.000 ]]> <![CDATA[ 1.000 ]]>

[0092] The key to this method's ability to achieve a performance comparable to the aforementioned best methods lies in the closed-loop optimization formed by OT-GND, HCAM, and contrastive learning constraints: OT-GND precisely aligns the geometric distributions of the two domains, significantly suppressing the probability of misclassification across different classes; HCAM progressively fuses color image textures with OCT depth structures, ensuring that complementary semantics are fully manifested; and InfoNCE solidifies class boundaries in the global angular domain, further enhancing discrimination robustness. The synergy of these three elements enables the model to achieve a perfect score of 1.000ACC / 1.000QWK without requiring a surge in additional parameters or strong priors, while maintaining good interpretability and cross-dataset transfer potential.

[0093] 4. Robustness verification Table 3 shows the Gaussian noise added to Fundus on the GAMMA dataset.

[0094] Table 4 shows the Gaussian noise addition applied to OCT on the GAMMA dataset.

[0095] As shown in Tables 3 and 4, and Figure 6-9 As shown, on the GAMMA dataset, this invention injects zero-mean Gaussian noise (σ∈{0.00,0.02,…,0.10}) into the color photo branch (Fundus,F) and the OCT branch (O), respectively, and records the ACC and double-weighted Kappa. Without noise, the performance of the two paths is completely identical (0.860 / 0.893). Once light noise (σ=0.02) is added, the Kappa drops by 7.9pp and 4.2pp on the F-side and O-side, respectively; when the noise increases to σ=0.04, both curves show a "slight rebound"—the F-side rebounds to 0.800 / 0.835, and the O-side rebounds to 0.820 / 0.861, then monotonically decays with increasing noise intensity, dropping to 0.750 / 0.814 (F) and 0.760 / 0.824 (O) at σ=0.10. Overall trend analysis: Mild random perturbations initially degrade high-frequency textures, followed by minor recovery through regularization at moderate intensity (σ=0.04). Further increases irreversibly damage discriminative features. Throughout the comparison, the O-side consistently outperforms the F-side, and in the mid-noise range (σ=0.04–0.06), the Kappa only slightly decreases by 0.003, demonstrating stronger robustness to level consistency.

[0096] The root of the difference lies in the dual role of the modal's inherent signal-to-noise structure and the model coupling mechanism. After the OCT features are densely connected and voxel pooled by 3DDenseNet, the noise is averaged out during hierarchical accumulation, exhibiting a natural filtering effect; the ViT-Fundus stream, on the other hand, is highly sensitive to high-frequency textures, and local perturbations are easily amplified by self-attention weights. Furthermore, OT-GND assigns larger edge weights to the OCT morphological features, enabling them to play a "supporting" role in the overall representation during the graph diffusion stage; the cross-modal symmetric InfoNCE also shortens the distance between similar samples, forming a stable pull from OCT to Fundus. Therefore, under light to moderate noise, the color photo branch can be corrected from OCT through collaborative attention and contrast constraints, but when σ≥0.08, local high-frequency distortion exceeds the adaptive range of cross-attention, Fundus is rapidly damaged, and OCT also decays synchronously due to the accumulation of noise from neighborhood aggregation. Experimental results verify the overall robustness of the OT-GND+HCAM architecture in resisting input noise and also suggest that future work can specifically optimize high-frequency filtering and noise perception for the color photo stream.

[0097] Table 5 shows the Gaussian noise added to Fundus on the OLIVES dataset.

[0098] Table 6 shows the Gaussian noise addition applied to OCT on the OLIVES dataset.

[0099] As shown in Tables 5 and 6, Figure 10-13 As shown, on the OLIVES dataset, this invention progressively increases the zero-mean Gaussian noise variance σ from 0.0 to 0.5 to observe the extreme disturbance rejection limits of the two branches. The color imaging path (F) remains almost unaffected when σ ≤ 0.1, with both ACC and QWK maintaining 1.000; even with the noise increasing to 0.5, its performance remains at 0.984 / 0.981, only declining by 1.6 and 1.9 percentage points from the baseline. The OCT path (O), however, exhibits a steep decay curve: when the noise variance increases from 0.0 to 0.1, it drops to 0.962 / 0.925; further increases to 0.5 cause ACC and QWK to decrease to 0.788 and 0.605 respectively, a cumulative drop of over 20 percentage points. The divergence between the two curves illustrates that in high-resolution, stable lighting OLIVES scenes, the color branch has extremely strong tolerance to Gaussian noise, while the OCT branch is more vulnerable to large pixel disturbances, and its level consistency is severely compromised.

[0100] This contrast stems from the interaction between the inherent signal-to-noise characteristics of the modality and the internal coupling strategy of the model. After equalization of illumination and color correction, OLIVES color images have richer and more stable high-frequency textures and color gradients than GAMMA; ViT's global self-attention is easy to focus on large structures (optic disc and vascular network) in this scene, thus it has natural immunity to pixel-level noise. In contrast, the inter-layer grayscale contrast of OLIVESOCT slices is inherently weak. When α ≥ 0.2, the noise variance is already on the same order of magnitude as the real layer thickness texture, and 3DDenseNet is difficult to completely average during voxel convergence; at the same time, the edge weight calculation of OT-GND depends on Euclidean distance, and noise causes the cost matrix between OCT nodes to expand, making the graph diffusion stage more inclined to color image flow and weakening backpropagation, ultimately forming a performance cliff at the OCT end. Although HCAM can partially correct OCT through cross-attention, when the noise exceeds 0.3, the similarity distribution of cross-query / key is already distorted, and the marginal benefit of cross-modal correction decreases rapidly. Experimental results show that on high-quality datasets, the color stream becomes a noise barrier, while the bottleneck of model robustness shifts to the OCT stream, suggesting that future work could explore adaptive voxel denoising or waveform reprojection to further improve the disturbance resistance limit of 3D paths.

[0101] 5. Ablation test Table 7 Ablation experiments on the GAMMA dataset

[0102] The ablation results in Table 7 exhibit a "three-stage progressive" pattern: independently activating any one of the OT-GND, HCAM, or CL modules can improve accuracy from 0.750 to 0.760–0.770, but the increase in weighted Kappa is much greater than that in ACC—OT-GND and HCAM alone bring a QWK improvement of ≥0.11 (0.817 / 0.823), while CL, although contributing little to ACC, raises QWK from 0.709 to 0.788, indicating that it mainly reshapes class geometry rather than simple hit rate. Further pairwise combinations show a significant synergistic effect between modules: OT-GND+HCAM increases ACC by 0.02 compared to individual modules, reaching QWK of 0.832; when CL is paired with OT-GND or HCAM, QWK rises to 0.837 and 0.861 respectively, indicating that contrastive constraints can "amplify" the feature differences created by geometric alignment or semantic interaction.

[0103] OT-GND explicitly shrinks cross-modal distribution differences through graph diffusion, first reducing large "cross-level" errors, thus bringing a leap in QWK. HCAM captures complementary semantics with layered progressive cross-attention in the two-stream residual framework, further shortening the distance between samples of the same level and improving the balance between ACC and QWK. CL has the effect of "pull similarities and repel dissimilarities", geometrically compressing the class edges, with particularly outstanding effects on Kappa. When OT-GND and HCAM are used simultaneously, the model achieves both geometric alignment and semantic complementarity, forming a stable increase. Once CL is added, the contrastive loss pushes color photos, OCT, and fused features of the same class in the same direction, while dispersing features of different classes in directions away from each other, significantly suppressing the remaining misclassification gradient, thus improving QWK to 0.893.

[0104] Ultimately, the three modules, operating in parallel, achieved peak scores of 0.860 and 0.893, representing an ACC improvement of 11.0 percentage points and a QWK surge of 18.4 percentage points compared to the baseline without any components. This result not only validates the complementary roles of each submodule—OT-GND for alignment, HCAM for fusion, and CL for edge compression—but also demonstrates that any missing component will lead to a broken information chain or insufficient penalty term, making it difficult to maintain a perfect close-range performance under strict level penalties.

[0105] Table 8 shows the temperature parameters learned from the comparative learning on the GAMMA dataset.

[0106] In contrastive learning, the temperature φ determines the scaling of the logits of the similarity between positive and negative samples, thus directly controlling the "sharpness" of the gradient. When φ is too small (0.02–0.05), the scores of positive samples are extremely amplified by the exponential function, and the network focuses more on a small number of easily distinguishable sample pairs while ignoring "difficult pairs." As a result, the AUC is acceptable (0.901–0.903), but ACC and Kappa are limited to moderate levels of 0.810–0.820 / 0.843–0.856, indicating that although the local geometry is tightened, it fails to globally correct cross-modal misclassification. As φ increases to 0.07, the gradient distribution reaches equilibrium: positive samples still have sufficient attraction, and negative samples can also provide effective repulsion. ACC and Kappa simultaneously reach their optimal values ​​(0.860 / 0.893), and the AUC also increases to 0.911, indicating that the model achieves the optimal trade-off between discriminability and geometric consistency.

[0107] Increasing the temperature further to 0.10 and 0.20 significantly diminishes the difference in positive and negative contrast scores, leading to overly smoothed gradients. At this point, the representation clusters in the projection space begin to loosen: ACC drops to 0.790–0.810, Kappa declines to 0.836–0.864, and AUC continuously decreases to 0.890. Higher temperatures weaken the "pulling" effect on similar samples, blurring class boundaries again and preventing sufficient geometric compression of cross-modal alignment; simultaneously, the negative sample repulsion is insufficient, leading to aliasing. Overall, α=0.07 is precisely at the critical point between "overly sharp gradients leading to undergeneralization" and "overly blunt gradients leading to discriminative dilution," adjusting the effective range of the InfoNCE loss to ensure that contrastive learning can clearly differentiate between different classes without making the feature distribution too scattered, thus achieving global optimum across all three core metrics.

[0108] Table 9 shows the ablation experiments on the Olives dataset.

[0109] On the OLIVES dataset, the base model achieved 0.949ACC / 0.957QWK, but the three functional modules still showed significant and complementary individual improvements. Enabling OT-GND improved the metrics to 0.962 / 0.973, indicating that the cross-modal geometric alignment brought by optimal transmission can effectively reduce large-scale misclassification. Enabling HCAM yielded 0.955 / 0.965, demonstrating that hierarchical collaborative attention can uncover fine-grained complementary semantics between color images and OCT, thereby fine-tuning the discrimination boundary. Contrastive learning (CL) alone boosted the metrics to 0.968 / 0.977, highlighting the "sameness-repulsion" distance compression constructed by InfoNCE, which is particularly beneficial for improving the second-weighted Kappa. These three sets of results show that geometric alignment, semantic interaction, and geometric compression each provide different dimensions of discriminative information under single-path conditions.

[0110] The combination of two modules brings near-saturation performance: OT-GND+HCAM reaches 0.992 / 0.985, showing that progressive semantic fusion based on geometric alignment can further stabilize class boundaries; OT-GND+CL improves ACC to 0.995, but due to the lack of semantic progression, QWK is slightly lower than HCAM+CL's 0.988.

[0111] When the three modules operate in parallel, they form a complete closed loop, achieving a model score of 1.000 ACC / 1.000 QWK. OT-GND provides a cross-modal geometric scaffold, HCAM overlays hierarchical semantic alignment on the scaffold, and CL shortens the distance between similar categories and widens the distance between different categories. The synergy of these three components simultaneously optimizes distribution consistency, semantic complementarity, and geometric discriminability. If any one of these components is missing, the closed loop is disrupted, and performance exhibits a step-like decline, validating the necessity and sufficiency of this triple mechanism for achieving extreme hierarchical consistency.

[0112] This invention proposes a unified framework based on fundus color imaging-OCT multimodal hierarchical modeling: Optimal-Transport Graph Neural Diffusion (OT-GND) – Hierarchical Co-Attention Module (HCAM) – contrastive learning loop. Unlike existing methods that only focus on a single point in the geometric, semantic, or discriminative dimensions, this invention theoretically proves first that optimal transport can map cross-modal distributions to a shared space and couple with graph neural networks (ODEs) to form a differentiable and reversible cross-domain alignment operator. Subsequently, it achieves complementary semantic distillation of "intramodal fidelity – cross-progression" through hierarchical multi-head collaborative attention. Finally, it uses intramodal and cross-modal symmetric InfoNCEs to solidify category edges and align geometric directions, thereby simultaneously optimizing distribution consistency, semantic complementarity, and discriminative geometry across the three links.

[0113] System experiments fully validated the effectiveness and robustness of the framework: On the GAMMA dataset, the model of this invention achieved an overall accuracy on par with the best publicly available method LMSF (0.860), but the second-weighted Kappa improved to 0.893, a further increase of 0.028 over the previous record; on the OLIVES dataset with higher image resolution, it achieved a perfect score of 1.000ACC / 1.000QWK, matching the best results without relying on parameter spikes or manual priors. Noise robustness experiments showed that the color image path maintained a score of 0.984 / 0.981 at Gaussian variance R=0.5, while the OCT path, although more susceptible to perturbations, showed significant reduction in degradation under the framework's guidance, indicating that the geometric skeleton provided by OT-GND and the multi-level correction brought by HCAM-CL have a natural inhibitory effect on input distortion. Ablation studies further reveal that all three modules are indispensable; enabling any one component alone can bring an ACC improvement of 1.0–2.4pp and a larger Kappa improvement, with the combination of the two showing a superlinear gain. The closed-loop three-module model boosts the Kappa of the GAMMA dataset by 18.4pp, while achieving a perfect score on the OLIVES dataset. Temperature sensitivity analysis shows that φ=0.07 is the optimal balance point for contrast regularization, balancing the sharpness of the "like-dislike" gradient with the stability of the representation.

[0114] In summary, the closed-loop mechanism formed by coupling OT-GND, HCAM, and contrastive learning provides an end-to-end, interpretable, and robust solution for multimodal medical image grading. This framework not only refreshes or matches the current state-of-the-art performance on standard datasets but also demonstrates excellent stability under extreme noise and cross-modal mismatch scenarios, laying an algorithmic foundation for practical clinical deployment. Future work will expand along two paths: first, combining graph diffusion with adaptive voxel denoising to introduce learnable noise-resistant priors to further improve the extreme fault tolerance of the OCT pathway; second, extending layer-by-layer collaborative attention to temporal domain or multi-lesion target detection tasks, exploring its generalization ability in multi-timepoint and multi-pathological annotation scenarios.

[0115] The English terms used in the embodiments and accompanying drawings of this invention are translated as follows: Figure 1In Chinese: 2D Fundus; 3D OCT; Space Mapping; Multimodal Feature Graph; Optimal Transport Plan; Time Step; Adjacency Matrix; Graph Neural ODE Diffusion; Hierarchical Co-Attention Module; classifier; 2D Encoder; 3D Encoder; Contrastive Learning; 2D modal feature vector flow; 3D modal feature vector flow; Fusion feature vector flow; Conduct contrastive learning flow.

[0116] Figure 2In Chinese: Temporal Step Network (TPM), Message MLP (Message Message Passing Network), Time Embedding (Time Embedding), Node MLP (Node Fusion), Constructing a multimodal feature graph (Constructing a Multimodal Feature Graph), Constructing an Adjacency Matrix (Constructing an Adjacency Matrix), Computing the Optimal Transport Plan (Computing the Optimal Transport Plan), Timestep Forecasting (Timestep Forecasting), Graph Neural ODEs (Graph Neural ODEs), Optimal Transportation Matrix (Optimal Transportation Matrix), Attention Matrix (Attention Matrix), Adjacency Matrix (Adjacency Matrix), Time Step (Time Step), Finalmoment Feature (Final Moment Feature), Concatenate (Concatenate), Add (Add), Sinkhorn Algorithm (Sinkhorn Algorithm), Time Series (Time Series), The Rate of Change of Node Features overtime (The Rate of Change of Node Features over Time), 2D Modal Feature Vector (Two-Dimensional Modal Feature Vector), 3D Modal Feature Vector (Three-Dimensional Modal Feature Vector), Fusion Feature Vector (Fusion Modal Feature Vector).

[0117] Figure 3 In Chinese: Self-attentiom: Self-attention; Cross-attentiom: Cross-attention; Feedforward Network: Feedforward Network; Softmax: Activation function; 2D modal feature: 2D modal feature; 3D modal feature: 3D modal feature; 2D modal feature vector: 2D modal feature vector; Fusion feature vector: Fusion modal feature vector; 3D modal feature vector: 3D modal feature vector; Concatenate: Concatenate.

[0118] Figure 4In Chinese: Reduce the distance between positive samples: Shrink the distance between positive samples. Increase the distance between negative samples: Expand the distance between negative samples. Positive sample: Positive sample. Negative sample: Negative sample.

[0119] Figure 5 In Chinese: Performance Comparison Accuracy vs Cohen's Kappa: Performance Comparison: Accuracy vs. Double-Weighted Kappa Cohen's Kappa: Double-Weighted Kappa Accuracy: Accuracy.

[0120] Figure 6 , Figure 7 , Figure 10 , Figure 11 Chinese: Gaussian Noise; Original Image.

[0121] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0122] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this hardware-software interchangeability, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized form in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this invention. The various illustrative logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein can be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in alternatives, the processor may be any conventional processor, battery compartment control board, micro battery compartment control board, or state machine. The processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration. The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by the processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read and write information from / to the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside as discrete components in the user terminal. In one or more exemplary embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functionality may be stored as one or more instructions or code on or transmitted via a computer-readable medium. Computer-readable media include both computer storage media and communication media, encompassing any medium that facilitates the transfer of computer programs from one location to another. Storage media can be any available medium that can be accessed by a computer.By way of example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Any connection is also properly referred to as computer-readable media. For example, if software is transmitted from a website, central computer, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then such coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. As used herein, disk and disc include compact discs (CDs), laser discs, optical discs, digital multi-purpose discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, and discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0123] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any indirect modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A multimodal ophthalmic image analysis method based on optimal transmission map diffusion, characterized in that, The method includes at least the following steps: Step S100, explicitly calculating the distance between fundus color images and OCT image modalities using an optimal transmission algorithm, and combining graph neural networks and neural network frequent differential equation models to perform co-evolution of features in the continuous time domain; Step S200, applying self-attention and cross-attention in parallel to preserve modal specificity at a low-level fine-grained level and progressively fuse color image textures and OCT deep structures at a high level to achieve deep semantic fusion with long-range dependencies across layers; Step S300, simultaneously applying symmetric InfoNCE contrast constraints in the projection space of fundus color images, OCT, and fused features to explicitly promote geometric consistency between modalities; and Step S400, building a multimodal ophthalmic image analysis model based on optimal transmission graph diffusion based on steps S100-S300, and then analyzing multimodal ophthalmic images.

2. The multimodal ophthalmic image analysis method based on optimal transmission map diffusion according to claim 1, characterized in that, In step S100, the two modalities first need to be projected into a unified semantic space. Given a 2D fundus image and a 3D fundus OCT, they are represented as follows: ,in For batch size, , Given the number of tokens in both modalities, a symmetric encoder with shared parameters projects both into a unified latent space. : ; The encoder uses a standard linear transform-normalization-activation structure: ; Parameter sharing ensures that the two modalities are represented in the same semantic space, providing a consistent metric basis for subsequent optimal transmission calculations. The corresponding decoder, Dec: Symmetric to it (Linear–LN–GELU–Linear), ensuring approximate invertibility and facilitating return to the original dimension; optimal transport theory establishes accurate cross-modal correspondences by solving for the "minimum transport cost," its physical meaning being: how to transform one distribution into another with minimal cost. In a unified space, the Euclidean distance cost of all cross-modal feature pairs is first calculated: ; Each element of cost matrix C Let represent the geometric distance between the i-th fundus feature and the j-th OCT feature. To avoid instability caused by hard matching, an optimal transmission problem with entropy regularization is introduced. Assume that the edge distributions of the two modalities are uniform vectors. The equilibrium OT problem with entropy regularization ε>0 is: ； in For the optimal transfer matrix, It is the total transmission cost, and the entropy regularization term. To ensure the smoothness and numerical stability of the solution, the Sinkhorn iterative algorithm is used for efficient solution in the logarithmic field. (Definition...) Then the iterative formula is: ； Converged transfer matrix The soft correspondence between fundus features and OCT features is provided, and the corresponding Wasserstein distance is: ； This distance can be used as a measure of cross-modal alignment, based on The alignment is reconstructed as , representing the transmission alignment result of 2D fundus images to OCT and its reverse alignment, respectively, to obtain the optimal transmission matrix. Next, the co-evolution of features needs to be achieved on a structured graph. The core of this step is to construct a dynamic graph structure that simultaneously encodes semantic similarity and geometric correspondence, and to achieve continuous-time feature diffusion through neural ordinary differential equations; a graph is then built on the set of nodes in the modality concatenation: Let For each attention head : ； The semantic adjacency matrix is ​​obtained after softmax normalization: ； Simultaneously, introduce OT-induced geometric edges (bidirectional symmetry) and add self-loops: ； coefficient By balancing semantic and geometric information sources to ensure that there is no over-reliance on any one connection pattern, a row-normalized adjacency matrix is ​​obtained that combines semantic similarity and geometric alignment information. On the constructed graph structure, a continuous-time feature evolution process is defined, assuming the initial node features are... ,exist Define the constant differential equation of God above: ； The derivative function can be obtained by combining the current node state, graph structure, and time information. : ； in For the neighborhood aggregation information of node i, the MLP consists of two fully connected subnetworks: message passing and node update. For temporal embedding, for adaptive control of evolution duration, the termination time is predicted based on the complexity of the input features: ； Generate discrete time point sequences Iterate using the Dormand-Prince solver to... This allows for the smooth fusion of the two modal features within a continuous domain. Finally, through: ； The weighted aggregated features from the two evolution paths are then processed by a symmetric decoder. Mapping back to the original dimension, the final cross-modal fusion representation is output. The entire process utilizes the optimal transport explicit alignment distribution and uses graph neural ODE to achieve semantic collaborative diffusion, taking into account both geometric matching and deep fusion.

3. The multimodal ophthalmic image analysis method based on optimal transmission map diffusion according to claim 2, characterized in that, In step S100, for each batch Estimate an entropy-regularized optimal transfer matrix to provide fundus color image features. With OCT features The soft coupling between them, specifically, firstly, constructing the cost matrix Its elements are given by Euclidean distance. ,right Perform maximum value normalization To improve the numerical condition number and ensure the stability of GPU computation, the source and target marginal distributions are then initialized with equal weights, and the entropy regularization term is applied. Under the constraints, the solution is obtained through Sinkhorn–Knopp iteration: ； in, It is a double random coupled polyhedron. For Shannon entropy, this iteration is in It achieves convergence within a time complexity while maintaining gradient analytical differentiability, allowing the process to be seamlessly embedded into an end-to-end backpropagation chain; when all batches obtain their respective... Then, by superimposing them along the batch dimension, a four-dimensional tensor is obtained. The elements in this tensor It measures both geometric distance and implicitly contains uncertainty in cross-modal pairing, by... Element-wise fusion with the self-attention similarity matrix generates an adjacency matrix that simultaneously embeds geometric consistency and contextual dependency.

4. The multimodal ophthalmic image analysis method based on optimal transmission map diffusion according to claim 3, characterized in that, In step S100, after obtaining the batch node representation... Next, it is first linearly mapped to the multi-head query-key space, resulting in... and Calculate the scaled dot product for each attention head. Then, the mean value is calculated along the first dimension to form the basic similarity matrix. Softmax operation will Normalization to a probability distribution provides an unbiased, numerically stable set of adjacency weights. The preliminary stage provides the optimal transport tensor. Further construct sparse matrices Then press Integrating geometric priors and contextual similarity, then... Add an identity matrix to avoid isolated nodes, and arrange them according to row degree vectors. Right normalization yields the normalized adjacency matrix in random walk form. This allows the global statistical relationships captured by self-attention to be embedded in the graph topology along with the provided cross-modal first-order geometric consistency, thereby improving the discriminativeness of adjacency weights.

5. The multimodal ophthalmic image analysis method based on optimal transmission map diffusion according to claim 4, characterized in that, In step S100, set the node state. Satisfying first-order ordinary differential equations Given the solver at time t and the adjacency matrix First, a learnable temporal embedder is used. After feeding the scalar t into a higher dimension, broadcast it to each node to obtain... Then, the neighbor features of the i-th node are aggregated row by row: and cascade with its own characteristics to form This tensor is processed by message MLP. After mapping, the data is aggregated into a global message tensor. , Will With time embedding Element-wise addition, followed by concatenation with the original feature, yields: ； Where h is the current node feature M, which is the message obtained by aggregating from neighbors and passing through the MLP, and T is the broadcast with time embedding. First, M+T is added element-wise, then it is concatenated with h in the channel dimension to form Z, and then sent to the node-level MLP. Output derivative The neural ODE constructed in this way outputs within each adaptive step size It inherits the local dependency modeling capability of graph networks and endows the diffusion process with reversible and time-continuous properties, providing a stable evolution for subsequent solvers.

6. A multimodal ophthalmic image analysis method based on optimal transmission map diffusion according to any one of claims 2-5, characterized in that, In step S200, the color image vector With OCT vector Pull into a unified metric domain, and set the number of layers as . , No. Layer input is denoted as The two vectors are first processed through a linear embedding mapping. Upgrade to Dimension Then it enters the parallel intramodal self-attention and cross-attention pathways: ； in, and It is the representation after self-attention is performed on each of the two modalities; and Cross-attention allows OCT←fundus and fundus←OCT ​​to retrieve supplementary information from each other's representations. Self-attention maintains the discrimination patterns within each modality, while cross-attention exchanges context, enabling the color image path to "question" the OCT's in-depth structure, and vice versa. These two outputs, after residual connections and layer normalization, enter a parallel feedforward network. This ultimately generates the input for the next layer: ； Among them, the first Representation after cross-modal attention , The feedforward network of this layer is fed in place by place. (MLP) performs a nonlinear mapping, and the output is the two-mode representation of the next layer. , The progression between layers allows the network to retain modality-specific fine-grained features at lower levels and gradually inject cross-modal complementary information at higher levels. At the same time, the parallel splitting of multi-head attention enables the model to capture multi-scale correspondences in different subspaces, fusing from global contours to local lesions step by step. The final output of two vectors is concatenated at the channel level and subjected to lightweight mapping to obtain a fused representation. This vector retains both the exclusive criteria of the two modalities and cross-modal consistent semantics.

7. The multimodal ophthalmic image analysis method based on optimal transmission map diffusion according to claim 6, characterized in that, In step S300, color photo features OCT features With fusion features After sharing two projection heads Later normalized to : In this projection space, intramodal InfoNCE is applied to each modality to bring similar samples closer together and push dissimilar samples further apart: ； Where m∈{f,o,z} represents the modality (fundus / OCT / fusion). For the set of positive sample pairs within this modality, For its quantity, (i,j) is one of the pairs. It is a projected representation of the sample. The similarity is calculated as (dot product / cosine), where τ is the temperature coefficient. The denominator is summed exponentially over all k (negative samples) of the same mode except itself. The overall similarity is averaged over all positive pairs across the three modes. Simultaneously, in the three sets of mode pairs... Apply cross-modal InfoNCE: ; where (m,n) enumerates the pairings of the three modes {f,o,z} (e.g., (f,o), (f,z), (o,z)). , For the projection representation of the corresponding mode, This represents a directed InfoNCE with modality m as the query and n as the key value. The sum of the two losses yields the total comparison target. And linearly weighted with the cross-entropy classification loss to form the final training criterion: .

8. A multimodal ophthalmic image analysis model based on optimal transport map diffusion, characterized in that, The multimodal ophthalmic image analysis method based on optimal transmission map diffusion according to any one of claims 1-7 includes at least the following: The OT-GND module explicitly calculates the distance between fundus color images and OCT image modalities through an optimal transmission algorithm, and combines graph neural networks and neural network frequent differential equation models to perform co-evolution of features in the continuous time domain; the hierarchical multi-head collaborative attention fusion module applies self-attention and cross-attention in parallel, preserving modal specificity at the low-level fine-grained level and progressively fusing color image texture and OCT deep structure at the high-level level, achieving deep semantic fusion with long-range dependencies across layers; the contrastive learning module applies symmetric InfoNCE contrast constraints simultaneously in the projection space of fundus color images, OCT, and fused features, explicitly promoting geometric consistency between modalities.

9. A system, characterized in that, The method includes a memory, a control processor, and a computer program stored in the memory and executable on the control processor, wherein the control processor executes the program to implement the multimodal ophthalmic image analysis method based on optimal transmission map diffusion as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for implementing the multimodal ophthalmic image analysis method based on optimal transmission map diffusion as described in any one of claims 1-7.

Images