Glaucoma grading system based on multi-modal conditioned state space fusion

Abstract

The application relates to the technical field of glaucoma grading, and discloses a glaucoma grading system based on multi-modal conditional state space fusion, which comprises the following steps: acquiring CFP images and OCT images, constructing a glaucoma grading network model, the model comprising a circular structure perception attention module, a layer prior enhanced axial attention module and a conditional state space fusion module; the circular structure perception attention module enhances the representation ability of a circular key pathological change region in the CFP image; the layer prior enhanced axial attention module enhances the utilization of key pathological change information in the OCT image; the conditional state space fusion module dynamically modulates the CFP spatial features through OCT global information, realizes cross-modal feature interaction, and uses the trained network model to perform glaucoma grading. The application can effectively fuse multi-modal information, enhance the feature extraction capability of glaucoma-related pathological features, and improve the performance of glaucoma grading.

Description

Technical Field

[0001] This invention relates to the field of glaucoma classification technology, and in particular to a glaucoma classification system based on multimodal conditional state space fusion. Background Technology

[0002] Glaucoma is a chronic neurodegenerative disease and one of the leading causes of irreversible but preventable blindness worldwide. Color fundus photography (CFP) and optical coherence tomography (OCT) are widely used imaging techniques in clinical glaucoma screening. CFP, through non-invasive high-resolution fundus imaging, assists doctors in observing macroscopic structural changes such as the cup-to-disc ratio of the optic disc, serving as a fundamental method for initial glaucoma screening. OCT not only quantitatively measures the thickness of the retinal nerve fiber layer (RNFL) but also precisely acquires microstructural changes in the macula (the area with the highest density of ganglion cell bodies)—changes that have been proven to be sensitive biomarkers for earlier stages of glaucoma. Combining CFP and OCT images for clinical glaucoma grading covers both the morphological characteristics of the optic disc and captures early lesion signals in the macula, making it a key technological combination for early glaucoma screening, disease grading, and progression monitoring. Clinicians can determine the stage and progression of glaucoma by observing changes in the cup-to-disc ratio of the optic disc in CFP and characteristics such as RNFL thickness and macular structure changes in OCT. However, clinical glaucoma grading methods still rely heavily on the subjective interpretation of physicians, which introduces errors and uncertainties, and the information fusion efficiency of multimodal data is relatively low.

[0003] In recent years, deep learning (DL)-based methods have been widely applied in medical image processing and analysis. Convolutional Neural Networks (CNNs), Transformers, and the Selective State Diffusion (SSD) module in the Mamba architecture have demonstrated significant application potential in various medical image processing and analysis tasks. With the development of these technologies, the accuracy and efficiency of medical image processing and analysis have been significantly improved: CNNs excel at capturing local features, Transformers can model long-distance dependencies, and Mamba's SSD module can more efficiently filter key information when processing high-dimensional medical images through the selective state diffusion mechanism, balancing computational cost and feature representation capabilities. These three technologies complement each other in improving the accuracy and speed of image processing, classification, and segmentation tasks, driving the development of medical image analysis towards automation and intelligence.

[0004] In existing technologies, CNN-based models have achieved relatively good automatic segmentation of the optic disc region in glaucoma CFP images, while Transformer-based methods can extract features from OCT 3D data to assist in assessing the degree of optic nerve damage. Furthermore, some deep learning models achieve glaucoma screening or classification using single-modal data (CFP only or OCT only), while others utilize multimodal data from CFP and OCT for glaucoma classification. Despite the significant progress made by deep learning technology in the medical field, its application in automatic glaucoma classification still has the following obvious shortcomings:

[0005] 1. There is a problem of insufficient multimodal fusion.

[0006] On the one hand, most existing methods are based on single-modal medical images, such as using only CFP or OCT, and cannot fully utilize the complementary information of multimodal medical images. On the other hand, although existing methods have attempted to fuse multimodal information such as CFP and OCT, current fusion methods mostly employ shallow fusion strategies such as simple feature stitching or weighted averaging, failing to fully consider the semantic associations and spatial correspondences between different modalities, and unable to effectively model the conditional dependencies between the spatial feature maps of CFP and the global semantic features of OCT. In addition, existing fusion methods lack the ability to model dynamic interactions between modalities, making it difficult to adaptively adjust fusion weights according to input samples, resulting in unsatisfactory fusion effects, failing to fully leverage the synergistic advantages of multimodal data, and limiting the model's comprehensive understanding of the complex pathological features of glaucoma.

[0007] 2. There are shortcomings in the extraction of glaucoma-related pathological features.

[0008] First, in fundus image analysis, circular structures (such as the optic disc and optic cup) are important pathological features. However, existing convolutional neural networks and attention mechanisms mostly use rectangular convolutional kernels, lacking specialized modeling for circular structures. This makes it difficult to fully utilize the geometric prior knowledge of the inherent circular structures like the optic disc and optic cup in fundus images, hindering the effective extraction of feature information from important glaucoma pathological changes such as the optic disc and optic cup. Second, OCT images contain rich information on the layered structure of retinal tissues. The thickness and morphological changes of different tissue layers (such as the retinal nerve fiber layer and ganglion cell layer) are important criteria for glaucoma grading. However, existing methods mostly use simple average pooling or max pooling for feature aggregation when processing OCT images, failing to fully utilize the prior knowledge of the layered structure in OCT images. This makes it difficult for the model to effectively capture the key features of glaucoma-related pathological changes in OCT images, especially for the subtle layered structural changes in early glaucoma, resulting in limited grading performance. Summary of the Invention

[0009] Therefore, the technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a glaucoma grading system based on multimodal conditional state space fusion, which can effectively fuse multimodal information, enhance the feature extraction capability of glaucoma-related pathological features, and improve the performance of glaucoma grading.

[0010] To address the aforementioned technical problems, this invention provides a glaucoma classification system based on multimodal conditional state-space fusion, comprising:

[0011] The data acquisition module is used to acquire CFP images and OCT images;

[0012] A glaucoma grading network model construction module is used to construct a glaucoma grading network model based on multimodal conditional state space fusion. This model includes a circular structure perception attention module, a layer prior enhanced axial attention module, and a conditional state space fusion module. The circular structure perception attention module enhances the representation ability of circular key pathological change regions in CFP images through multi-radius soft circular convolutional kernels. The layer prior enhanced axial attention module enhances the capture ability of long-range dependencies in the vertical direction and improves the utilization of key pathological change information in OCT images. The conditional state space fusion module models long-range dependencies of spatial sequences through a selective scanning mechanism and combines this with a mechanism that dynamically modulates CFP spatial features using global OCT information to achieve fine-grained, adaptive cross-modal feature interaction.

[0013] The model training module is used to train the glaucoma hierarchical network model based on multimodal conditional state space fusion.

[0014] The glaucoma grading module is used to input the CFP image and OCT image to be graded into the trained glaucoma grading network model based on multimodal conditional state space fusion, and obtain the glaucoma grading result.

[0015] Furthermore, the circular structure perception attention module enhances the representation ability of circular key pathological change regions in CFP images through multi-radius soft circular convolution kernels, including:

[0016] Important feature channels are highlighted through channel attention to obtain enhanced feature maps. Multi-radius soft circular spatial attention mechanism is used on the enhanced feature maps. By constructing three parallel circular attention branches, multi-scale information is captured to obtain spatial attention weights, thereby enhancing the representation ability of key circular regions such as the visual disk and visual cup in CFP images.

[0017] The spatial attention weights are multiplied element-wise with the channel-enhanced feature map to obtain the output feature map of the circular structure perception attention module.

[0018] Furthermore, the enhanced feature map of the channel employs a multi-radius soft circular spatial attention mechanism, which captures multi-scale information by constructing three parallel circular attention branches to obtain spatial attention weights, including:

[0019] Max pooling and average pooling are performed on the channel-enhanced feature map in the channel dimension to obtain two spatial feature maps. The two spatial feature maps are then concatenated in the channel dimension to obtain a dual-channel feature map.

[0020] Multiple parallel soft circular attention branches are constructed, each using a soft circular convolution kernel to capture circular structures at different scales; the learnable parameters of each soft circular attention branch include radius and boundary smoothing coefficient, wherein the radius is the radius parameter of the soft circular mask used to control the range covered by the circular attention, and the boundary smoothing coefficient controls the decay rate of the circular boundary from the center to the periphery;

[0021] For each soft circular attention branch, with the center of the convolution kernel as the origin, calculate the distance from each position to the center and generate a soft circular mask for each position; multiply the standard convolution weights element-wise with the soft circular mask for each position to obtain the effective weights of each soft circular attention branch.

[0022] The effective weights of each soft circular attention branch are used to perform a convolution operation on the dual-channel feature map to obtain the circular attention weights corresponding to each soft circular attention branch.

[0023] The circular attention weights corresponding to all soft circular attention branches are concatenated along the channel dimension, then fused through convolution, and finally generated by an activation function to produce the spatial attention weights.

[0024] Furthermore, the layer prior enhanced axial attention module enhances the ability to capture long-range dependencies in the vertical direction and improves the utilization of key pathological changes in OCT images, including:

[0025] The layer prior enhanced axial attention module includes a vertical branch and a horizontal branch. The input to the layer prior enhanced axial attention module is denoted as... ,right One-dimensional axial attention calculation is performed along the vertical direction to obtain the output features of the vertical branch;

[0026] The output features of the vertical branch are subjected to one-dimensional axial attention along the horizontal direction to obtain the output feature map of the layer prior enhancement axial attention module.

[0027] Furthermore, the aforementioned One-dimensional axial attention calculation is performed along the vertical direction to obtain the output features of the vertical branch, including:

[0028] right Perform dimensional permutation and reshaping operations, and the resulting sequence feature map is denoted as... , Each column in the table corresponds to an independent sequence;

[0029] Using a multi-head attention mechanism The mapping is done as query vector, key vector, and value vector, and the content similarity score is calculated by combining the query vector, key vector, and value vector.

[0030] The relative position offset is obtained by using a direct index offset table scheme for relative position encoding.

[0031] Use vertical strip convolution kernels Convolution is performed, and the convolution result is enhanced by an activation function to reshape it into a sequence format, yielding the layer prior features, denoted as... , For the layer prior features corresponding to the i-th attention head, , For the number of heads;

[0032] Applying normalization along the sequence dimension yields a normalized layer prior sequence, denoted as... , Let be the normalized layer prior sequence corresponding to the i-th attention head; Expanded to a bias tensor that matches the content similarity score dimension, denoted as , Let i be the bias tensor corresponding to the i-th attention head;

[0033] The relative position bias and bias tensor are added to the content similarity score to obtain the bias attention weight in the vertical direction.

[0034] The vertical bias attention weights are multiplied by the value vector to obtain the output of each attention head;

[0035] The outputs of all attention heads are concatenated along the channel dimension and then fused. The fusion result is then dimensionally reshaped to obtain the output features of the vertical branch.

[0036] Furthermore, the method for calculating the vertical bias attention weight is as follows:

[0037] ,

[0038] In the formula, Let be the bias attention weight in the vertical direction corresponding to the i-th attention head. For the Softmax function, Let be the content similarity score of the i-th attention head. Let λ be the relative position offset of the i-th attention head, and λ be an adjustable weight coefficient. Let be the bias tensor corresponding to the i-th attention head.

[0039] Furthermore, the conditional state space fusion module models long-distance dependencies of spatial sequences through a selective scanning mechanism, and combines this with a mechanism that dynamically modulates CFP spatial features using OCT global information to achieve fine-grained, adaptive cross-modal feature interaction, including:

[0040] The output feature map of the layer prior enhancement axial attention module is used to generate modulation features through a multilayer perceptron and a broadcast mechanism;

[0041] The output feature map of the layer prior enhancement axial attention module is used to generate a state transition matrix and a time step through two parallel, parameter-independent multilayer perceptrons.

[0042] A selective scanning mechanism is adopted to model the modulation features in one-dimensional state space along both row and column directions based on the state transition matrix and time step. The dynamic interaction of cross-modal features is realized through the state space conditionalization mechanism to obtain row scanning features and column scanning features.

[0043] By fusing row scan features and column scan features, multimodal fusion features are obtained.

[0044] Furthermore, the output feature map of the layer prior enhancement axial attention module is used to generate modulation features through a multilayer perceptron and a broadcast mechanism, specifically as follows:

[0045] The output feature map of the layer prior enhanced axial attention module is used to generate FiLM parameters through a multilayer perceptron. The FiLM parameters are then dimensionally reshaped to obtain scaling and offset parameters. These parameters are then used to modulate the output feature map of the circular structure perceptual attention module via a broadcast mechanism, resulting in the modulated features:

[0046] ,

[0047] In the formula, For modulation features, ⊙ indicates element-wise multiplication. This is the output feature map of the circular structure perception attention module. For scaling parameters, This is the offset parameter.

[0048] Further, the calculation method of the state transition matrix is ​​as follows: The output feature map of the layer prior enhancement axial attention module is subjected to MLP transformation to generate a state increment adjustment value; the state increment adjustment value is then dimensionally reshaped to obtain the dimensionally reshaped state increment adjustment value, denoted as... Introducing learnable initial bias parameters and performing dimensional reshaping yields the dimensional reshaping bias parameters, denoted as... ;Will and Add them together and obtain the state transition matrix through an exponential function;

[0049] The time step is calculated as follows: The output feature map of the layer prior enhancement axial attention module is subjected to MLP transformation to generate a step size increment adjustment value. The MLP internal parameters of the MLP transformation used to generate the step size increment adjustment value and the MLP transformation used to generate the state increment adjustment value are not shared. The step size increment adjustment value is then dimensionally reshaped to obtain the dimensionally reshaped step size increment adjustment value, denoted as... ;right After the function is activated, the time step is multiplied by the step scaling factor to obtain the time step.

[0050] Furthermore, when performing one-dimensional state-space modeling of the modulation features along both row and column directions, the modulation features are scanned in rows to obtain the row scanning result, specifically as follows:

[0051] The modulation features are reshaped into a sequence format row by row to obtain a row sequence, denoted as . After applying layer normalization to each sequence in the row sequence, a selective scan is performed to obtain the row scan result:

[0052] ,

[0053] In the formula, Indicates the row scan results. Representation layer normalization, Represents the state transition matrix. Indicates the time step. Indicates selective scanning;

[0054] The state update equation for selective scanning is:

[0055] ,

[0056] In the formula, This represents the row scan result at time t. This represents the state vector at time t. For learnable parameters, Represents the input and corresponding time t. The t-th element in The calculation method is as follows:

[0057] ,

[0058] In the formula, This represents the natural exponential function. This represents element-wise product.

[0059] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0060] This invention uses a circular structure perception attention module to better capture circular structural features in fundus images, and a layer prior enhancement axial attention module to capture feature changes in different tissue layers, thereby more accurately identifying key pathological change areas and improving the ability to identify subtle tissue layer structural changes in early glaucoma. Furthermore, the conditional state space fusion module fully explores complementary information between different modalities, achieving effective fusion of multimodal information and improving the performance of glaucoma grading. Attached Figure Description

[0061] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0062] Figure 1 This is a diagram showing the overall structure of a glaucoma hierarchical network model based on multimodal conditional state space fusion in a preferred embodiment of the present invention.

[0063] Figure 2 This is a structural diagram of the circular structure perception and attention module in a preferred embodiment of the present invention.

[0064] Figure 3 This is a structural diagram of the layer prior enhancement axial attention module in a preferred embodiment of the present invention.

[0065] Figure 4This is a structural diagram of the conditional state space fusion module in a preferred embodiment of the present invention.

[0066] Figure 5 This is a structural diagram of the feature-level linear modulation module in a preferred embodiment of the present invention.

[0067] Figure 6 This is a structural diagram of the state space parameter conditionalization module in a preferred embodiment of the present invention. Detailed Implementation

[0068] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0069] This invention discloses a glaucoma classification system based on multimodal conditional state space fusion, including a data acquisition module, a glaucoma classification network model construction module, a model training module, and a glaucoma classification module. The process of glaucoma classification in each module specifically includes the following steps:

[0070] S1: Data acquisition module, used to acquire CFP images and OCT images.

[0071] S2: Glaucoma Grading Network Model Construction Module, used to construct a glaucoma grading network model based on multi-modal conditional state-space fusion (GLA-Net). The glaucoma grading network model based on multi-modal conditional state-space fusion includes a Circular-Structure Perception Module (CSPM), a Layer-prior Augmented Axial Attention Module (LA3), and a Conditional State-Space Fusion Module (CSSF). The Circular-Structure Perception Module extracts features from the CFP image to obtain first-modal space features representing circular structures such as the optic disc and optic cup; the Layer-prior Augmented Axial Attention Module extracts features from the OCT image to obtain second-modal features representing structural changes in different tissue layers; the Conditional State-Space Fusion Module performs state-space modeling and fusion of the first-modal space features and the second-modal features to obtain multi-modal fused features.

[0072] The circular structure perception attention module enhances the representation ability of key circular pathological change regions such as the optic disc and optic cup in CFP images through multi-radius soft circular convolution kernels. The layer prior enhancement axial attention module enhances the ability to capture long-range dependencies in the vertical direction and enhances the utilization of key pathological change information such as the retinal tissue layer structure in OCT images. The conditional state space fusion module is based on the idea of ​​selective state space model. It models the long-range dependencies of spatial sequences through selective scanning mechanism. Combined with conditionalization mechanism, it realizes fine-grained and adaptive cross-modal feature interaction through a mechanism that dynamically modulates CFP spatial features with global OCT information.

[0073] like Figure 1 As shown, the structure of the glaucoma classification network model based on multimodal conditional state-space fusion includes: a CFP feature extraction branch, comprising a lightweight backbone network EfficientNet-B0 and a circular perceptual attention module, used to extract CFP image features; an OCT feature extraction branch, comprising EfficientNet-B0 and a layer prior-enhanced axial attention module, used to extract OCT image features; the encoder consists of two parallel branches, the CFP feature extraction branch and the OCT feature extraction branch, and the features extracted by the two branches are fused through the conditional state-space fusion module. Employing the idea of ​​a selective state-space model, through selective scanning and conditionalization mechanisms, the OCT global information is dynamically modulated onto the CFP spatial features. Finally, the glaucoma classification result is output through the classification head.

[0074] CFP images are processed by EfficientNet-B0 and then input into the circular structure perception attention module. OCT images are processed by EfficientNet-B0 and then input into the layer prior enhancement axial attention module. EfficientNet-B0 is a lightweight convolutional neural network that achieves a good balance between accuracy and efficiency through a compound scaling strategy. The network structure of EfficientNet-B0 includes an initial convolutional layer and seven Mobile Inverted Bottleneck Blocks (MBConv). Each block includes a 1×1 extended convolution, a depthwise separable convolution, a squeeze-and-excitation (SE) attention module, a 1×1 projective convolution, and residual connections.

[0075] S2-1: The feature extraction process of the CFP feature extraction branch is as follows:

[0076] S2-1-1: Denote the CFP image as... , , The feature map obtained after EfficientNet-B0 is denoted as follows: , ,Will Input circular structure to perceive attention module.

[0077] S2-1-2: The circular structure perception attention module enhances the representation ability of key circular pathological change areas such as the optic disc and optic cup in CFP images through multi-radius soft circular convolution kernels.

[0078] The circular structure perception attention module is specifically designed to capture circular structural features in fundus images, such as key circular regions like the optic disc and optic cup. This module uses multi-radius soft circular convolutional kernels instead of traditional rectangular convolutional kernels, better adapting to the inherent circular geometry in fundus images and enhancing the representation ability of circular structures at different scales through its multi-radius design. Figure 2 As shown, the circular structure perception attention module consists of two parts: a channel attention module and a spatial attention module.

[0079] S2-1-2-1: The input to the circular structure perception attention module is... Highlighting attention through channels Important feature channels are used to obtain enhanced feature maps, denoted as... , .

[0080] S2-1-2-1-1: The input feature map is processed using Global Average Pooling (GAP) and Global Max Pooling (GMP). Spatial dimensionality compression is performed to obtain the features. and , , .

[0081] S2-1-2-1-2: and Feature transformation is performed using a multilayer perceptron (MLP) with two fully connected layers. The first fully connected layer compresses the number of channels from 1280 to 80, and the second fully connected layer expands the number of channels back to 1280. A ReLU activation function is used between the two fully connected layers to obtain two transformed features. and , , ,Will and The values ​​are added together, and then activated by the Sigmoid activation function to generate channel attention weights, denoted as... , .

[0082] S2-1-2-1-3: Adjust the channel attention weights ( ) and input feature map ( ) Element-wise multiplication yields the channel-enhanced feature map, denoted as , Specifically:

[0083] ,

[0084] In the formula, denoted as Sigmoid activation function, GAP() represents global average pooling (pooling on spatial dimension H×W), GMP() represents global max pooling (pooling on spatial dimension H×W), and MLP is a multilayer perceptron (bottleneck structure, 1280→80→1280). This indicates element-wise multiplication.

[0085] S2-1-2-2: Feature map after channel enhancement ( This employs a multi-radius soft circular spatial attention mechanism, constructing three parallel circular attention branches to capture multi-scale information, and obtaining the spatial attention weights, denoted as... This enhances the ability to represent key circular regions such as the visual disc and visual cup in CFP images.

[0086] S2-1-2-2-1: In order to preserve rich spatial context information, the feature map after channel enhancement ( Feature compression and stitching are performed: First, the enhanced feature maps of the channels are compressed and stitched together along the channel dimension. Max pooling and average pooling are performed to obtain two spatial feature maps, denoted as . and , , Then, splice them along the channel dimension. and The dual-channel feature map is obtained, denoted as , ,Right now:

[0087] ,

[0088] In the formula, and These represent the max pooling and average pooling operations in channel-wise pooling, respectively.

[0089] S2-1-2-2-2: Construct multiple parallel soft circular attention branches. Each soft circular attention branch uses a soft circular convolution kernel to capture circular structures of different scales. The kernel size of each branch is K×K, used to define the receptive field of the convolution operation. The core of each soft circular attention branch consists of two learnable parameters: radius and boundary smoothing coefficient. The radius is the radius parameter of the soft circular mask, used to control the range of circular attention coverage. In this embodiment, three parallel soft circular attention branches are constructed, and the initial values ​​of the radii of the three branches are set to... , , This allows the three branches to capture circular structures of different scales, thereby enhancing the representation of key circular regions such as the optic disc and optic cup of different sizes in CFP images. The boundary smoothing coefficient controls the attenuation rate of the circular boundary from the center to the periphery, with an initial value uniformly set to 1.0; the larger the boundary smoothing coefficient, the smoother the transition of the circular boundary; the smaller the soft boundary, the sharper the circular boundary.

[0090] S2-1-2-2-3: For each soft circular attention branch, calculate the distance from each location to the center, with the center of the convolution kernel as the origin:

[0091] ,

[0092] In the formula, This represents the distance from the i-th position in the n-th soft circular attention branch to the center.

[0093] n=1,2,…,N, where N is the number of soft circular attention branches. In this embodiment, N=3. , ) represents the coordinates of the center of the convolution kernel. , Let be the coordinates of the i-th position;

[0094] The soft circular mask at position i in the nth soft circular attention branch is generated using the Sigmoid function as follows:

[0095] ,

[0096] In the formula, This represents the soft circular mask at position i in the nth soft circular attention branch. σ represents the Sigmoid function. This represents the radius of the nth soft circular attention branch. Represents the boundary smoothing coefficient of the nth soft circular attention branch;

[0097] For the nth soft circular attention branch, the standard convolution weights (denoted as...) are... , ) and the soft circular mask for each position ( ) Multiply element by element to obtain the effective weight of the nth soft circular attention branch, denoted as . , ;

[0098] Use the effective weights of each soft circular attention branch to evaluate the two-channel feature map ( Perform a convolution operation to obtain the circular attention weights corresponding to each soft circular attention branch:

[0099] ,

[0100] In the formula, The circular attention weights corresponding to the nth soft circular attention branch. ,⊙ represents element-wise multiplication, and Conv represents the convolution operation. This represents the effective weight of the nth soft circular attention branch.

[0101] S2-1-2-2-4: Circular attention weights for all soft circular attention branches (i.e. The data are concatenated along the channel dimension, then fused using a 1×1 convolution, and finally processed by a Sigmoid activation function to generate the spatial attention weights, denoted as . , ,Right now:

[0102] ,

[0103] In the formula, σ represents the Sigmoid function. Represents a 1×1 convolution. This represents the circular attention weights of the first soft circular attention branch. This represents the circular attention weights of the second soft circular attention branch. This represents the circular attention weight of the third soft circular attention branch.

[0104] S2-1-2-3: Adjust spatial attention weights ( ) and the feature map after channel enhancement ( ) Perform element-wise multiplication to obtain the output feature map (i.e., the first modality space feature) of the circular structure perceptual attention module, denoted as . , ,Right now:

[0105] ,

[0106] In the formula, This indicates element-wise multiplication.

[0107] S2-2: The feature extraction process of the OCT feature extraction branch is as follows:

[0108] S2-2-1: Denote the OCT image as... , The feature map obtained after EfficientNet-B0 is denoted as follows: , ,Will Input layer prior enhancement axial attention module;

[0109] S2-2-2: The layer prior enhanced axial attention module enhances the ability to capture long-range dependencies in the vertical direction and enhances the utilization of key pathological changes such as the retinal tissue layer structure in OCT images.

[0110] The layer prior enhanced axial attention module is specifically designed to extract layer structure features from OCT images. Through axial attention mechanisms and layer prior knowledge, it can effectively identify feature variations across different tissue layers. For example... Figure 3 As shown, the layer prior-enhanced axial attention module consists of two parts: an H-axis (vertical) branch and a W-axis (horizontal) branch. Based on the long-range spatial dependencies in the vertical direction among different retinal tissue layers (such as the retinal nerve fiber layer and ganglion cell layer), the H-axis branch enhances the model's ability to extract long-range dependencies in the vertical direction by injecting layer prior biases, effectively improving the recognition performance of tissue layer structures in OCT images. The W-axis branch employs a standard axial attention mechanism to extract feature dependencies in the horizontal direction.

[0111] S2-2-2-1: The layer prior enhanced axial attention module includes a vertical branch and a horizontal branch. The input to the layer prior enhanced axial attention module is denoted as... ,right One-dimensional axial attention calculation is performed along the vertical direction (H-axis) to obtain the output features of the vertical direction (H-axis) branch, denoted as... .

[0112] S2-2-2-1-1: Regarding Perform dimensional permutation and reshaping operations, and the resulting sequence feature map is denoted as... , , Each column in the array corresponds to an independent sequence with a sequence length of H=32.

[0113] S2-2-2-1-2: Employing a multi-head attention mechanism to... The input feature map, with C=1280 channels, is decomposed into h attention heads by a linear transformation layer, mapping the input to query vector, key vector, and value vector respectively. Each attention head has a dimension of... Where h is the number of attention heads. After reshaping into a multi-head format, the content similarity score is calculated by combining the query vector, key vector, and value vector; for each attention head, the content similarity score is:

[0114] ,

[0115] In the formula, Let be the content similarity score of the i-th attention head. , For the number of attention heads, , Let i be the query vector for the i-th attention head. Let i be the key vector of the i-th attention head. Let T be the vector dimension of the i-th attention head, and let T denote the transpose.

[0116] S2-2-2-1-3: Relative Position Bias. The layer prior enhancement axial attention module uses a Direct Index Bias Table scheme for relative position encoding, assigning a learnable bias scalar to each relative distance. All attention heads share a single bias parameter table, denoted as... , ,in This is the maximum sequence length. For each query position... and corresponding key positions Calculate relative position ,pass The offset value is directly indexed, and then the relative position offset is obtained through a broadcast mechanism, denoted as... , .

[0117] S2-2-2-1-4: Layer prior bias, from Extracting layer prior information. First, using... Vertical strip convolution kernel pairs Convolution is used to capture intensity changes in the vertical direction. The height of the vertical strip convolution kernel is represented; the nonlinearity is enhanced by the Softplus activation function and reshaped into a sequence format to obtain the layer prior features, which are denoted as . , ,Right now:

[0118] ,

[0119] In the formula, For the layer prior features corresponding to the i-th attention head, , For the number of heads;

[0120] Then, a standardization operation (subtracting the mean and dividing by the standard deviation) is applied along the sequence dimension to obtain the standardized layer prior sequence, denoted as . , , Let be the normalized layer prior sequence corresponding to the i-th attention head; Expanded to a bias tensor that matches the content similarity score dimension, denoted as , , Let the bias tensor corresponding to the i-th attention head be such that each query position The vertical prior values ​​are copied to all corresponding key positions. .

[0121] S2-2-2-1-5: Offset the relative position ( ) and bias tensor ( Add to content similarity score ( The vertical bias attention weights are calculated using the Softmax function.

[0122] S2-2-2-1-6: Imposing vertical bias attention weights ( Multiplying the value vector by the value vector yields the output of each attention head:

[0123] ,

[0124] In the formula, For the output of the i-th attention head, , Let be the bias attention weight in the vertical direction corresponding to the i-th attention head. Represents matrix multiplication. Let i be the value vector of the attention head. The method for calculating the vertical bias attention weight is as follows:

[0125] ,

[0126] In the formula, Let be the bias attention weight in the vertical direction corresponding to the i-th attention head. , For the Softmax function, Let be the content similarity score of the i-th attention head. λ represents the relative position offset of the i-th attention head, and λ is an adjustable weight coefficient whose value is adjusted according to the actual situation. Let be the bias tensor corresponding to the i-th attention head.

[0127] The outputs of all attention heads are concatenated along the channel dimension and then fused through a projection layer. The resulting feature is denoted as . , ,Right now:

[0128] ,

[0129] In the formula, FC represents the projection layer;

[0130] right Dimensional reshaping yields the output features of the vertical (H-axis) branch, denoted as... , .

[0131] S2-2-2-2: Output characteristics of the vertical (H-axis) branch ( One-dimensional axial attention is performed along the horizontal direction (W axis) to obtain the output feature map (i.e., the second modality feature) of the layer prior enhanced axial attention module, denoted as... .

[0132] S2-2-2-2-1: Output characteristics of the vertical (H-axis) branch ( Perform dimensional permutation and reshaping operations to obtain the sequence feature map, denoted as . , , Each column in the array corresponds to an independent sequence with a sequence length of W=32;

[0133] S2-2-2-2-2: Employs the same multi-head attention mechanism and relative position encoding scheme as the vertical direction (H-axis), but without injecting layer prior bias. The specific process includes: The mapping is performed as query vector, key vector, and value vector, and content similarity scores are calculated. A relative position bias is added, and horizontal bias attention weights are calculated using Softmax. The value vectors are then weighted and summed. The outputs of all attention heads are concatenated and fused through a projection layer to obtain the resulting feature, denoted as [feature name missing]. , Finally, Dimension reshaping and adaptive average pooling are performed to obtain the output feature map of the layer prior enhanced axial attention module. ), .

[0134] The layer prior enhanced axial attention module, while maintaining linear computational complexity O(H×W×(H+W)), explicitly enhances the capture of vertical dimension contextual information, enabling the model to learn more discriminative directional features from OCT images.

[0135] The feature extraction processes of the CFP feature extraction branch in step S2-1 and the OCT feature extraction branch in step S2-2 are not sequential. For ease of description, this embodiment uses steps S2-1 and S2-2 to illustrate the process.

[0136] S2-3: As Figure 4 As shown, the conditional state space fusion module will convert the spatial feature map of the CFP branch. Global features of the OCT branch Conditional fusion is performed to achieve deep collaboration of multimodal features. The conditional state space fusion module is based on the idea of ​​selective state space model. It models the long-distance dependency of spatial sequence through selective scanning mechanism. Combined with conditionalization mechanism, it achieves fine-grained and adaptive cross-modal feature interaction through a mechanism that dynamically modulates CFP spatial features with global information from OCT.

[0137] S2-3-1: As Figure 5 As shown, OCT global information is injected into the CFP spatial features through a feature-wise linear modulation (FiLM) module. Specifically, the output feature map of the layer prior enhancement axial attention module (FiLM) is processed... The FiLM parameters are generated through a multilayer perceptron, and the scaling parameters (denoted as ) are obtained by dimensional reshaping the FiLM parameters. , ) and offset parameter (denoted as , The output feature map of the circular structure perception attention module is broadcast through a broadcast mechanism. Modulation is performed to obtain the following modulation characteristics:

[0138] ,

[0139] In the formula, Modulation characteristics, ,⊙ indicates element-wise multiplication. This is the output feature map of the circular structure perception attention module. For scaling parameters, This is the offset parameter.

[0140] S2-3-2: Constructing as follows Figure 6 The state-space parameter conditionalization module shown above, for the output feature map of the layer prior enhanced axial attention module ( The state transition matrix (denoted as ) is generated by two parallel, parameter-independent multilayer perceptrons. ) and time step (denoted as ).

[0141] S2-3-2-1: The method for calculating the state transition matrix is ​​as follows:

[0142] The output feature map of the layer prior enhancement axial attention module ( Perform MLP transformation to generate state increment adjustment values, denoted as... , The state increment adjustment value is then dimension-reshaped to obtain the dimension-reshaped state increment adjustment value, denoted as . , Introduce learnable initial bias parameters, denoted as... , ),right The dimensionality reshaping process yields the reshaped bias parameters, denoted as... , ;Will and Adding them together, the state transition matrix obtained by the exponential function is:

[0143] ,

[0144] In the formula, Here is the state transition matrix. , It is a natural exponential function. The state transition matrix enables the state space parameters to adaptively adjust according to the global information of OCT, capturing feature changes at different scales.

[0145] S2-3-2-2: The method for calculating the time step is as follows:

[0146] The output feature map of the layer prior enhancement axial attention module ( Perform MLP transformation to generate step size increment adjustment value, denoted as... , The MLP internal parameters of the MLP transformation used to generate the step size increment adjustment value and the MLP transformation used to generate the state increment adjustment value are not shared; the step size increment adjustment value is dimensionally reshaped to obtain the dimensionally reshaped step size increment adjustment value, denoted as... , ;right Activate using the Softplus function, ensure the output value is positive, and then multiply by the step scaling factor (denoted as ). ), to obtain the time step, denoted as , ,Right now:

[0147] ,

[0148] In the formula, The method of calculating the time step for Softplus function activation allows the time step to be adaptively adjusted based on global information from OCT, thereby controlling the update rate of the state-space model.

[0149] S2-3-3: Selective Scanning. The conditional state space fusion module employs a selective scanning mechanism, scanning the modulation features ( ) along both row and column directions based on the state transition matrix and time step. One-dimensional state space modeling is performed, and dynamic interaction of cross-modal features is achieved through state space conditionalization mechanism to obtain row scan features and column scan features.

[0150] S2-3-3-1: Regarding the modulation features ( Perform a row-wise scan to obtain the row scan results, specifically:

[0151] The modulation features ( Reshape the rows into a sequence format to obtain the row sequence, denoted as . , After applying layer normalization to each sequence in the row sequence, a selective scan is performed to obtain the row scan result:

[0152] ,

[0153] In the formula, Indicates the row scan results. , Representation layer normalization, Represents the state transition matrix. Indicates the time step. Indicates selective scanning;

[0154] The state update equation for selective scanning is:

[0155] ,

[0156] In the formula, This represents the row scan result at time t. This represents the state vector at time t. , For learnable parameters, , Represents the input and corresponding time t. The t-th element in , The calculation method is as follows:

[0157] ,

[0158] In the formula, This represents the natural exponential function. This represents element-wise multiplication; the initial state in this embodiment... It is a zero vector.

[0159] S2-3-3-2: Regarding the modulation features ( Perform a column-wise scan to obtain the column scan results.

[0160] Using the modulation features ( The modulation features are scanned using the same method as the row scan. The column scan is performed to obtain the column scan result. That is, the modulation features ( Reshape the columns into a sequence format to obtain a column sequence, denoted as . After applying layer normalization to each sequence in the column sequence, a selective scan is performed to obtain the column scan result:

[0161] ,

[0162] In the formula, This indicates the column scan results. .

[0163] S2-3-4: Combine row scan features and column scan features to obtain multimodal fusion features.

[0164] The row scan results ( ) and column scan results ( The features are fused and projected using a 1×1 convolution to obtain the multimodal fusion features:

[0165] ,

[0166] In the formula, Indicates multimodal fusion features, , This represents a 1×1 convolution, where k1 is the fusion coefficient, and in this embodiment, k1=0.5.

[0167] This fusion strategy can simultaneously utilize spatial dependencies in both horizontal and vertical directions, enhancing the modeling capability of CFP spatial features. The output is a fused feature map. It contains spatial structural information of CFP and global semantic information of OCT.

[0168] S2-4: Multimodal fusion features ( After adaptive average pooling, the results are mapped to the glaucoma grading results through a classification head. The classification head consists of a Dropout layer, a fully connected layer, and a Softmax activation function. The three categories of glaucoma grading correspond to: normal, early, and intermediate-to-late stage.

[0169] S3: Model training module, used to train the glaucoma hierarchical network model based on multimodal conditional state space fusion.

[0170] Because the number of early-stage glaucoma samples is relatively small while the number of normal samples is relatively large, class imbalance exists in the automatic glaucoma classification. To alleviate this problem, this invention employs Focal Loss as the loss function when training the glaucoma classification network model based on multimodal conditional state space fusion. Focal Loss dynamically adjusts the weights of easy and difficult samples, enabling the model to pay more attention to difficult-to-classify samples and minority class samples.

[0171] S4: Glaucoma grading module, used to input the CFP image and OCT image to be graded into the trained glaucoma grading network model based on multimodal conditional state space fusion, and obtain the glaucoma grading result.

[0172] The present invention also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the functions of each module in a glaucoma classification system based on multimodal conditional state space fusion.

[0173] The present invention also discloses an apparatus including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the functions of each module in a glaucoma classification system based on multimodal conditional state space fusion.

[0174] This invention effectively addresses the problems of insufficient multimodal fusion, limited feature extraction capabilities, and class imbalance encountered in automatic glaucoma classification. It improves the model's glaucoma classification performance by designing a circular perceptual attention module, a layer-prior-enhanced axial attention module, and a conditional state-space fusion module, combined with the Focal Loss loss function and effective data augmentation strategies. Compared with existing technologies, the advantages of this invention are as follows:

[0175] 1. Effective Fusion of Multimodal Information. To address the shortcomings of existing methods in multimodal fusion, this invention designs a conditional state-space fusion module. This module effectively models the conditional dependencies between the CFP spatial feature map and the OCT global semantic features. The conditional state-space fusion module adopts the idea of ​​a selective state-space model, using a selective scanning mechanism to model the long-term dependencies of spatial sequences. It uses the global features of the OCT as conditional information to dynamically modulate the CFP spatial feature map, achieving deeper multimodal feature fusion. Specifically, by combining FiLM modulation and state-space parameter conditionalization mechanisms, it achieves fine-grained, adaptive cross-modal feature interaction by dynamically modulating CFP spatial features with OCT global information. This conditional fusion mechanism allows the model to adaptively adjust its focus on CFP spatial features based on the global semantic information provided by the OCT, thereby fully utilizing the complementary advantages of multimodal data. Compared with the shallow fusion strategies such as simple feature concatenation or weighted averaging used in existing methods, this module effectively models the conditional dependencies between the CFP spatial feature map and the OCT global semantic vector, achieving deeper multimodal feature fusion and significantly improving glaucoma classification performance.

[0176] The conditional state-space fusion module applies a state-space model to multimodal fusion tasks, combining a feature-level linear modulation module and a state-space parameter conditionalization module to achieve dynamic interaction of cross-modal features. This conditional fusion mechanism enables the model to adaptively adjust its focus on CFP spatial features based on the global semantic information provided by OCT, thereby fully utilizing the complementary advantages of multimodal data and improving the accuracy of glaucoma classification. Through deep collaboration between the CFP and OCT branches, CFP provides spatial structure information, and OCT provides global semantic information. The two are conditionally fused through the conditional state-space fusion module, allowing the model to simultaneously utilize the advantages of both modalities to form a more comprehensive and accurate glaucoma classification judgment.

[0177] 2. Enhanced feature extraction capability. To address the shortcomings of existing methods in extracting key fundus pathological changes related to glaucoma, this invention makes improvements in two aspects.

[0178] First, to fully extract key fundus changes in glaucoma, such as the optic disc and optic cup, from fundus images, a circular structure perception attention module was designed. This module uses circular convolutional kernels instead of traditional rectangular convolutional kernels, which can better capture the circular structural features of the optic disc and optic cup in fundus images. A multi-radius design enhances the representation ability of circular structures at different scales. The circular structure perception attention module, through a circular spatial attention mechanism, enables the model to specifically focus on regions with circular geometric features, thereby improving the recognition accuracy of key pathological change regions such as the optic disc and optic cup.

[0179] Secondly, to address the issue of insufficient utilization of retinal tissue layer structure information in OCT images by existing methods, a layer-prior-enhanced axial attention module was designed. This module is specifically designed for the characteristics of retinal tissue layer structure in OCT images. Through axial attention mechanisms and layer prior knowledge, it can effectively capture feature changes such as the thickness of different tissue layers, explicitly enhancing the model's ability to capture long-range dependencies in the vertical direction. This enables the model to more accurately identify key pathological changes such as tissue layer thickness in OCT images, particularly demonstrating stronger recognition capabilities for subtle tissue layer structure changes in early glaucoma.

[0180] 3. Excellent glaucoma grading performance. The design of the circular structure perception attention module and the layer prior enhanced axial attention module fully utilizes the inherent circular geometric prior knowledge in fundus images and the tissue layer structure prior in OCT images, enabling the model to more accurately identify key pathological change areas. In particular, it has a stronger ability to identify subtle tissue layer structure changes in early glaucoma, significantly improving the model's feature extraction capability.

[0181] 4. Flexible Modular Design. The circular structure perception attention module, layer prior enhanced axial attention module, and conditional state space fusion module in this invention possess strong modularity. These modules can be easily integrated into different deep learning architectures to solve problems such as multimodal fusion and feature extraction. The circular structure perception attention module can be applied to other image analysis tasks that require capturing circular structures; the layer prior enhanced axial attention module can be applied to other tasks that require capturing hierarchical structures or long-range vertical dependencies; and the conditional state space fusion module can be applied to other multimodal fusion tasks. This flexibility enables the technology of this invention to have a wide range of application scenarios, suitable for different datasets and task requirements, and has high scalability and versatility.

[0182] In summary, the GLA-Net designed in this invention achieves fine-grained, adaptive cross-modal feature interaction through a conditional state space fusion module, and enhances feature extraction capabilities through a circular structure perceptual attention module and a layer prior-enhanced axial attention module. It demonstrates significant technical advantages in solving key problems in glaucoma classification tasks. In particular, while addressing issues such as insufficient multimodal fusion and limited feature extraction capabilities, it achieves effective control over parameter quantity and computational resources through lightweight backbone network design and efficient feature extraction, and has good application prospects.

[0183] To further demonstrate the beneficial effects of this invention, experimental verification was conducted on the publicly available dataset GAMMA in this embodiment. The GAMMA dataset contains 300 pairs of CFP and OCT multimodal images, labeled with a three-level classification of glaucoma, divided into three categories: no glaucoma (normal), early glaucoma, and intermediate-to-late-stage glaucoma. CFP images include two sizes: 2000×2992 pixels and 1934×1956 pixels, in RGB color format. OCT images are scanned with the macula as the center, and each OCT volume contains 256 consecutive two-dimensional cross-sectional images, each cross-sectional image measuring 992×512 pixels.

[0184] The dataset was divided into a training set (100 pairs), a validation set (100 pairs), and a test set (100 pairs). The same random seed and preprocessing strategy were used in all experiments to ensure fairness in training and testing.

[0185] Due to the limited number of samples in the original GAMMA dataset, different data augmentation strategies were employed for CFP and OCT images in the experiment to prevent overfitting and enhance the generalization ability of the model.

[0186] The data augmentation strategies for CFP images are as follows: 1. Random horizontal flipping: The image is flipped horizontally with a probability of 0.5, which helps the model learn the symmetrical fundus structure features; 2. Random rotation: The rotation angle θ is randomly selected within the range of [-30°, +30°] to rotate the image, which helps the model to be robust to image rotation; 3. Center cropping: The image is cropped to a fixed size of 512×512 to ensure that all input images have a uniform size; 4. Normalization: The image is normalized using the mean and standard deviation of the ImageNet dataset.

[0187] The OCT images were in grayscale format and converted to RGB format by copying channels to adapt to the EfficientNet-B0 network. Sixteen slices were selected from each 3D OCT dataset (eight slices before and after the slice with index 130). The data augmentation strategies for the OCT images were: 1. Random horizontal flipping: The image was horizontally flipped with a probability of 0.5, which helps the model learn the left-right symmetrical OCT tissue layer structure features; 2. Random affine transformation: The image was randomly translated horizontally and vertically (translation range ±5%) and randomly scaled (scaling range [0.94, 1.06]), which helps the model be robust to changes in the position and scale of the OCT image; 3. Normalization: The image was normalized using the mean and standard deviation of the ImageNet dataset.

[0188] To objectively evaluate the glaucoma grading performance of GLA-Net in this invention, accuracy, F1 score, recall, precision, specificity, area under the ROC curve (AUC), and weighted Kappa coefficient were used as evaluation indicators in comparative and ablation experiments.

[0189] Glaucoma classification was performed using GLA-Net from this invention and several current advanced multimodal fusion methods. The experimental results are shown in Table 1. The comparison methods included those used by the top five teams on the 2021 MICCAI GAMMA Challenge official leaderboard (Smart DSP, Voxelcloud, EyeStar, HZL, IBME) and other multimodal fusion methods (MSAN, MM-CNN, DeepGuide, CRD-Net, EyeMost, MATCNN, M2AFNet). To ensure experimental fairness, all comparison methods were reproduced using their original best-in-class configurations and training strategies, maintaining their optimal state.

[0190] Table 1. Results of glaucoma classification using the present invention and comparative methods.

[0191]

[0192] As can be seen from Table 1, the present invention performs well in several key indicators, especially in the two core indicators of accuracy and weighted Kappa coefficient, which are both optimal, reaching 89.00% and 92.42% respectively, demonstrating excellent performance in comparison with other models.

[0193] The top five teams on the official leaderboard of the 2021 MICCAI GAMMA Challenge (Smart DSP, Voxelcloud, EyeStar, HZL, and IBME) demonstrated good accuracy, with HZL achieving 83.00% accuracy and Smart DSP achieving a weighted Kappa coefficient of 85.49%. However, these methods may still employ relatively simple feature concatenation or weighted averaging as shallow fusion strategies for multimodal fusion, failing to fully utilize the deep semantic connections and conditional dependencies between the CFP and OCT modalities, thus limiting further improvements in model performance.

[0194] MSAN employs a modality-specific attention network, extracting modality-specific features by designing specialized attention mechanisms for different modalities. This method achieves an accuracy of 70.00%, an F1 score of 64.80%, and a weighted Kappa coefficient of 71.51%. While MSAN considers the characteristics of different modalities, its attention mechanism may fail to effectively model the dynamic interactions between modalities, resulting in limited multimodal fusion performance.

[0195] MM-CNN employs a two-stream CNN architecture, processing features from different modalities through two parallel CNN branches before feature fusion. This method achieves an accuracy of 78.00%, an F1 score of 71.38%, and a weighted Kappa coefficient of 78.84%. While MM-CNN's two-stream design effectively extracts features from each modality, its fusion strategy may be relatively simple, failing to fully consider the semantic correspondences between different modalities, particularly in handling the conditional dependencies between CFP spatial feature maps and OCT global semantic vectors.

[0196] DeepGuide employs a deep multimodal guidance strategy, training modality-specific classifiers and then using a guidance model to direct the learning of weaker modalities. This method achieved an accuracy of 80.00%, an F1 score of 74.02%, and a weighted Kappa coefficient of 81.77%. While DeepGuide's guidance mechanism can improve model performance to some extent, its guidance strategy may not fully utilize the dynamic modulation capabilities of OCT global information on CFP spatial features, limiting the depth of cross-modal feature interaction.

[0197] CRD-Net employs a cross-modal attention network to model the relationships between different modalities through an attention mechanism. This method achieves an accuracy of 79.00%, an F1 score of 72.84%, a weighted Kappa coefficient of 82.44%, and an AUC of 91.95%. While CRD-Net's cross-modal attention mechanism can capture intermodal relationships, its attention computation may be based on global features, failing to effectively model the fine-grained conditional dependencies between spatial feature maps and global semantic vectors.

[0198] EyeMost employs a confidence-aware multimodal learning strategy, balancing the contributions of different modalities by introducing a confidence mechanism. This method achieves an accuracy of 74.00%, an F1 score of 54.29%, and a weighted Kappa coefficient of 83.01%. Although EyeMost's weighted Kappa value is relatively high, its accuracy and F1 score are relatively low. This may be because its confidence mechanism performs less effectively in handling class imbalance, resulting in limited recognition ability for certain classes.

[0199] MATCNN employs a fusion method based on multi-scale CNNs and attention Transformers, originally designed for the fusion of infrared and visible light images. In a glaucoma grading task, MATCNN achieved an accuracy of 64.00%, an F1 score of 56.86%, and a weighted Kappa coefficient of 68.81%. The relatively low performance of MATCNN may be due to its initial design focus on the fusion of infrared and visible light images. When handling multimodal fusion tasks in medical images, it failed to adequately consider the unique structural features of CFP and OCT images (such as the circular structure in fundus images and the tissue layer structure in OCT images), resulting in poor feature extraction and fusion performance.

[0200] M2AFNet employs a Mamba-based multimodal CLIP-assisted fusion network, combining the selective state-space model of the Mamba architecture with the visual-language alignment capability of CLIP. This method achieves an accuracy of 81.00%, an F1 score of 76.20%, a weighted Kappa coefficient of 78.37%, and an AUC of 93.91%. While M2AFNet exhibits relatively good performance, its fusion strategy may not fully utilize the dynamic modulation capability of OCT global information on CFP spatial features, particularly in the design of the conditionalization mechanism, which may not be as refined as the conditional state-space fusion module of GLA-Net.

[0201] To verify the impact of the circular structure perception attention module, layer prior enhanced axial attention, and conditional state space fusion module on model performance, relevant ablation experiments were conducted. Table 2 shows the ablation experiment results for the circular structure perception attention module, layer prior enhanced axial attention, and conditional state space fusion module. The baseline model is the model without using any modules.

[0202] Table 2 Ablation Experiment Results for Different Modules

[0203]

[0204] In Table 2, "×" indicates that the corresponding module is not used, and "√" indicates that the corresponding module is used. Table 2 shows that the baseline model achieved 73.00% accuracy and 77.62% weighted Kappa value. Introducing CSPM, LA3, or CSFF individually all improved model performance. When all three modules were introduced simultaneously, the model achieved 89.00% accuracy and 92.42% weighted Kappa value, representing improvements of 16.00% and 14.80% compared to the baseline model. These results demonstrate that the synergistic effect of the three modules significantly improves model performance. The circular structure perception attention module enhances the CFP branch's ability to perceive circular structures, the layer prior enhancement axial attention module enhances the OCT branch's ability to represent retinal tissue layer structures, and the conditional state space fusion module achieves deep fusion of features from the two branches. All three are indispensable and together constitute the core advantages of GLA-Net.

[0205] To verify the impact of different feature extraction backbone networks on model performance, ablation experiments were conducted with different backbone networks while keeping CSPM, LA3, and CSFF constant. The results of the ablation experiments with different backbone networks are shown in Table 3. The backbone networks used included ResNet50, ResNext50, MobileNetV3, ConvNext, ViT-Base / 16, DeiT-Base / 16, SwinTransformer, Transformer-UNet, and EfficientNet-B0.

[0206] Table 3 Ablation Experiment Results for Different Backbone Networks

[0207]

[0208] As shown in Table 3, traditional ResNet and ResNext perform reasonably well in terms of accuracy and weighted Kappa, but have high parameter counts and FLOPs. MobileNetV3, while the most lightweight, shows a slight performance decrease. Transformer backbones such as ViT-Base / 16, DeiT-Base / 16, SwinTransformer, and Transformer-UNet have significantly higher parameter counts and computational complexity than convolutional backbones, yet fail to deliver proportional performance improvements. Considering accuracy, weighted Kappa coefficients, parameter count, and floating-point operations (FLOPs), EfficientNet-B0 achieves an accuracy of 89.00% and a weighted Kappa coefficient of 92.42% while maintaining 37.11M parameters and approximately 201G FLOPs, achieving the optimal balance between performance and efficiency. Therefore, this invention selects EfficientNet-B0 as the feature extraction backbone network, achieving a balance between model lightweighting and computational efficiency while maintaining high performance.

[0209] Experimental results show that by effectively addressing the problems of insufficient multimodal fusion and limited feature extraction capabilities, this invention demonstrates superior grading performance in glaucoma grading tasks, with all indicators outperforming other advanced grading methods, especially in weighted Kappa.

[0210] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0211] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0212] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0213] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0214] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A glaucoma classification system based on multimodal conditional state-space fusion, characterized in that, include: The data acquisition module is used to acquire CFP images and OCT images; A glaucoma grading network model construction module is used to construct a glaucoma grading network model based on multimodal conditional state space fusion. This model includes a circular structure perception attention module, a layer prior enhanced axial attention module, and a conditional state space fusion module. The circular structure perception attention module enhances the representation ability of circular key pathological change regions in CFP images through multi-radius soft circular convolutional kernels. The layer prior enhanced axial attention module enhances the capture ability of long-range dependencies in the vertical direction and improves the utilization of key pathological change information in OCT images. The conditional state space fusion module models long-range dependencies of spatial sequences through a selective scanning mechanism and combines this with a mechanism that dynamically modulates CFP spatial features using global OCT information to achieve fine-grained, adaptive cross-modal feature interaction. The model training module is used to train the glaucoma hierarchical network model based on multimodal conditional state space fusion. The glaucoma grading module is used to input the CFP image and OCT image to be graded into the trained glaucoma grading network model based on multimodal conditional state space fusion, and obtain the glaucoma grading result.

2. The glaucoma classification system based on multimodal conditional state-space fusion according to claim 1, characterized in that: The circular structure perception attention module enhances the representation ability of circular key pathological change regions in CFP images through multi-radius soft circular convolution kernels, including: Important feature channels are highlighted through channel attention to obtain enhanced feature maps. Multi-radius soft circular spatial attention mechanism is used on the enhanced feature maps. By constructing three parallel circular attention branches, multi-scale information is captured to obtain spatial attention weights, thereby enhancing the representation ability of key circular regions such as the visual disk and visual cup in CFP images. The spatial attention weights are multiplied element-wise with the channel-enhanced feature map to obtain the output feature map of the circular structure perception attention module.

3. The glaucoma classification system based on multimodal conditional state-space fusion according to claim 2, characterized in that: The enhanced feature map of the channel employs a multi-radius soft circular spatial attention mechanism, which captures multi-scale information by constructing three parallel circular attention branches to obtain spatial attention weights, including: Max pooling and average pooling are performed on the channel-enhanced feature map in the channel dimension to obtain two spatial feature maps. The two spatial feature maps are then concatenated in the channel dimension to obtain a dual-channel feature map. Multiple parallel soft circular attention branches are constructed, each using a soft circular convolution kernel to capture circular structures at different scales; the learnable parameters of each soft circular attention branch include radius and boundary smoothing coefficient, wherein the radius is the radius parameter of the soft circular mask used to control the range covered by the circular attention, and the boundary smoothing coefficient controls the decay rate of the circular boundary from the center to the periphery; For each soft circular attention branch, with the center of the convolution kernel as the origin, calculate the distance from each position to the center and generate a soft circular mask for each position; multiply the standard convolution weights element-wise with the soft circular mask for each position to obtain the effective weights of each soft circular attention branch. The effective weights of each soft circular attention branch are used to perform a convolution operation on the dual-channel feature map to obtain the circular attention weights corresponding to each soft circular attention branch. The circular attention weights corresponding to all soft circular attention branches are concatenated along the channel dimension, then fused through convolution, and finally generated by an activation function to produce the spatial attention weights.

4. The glaucoma classification system based on multimodal conditional state-space fusion according to claim 1, characterized in that: The layer-prior enhanced axial attention module enhances the ability to capture long-range dependencies in the vertical direction and improves the utilization of key pathological changes in OCT images, including: The layer prior enhanced axial attention module includes a vertical branch and a horizontal branch. The input to the layer prior enhanced axial attention module is denoted as... ,right One-dimensional axial attention calculation is performed along the vertical direction to obtain the output features of the vertical branch; The output features of the vertical branch are subjected to one-dimensional axial attention along the horizontal direction to obtain the output feature map of the layer prior enhancement axial attention module.

5. The glaucoma classification system based on multimodal conditional state-space fusion according to claim 4, characterized in that: The pair One-dimensional axial attention calculation is performed along the vertical direction to obtain the output features of the vertical branch, including: right Perform dimensional permutation and reshaping operations, and the resulting sequence feature map is denoted as... ; Using a multi-head attention mechanism The mapping is done as query vector, key vector, and value vector, and the content similarity score is calculated by combining the query vector, key vector, and value vector. The relative position offset is obtained by using a direct index offset table scheme for relative position encoding. Use vertical strip convolution kernels Convolution is performed, and the convolution result is enhanced by an activation function to reshape it into a sequence format, yielding the layer prior features, denoted as... , For the layer prior features corresponding to the i-th attention head, , For the number of heads; Applying normalization along the sequence dimension yields a normalized layer prior sequence, denoted as... , Let be the standardized layer prior sequence corresponding to the i-th attention head; Expanded to a bias tensor that matches the content similarity score dimension, denoted as , Let i be the bias tensor corresponding to the i-th attention head; The relative position bias and bias tensor are added to the content similarity score to obtain the bias attention weight in the vertical direction. The vertical bias attention weights are multiplied by the value vector to obtain the output of each attention head; The outputs of all attention heads are concatenated along the channel dimension and then fused. The fusion result is then dimensionally reshaped to obtain the output features of the vertical branch.

6. The glaucoma classification system based on multimodal conditional state-space fusion according to claim 5, characterized in that: The method for calculating the vertical bias attention weight is as follows: ， In the formula, Let be the bias attention weight in the vertical direction corresponding to the i-th attention head. For the Softmax function, Let be the content similarity score of the i-th attention head. Let λ be the relative position offset of the i-th attention head, and λ be an adjustable weight coefficient. Let be the bias tensor corresponding to the i-th attention head.

7. The glaucoma classification system based on multimodal conditional state-space fusion according to claim 1, characterized in that: The conditional state space fusion module models long-distance dependencies in spatial sequences through a selective scanning mechanism. Combined with a mechanism that dynamically modulates CFP spatial features using OCT global information, it achieves fine-grained, adaptive cross-modal feature interaction, including: The output feature map of the layer prior enhancement axial attention module is used to generate modulation features through a multilayer perceptron and a broadcast mechanism; The output feature map of the layer prior enhancement axial attention module is used to generate a state transition matrix and a time step through two parallel, parameter-independent multilayer perceptrons. A selective scanning mechanism is adopted to model the modulation features in one-dimensional state space along both row and column directions based on the state transition matrix and time step. The dynamic interaction of cross-modal features is realized through the state space conditionalization mechanism to obtain row scanning features and column scanning features. By fusing row scan features and column scan features, multimodal fusion features are obtained.

8. The glaucoma classification system based on multimodal conditional state-space fusion according to claim 7, characterized in that: The output feature map of the layer prior enhancement axial attention module is used to generate modulation features through a multilayer perceptron and a broadcast mechanism, specifically: The output feature map of the layer prior enhanced axial attention module is used to generate FiLM parameters through a multilayer perceptron. The FiLM parameters are then dimensionally reshaped to obtain scaling and offset parameters. These parameters are then used to modulate the output feature map of the circular structure perceptual attention module via a broadcast mechanism, resulting in the modulated features: ， In the formula, For modulation features, ⊙ indicates element-wise multiplication. This is the output feature map of the circular structure perception attention module. For scaling parameters, This is the offset parameter.

9. The glaucoma classification system based on multimodal conditional state-space fusion according to claim 7, characterized in that: The state transition matrix is ​​calculated as follows: the output feature map of the layer prior enhancement axial attention module is subjected to MLP transformation to generate state increment adjustment values; the state increment adjustment values ​​are then dimensionally reshaped to obtain the dimensionally reshaped state increment adjustment values, denoted as . Introducing learnable initial bias parameters and performing dimensional reshaping yields the dimensional reshaping bias parameters, denoted as... ,Will and Add them together and obtain the state transition matrix through an exponential function; The time step is calculated as follows: The output feature map of the layer prior enhancement axial attention module is subjected to MLP transformation to generate a step size increment adjustment value. The MLP internal parameters of the MLP transformation used to generate the step size increment adjustment value and the MLP transformation used to generate the state increment adjustment value are not shared. The step size increment adjustment value is then dimensionally reshaped to obtain the dimensionally reshaped step size increment adjustment value, denoted as... ;right After the function is activated, the time step is multiplied by the step scaling factor to obtain the time step.

10. The glaucoma classification system based on multimodal conditional state-space fusion according to claim 7, characterized in that: When performing one-dimensional state-space modeling of the modulation features along both row and column directions, the modulation features are scanned in rows to obtain the row scanning result, specifically as follows: The modulation features are reshaped into a sequence format row by row to obtain a row sequence, denoted as . After applying layer normalization to each sequence in the row sequence, a selective scan is performed to obtain the row scan result: ， In the formula, Indicates the row scan results. Representation layer normalization, Represents the state transition matrix. Indicates the time step. Indicates selective scanning; The state update equation for selective scanning is: ， In the formula, This represents the row scan result at time t. This represents the state vector at time t. For learnable parameters, Represents the input and corresponding time t. The t-th element in The calculation method is as follows: ， In the formula, This represents the natural exponential function. This represents element-wise product.

Images