A multi-modal glaucoma grading diagnostic method

Abstract

The present application relates to the field of medical artificial intelligence and computer-aided diagnosis, in particular to a multi-modal glaucoma grading diagnosis method, comprising: first, generating the representation of the missing modality at the feature semantic level through the cross-modal adversarial generative network, and constructing a complete multi-modal feature set; second, mapping the features to the anatomical region based on the pre-defined structured anatomical atlas, generating the integrated sign description with confidence through local sign recognition and graph neural network inference; finally, adopting the hierarchical fusion framework based on evidence theory, synthesizing the evidence in the clinical sub-dimensions such as optic disc morphology, retinal nerve fiber layer structure and visual field function, and then performing global fusion according to the uncertainty of each sub-dimension to output the grading diagnosis conclusion and the structured diagnosis report. The present application significantly improves the diagnosis robustness under the condition of missing modality, and provides transparent and verifiable clinical decision basis.

Description

Technical Field

[0001] This invention relates to the field of medical artificial intelligence and computer-aided diagnostic technology, specifically a multimodal glaucoma grading diagnostic method. Background Technology

[0002] Accurate staging of glaucoma is crucial for clinical treatment decisions. Currently, standard diagnostic procedures rely on ophthalmologists' comprehensive interpretation of multimodal information, including color fundus photography, optical coherence tomography (OCT), and visual field testing. However, this process is highly dependent on physician experience, exhibiting drawbacks such as strong subjectivity and time-consuming nature. Existing AI-based assisted diagnostic solutions face fundamental challenges in real-world clinical scenarios: First, most models require complete multimodal data input, which is severely out of sync with the prevalent incomplete data modality in primary healthcare institutions within a tiered healthcare system; second, existing feature fusion methods fail to incorporate medical knowledge, unable to simulate the reasoning process of physicians using anatomical and functional correlations for cross-validation; and finally, most mainstream models are "black boxes," lacking transparency in their decision-making logic.

[0003] Therefore, there is an urgent need for an intelligent grading and diagnostic method for glaucoma that can proactively address modality loss, ensure the fusion process aligns with clinical understanding, and provide transparent and traceable decision-making criteria. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal glaucoma grading and diagnostic method to solve the problems mentioned in the background art.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a multimodal glaucoma grading diagnostic method, comprising the following steps: S1: Multimodal data reception and availability marking: Receive input fundus examination data, which includes at least one modality among color fundus photography, optical coherence tomography, and visual field examination; automatically detect and mark the availability status of each modality; S2: Deep feature extraction and cross-modal semantic completion: Extract deep feature vectors from available modal data and input them into a pre-trained cross-modal adversarial generative network to generate semantic feature vectors of missing modalities, thus forming a complete set of multimodal semantic features; S3: Hierarchical feature extraction and encoding based on anatomical atlas: The semantic feature set is mapped to a predefined structured anatomical atlas region; local feature description vectors for each region are generated through a local network, and graph attention network is used to aggregate and generate integrated feature descriptions with confidence. S4: Evidence synthesis of clinical sub-dimensions based on evidence theory: The integrated description of signs is classified into clinical sub-dimensions that include at least optic disc morphology, retinal nerve fiber layer structure and visual field function. Within each sub-dimension, Dempster-Shafer evidence theory is used to synthesize evidence to obtain the reliability allocation and uncertainty measure of each sub-dimension. S5: Uncertainty-weighted global fusion and diagnostic report generation: Calculate the decision weight of each sub-dimension based on its uncertainty measure, perform weighted fusion of the sub-dimension reliability assignments to obtain the global reliability assignment, determine the hierarchical diagnostic conclusion based on this, and generate a structured diagnostic report containing the conclusion, basis and uncertainty explanation.

[0006] Furthermore, the cross-modal adversarial generative network in step S2 adopts a training strategy that generates a third modal feature based on any two modal features, and is optimized through adversarial loss and cycle consistency loss.

[0007] Furthermore, in step S3, the structured anatomical atlas uses a polar coordinate grid centered on the optic disc to divide the retinal region; the graph attention network is constructed with local signs as nodes and anatomical adjacency relationships between regions as edges.

[0008] Furthermore, in step S4, the uncertainty measure of each clinical sub-dimension is quantified by the "unknown" reliability value generated after the evidence is synthesized within that sub-dimension.

[0009] Furthermore, in step S5, the calculation of the sub-dimension decision weight is inversely proportional to the uncertainty measure of that sub-dimension.

[0010] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: by performing adversarial generative completion in the feature space, the problem of model dependence on data integrity is fundamentally solved, and the performance degradation in the case of modality missing is significantly lower than that of traditional methods; through the chain reasoning of "feature → local signs → integrated signs → sub-dimensional evidence → global decision", the entire decision-making path is completely transparent and auditable, and the generated structured diagnostic report greatly improves clinical adoption; the system not only provides staging, but also points out the core lesion area and the main damaged modalities, providing a direct basis for formulating targeted treatment and follow-up plans. Attached Figure Description

[0011] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating the overall system workflow of the method of this invention; Figure 2 This is a schematic diagram illustrating the structure and operation of a cross-modal adversarial generative network. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] Please see Figure 1-2 The present invention provides a technical solution: a multimodal glaucoma grading diagnosis method, comprising the following steps: S1: Multimodal Data Reception and Availability Marker It receives input fundus examination data, which includes at least one modality from color fundus photography, optical coherence tomography, and visual field examination; it automatically detects and marks the availability status of each modality, and records the missing status of unavailable modalities.

[0014] S2: Deep Feature Extraction and Cross-Modal Semantic Completion For each modality data marked as available in step S1, a high-dimensional deep feature vector is extracted through its corresponding pre-trained deep encoder network; The extracted deep feature vectors of all available modalities are input into a pre-trained cross-modal adversarial generative network. The network infers and generates semantic feature vectors corresponding to the missing modalities based on the input available modal features. All the real extracted and generated completed semantic feature vectors are normalized and combined to form a complete and standardized multimodal semantic feature set.

[0015] S3: Hierarchical Sign Extraction and Encoding Based on Anatomical Atlas A predefined digital structured anatomical atlas divides the posterior pole of the retina into multiple clinically significant regions based on anatomical structures. The multimodal semantic feature set obtained in step S2 is mapped to the corresponding region of the structured anatomical atlas according to its physical origin; Within each atlas region, a lightweight local feature recognition network is used to fuse and analyze the multimodal semantic features mapped to that region, and output a set of quantified local feature description vectors describing the microscopic pathological changes in that region. Using the local sign description vectors of all regions as nodes and the anatomical adjacency relationships and functional transmission paths between regions as edges, a sign relationship graph is constructed. The graph attention network is used to propagate and aggregate information from the sign relationship graph to generate a set of integrated sign descriptions that can reflect cross-regional pathological association patterns and are accompanied by confidence scores.

[0016] S4: Evidence Synthesis in the Clinical Subdimension Based on Evidence Theory Define multiple clinical decision-making sub-dimensions, which include at least the optic disc morphology sub-dimension, the retinal nerve fiber layer structure sub-dimension, and the visual field function sub-dimension; Each integrated sign description generated in step S3 is categorized into one or more corresponding sub-dimensions based on its clinical significance. Within each clinical sub-dimension, the Dempster-Shafer evidence theory is used to convert all integrated symptom descriptions belonging to that sub-dimension into basic probability assignment functions and synthesize evidence to obtain the reliability assignment and uncertainty measure of that sub-dimension for different glaucoma grades.

[0017] S5: Uncertainty-Weighted Global Fusion and Diagnostic Report Generation Based on the uncertainty measure of each clinical sub-dimension calculated in step S4, the weight of each sub-dimension in the final decision is dynamically calculated, and the sub-dimension with higher uncertainty is assigned a lower weight. Using the calculated weights, a weighted average is applied to the reliability distribution of each sub-dimension to obtain a global comprehensive reliability distribution. Based on the overall reliability allocation, the glaucoma grade with the highest reliability was selected as the final grade diagnosis conclusion; Based on the graded diagnostic conclusion, key integrated signs description, and the reliability and uncertainty of each sub-dimension, a structured diagnostic report is automatically generated. The report includes at least the graded conclusion, overall confidence level, a list of core diagnostic criteria, and explanations of missing or highly uncertain evidence.

[0018] 1. System Construction and Deployment Data preparation and module training Data: Training and validation were performed using a complete trimodal fundus dataset of 8,000 cases. All data were labeled with glaucoma staging (0: normal, 1: early, 2: intermediate, 3: late) as confirmed by an expert committee.

[0019] Training cross-modal adversarial generative networks: Structure: The encoder uses ResNet-50, and the generator is a fully connected network with three layers of residual blocks.

[0020] Training strategy: A cycle-consistent training strategy is adopted. One modality is randomly selected as the "missing" modality, and the generator is trained to generate features based on the features of the other two modalities. A discriminator is used to ensure the authenticity of the generated feature distribution.

[0021] Loss function: L = λ1 * L recon + λ2 * L adv+ λ3 * L cycle L recon For reconstruction loss, L adv To combat the losses, L cycle The loss is for cycle consistency. We set λ1=1.0, λ2=0.1, and λ3=0.5.

[0022] Structured anatomical atlas and physical sign recognition network: Atlas definition: Using a polar coordinate system, it is divided into 4 concentric rings (corresponding to the area from the center of the visual disc to the pericenter) and 12 sectors (one every 30 degrees), for a total of 48 basic regions.

[0023] Local feature recognition network: Each region corresponds to the same multilayer perceptron. The input is the spliced ​​multimodal features (256 dimensions in total), and the output is the probability of 5 preset local features (such as "disk edge notch", "RNFL thinning", etc.).

[0024] Evidence fusion rule initialization: The basic probability assignment function is implemented through a learnable matrix, which is initialized by senior ophthalmologists based on clinical experience to score the correlation between common signs and staging (0-1 points), and fine-tuned during training.

[0025] 2. Application Examples Suppose we are processing data from a patient whose only available data are color fundus photography (CFP) and optical coherence tomography (OCT), but visual field testing (VF) is missing.

[0026] Execute S1: The system receives CFP and OCT images and marks the VF mode as "missing".

[0027] Execute S2: Feature F was extracted using both the CFP encoder and the OCT encoder. cfp and F oct .

[0028] F cfp and F oct Input to the generator network to generate pseudo-view features F vf .

[0029] For [F] cfp , F oct , F vf Normalization is performed to obtain a standardized feature set.

[0030] Execute S3: The features are mapped to a 48-zone map. For example, in the “lower temporal quadrant” region, the local network output is: “Disk edge loss: 0.92”, “RNFL thickness value (68μm): 0.05 (very low percentile)”.

[0031] After graph network aggregation, an integrated feature description was generated: "There is clear and consistent evidence of loss of disc rim morphology and structural thinning of the RNFL in the inferior temporal quadrant", with a confidence level of 0.95.

[0032] Execute S4: The aforementioned physical characteristics were simultaneously fed into the "morphological sub-dimension" and the "structural sub-dimension".

[0033] Within the "morphological sub-dimension," this trait supports a confidence level of 0.7 for the "intermediate" phase and an uncertainty level of 0.2.

[0034] Within the "structural sub-dimension", the confidence level supporting the "interim" is 0.8, and the uncertainty is 0.15.

[0035] The output of the "functional sub-dimension" is highly uncertain (close to 0.9) due to the lack of reliable evidence.

[0036] Execute S5: Calculation of weights: Due to the extremely high uncertainty of the functional sub-dimension, its weight is reduced to close to 0; the morphological and structural sub-dimensions have higher weights.

[0037] After weighted fusion, the global confidence level is distributed as follows: early stage: 0.15, middle stage: 0.78, late stage: 0.07.

[0038] Output: Diagnostic conclusion: Glaucoma, intermediate stage.

[0039] Overall confidence level: 83%.

[0040] Structured diagnostic report: "The core diagnostic criteria are clear disc margin loss in the inferotemporal quadrant and significant thinning of the retinal nerve fiber layer. Visual field function assessment could not be performed due to missing data, so the conclusions in this part are highly uncertain. We recommend completing visual field examination." 3. Implementation methods for electronic devices and storage media The computer program that implements the above method can be stored on a computer-readable storage medium (such as a USB flash drive, hard disk, optical disc, or server). When the program is read and executed by a processor (such as a general-purpose CPU or GPU), it enables the electronic device (such as a workstation, server, or cloud computing platform) to complete all steps S1 to S5 above, thereby completing the graded diagnosis of glaucoma.

[0041] Finally, it should be noted that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal glaucoma grading diagnostic method, characterized in that, Includes the following steps: S1: Multimodal data reception and availability marking: Receive input fundus examination data, which includes at least one modality among color fundus photography, optical coherence tomography, and visual field examination; automatically detect and mark the availability status of each modality; S2: Deep Feature Extraction and Cross-Modal Semantic Completion: Extract deep feature vectors from available modal data and input them into a pre-trained cross-modal adversarial generative network to generate semantic feature vectors for missing modalities, thus forming a complete set of multimodal semantic features; S3: Hierarchical feature extraction and encoding based on anatomical atlas: The semantic feature set is mapped to a predefined structured anatomical atlas region; local feature description vectors for each region are generated through a local network, and graph attention network is used to aggregate and generate integrated feature descriptions with confidence. S4: Evidence synthesis of clinical sub-dimensions based on evidence theory: The integrated description of signs is classified into clinical sub-dimensions that include at least optic disc morphology, retinal nerve fiber layer structure and visual field function. Within each sub-dimension, Dempster-Shafer evidence theory is used to synthesize evidence to obtain the reliability allocation and uncertainty measure of each sub-dimension. S5: Uncertainty-weighted global fusion and diagnostic report generation: Calculate the decision weight of each sub-dimension based on its uncertainty measure, perform weighted fusion of the sub-dimension reliability assignments to obtain the global reliability assignment, determine the hierarchical diagnostic conclusion based on this, and generate a structured diagnostic report containing the conclusion, basis and uncertainty explanation.

2. The multimodal glaucoma grading diagnostic method according to claim 1, characterized in that, The cross-modal adversarial generative network in step S2 employs a training strategy that generates a third modal feature based on any two modal features, and is optimized using adversarial loss and cycle consistency loss.

3. The multimodal glaucoma grading diagnostic method according to claim 1, characterized in that, The structured anatomical atlas in step S3 uses a polar coordinate grid centered on the optic disc to divide the retinal region; the graph attention network is constructed with local signs as nodes and anatomical adjacency relationships between regions as edges.

4. The multimodal glaucoma grading diagnostic method according to claim 1, characterized in that, In step S4, the uncertainty measure of each clinical sub-dimension is quantified by the "unknown" reliability value generated after the evidence is synthesized within that sub-dimension.

5. The multimodal glaucoma grading diagnostic method according to claim 1, characterized in that, In step S5, the calculation of the decision weight of the sub-dimension is inversely proportional to the uncertainty measure of that sub-dimension.

Images