Intelligent labeling method and diagnosis system for fundus lesions based on three-dimensional reconstruction

Abstract

The application relates to the technical field of ophthalmic medical diagnosis, and discloses an intelligent fundus lesion marking method based on three-dimensional reconstruction and a diagnosis system.The method comprises the following steps: receiving fundus color photos, OCT images and FFA images and other multi-modal image data streams of ophthalmic patients; using a pre-trained lesion feature fusion model to perform spatial registration and feature fusion, generating a three-dimensional lesion probability distribution graph and a lesion category confidence matrix; constructing an adaptive marking threshold model to generate a multi-modal marking instruction set; simulating lesion development based on a lesion development chain model and optimizing the parameters of the marking instruction set; and through iterative optimization of a distributed reinforcement learning framework, outputting a lesion marking action sequence to an ophthalmic diagnosis platform.The application can integrate multi-modal image information, improve diagnosis accuracy and efficiency, realize personalized diagnosis, rationally utilize resources and provide strong support for ophthalmic disease diagnosis.

Description

Technical Field

[0001] This invention relates to the field of ophthalmic medical diagnostic technology, specifically to a method and diagnostic system for intelligent annotation of fundus lesions based on three-dimensional reconstruction. Background Technology

[0002] Eye diseases pose a serious threat to human visual health, affecting a vast number of people worldwide. Early and accurate diagnosis is crucial for effective treatment and preventing irreversible vision loss. With advancements in medical technology, multimodal ophthalmic imaging techniques, such as fundus photography, optical coherence tomography (OCT), and fluorescein fundus angiography (FFA), are widely used in clinical practice. These technologies present the structure and characteristics of the eye from different perspectives, providing rich information for disease diagnosis, but also presenting numerous challenges.

[0003] In clinical practice, the integration and analysis of multimodal images faces challenges. On one hand, different modalities of imaging differ significantly in their imaging principles, resolution, and focus of observation. Color fundus photography can directly display changes in the color and morphology of retinal blood vessels; OCT can clearly present the microscopic structure of various layers of ocular tissue; and FFA focuses on blood circulation and leakage in retinal blood vessels. Doctors must interpret these images separately and then make a comprehensive judgment based on experience, a cumbersome process. Furthermore, due to varying levels of professional expertise and clinical experience among doctors, the understanding and judgment of images are subjective, leading to inconsistent and inaccurate diagnostic results and potentially delaying treatment.

[0004] On the other hand, existing diagnostic methods are mostly based on two-dimensional image analysis, which cannot fully explore the three-dimensional spatial information of multimodal images. Two-dimensional annotation is difficult to accurately determine the depth and spatial relationship of lesions in ocular tissues, which can easily lead to missed diagnoses of deep lesions or lesions with complex three-dimensional structures. Moreover, two-dimensional analysis cannot dynamically track the development of lesions and predict their future trends, which is not conducive to the formulation of scientific treatment plans and long-term disease monitoring plans.

[0005] Furthermore, there are many types of ophthalmic diseases, and the characteristics of lesions are often similar. For example, early-stage diabetic retinopathy and retinal vein occlusion may have similar fundus imaging features, making accurate differentiation difficult using traditional imaging analysis methods alone, which poses a significant challenge to clinical diagnosis. At the same time, traditional diagnostic processes lack effective resource management, often resulting in irrational resource allocation during labeling and diagnosis, leading to resource waste and increased medical costs.

[0006] With the rapid development of artificial intelligence technology, its application in the diagnosis of ophthalmic diseases has become a research hotspot. However, existing intelligent diagnostic solutions still have many shortcomings and cannot fully meet clinical needs. Therefore, it is urgent to develop a fundus lesion diagnostic technology that can efficiently integrate multimodal image information, achieve intelligent annotation and accurate diagnosis, and make rational use of medical resources. This technology is of great significance for improving the level of ophthalmological medical care and protecting patients' visual health. Summary of the Invention

[0007] The purpose of this invention is to provide a method and diagnostic system for intelligent annotation of fundus lesions based on three-dimensional reconstruction, so as to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a method and diagnostic system for intelligent annotation of fundus lesions based on three-dimensional reconstruction, the method comprising:

[0009] Receive multimodal image data streams from ophthalmology patients, the data streams including fundus color images, optical coherence tomography (OCT) images, and fluorescein fundus angiography (FFA) images;

[0010] Based on a pre-trained lesion feature fusion model, spatial registration and feature fusion are performed on the multimodal image data to generate a three-dimensional lesion probability distribution map and a lesion category confidence matrix.

[0011] Based on the three-dimensional lesion probability distribution map, an adaptive annotation threshold model is constructed to generate a multimodal annotation instruction set, which includes an annotation region topology network and a dynamic allocation strategy for annotation levels.

[0012] Based on a pre-trained chain model of lesion development, the cascading effect of lesion development within a preset time window is simulated, and the collaborative decision parameters in the labeled instruction set are optimized.

[0013] The collaborative decision parameters are iteratively optimized using a distributed reinforcement learning framework, and the lesion-labeled action sequence is output to the ophthalmology diagnostic platform.

[0014] Preferably, the steps for constructing the lesion feature fusion model include:

[0015] Collect a database of fundus lesion cases and construct a multidimensional feature training set that includes abnormal imaging patterns, physiological structure data, and lesion triggering conditions.

[0016] The multidimensional feature training set is decoupled by latent variables through a deep variational autoencoder to extract independent representations of the dominant and secondary features of the lesion.

[0017] By combining the lesion evolution dynamics equation, differential constraints on the nonlinear coupling relationship between features are constructed;

[0018] The differential constraints are embedded into a graph convolutional network to generate the lesion feature fusion model that supports incremental learning.

[0019] Preferably, the adaptive labeling threshold model includes:

[0020] Based on the gradient changes in the three-dimensional lesion probability distribution map, the lesion probability density levels are dynamically classified.

[0021] Based on the complexity of the patient's ocular physiological structure and the risk index of past medical history, the sensitivity score of lesion impact is calculated;

[0022] The sensitivity score and probability density level are non-linearly mapped using a logistic function to generate an individual-specific annotation threshold.

[0023] The activation condition for triggering the multi-level annotation response protocol is based on the threshold.

[0024] Preferably, the steps for constructing the lesion development chain model include:

[0025] Collect secondary lesion association data from historical fundus lesion events to construct a lesion causal graph dataset;

[0026] The transmission probability and delay time parameters between lesion events are extracted using a causal inference algorithm;

[0027] By combining complex network theory, a directed acyclic graph of lesion propagation is constructed to quantify the vulnerability dependency strength between nodes;

[0028] The dependence strength and real-time physiological perturbation factors are input into the spatiotemporal capsule network to generate the lesion development chain model.

[0029] Preferably, the method further includes:

[0030] Based on the simulation results of the lesion development chain model, key cascading interruption nodes are identified;

[0031] Configure a blocking and intervention annotation strategy for the node in the annotation instruction set;

[0032] Based on the aforementioned blocking and intervention annotation strategy, a cross-modal collaborative annotation scheme is automatically generated, including annotation priorities and annotation fusion rules for different image modalities.

[0033] Preferably, the calculation of the lesion impact sensitivity score includes:

[0034] Obtain a real-time matrix of ocular physiological function indicators and a heatmap of genetic risk factors to construct an individual resistance cube for lesion resilience assessment;

[0035] High-order correlation weights between multidimensional features are calculated using a hypergraph attention network.

[0036] The correlation weights are combined with the evaluation cube using tensor shrinking to obtain a comprehensive sensitivity score.

[0037] The formula for calculating the comprehensive sensitivity score is as follows:

[0038]

[0039] In the formula, S represents the comprehensive sensitivity score, and α i β represents the vulnerability index of the i-th type of physiological structure. i γ represents the priority weight for the i-th type of physiological structure. c This represents a constant representing an individual's baseline resistance to lesions. Let represent the tensor Kronecker product, and n represent the total number of physiological structural categories.

[0040] Preferably, the embedding of the differential constraint conditions includes:

[0041] Lagrange stability analysis was performed on the decoupling results of the hidden variables to screen physically realizable coupling modes;

[0042] Characteristic evolution trajectories that conform to dynamic constraints are generated by Markov chain Monte Carlo sampling;

[0043] The edge weights of the graph convolutional network are trained using trajectory data to ensure that the model output conforms to the physical laws of lesion development.

[0044] Preferably, the execution of the distributed reinforcement learning framework includes:

[0045] Define a reward function for multi-agent collaborative decision-making, which includes the dual objectives of lesion labeling accuracy and resource utilization efficiency;

[0046] The policy network of each modality-labeled agent is trained using a hierarchical meta-learning strategy.

[0047] In each round of training, the credit allocation weights among agents are dynamically adjusted according to the degree of target conflict.

[0048] Output the lesion-labeled action sequence that satisfies the Pareto optimality condition;

[0049] The reward function is as follows:

[0050] R=ω·C a +(1-ω)·η r

[0051] In the formula, R represents the reward value, and C a Indicates the accuracy of lesion labeling and scoring, η r This represents the resource utilization efficiency score, where ω is the dynamic balance factor;

[0052] The quantification of the resource utilization efficiency target includes:

[0053] Establish a spatiotemporal utility decay model for multi-type labeled resources and define a resource idle penalty function;

[0054] Based on the lesion evolution phase diagram, calculate the marginal utility value of resource scheduling in each time period;

[0055] The marginal utility value is used as the dynamic gain coefficient of the reward function;

[0056] The calculation formula for the attenuation model is as follows:

[0057]

[0058] In the formula, K j τ represents the initial utility value of the j-th labeled resource. j x represents the aging decay coefficient. j (t) represents the indicator function for the enabled status of labeled resources at time t, and m represents the total number of labeled resource types.

[0059] Preferably, the method further includes:

[0060] After configuring the blocking intervention labeling strategy, the state transition probability of the cascaded interruption nodes is monitored in real time;

[0061] If the transition probability exceeds the preset critical value, the simulated annealing optimization mechanism is triggered to re-plan the spatiotemporal collaborative scheme for cross-modal annotation.

[0062] Preferably, the present invention also includes an intelligent diagnostic system for fundus lesions based on three-dimensional reconstruction, the system comprising:

[0063] Data receiving module: used to receive multimodal image data streams from ophthalmology patients, including fundus color images, optical coherence tomography (OCT) images, and fluorescein fundus angiography (FFA) images;

[0064] Feature fusion and analysis module: Based on a pre-trained lesion feature fusion model, spatial registration and feature fusion are performed on the multimodal image data to generate a three-dimensional lesion probability distribution map and a lesion category confidence matrix;

[0065] Adaptive annotation module: Based on the three-dimensional lesion probability distribution map, construct an adaptive annotation threshold model and generate a multimodal annotation instruction set, which includes an annotation region topology network and a dynamic allocation strategy for annotation levels;

[0066] Lesion development simulation module: Based on a pre-trained lesion development chain model, it simulates the cascading effect of lesion development within a preset time window in the future, and optimizes the collaborative decision parameters in the labeled instruction set;

[0067] Decision optimization and output module: Iteratively optimizes the collaborative decision parameters through a distributed reinforcement learning framework, and outputs the lesion-annotated action sequence to the ophthalmology diagnostic platform.

[0068] Compared with the prior art, the beneficial effects of the present invention are:

[0069] Regarding diagnostic accuracy, the system receives multimodal image data streams and utilizes a pre-trained lesion feature fusion model for spatial registration and feature fusion, generating a three-dimensional lesion probability distribution map and a lesion category confidence matrix. This approach integrates the advantageous information from different modalities of imagery, enabling precise presentation of lesion characteristics from multiple dimensions. Taking glaucoma diagnosis as an example, traditional methods may rely solely on a single intraocular pressure index or two-dimensional fundus optic nerve images for judgment, easily leading to misdiagnosis. However, this invention, by fusing fundus color photography, OCT images, and FFA images, comprehensively analyzes changes in optic nerve fiber layer thickness, abnormal fundus vascular hemodynamics, and retinal ganglion cell damage in three-dimensional space, significantly improving the accuracy of early glaucoma diagnosis and reducing misdiagnosis and missed diagnosis rates.

[0070] In terms of diagnostic efficiency, the adaptive annotation threshold model generates a multimodal annotation instruction set based on the 3D lesion probability distribution map, including an annotation region topology network and a dynamic annotation level allocation strategy. This enables the system to automatically and quickly locate key annotation regions and allocate annotation levels according to the importance of lesions, eliminating the need for doctors to manually review a large number of images one by one. Simultaneously, the lesion development chain model simulates the cascading effects of future lesion development, providing doctors with valuable predictive information in advance and assisting them in making rapid diagnostic decisions, greatly saving diagnostic time. This is particularly significant in dealing with emergency patients or large-scale screenings, where it can significantly improve work efficiency.

[0071] For personalized diagnosis, the system fully considers individual patient differences. When calculating the sensitivity score for lesion impact, it constructs an individual lesion resilience assessment cube by combining a real-time ocular physiological function index matrix and a genetic risk factor heatmap. A hypergraph attention network is then used to calculate the higher-order correlation weights between multidimensional features, thereby generating individual-specific annotation thresholds. This process ensures that annotation and diagnosis closely align with each patient's physiological condition and genetic background, providing customized diagnostic plans and treatment recommendations, achieving precision medicine, and improving treatment outcomes and patient satisfaction.

[0072] In terms of resource utilization, the distributed reinforcement learning framework defines a reward function that balances lesion labeling accuracy with resource utilization efficiency. By establishing a spatiotemporal utility decay model for multiple types of labeled resources and a resource idle penalty function, the marginal utility value of resource scheduling at each time period is dynamically calculated based on the lesion evolution phase map. This marginal utility value is then used to adjust the dynamic gain coefficient of the reward function, thereby optimizing the credit allocation weights among agents. For example, when diagnostic equipment resources are limited, computing resources are prioritized for urgent and complex cases, avoiding resource idleness and waste, reducing medical costs, improving overall resource utilization efficiency, and enabling a more rational allocation of medical resources.

[0073] Furthermore, based on the simulation results of the chain model of lesion development, key cascade interruption nodes can be identified, blocking intervention annotation strategies can be configured, and cross-modal collaborative annotation schemes can be automatically generated. This enables real-time tracking of lesion development, timely detection of potential risk points, and provides key references for clinical treatment. It helps doctors develop more targeted and forward-looking treatment plans, effectively improve patient prognosis, and enhance the quality of ophthalmological medical services. Attached Figure Description

[0074] Figure 1 This is a schematic diagram illustrating the working principle of the intelligent annotation method for fundus lesions based on three-dimensional reconstruction described in this invention.

[0075] Figure 2 Flowchart for constructing a lesion feature fusion model;

[0076] Figure 3 A flowchart for constructing a chain model of lesion development;

[0077] Figure 4 This is a flowchart illustrating the calculation and application of sensitivity scores for lesion impact. Detailed Implementation

[0078] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0079] Please see Figures 1-4 This invention provides a method and diagnostic system for intelligent annotation of fundus lesions based on three-dimensional reconstruction, the specific implementation steps of which are as follows:

[0080] Receive multimodal image data streams from ophthalmology patients: Acquire multimodal image data from ophthalmology patients, including fundus color images, optical coherence tomography (OCT) images, and fluorescein fundus angiography (FFA) images. These images reflect the physiological structure and pathological conditions of the eye from different angles and levels, providing a rich data foundation for subsequent analysis and diagnosis. Fundus color images can visually present the morphological and color changes of blood vessels, retina, and other tissues in the fundus; OCT images can clearly show the fine structure of various layers of tissues in the eye, such as the layered structure of the retina; and FFA images help to observe blood circulation and leakage in the fundus vessels.

[0081] Based on a pre-trained lesion feature fusion model, spatial registration and feature fusion are performed: The pre-trained lesion feature fusion model is used to process the acquired multimodal image data. Spatial registration aligns images of different modalities in space, enabling them to be analyzed in the same coordinate system. Simultaneously, image features are fused to fully integrate the advantageous information from each modality, generating a 3D lesion probability distribution map and a lesion category confidence matrix. The 3D lesion probability distribution map visually illustrates the possible distribution of lesions in 3D space, while the lesion category confidence matrix reflects the reliability of the judgment of different lesion categories.

[0082] An adaptive annotation threshold model is constructed to generate a multimodal annotation instruction set: Based on the generated 3D lesion probability distribution map, an adaptive annotation threshold model is constructed. This model generates a multimodal annotation instruction set by analyzing the characteristics of the probability distribution map. The instruction set includes an annotation region topology network to determine the regions to be annotated and their interrelationships; and a dynamic annotation level allocation strategy to dynamically allocate annotation levels according to the different conditions and importance of lesions, thereby improving the accuracy and efficiency of annotation.

[0083] Based on a pre-trained lesion development chain model, lesion development is simulated and parameters are optimized: The pre-trained lesion development chain model simulates the cascading effects of lesion development within a preset time window. This simulation provides a deeper understanding of lesion development trends and potential changes. Simultaneously, the collaborative decision parameters in the annotation instruction set are optimized based on the simulation results, enabling the annotations to better adapt to lesion development.

[0084] The collaborative decision-making parameters are iteratively optimized using a distributed reinforcement learning framework to output labeled action sequences. By continuously adjusting the parameters, the annotation results are made more efficient while maintaining accuracy. Finally, the optimized lesion-annotated action sequences are output to an ophthalmology diagnostic platform, providing strong support for doctors' diagnoses.

[0085] The implementation of the present invention will be further described below with reference to Examples 1 to 5.

[0086] Example 1:

[0087] In this embodiment, the construction of the lesion feature fusion model and the embedding process of differential constraints are described in detail.

[0088] First, a database of fundus lesion cases was collected, containing a large amount of data related to fundus lesions. The image abnormality patterns record the unique characteristics of various ophthalmic diseases on different modalities of imaging. For example, retinal lesions may appear as abnormal morphologies such as hemorrhages and exudates on color fundus photography, while on OCT images they may show changes in the interlaminar structure of the retina. Physiological structural data covers normal physiological parameters of the eye, such as retinal thickness and optic nerve fiber layer thickness. This data provides a reference standard for judging whether a lesion has occurred and its severity. Lesion triggering conditions record factors that may lead to the occurrence of lesions, such as the association between systemic diseases like hypertension and diabetes and fundus lesions. Based on this data, a training set containing the above multidimensional features was constructed.

[0089] A deep variational autoencoder (DUE) is used to decouple latent variables from the multidimensional feature training set. By learning the latent distribution of the data, the DUE separates the dominant and secondary features of lesions in the multidimensional features, extracting their independent representations. For example, in diabetic retinopathy, the dominant features directly related to the lesion, such as abnormal vascular dilation and neovascularization, are decoupled from some secondary features, such as minor ocular inflammation that may exist but does not directly cause the lesion.

[0090] By combining the lesion evolution dynamics equation, differential constraints are constructed to establish the nonlinear coupling relationships between features. The lesion evolution dynamics equation describes the dynamic process of lesion development over time. By analyzing the interactions and influences between different features, the nonlinear coupling relationships between them are established. For example, when a microaneurysm (a common fundus lesion) appears on the retina, the changes in hemodynamics of the surrounding blood vessels interact with the metabolic needs of the retinal tissue. This interaction can be described by differential equations, thus constructing the nonlinear coupling relationships between features.

[0091] When embedding differential constraints, Lagrangian stability analysis is performed on the decoupling results of latent variables. Lagrangian stability analysis is used to determine whether a system can remain stable after being subjected to small perturbations, and it filters out physically realizable coupling modes. For example, when analyzing the coupling relationship between retinal vessels and neural tissue, coupling modes that are physically unrealistic are excluded to ensure the model's rationality.

[0092] Markov chain Monte Carlo sampling is used to generate feature evolution trajectories that conform to dynamic constraints. Markov chain Monte Carlo sampling is a method of random sampling in high-dimensional space that can simulate the evolution of features over time under dynamic constraints. These trajectory data are then used to regularize the edge weights of a graph convolutional network (GCNN). The GCNN learns the features of the data by processing information from nodes and edges. Regularization training ensures that the model output conforms to the physical laws of lesion development. For example, during training, the edge weights representing the relationship between blood vessels and retinal tissue in the GCNN are adjusted according to the feature evolution trajectories, enabling the model to more accurately reflect the interaction between the two during lesion development.

[0093] Example 2:

[0094] Based on the gradient changes in the three-dimensional lesion probability distribution map, the lesion probability density levels are dynamically classified. In the three-dimensional lesion probability distribution map, the gradient changes in probability density reflect the changing trend of lesion distribution. By analyzing the gradient, the probability density is divided into different levels. For example, areas with large gradient changes indicate more drastic changes in lesion distribution and may contain more important lesion information; these are classified as high probability density levels. Conversely, areas with smaller gradient changes indicate relatively stable lesion distribution and are classified as low probability density levels.

[0095] A real-time ocular physiological function index matrix and a heatmap of genetic risk factors were acquired to construct an individual lesion resistance resilience assessment cube. The real-time ocular physiological function index matrix included data reflecting the current physiological state of the eye, such as intraocular pressure and electroretinography (ERG). The genetic risk factor heatmap, through analysis of the patient's family history of genetic diseases, presented genetic risk factors related to ophthalmic diseases in the form of a heatmap, with darker colors indicating higher risks. These data were combined to construct the individual lesion resistance resilience assessment cube, comprehensively evaluating an individual's ability to resist lesions from multiple dimensions.

[0096] Hypergraph attention networks are used to calculate higher-order correlation weights between multidimensional features. Hypergraph attention networks can capture complex relationships between multiple features, and can be used to calculate higher-order correlation weights between multidimensional features such as physiological function indicators and genetic risk factors. For example, regarding the relationship between intraocular pressure (IOP) and glaucoma, hypergraph attention networks can analyze the influence weight of IOP on the occurrence and development of glaucoma under different genetic backgrounds and other physiological indicators.

[0097] The overall sensitivity score is obtained by performing a tensor shrinking operation on the association weights and the evaluation cube. The formula for calculating the overall sensitivity score is as follows:

[0098]

[0099] Where S represents the overall sensitivity score, which reflects an individual's sensitivity to lesions. α i This represents the vulnerability index of the i-th type of physiological structure, such as the cellular structures at different levels in the retina. The vulnerability index varies depending on the cell type and function. β i γ represents the priority weight for the i-th type of physiological structure, determined based on the importance of different physiological structures in the diagnosis of ophthalmic diseases. c It represents an individual's baseline resistance to lesions, which comprehensively considers the basic influence of an individual's overall health status and genetic factors on their ability to resist lesions. represents the tensor Kronecker product, used to calculate the product relationship between different tensors. n represents the total number of physiological structure categories, covering all physiological structure categories of the eye involved in the assessment.

[0100] Sensitivity scores and probability density levels are non-linearly mapped using a logistic function to generate individual-specific annotation thresholds. The logistic function maps values ​​from different ranges to a suitable interval. By mapping sensitivity scores and probability density levels using this function, annotation thresholds for each individual are obtained. For example, for regions with high sensitivity scores and high probability density levels, lower annotation thresholds are generated to more promptly annotate potential lesions; conversely, for regions with low sensitivity scores and low probability density levels, higher annotation thresholds are generated to reduce unnecessary annotations.

[0101] The activation conditions for the multi-level annotation response protocol are triggered based on a threshold. When the calculated annotation threshold is compared with the actual lesion probability density, if the lesion probability exceeds the annotation threshold, the corresponding annotation response is triggered. A multi-level annotation response protocol is set according to different threshold ranges. For example, when the lesion probability exceeds a lower threshold, preliminary annotation is initiated, marking the approximate area where lesions may exist; when the lesion probability exceeds a higher threshold, more detailed annotation is performed, including the specific morphology and boundaries of the lesions.

[0102] Example 3:

[0103] This embodiment describes the construction process of a chain model of lesion development. Specifically, it includes:

[0104] Secondary lesion association data were collected from historical fundus lesion events to construct a lesion causal graph dataset. Historical fundus lesion events contain a large amount of patient case data, from which secondary lesion association data was extracted—that is, the correlation information between the appearance of one lesion and the subsequent occurrence of other lesions. For example, in diabetic retinopathy, early microaneurysms may trigger subsequent retinal neovascularization. This association data was compiled into a lesion causal graph dataset.

[0105] The causal inference algorithm extracts the transmission probability and delay time parameters between lesion events. This algorithm analyzes the causal relationships between different lesion events in a causal graph dataset, calculating the transmission probability of one lesion leading to another, and the delay time between the occurrence of two lesions. For example, analysis reveals that in a certain ophthalmic disease, the probability of lesion B developing within 3-6 months after the appearance of lesion A is 70%. Here, 70% is the transmission probability, and 3-6 months is the delay time parameter.

[0106] By combining complex network theory with the construction of a directed acyclic graph (DAG) of lesion propagation, the vulnerability dependency strength between nodes is quantified. Complex network theory is used to study the interrelationships between nodes in complex systems. Fundus lesions are treated as nodes, and a DAG is constructed based on causal relationships. In this graph, the vulnerability dependency strength between nodes is quantified, that is, the degree to which a change in one lesion affects other lesions. For example, in a retinal vascular disease network, the impact of lesions in central vascular nodes on peripheral vascular nodes can be quantified by calculating the connection weights between nodes.

[0107] By inputting vulnerability dependence strength and real-time physiological perturbation factors into a spatiotemporal capsule network, a chain model of lesion development is generated. The spatiotemporal capsule network can simultaneously process information in both spatial and temporal dimensions. Quantified vulnerability dependence strength and real-time physiological perturbation factors (such as current intraocular pressure changes, blood glucose fluctuations, and other factors affecting ocular physiological state) are input into the network. Through network learning and training, a chain model of lesion development is generated. This model can simulate the cascading effects of lesion development within a predetermined time window, providing a basis for subsequent annotation and diagnosis.

[0108] Based on the simulation results of the lesion development chain model, key cascade interruption nodes were identified. In the process of simulating the cascade effects of future lesion development in the lesion development chain model, some nodes emerge that play a crucial role in the entire lesion development process. If these nodes are blocked or intervened in, they may alter the direction of lesion development or slow its rate of development. For example, in the cascade reaction of retinal angiogenesis, a key cellular node that promotes the overexpression of vascular endothelial growth factor is crucial for angiogenesis and has been identified as a key cascade interruption node.

[0109] Configure blocking intervention annotation strategies for key cascade disruption nodes in the annotation instruction set. Once a key cascade disruption node is identified, configure the corresponding blocking intervention annotation strategy for it in the annotation instruction set. For example, for the nodes that promote the overexpression of vascular endothelial growth factor (VEGF), the annotation strategy may include more precise annotation of the region where the node is located, recording its detailed physiological status information, while reminding doctors to pay attention to the potential subsequent lesions caused by the node, and considering appropriate intervention measures, such as drug treatment to inhibit the expression of the factor.

[0110] Based on the blocking intervention annotation strategy, a cross-modal collaborative annotation scheme is automatically generated, including annotation priorities and annotation fusion rules for different imaging modalities. Different imaging modalities have different advantages in observing lesions. The annotation priorities of different imaging modalities are determined according to the needs of the blocking intervention annotation strategy. For example, for key cascade interruption nodes, OCT images may be more helpful in observing their microstructural changes; therefore, the annotation priority of OCT images is increased. Simultaneously, annotation fusion rules are formulated to fuse the annotation information from different imaging modalities. For instance, combining the macroscopic vascular morphology information shown in fundus color imaging with the microstructural information of the same region in OCT images provides a more comprehensive presentation of the lesion.

[0111] After configuring the blocking intervention labeling strategy, the state transition probability of cascade interruption nodes is monitored in real time. By continuously monitoring the state changes of key cascade interruption nodes, their state transition probability, i.e., the possibility of transitioning from the current state to other states, is obtained. For example, the probability of the aforementioned key cell nodes transitioning from a state promoting vascular endothelial growth factor overexpression to a normal state or other abnormal states is monitored.

[0112] If the transition probability exceeds a preset threshold, a simulated annealing optimization mechanism is triggered to redesign the spatiotemporal collaborative scheme for cross-modal annotation. When the state transition probability exceeds the preset threshold, it indicates that the development of the lesion may have changed significantly, and the original annotation scheme may no longer be applicable. At this time, the simulated annealing optimization mechanism is triggered. This mechanism randomly searches for a better solution within a certain range by simulating the physical annealing process. By redesigning the spatiotemporal collaborative scheme for cross-modal annotation, the annotation priority and fusion rules are adjusted to adapt to the new lesion development, ensuring the accuracy and effectiveness of the annotation.

[0113] Example 4:

[0114] This embodiment specifically describes the execution process of the distributed reinforcement learning framework. Specifically, it includes:

[0115] Define a reward function for multi-agent collaborative decision-making, incorporating both lesion labeling accuracy and resource utilization efficiency as dual objectives. The reward function is:

[0116] R=ω·C a +(1-ω)·η r

[0117] Here, R represents the reward value, which comprehensively reflects the quality of the labeling decision. C a This represents the accuracy score for lesion annotation, used to measure the degree of match between the annotation results and the actual lesion situation. A higher score indicates more accurate annotation. η rThis represents the resource utilization efficiency score, reflecting the efficiency of resource utilization during the annotation process. ω is a dynamic balancing factor used to adjust the relative importance of annotation accuracy and resource utilization efficiency in the reward function. For example, when resources are abundant, the value of ω can be appropriately increased to focus more on annotation accuracy; while when resources are limited, the value of ω should be decreased to focus more on resource utilization efficiency.

[0118] A spatiotemporal utility decay model for various types of labeled resources is established, and a resource idleness penalty function is defined. Labeled resources include computing resources, storage resources, and manpower, etc. The calculation formula for the spatiotemporal utility decay model is as follows:

[0119]

[0120] Among them, K j τ represents the initial utility value of the j-th type of labeled resource. Different types of resources have different initial utilities; for example, high-performance computing devices have relatively high initial utility values. j This represents the time-depreciation coefficient, reflecting the rate at which the utility of a resource decays over time. For example, the time-depreciation coefficient of storage resources may be low because the stored data still has value over a longer period. j (t) represents the annotation resource activation status indicator function at time t, when x j When (t) = 1, it means that the resource is enabled at time t, and when x... j When (t) = 0, it indicates that the resource is idle at time t. The resource idle penalty function then penalizes the idle resource, prompting the system to use resources more rationally.

[0121] Based on the lesion evolution phase map, the marginal utility value of resource allocation at each time period is calculated. The lesion evolution phase map shows the characteristics and trends of lesions at different development stages. By analyzing the phase map, the marginal utility value of resource allocation at different time periods is calculated. For example, in the rapid development stage of the lesion, investing more computing resources in real-time analysis of image data may yield a higher marginal utility value, because timely and accurate annotation is crucial for diagnostic and treatment decisions.

[0122] The marginal utility value is used as the dynamic gain coefficient of the reward function. Based on the calculated marginal utility value, the weight of resource utilization efficiency in the reward function is adjusted. When the marginal utility value is high, the weight of resource utilization efficiency in the reward function is increased to incentivize the system to utilize resources more efficiently during that period; when the marginal utility value is low, the weight of resource utilization efficiency is appropriately reduced, with greater emphasis placed on labeling accuracy.

[0123] A hierarchical meta-learning strategy is used to train the policy networks of each modality-annotating agent. This strategy divides the learning process into multiple layers, each focusing on a different learning task. When training the policy networks of each modality-annotating agent, higher-level learning tasks can involve learning collaborative patterns between different modalities, while lower-level learning tasks can involve optimizing annotations for a single modality. This hierarchical learning improves the learning efficiency and adaptability of the agents.

[0124] In each training round, the credit allocation weights among agents are dynamically adjusted based on the degree of target conflict. During multi-agent collaborative annotation, different agents may have conflicting objectives; for example, one agent may prioritize annotation speed while another prioritizes accuracy. The credit allocation weights among agents are dynamically adjusted based on the degree of target conflict. When the conflict is high, the weights are adjusted to balance the objectives of different agents; when the conflict is low, the weights are further optimized based on the actual situation to improve the overall annotation performance. Finally, the output is a lesion annotation action sequence that satisfies the Pareto optimal condition, i.e., an annotation scheme that cannot further improve any performance metric without reducing other performance metrics.

[0125] Example 5:

[0126] This embodiment provides a detailed description of each module of the intelligent diagnostic system for fundus lesions based on three-dimensional reconstruction.

[0127] The data receiving module receives multimodal image data streams from ophthalmology patients, including fundus photography, optical coherence tomography (OCT) images, and fluorescein fundus angiography (FFA) images. The module boasts efficient data transmission and processing capabilities, enabling rapid and accurate acquisition of image data from various devices. It connects to ophthalmic examination equipment via a network interface, receiving image data in real time and performing preliminary preprocessing, such as data format conversion and data integrity checks, to ensure smooth use of the data by subsequent modules.

[0128] The feature fusion and analysis module, based on a pre-trained lesion feature fusion model, performs spatial registration and feature fusion on multimodal image data. This module first calls the pre-trained lesion feature fusion model to accurately register fundus color images, OCT images, and FFA images in space, ensuring alignment of images from different modalities for unified analysis. Then, feature fusion is performed on the registered images to extract lesion-related features from each modality, generating a 3D lesion probability distribution map and a lesion category confidence matrix. When generating the 3D lesion probability distribution map, the location and morphology of lesions in different modalities are comprehensively considered to construct the probability distribution of lesions in 3D space, thus intuitively showing the possible areas where lesions may appear. The lesion category confidence matrix, through in-depth analysis of the fused features and utilizing the model's classification capabilities, calculates the confidence level of different lesion categories, providing a basis for subsequent diagnosis.

[0129] The adaptive annotation module constructs an adaptive annotation threshold model based on the 3D lesion probability distribution map, generating a multimodal annotation instruction set. This module first analyzes the characteristics of the 3D lesion probability distribution map, such as the distribution and trend of probability density. Based on these analysis results, combined with the complexity of the patient's ocular physiological structure and the risk index of past medical history, an adaptive annotation threshold model is constructed. By dynamically dividing the lesion probability density levels, calculating the lesion impact sensitivity score, and performing a nonlinear mapping between the two to generate an individual-specific annotation threshold. Based on this threshold, the activation conditions of the multi-level annotation response protocol are triggered, thereby generating a multimodal annotation instruction set containing the annotation region topology network and a dynamic allocation strategy for annotation levels, achieving accurate and efficient annotation.

[0130] The lesion development simulation module, based on a pre-trained lesion development chain model, simulates the cascading effects of lesion development within a preset time window and optimizes the collaborative decision-making parameters in the annotation instruction set. This module calls the pre-trained lesion development chain model, inputting current lesion information and related physiological parameters. The model predicts lesion changes over a future period by simulating the cascading effects of lesion development, including lesion enlargement and the emergence of new lesions. Based on the simulation results, the collaborative decision-making parameters in the annotation instruction set are optimized, such as adjusting annotation priority and level of detail, so that the annotations better adapt to the lesion development trend, providing doctors with more forward-looking diagnostic support.

[0131] The decision optimization and output module iteratively optimizes collaborative decision-making parameters using a distributed reinforcement learning framework, outputting lesion-annotated action sequences to the ophthalmology diagnostic platform. This module defines the reward function for multi-agent collaborative decision-making, incorporating both lesion annotation accuracy and resource utilization efficiency as dual objectives. A hierarchical meta-learning strategy is used to train the policy network of each modality's annotation agents. During training, the credit allocation weights between agents are dynamically adjusted based on the degree of objective conflict. Through continuous iterative optimization, the collaborative decision-making parameters are brought to their optimal state, ultimately outputting lesion-annotated action sequences that satisfy Pareto optimality. These annotated action sequences are transmitted to the ophthalmology diagnostic platform, allowing doctors to visually view the annotation results and assist in the diagnosis and treatment planning of ophthalmic diseases.

[0132] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0133] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for intelligent annotation of fundus lesions based on three-dimensional reconstruction, characterized in that, include: Receive multimodal image data streams from ophthalmology patients, the data streams including fundus color images, optical coherence tomography (OCT) images, and fluorescein fundus angiography (FFA) images; Based on a pre-trained lesion feature fusion model, spatial registration and feature fusion are performed on the multimodal image data to generate a three-dimensional lesion probability distribution map and a lesion category confidence matrix. Based on the three-dimensional lesion probability distribution map, an adaptive annotation threshold model is constructed to generate a multimodal annotation instruction set, which includes an annotation region topology network and a dynamic allocation strategy for annotation levels. The adaptive annotation threshold model generates a multimodal annotation instruction set based on the three-dimensional lesion probability distribution map, which includes the annotation region topology network and the dynamic allocation strategy of annotation level; The adaptive annotation threshold model includes: Based on the gradient changes in the three-dimensional lesion probability distribution map, the lesion probability density levels are dynamically classified. Based on the complexity of the patient's ocular physiological structure and the risk index of past medical history, the sensitivity score of lesion impact is calculated; The sensitivity score and probability density level are non-linearly mapped using a logistic function to generate an individual-specific annotation threshold. The activation condition for triggering the multi-level annotation response protocol is based on the threshold. Based on a pre-trained chain model of lesion development, the cascading effect of lesion development within a preset time window is simulated, and the collaborative decision parameters in the labeled instruction set are optimized. The steps for constructing the chain model of lesion development include: Collect secondary lesion association data from historical fundus lesion events to construct a lesion causal graph dataset; The transmission probability and delay time parameters between lesion events are extracted using a causal inference algorithm; By combining complex network theory, a directed acyclic graph of lesion propagation is constructed to quantify the vulnerability dependency strength between nodes; The dependence strength and real-time physiological perturbation factors are input into the spatiotemporal capsule network to generate the lesion development chain model. Based on the simulation results of the lesion development chain model, key cascading interruption nodes are identified; Configure a blocking and intervention annotation strategy for the node in the annotation instruction set; Based on the aforementioned blocking intervention annotation strategy, a cross-modal collaborative annotation scheme is automatically generated, including annotation priorities and annotation fusion rules for different image modalities. The collaborative decision-making parameters include the priority of the annotations and the level of detail of the annotations; The collaborative decision parameters are iteratively optimized using a distributed reinforcement learning framework, and the lesion-labeled action sequence is output to the ophthalmology diagnostic platform.

2. The intelligent annotation method for fundus lesions based on three-dimensional reconstruction according to claim 1, characterized in that, The steps for constructing the lesion feature fusion model include: Collect a database of fundus lesion cases and construct a multidimensional feature training set that includes abnormal imaging patterns, physiological structure data, and lesion triggering conditions. The multidimensional feature training set is decoupled by latent variables through a deep variational autoencoder to extract independent representations of the dominant and secondary features of the lesion. By combining the lesion evolution dynamics equation, differential constraints on the nonlinear coupling relationship between features are constructed; The differential constraints are embedded into a graph convolutional network to generate the lesion feature fusion model that supports incremental learning.

3. The intelligent annotation method for fundus lesions based on three-dimensional reconstruction according to claim 2, characterized in that, The calculation of the sensitivity score for the lesion impact includes: Obtain a real-time matrix of ocular physiological function indicators and a heatmap of genetic risk factors to construct an individual resistance cube for lesion resilience assessment; High-order correlation weights between multidimensional features are calculated using a hypergraph attention network. The correlation weights are combined with the evaluation cube using tensor shrinking to obtain a comprehensive sensitivity score. The formula for calculating the comprehensive sensitivity score is as follows: In the formula, This represents the overall sensitivity score. Indicates the first Fragility index of physiological structures Indicates the first Priority weighting for physiological structures This represents a constant representing an individual's baseline resistance to lesions. Represents the tensor Kronecker product. This represents the total number of physiological structural categories.

4. The intelligent annotation method for fundus lesions based on three-dimensional reconstruction according to claim 3, characterized in that, The embedding of the differential constraint conditions includes: Lagrange stability analysis was performed on the decoupling results of the hidden variables to screen physically realizable coupling modes; Characteristic evolution trajectories that conform to dynamic constraints are generated by Markov chain Monte Carlo sampling; The edge weights of the graph convolutional network are trained using trajectory data to ensure that the model output conforms to the physical laws of lesion development.

5. The intelligent annotation method for fundus lesions based on three-dimensional reconstruction according to claim 4, characterized in that, The execution of the distributed reinforcement learning framework includes: Define a reward function for multi-agent collaborative decision-making, which includes the dual objectives of lesion labeling accuracy and resource utilization efficiency; The policy network of each modality-labeled agent is trained using a hierarchical meta-learning strategy. In each round of training, the credit allocation weights among agents are dynamically adjusted according to the degree of target conflict. Output the lesion-labeled action sequence that satisfies the Pareto optimality condition; The reward function is as follows: In the formula, Indicates the reward value. This indicates that the lesion is accurately marked and scored. Indicates the resource utilization efficiency score. As a dynamic equilibrium factor; The quantification of the resource utilization efficiency target includes: Establish a spatiotemporal utility decay model for multi-type labeled resources and define a resource idle penalty function; Based on the lesion evolution phase diagram, calculate the marginal utility value of resource scheduling in each time period; The marginal utility value is used as the dynamic gain coefficient of the reward function; The calculation formula for the attenuation model is as follows: In the formula, Indicates the first The initial utility value of the class-labeled resource. Indicates the aging decay coefficient. express The function indicating the enabled status of the labeled resource at any given time. This indicates the total number of labeled resource types.

6. The intelligent annotation method for fundus lesions based on three-dimensional reconstruction according to claim 5, characterized in that, Also includes: After configuring the blocking intervention labeling strategy, the state transition probability of the cascaded interruption nodes is monitored in real time; If the transition probability exceeds the preset critical value, the simulated annealing optimization mechanism is triggered to re-plan the spatiotemporal collaborative scheme for cross-modal annotation.

7. A three-dimensional reconstruction-based intelligent diagnostic system for fundus lesions, used to implement the three-dimensional reconstruction-based intelligent annotation method for fundus lesions as described in any one of claims 1-6, characterized in that, include: Data receiving module: used to receive multimodal image data streams from ophthalmology patients, including fundus color images, optical coherence tomography (OCT) images, and fluorescein fundus angiography (FFA) images; Feature fusion and analysis module: Based on a pre-trained lesion feature fusion model, spatial registration and feature fusion are performed on the multimodal image data to generate a three-dimensional lesion probability distribution map and a lesion category confidence matrix; Adaptive annotation module: Based on the three-dimensional lesion probability distribution map, construct an adaptive annotation threshold model and generate a multimodal annotation instruction set, which includes an annotation region topology network and a dynamic allocation strategy for annotation levels; Lesion development simulation module: Based on a pre-trained lesion development chain model, it simulates the cascading effect of lesion development within a preset time window in the future, and optimizes the collaborative decision parameters in the labeled instruction set; Decision optimization and output module: Iteratively optimizes the collaborative decision parameters through a distributed reinforcement learning framework, and outputs the lesion-annotated action sequence to the ophthalmology diagnostic platform.

Images