Intelligent labeling method and diagnosis system for fundus focus based on three-dimensional reconstruction

Abstract

The invention relates to the technical field of ophthalmology medical diagnosis, and discloses a three-dimensional reconstruction-based fundus focus intelligent labeling method and diagnosis system. The method comprises the following steps: receiving multi-modal image data streams such as fundus color photos, OCT images and FFA images of an ophthalmological patient; performing spatial registration and feature fusion by using a pre-trained lesion feature fusion model to generate a three-dimensional lesion probability distribution diagram and a lesion category confidence matrix; constructing an adaptive annotation threshold model to generate a multi-modal annotation instruction set; based on the focus development chain model, focus development is simulated, and instruction set parameters are optimized and labeled; and iteratively optimizing through a distributed reinforcement learning framework, and outputting the focus labeling action sequence to an ophthalmology diagnosis platform. According to the method, multi-modal image information can be integrated, the diagnosis accuracy and efficiency are improved, personalized diagnosis is realized, resources are reasonably utilized, and powerful support is provided for ophthalmic disease diagnosis.

Description

Technical Field

[0001] The present invention relates to the technical field of ophthalmic medical diagnosis, and particularly to an intelligent annotation method and diagnosis system for fundus lesions based on three-dimensional reconstruction. Background Art

[0002] Ophthalmic diseases seriously threaten human visual health. The number of patients with ophthalmic diseases is huge globally. Early and accurate diagnosis is the key to effective treatment and avoiding irreversible vision damage. With the progress of medical technology, multimodal ophthalmic imaging technologies such as fundus color photographs, optical coherence tomography (OCT) images, and fluorescein fundus angiography (FFA) images are widely used in clinical practice. These technologies present the ocular structure and lesion characteristics from different dimensions, providing rich information for disease diagnosis, but also bringing many challenges.

[0003] In clinical practice, the integration and analysis of multimodal images face difficulties. On the one hand, the imaging principles, resolutions, and observation focuses of different modal images are significantly different. Fundus color photographs can intuitively show the changes in the blood vessels, retina color, and morphology of the fundus; OCT can clearly present the microscopic structure of each layer of ocular tissues; FFA focuses on the blood circulation and leakage of the fundus blood vessels. Doctors need to interpret these images separately and then make a comprehensive judgment based on experience. The process is cumbersome. Due to the uneven professional levels and clinical experiences of different doctors, the understanding and judgment of images are subjective, resulting in the lack of consistency and accuracy of diagnostic results and easily delaying the condition.

[0004] On the other hand, existing diagnostic methods are mostly based on two-dimensional image analysis and cannot fully exploit the three-dimensional spatial information of multimodal images. Two-dimensional annotation is difficult to accurately determine the depth position and spatial relationship of lesions in ocular tissues. For deep lesions or lesions with complex three-dimensional structures, it is extremely easy to cause missed diagnoses. Moreover, two-dimensional analysis cannot dynamically track the development of lesions and is difficult to predict their future change trends, which is not conducive to formulating scientific treatment plans and long-term disease monitoring plans.

[0005] In addition, there are many types of ophthalmic diseases, and the lesion characteristics are similar. For example, in the early stages of diabetic retinopathy and retinal vein occlusion, the fundus images may be similar. It is very difficult to accurately distinguish them only by traditional image analysis methods, which brings great difficulties to clinical diagnosis. At the same time, the traditional diagnostic process lacks effective management of resources, and there are often unreasonable phenomena in resource allocation during the annotation and diagnosis processes, resulting in resource waste and increasing medical costs.

[0006] With the rapid development of artificial intelligence technology, its application in the diagnosis of ophthalmic diseases has become a research hotspot. However, existing intelligent diagnosis solutions still have many deficiencies and cannot fully meet clinical needs. Therefore, it is urgent to develop a fundus lesion diagnosis technology that can efficiently integrate multi-modal imaging information, achieve intelligent annotation and accurate diagnosis, and rationally utilize medical resources, which is of great significance for improving the level of ophthalmic medical care and ensuring the visual health of patients. Summary of the Invention

[0007] The purpose of the present invention is to provide an intelligent annotation method and diagnosis system for fundus lesions based on three-dimensional reconstruction to solve the problems raised in the above background technology.

[0008] To achieve the above purpose, the present invention provides the following technical solutions: An intelligent annotation method and diagnosis system for fundus lesions based on three-dimensional reconstruction, the method includes:

[0009] Receiving multi-modal image data streams of ophthalmic patients, the data streams include fundus color photos, optical coherence tomography (OCT) images, and fluorescein fundus angiography (FFA) images;

[0010] Based on a pre-trained lesion feature fusion model, performing spatial registration and feature fusion on the multi-modal image data to generate a three-dimensional lesion probability distribution map and a lesion category confidence matrix;

[0011] According to the three-dimensional lesion probability distribution map, constructing an adaptive annotation threshold model to generate a multi-modal annotation instruction set, the instruction set includes an annotation area topology network and an annotation level dynamic allocation strategy;

[0012] Based on a pre-trained lesion development chain model, simulating the cascade effect of lesion development within a preset future time window and optimizing the collaborative decision-making parameters in the annotation instruction set;

[0013] Iteratively optimizing the collaborative decision-making parameters through a distributed reinforcement learning framework and outputting a lesion annotation action sequence to an ophthalmic diagnosis platform.

[0014] Preferably, the construction steps of the lesion feature fusion model include:

[0015] Collecting a fundus lesion case library and constructing a multi-dimensional feature training set including image abnormal patterns, physiological structure data, and lesion triggering conditions;

[0016] Performing latent variable decoupling on the multi-dimensional feature training set through a deep variational autoencoder to extract independent representations of the dominant and secondary features of the lesions;

[0017] Combining the lesion evolution dynamics equation to construct differential constraint conditions for the non-linear coupling relationship between features;

[0018] Embed the differential constraint conditions into the graph convolutional network to generate the lesion feature fusion model that supports incremental learning.

[0019] Preferably, the adaptive annotation threshold model includes:

[0020] Dynamically divide the lesion probability density levels according to the gradient change of the three-dimensional lesion probability distribution map;

[0021] Calculate the lesion impact sensitivity score based on the complexity of the patient's eye physiological structure and the risk index of previous medical history;

[0022] Non-linearly map the sensitivity score and the probability density level through the logistic function to generate an individual-specific annotation threshold;

[0023] Trigger the activation condition of the multi-level annotation response protocol according to the threshold.

[0024] Preferably, the construction steps of the lesion development chain model include:

[0025] Collect the secondary lesion association data in the historical fundus lesion events to construct a lesion causal graph data set;

[0026] Extract the transfer probability and delay time parameters between lesion events through the causal inference algorithm;

[0027] Combine the complex network theory to construct a directed acyclic graph of lesion propagation and quantify the vulnerability dependence strength between nodes;

[0028] Input the dependence strength and the real-time physiological perturbation factor into the spatio-temporal capsule network to generate the lesion development chain model.

[0029] Preferably, the method further includes:

[0030] Identify the critical cascade interruption nodes according to the simulation results of the lesion development chain model;

[0031] Configure a blocking intervention annotation strategy for the nodes in the annotation instruction set;

[0032] Based on the blocking intervention annotation strategy, automatically generate a cross-modal collaborative annotation scheme, including the annotation priority and annotation fusion rules of different imaging modalities.

[0033] Preferably, the calculation of the lesion impact sensitivity score includes:

[0034] Obtain the real-time eye physiological function index matrix and the genetic risk factor heat map, and construct an individual anti-lesion resilience evaluation cube;

[0035] Calculate the high-order association weights between multi-dimensional features through the hypergraph attention network;

[0036] Perform a tensor contraction operation on the associated weight and the evaluation cube to obtain a comprehensive sensitivity score;

[0037] Among them, the calculation formula for the comprehensive sensitivity score is:

[0038]

[0039] In the formula, S represents the numerical value of the comprehensive sensitivity score, α i represents the vulnerability index of the i-th type of physiological structure, β i represents the attention priority weight of the i-th type of physiological structure, γ c represents the individual's basic disease resistance ability constant, represents the tensor Kronecker product, and n represents the total number of physiological structure classifications.

[0040] Preferably, the embedding of the differential constraint conditions includes:

[0041] Perform Lagrangian stability analysis on the decoupled result of the latent variable to screen physically realizable coupling modes;

[0042] Generate a characteristic evolution trajectory that conforms to the dynamic constraints through Markov chain Monte Carlo sampling;

[0043] Use the trajectory data to regularize the edge weights of the graph convolutional network to ensure that the model output conforms to the physical laws of lesion development.

[0044] Preferably, the execution of the distributed reinforcement learning framework includes:

[0045] Define a reward function for multi-agent collaborative decision-making, including the dual objectives of lesion annotation accuracy and resource utilization efficiency;

[0046] Train the policy network of each modality annotation agent through a hierarchical meta-learning strategy;

[0047] In each round of training, dynamically adjust the credit assignment weights between agents according to the degree of goal conflict;

[0048] Output the lesion annotation action sequence that satisfies the Pareto optimal condition;

[0049] Among them, the reward function is:

[0050] R = ω·C a +(1 - ω)·η r

[0051] In the formula, R represents the reward value, C a represents the lesion annotation accuracy score, η r represents the resource utilization efficiency score, and ω is the dynamic balance factor;

[0052] The quantification of the resource utilization efficiency target includes:

[0053] Establish a spatio-temporal utility decay model for multi-type annotation resources and define a resource idle penalty function;

[0054] According to the lesion evolution phase diagram, calculate the marginal utility value of resource scheduling for each time period;

[0055] Use the marginal utility value as the dynamic gain coefficient of the reward function;

[0056] Among them, the calculation formula of the decay model is:

[0057]

[0058] In the formula, K j represents the initial utility value of the j-th type of annotation resource, τ j represents the time effect decay coefficient, x j (t) represents the annotation resource activation status indicator function at time t, and m represents the total number of annotation resource types.

[0059] Preferably, the above further includes:

[0060] After configuring the blocking intervention annotation strategy, monitor the state transition probability of the cascade interruption node in real time;

[0061] If the transition probability exceeds the preset critical value, trigger the simulated annealing optimization mechanism and re-plan the spatio-temporal coordination scheme for cross-modal annotation.

[0062] Preferably, the present invention further includes an intelligent diagnosis system for fundus lesions based on three-dimensional reconstruction, and the system includes:

[0063] Data receiving module: used to receive the multi-modal image data stream of ophthalmic patients, and the data stream includes fundus color photos, optical coherence tomography (OCT) images, and fundus fluorescein angiography (FFA) images;

[0064] Feature fusion and analysis module: based on a pre-trained lesion feature fusion model, perform spatial registration and feature fusion on the multi-modal image data to generate a three-dimensional lesion probability distribution map and a lesion category confidence matrix;

[0065] Adaptive annotation module: according to the three-dimensional lesion probability distribution map, construct an adaptive annotation threshold model and generate a multi-modal annotation instruction set, and the instruction set includes an annotation area topology network and an annotation level dynamic allocation strategy;

[0066] Lesion development simulation module: based on a pre-trained lesion development chain model, simulate the cascade effect of lesion development within a preset future time window and optimize the collaborative decision-making parameters in the annotation instruction set;

[0067] Decision optimization and output module: Iteratively optimize the collaborative decision-making parameters through a distributed reinforcement learning framework, and output the lesion annotation action sequence to the ophthalmic diagnosis platform.

[0068] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0069] In terms of diagnostic accuracy, the system receives multi-modal image data streams, uses a pre-trained lesion feature fusion model for spatial registration and feature fusion, and generates a three-dimensional lesion probability distribution map and a lesion category confidence matrix. This method integrates the advantageous information of different modal images and can accurately present lesion features from multiple dimensions. Taking glaucoma diagnosis as an example, traditional methods may only make judgments based on a single intraocular pressure index or two-dimensional fundus optic nerve images, which are prone to misdiagnosis. In contrast, the present invention fuses fundus color photographs, OCT images, and FFA images to comprehensively analyze the changes in the thickness of the optic nerve fiber layer, the abnormalities of fundus vascular hemodynamics, and the damage of retinal ganglion cells in three-dimensional space, greatly improving the accuracy of early glaucoma diagnosis and reducing the misdiagnosis and missed diagnosis rates.

[0070] In terms of diagnostic efficiency, the adaptive annotation threshold model generates a multi-modal annotation instruction set based on the three-dimensional lesion probability distribution map, including an annotation area topology network and an annotation level dynamic allocation strategy. This enables the system to automatically and quickly lock in key annotation areas and allocate annotation levels according to the importance of the lesions, without the need for doctors to manually examine a large number of images one by one. At the same time, the lesion development chain model simulates the cascading effect of future lesion development, providing valuable predictive information for doctors in advance to assist them in quickly making diagnostic decisions, greatly saving diagnostic time. Especially when dealing with emergency patients or large-scale screenings, it can significantly improve work efficiency.

[0071] For personalized diagnosis, the system fully considers individual patient differences. When calculating the lesion impact sensitivity score, an individual anti-lesion resilience assessment cube is constructed by combining the real-time ocular physiological function index matrix and the genetic risk factor heat map, and the hypergraph attention network is used to calculate the high-order correlation weights between multi-dimensional features, and then an individual-specific annotation threshold is generated. This process ensures that the annotation and diagnosis can closely fit the physiological conditions and genetic backgrounds of each patient, providing customized diagnostic plans and treatment suggestions for patients, achieving precision medicine, and improving the treatment effect and patient satisfaction.

[0072] In terms of resource utilization, the distributed reinforcement learning framework defines a reward function that takes into account both the accuracy of lesion annotation and the resource usage efficiency. By establishing a spatio-temporal utility decay model for multiple types of annotation resources and a resource idle penalty function, the marginal utility value of resource scheduling for each time period is dynamically calculated based on the lesion evolution phase diagram, and the dynamic gain coefficient of the reward function is adjusted accordingly to optimize the credit assignment weights among agents. For example, in the case of limited diagnostic device resources, computational resources are preferentially allocated to urgent and complex cases to avoid resource idleness and waste, reduce medical costs, improve the overall resource utilization efficiency, and enable a more reasonable allocation of medical resources.

[0073] In addition, based on the simulation results of the lesion development chain model, key cascade interruption nodes are identified, blocking intervention annotation strategies are configured, and cross-modal collaborative annotation schemes are automatically generated, which can track the lesion development in real time, timely detect potential risk points, provide key references for clinical treatment, help doctors formulate more targeted and forward-looking treatment plans, effectively improve the prognosis of patients, and enhance the quality of ophthalmic medical services. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 is the working principle diagram of the intelligent fundus lesion annotation method based on three-dimensional reconstruction according to the present invention;

[0075] Figure 2 is the flowchart for constructing the lesion feature fusion model;

[0076] Figure 3 is the flowchart for constructing the lesion development chain model;

[0077] Figure 4 is the flowchart for calculating and applying the lesion impact sensitivity score. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0078] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0079] Please refer to Figures 1-4 , the present invention provides an intelligent fundus lesion annotation method and diagnostic system based on three-dimensional reconstruction, and the specific implementation steps are as follows:

[0080] Receive the multi-modal image data stream of ophthalmic patients: Obtain the multi-modal image data of ophthalmic patients, where the data stream covers fundus color photos, optical coherence tomography (OCT) images, and fluorescein fundus angiography (FFA) images. These images reflect the physiological structure and pathological conditions of the eye from different angles and levels, providing a rich data basis for subsequent analysis and diagnosis. Fundus color photos can directly present the morphology and color changes of tissues such as blood vessels and the retina in the fundus; OCT images can clearly show the fine structures of various layers of tissues in the eye, such as the layered structure of the retina; FFA images are helpful for observing the blood circulation and leakage conditions of the fundus blood vessels.

[0081] Based on the pre-trained lesion feature fusion model, perform spatial registration and feature fusion: Use the pre-trained lesion feature fusion model to process the obtained multi-modal image data. Through spatial registration, align different modal images in space so that they can be analyzed in the same coordinate system. At the same time, fuse the image features, fully integrate the advantageous information of each modal image, and generate a three-dimensional lesion probability distribution map and a lesion category confidence matrix. The three-dimensional lesion probability distribution map intuitively shows the possible distribution of lesions in three-dimensional space, while the lesion category confidence matrix reflects the judgment credibility of different categories of lesions.

[0082] Construct an adaptive annotation threshold model to generate a multi-modal annotation instruction set: Based on the generated three-dimensional lesion probability distribution map, construct an adaptive annotation threshold model. This model generates a multi-modal annotation instruction set by analyzing the characteristics of the probability distribution map. The instruction set includes an annotation area topology network for determining the areas to be annotated and their mutual relationships; an annotation level dynamic allocation strategy that dynamically allocates the annotation levels according to different conditions and importance degrees of the lesions, improving the accuracy and efficiency of annotation.

[0083] Based on the pre-trained lesion development chain model, simulate lesion development and optimize parameters: With the help of the pre-trained lesion development chain model, simulate the cascade effect of lesion development within a preset future time window. Through this simulation, deeply understand the development trend and possible changes of the lesions. At the same time, optimize the collaborative decision-making parameters in the annotation instruction set according to the simulation results, so that the annotation can better adapt to the development of the lesions.

[0084] Iteratively optimize the collaborative decision-making parameters through a distributed reinforcement learning framework and output an annotation action sequence: Use the distributed reinforcement learning framework to iteratively optimize the collaborative decision-making parameters. By continuously adjusting the parameters, make the annotation results improve the resource utilization efficiency while ensuring the accuracy rate. Finally, output the optimized lesion annotation action sequence to the ophthalmic diagnosis platform to provide strong support for doctors' diagnosis.

[0085] The following further illustrates the implementation of the present invention in combination with Embodiments 1 to 5.

[0086] Example 1:

[0087] In this example, the construction of the lesion feature fusion model and the embedding process of the differential constraint conditions are elaborated in detail.

[0088] First, collect a fundus lesion case library, which contains a large amount of data related to fundus lesions. Among them, the imaging abnormal patterns record the unique features presented by various ophthalmic diseases on different modality images. For example, diabetic retinopathy may show abnormal forms such as bleeding points and exudates on fundus color photos, and may manifest as changes in the retinal layer structure on OCT images. The physiological structure data covers the normal physiological parameters of the eye, such as retinal thickness and optic nerve fiber layer thickness. These data provide a reference standard for judging whether a lesion has occurred and the degree of the lesion. The lesion trigger conditions record the factors that may lead to the appearance of lesions, such as the association between systemic diseases such as hypertension and diabetes and fundus lesions. Based on these data, a training set containing the above multi-dimensional features is constructed.

[0089] Use a deep variational autoencoder to decouple the latent variables of the multi-dimensional feature training set. The deep variational autoencoder separates the dominant lesion features and secondary features in the multi-dimensional features by learning the latent distribution of the data, and extracts their respective independent representations. For example, for diabetic retinopathy, the dominant features directly related to the lesion, such as abnormal blood vessel dilation and neovascularization, and secondary features such as mild eye inflammation that may exist but do not directly cause the lesion, are decoupled.

[0090] Combine the lesion evolution dynamics equation to construct differential constraint conditions for the non-linear coupling relationship between features. The lesion evolution dynamics equation describes the dynamic process of lesion development over time. By analyzing the interactions and influences between different features, a non-linear coupling relationship between them is established. For example, when microaneurysms (a common fundus lesion) appear on the retina, the changes in the hemodynamics of the surrounding blood vessels interact with the metabolic requirements of the retinal tissue, and this interaction can be described by a differential equation, thus constructing a non-linear coupling relationship between features.

[0091] When embedding the differential constraint conditions, perform Lagrangian stability analysis on the latent variable decoupling results. Lagrangian stability analysis is used to judge whether the system can remain stable after being slightly perturbed, and physical realizable coupling modes are selected through this analysis. For example, when analyzing the coupling relationship between retinal blood vessels and nerve tissues, exclude those coupling modes that do not conform to the actual situation physically to ensure the rationality of the model.

[0092] Generate the characteristic evolution trajectory that conforms to the kinetic constraints through Markov Chain Monte Carlo sampling. Markov Chain Monte Carlo sampling is a method for random sampling in a high-dimensional space, which can simulate the evolution process of characteristics over time under the condition of meeting the kinetic constraints. Use this trajectory data to regularize the training of the edge weights of the graph convolutional network. The graph convolutional network learns the characteristics of the data by processing the information of nodes and edges. Through regularization training, ensure that the model output conforms to the physical laws of the development of the lesion. For example, during the training process, adjust the edge weights representing the relationship between blood vessels and retinal tissue in the graph convolutional network according to the characteristic evolution trajectory, so that the model can more accurately reflect the interaction between the two during the development of the lesion.

[0093] Example 2:

[0094] Dynamically divide the lesion probability density levels according to the gradient change of the three-dimensional lesion probability distribution map. In the three-dimensional lesion probability distribution map, the gradient change of the probability density reflects the change trend of the lesion distribution. By analyzing the gradient, the probability density is divided into different levels. For example, in the area where the probability density gradient changes greatly, it indicates that the change of the lesion distribution is relatively drastic, and there may be relatively important lesion information, which is divided into the high probability density level; while in the area where the gradient change is small, the lesion distribution is relatively stable and is divided into the low probability density level.

[0095] Obtain the real-time eye physiological function index matrix and the genetic risk factor heat map, and construct an individual anti-lesion resilience evaluation cube. The real-time eye physiological function index matrix contains data such as intraocular pressure and electroretinogram that reflect the current physiological function state of the eye. The genetic risk factor heat map presents the genetic risk factors related to ophthalmic diseases in the form of a heat map through the analysis of the patient's family genetic history, and the darker the color, the higher the risk. Combine these data to construct an individual anti-lesion resilience evaluation cube to comprehensively evaluate an individual's ability to resist lesions from multiple dimensions.

[0096] Calculate the high-order correlation weights between multi-dimensional features through the hypergraph attention network. The hypergraph attention network can capture the complex correlation relationships between multiple features, and calculate the high-order correlation weights between multi-dimensional features such as physiological function indicators and genetic risk factors through this network. For example, for the relationship between intraocular pressure and glaucoma, the hypergraph attention network can analyze the influence weight of intraocular pressure on the occurrence and development of glaucoma under the influence of different genetic backgrounds and other physiological indicators.

[0097] Perform a tensor contraction operation on the correlation weight and the evaluation cube to obtain a comprehensive sensitivity score. The calculation formula for the comprehensive sensitivity score is:

[0098]

[0099] Among them, S represents the comprehensive sensitivity score value, which reflects the sensitivity of an individual to the lesion. α i represents the vulnerability index of the i-th type of physiological structure. For example, for different hierarchical cell structures in the retina, the vulnerability index will vary depending on cell type and function. β i represents the attention priority weight of the i-th type of physiological structure, which is determined according to the importance of different physiological structures in ophthalmic disease diagnosis. γ c represents the individual's basic anti-lesion ability constant, which comprehensively considers the overall health status of the individual and the basic impact of genetic factors on the ability to resist lesions. represents the tensor Kronecker product, which is used to calculate the product relationship between different tensors. n represents the total number of physiological structure classifications, covering all physiological structure categories involved in the eye assessment.

[0100] The sensitivity score and the probability density level are non-linearly mapped through a logistic function to generate an individual-specific annotation threshold. The logistic function can map numerical values in different ranges to a suitable interval. By mapping the sensitivity score and the probability density level through this function, an annotation threshold for each individual is obtained. For example, for areas with a high sensitivity score and a high probability density level, a lower annotation threshold is generated to more timely annotate possible lesions; while for areas with a low sensitivity score and a low probability density level, a higher annotation threshold is generated to reduce unnecessary annotations.

[0101] According to the threshold, the activation conditions of the multi-level annotation response protocol are triggered. When the calculated annotation threshold is compared with the actual lesion probability density, if the lesion probability exceeds the annotation threshold, the corresponding annotation response is triggered. According to different threshold ranges, a multi-level annotation response protocol is set. For example, when the lesion probability exceeds the lower threshold, a preliminary annotation is initiated to mark the approximate area where a lesion may exist; when the lesion probability exceeds the higher threshold, a more detailed annotation is carried out, including information such as the specific morphology and boundary of the lesion.

[0102] Example 3:

[0103] This example is used to describe the construction process of the lesion development chain model. Specifically, it includes:

[0104] Collect the associated data of secondary lesions in historical fundus lesion events to construct a lesion causal graph data set. Historical fundus lesion events contain a large amount of patient case data, from which the associated data of secondary lesions is extracted, that is, the association information between the occurrence of one lesion and the subsequent occurrence of other lesions. For example, in diabetic retinopathy, early microaneurysms may trigger subsequent retinal neovascularization. These associated data are sorted out to construct a lesion causal graph data set.

[0105] Extract the transmission probability and delay time parameters between lesion events through the causal inference algorithm. The causal inference algorithm is used to analyze the causal relationships between different lesion events in the causal graph dataset. Through this algorithm, the transmission probability of one lesion causing another lesion and the delay time between two lesions are calculated. For example, through analysis, it is found that in a certain ophthalmic disease, after lesion A appears, the probability of causing lesion B within 3 to 6 months is 70%. Here, 70% is the transmission probability, and 3 to 6 months is the delay time parameter.

[0106] Construct a directed acyclic graph of lesion propagation by combining complex network theory and quantify the vulnerability dependence strength between nodes. Complex network theory is used to study the mutual relationships between nodes in complex systems. Regarding fundus lesions as nodes, a directed acyclic graph is constructed according to the causal relationships. In this graph, the vulnerability dependence strength between nodes is quantified, that is, the degree of influence of the change of one lesion on other lesions. For example, in the retinal vascular lesion network, the influence strength of the lesion of the central vascular node on the peripheral vascular nodes can be quantified by calculating the connection weights between nodes.

[0107] Input the dependence strength and real-time physiological perturbation factors into the spatio-temporal capsule network to generate a chain model of lesion development. The spatio-temporal capsule network can process information in both spatial and temporal dimensions. Input the quantified vulnerability dependence strength and real-time physiological perturbation factors (such as current intraocular pressure changes, blood glucose fluctuations, and other factors affecting the ocular physiological state) into this network. Through the learning and training of the network, a chain model of lesion development is generated. This model can simulate the cascade effect of lesion development within a preset future time window and provide a basis for subsequent annotation and diagnosis.

[0108] Identify the key cascade interruption nodes according to the simulation results of the chain model of lesion development. During the process of the chain model of lesion development simulating the cascade effect of future lesion development, there will be some nodes that play a key role in the entire lesion development process. If these nodes are blocked or intervened, it may change the development direction of the lesion or slow down its development speed. For example, in the cascade reaction of retinal neovascularization, a key cell node that promotes the overexpression of vascular endothelial growth factor, its activity is crucial for the formation of new blood vessels, and it is identified as a key cascade interruption node.

[0109] Configure a blocking intervention annotation strategy for the key cascade interruption nodes in the annotation instruction set. Once the key cascade interruption nodes are determined, configure the corresponding blocking intervention annotation strategy for them in the annotation instruction set. For example, for the above-mentioned node that promotes the overexpression of vascular endothelial growth factor, the annotation strategy can include more precise annotation of the area where the node is located, recording its detailed physiological state information, while reminding the doctor to pay attention to the subsequent lesions that may be caused by this node and consider taking corresponding intervention measures, such as drug treatment to inhibit the expression of this factor.

[0110] Based on the blocking intervention annotation strategy, an automatic cross-modal collaborative annotation scheme is generated, including the annotation priorities of different imaging modalities and the annotation fusion rules. Different imaging modalities have different advantages in observing lesions. According to the requirements of the blocking intervention annotation strategy, the annotation priorities of different imaging modalities are determined. For example, for key cascade interruption nodes, OCT images may be more helpful in observing their microscopic structural changes, so the annotation priority of OCT images is increased. At the same time, annotation fusion rules are formulated to fuse the annotation information of different modality images. For example, the macroscopic morphological information of blood vessels shown in fundus color photos is combined with the microscopic structural information of this area in OCT images to present the lesion situation more comprehensively.

[0111] After configuring the blocking intervention annotation strategy, the state transition probability of cascade interruption nodes is monitored in real time. By continuously monitoring the state changes of key cascade interruption nodes, their state transition probability is obtained, that is, the possibility of changing from the current state to other states. For example, monitor the probability of the above-mentioned key cell nodes changing from the state of promoting the overexpression of vascular endothelial growth factor to the normal state or other abnormal states.

[0112] If the transition probability exceeds the preset critical value, the simulated annealing optimization mechanism is triggered to re-plan the spatio-temporal collaboration scheme of cross-modal annotation. When the state transition probability exceeds the preset critical value, it indicates that the development of the lesion may have changed significantly, and the original annotation scheme may no longer be applicable. At this time, the simulated annealing optimization mechanism is triggered. This mechanism searches for a better solution randomly within a certain range by simulating the physical annealing process. By re-planning the spatio-temporal collaboration scheme of cross-modal annotation, the annotation priorities and fusion rules are adjusted to adapt to the new lesion development situation and ensure the accuracy and effectiveness of the annotation.

[0113] Example 4:

[0114] This example specifically describes the execution process of the distributed reinforcement learning framework. Specifically, it includes:

[0115] Define the reward function for multi-agent collaborative decision-making, which includes the dual objectives of lesion annotation accuracy and resource utilization efficiency. The reward function is:

[0116] R = ω·C a +(1 - ω)·η r

[0117] Among them, R represents the reward value, which comprehensively reflects the quality of the annotation decision. C a represents the accurate score of lesion annotation, which is used to measure the matching degree between the annotation result and the actual lesion situation. The higher the score, the more accurate the annotation. η rIt represents the resource utilization efficiency score, which reflects the utilization efficiency of various resources during the annotation process. ω is the dynamic balance factor, used to adjust the relative importance of annotation accuracy and resource usage efficiency in the reward function. For example, when resources are relatively abundant, the value of ω can be appropriately increased to pay more attention to annotation accuracy; while when resources are limited, the value of ω is decreased to focus more on resource usage efficiency.

[0118] Establish a spatio-temporal utility decay model for multi-type annotation resources and define a resource idle penalty function. Annotation resources include computing resources, storage resources, and human resources, etc. The calculation formula of the spatio-temporal utility decay model is:

[0119]

[0120] Among them, K j represents the initial utility value of the j-th type of annotation resource. Different types of resources have different initial utilities. For example, the initial utility value of high-performance computing devices is relatively high. τ j represents the time-effect decay coefficient, which reflects the decay speed of resource utility over time. For example, the time-effect decay coefficient of storage resources may be relatively low because the stored data still has value for a long time. x j (t) represents the annotation resource activation status indicator function at time t. When x j (t) = 1, it means the resource is activated at time t. When x j (t) = 0, it means the resource is idle at time t. The resource idle penalty function penalizes the idle resources, prompting the system to utilize resources more reasonably.

[0121] According to the lesion evolution phase diagram, calculate the marginal utility value of resource scheduling for each time period. The lesion evolution phase diagram shows the characteristics and change trends of the lesion at different development stages. By analyzing the phase diagram, calculate the marginal utility value brought by scheduling resources at different time periods. For example, during the rapid development stage of the lesion, investing more computing resources for real-time analysis of image data may obtain a higher marginal utility value because timely and accurate annotation is crucial for diagnosis and treatment decisions.

[0122] Take the marginal utility value as the dynamic gain coefficient of the reward function. According to the calculated marginal utility value, adjust the weight of the resource utilization efficiency score in the reward function. When the marginal utility value is high, increase the proportion of resource utilization efficiency in the reward function to encourage the system to utilize resources more efficiently during this time period; when the marginal utility value is low, appropriately reduce the weight of resource utilization efficiency and pay more attention to annotation accuracy.

[0123] The policy network of each modality annotation agent is trained through a hierarchical meta - learning strategy. The hierarchical meta - learning strategy divides the learning process into multiple levels, and each level focuses on different learning tasks. When training the policy network of each modality annotation agent, the high - level learning task can be to learn the collaboration mode between different modalities, and the low - level learning task can be the annotation optimization for a single modality. Through this hierarchical learning, the learning efficiency and adaptability of the agent are improved.

[0124] In each round of training, the credit assignment weights between agents are dynamically adjusted according to the degree of goal conflict. In the process of multi - agent collaborative annotation, the goals of different agents may conflict. For example, one agent pays more attention to the annotation speed, while another agent pays more attention to the annotation accuracy. According to the degree of goal conflict, the credit assignment weights between agents are dynamically adjusted. When the conflict degree is high, the weights are adjusted to balance the goals of different agents; when the conflict degree is low, the weights are further optimized according to the actual situation to improve the overall annotation effect. Finally, a lesion annotation action sequence that meets the Pareto - optimal condition is output, that is, an annotation scheme that cannot further improve any performance index without reducing other performance indicators.

[0125] Embodiment 5:

[0126] This embodiment details each module of the intelligent fundus lesion diagnosis system based on three - dimensional reconstruction.

[0127] The data receiving module is used to receive the multi - modality image data stream of ophthalmic patients, and this data stream includes fundus color photos, optical coherence tomography (OCT) images, and fluorescein fundus angiography (FFA) images. The data receiving module has efficient data transmission and processing capabilities, and can quickly and accurately obtain image data from different devices. It can be connected to ophthalmic examination devices through a network interface, receive image data in real - time, and perform preliminary pre - processing on the data, such as data format conversion, data integrity check, etc., to ensure that the subsequent modules can use this data smoothly.

[0128] Based on the pre-trained lesion feature fusion model, the feature fusion and analysis module performs spatial registration and feature fusion on multi-modal imaging data. This module first calls the pre-trained lesion feature fusion model to precisely register fundus color photos, OCT images, and FFA images in space, enabling different modal images to be aligned for unified analysis. Then, it performs feature fusion on the registered images, extracts lesion-related features from each modal image, and generates a three-dimensional lesion probability distribution map and a lesion category confidence matrix. When generating the three-dimensional lesion probability distribution map, it comprehensively considers information such as the location and morphology of lesions in different modal images to construct the probability distribution of lesions in three-dimensional space, thus intuitively showing the areas where lesions may appear. The lesion category confidence matrix, on the other hand, calculates the confidence levels of different category lesions through in-depth analysis of the fused features and using the classification ability of the model, providing a basis for subsequent diagnosis.

[0129] Based on the three-dimensional lesion probability distribution map, the adaptive annotation module constructs an adaptive annotation threshold model and generates a multi-modal annotation instruction set. This module first analyzes the features of the three-dimensional lesion probability distribution map, such as the distribution and trend of probability density. Based on these analysis results, combined with the complexity of the patient's eye physiological structure and the risk index of previous medical history, it constructs an adaptive annotation threshold model. By dynamically dividing the lesion probability density levels, calculating the lesion impact sensitivity score, and performing a non-linear mapping of the two to generate an individual-specific annotation threshold. According to this threshold, it triggers the activation conditions of the multi-level annotation response protocol, and then generates a multi-modal annotation instruction set including the annotation area topology network and the dynamic allocation strategy of annotation levels to achieve precise and efficient annotation.

[0130] Based on the pre-trained lesion development chain model, the lesion development simulation module simulates the cascade effect of lesion development within a preset future time window and optimizes the collaborative decision-making parameters in the annotation instruction set. This module calls the pre-trained lesion development chain model and inputs the current lesion information and relevant physiological parameter data. The model predicts the changes in lesions in the future period by simulating the cascade effect of lesion development, including the expansion of lesions and the emergence of new lesions. According to the simulation results, it optimizes the collaborative decision-making parameters in the annotation instruction set, such as adjusting the annotation priority and the detail level of annotation, so that the annotation can better adapt to the development trend of lesions and provide more forward-looking diagnostic support for doctors.

[0131] The decision-making optimization and output module iteratively optimizes the collaborative decision-making parameters through a distributed reinforcement learning framework and outputs the lesion annotation action sequence to the ophthalmic diagnosis platform. This module defines a reward function for multi-agent collaborative decision-making, which includes two objectives: the accuracy of lesion annotation and the efficiency of resource utilization. The policy networks of each modality annotation agent are trained using a hierarchical meta-learning strategy. During the training process, the credit assignment weights between agents are dynamically adjusted according to the degree of goal conflict. Through continuous iterative optimization, the collaborative decision-making parameters are optimized to the best, and finally, the lesion annotation action sequence that meets the Pareto optimal conditions is output. These annotation action sequences are transmitted to the ophthalmic diagnosis platform, where doctors can visually view the annotation results to assist in the diagnosis of ophthalmic diseases and the formulation of treatment plans.

[0132] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device.

[0133] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent annotation method for fundus lesions based on three-dimensional reconstruction, characterized in that, Comprising: Receiving a multi-modal image data stream of ophthalmic patients, the data stream including fundus color photographs, optical coherence tomography (OCT) images, and fluorescein fundus angiography (FFA) images; Based on a pre-trained lesion feature fusion model, performing spatial registration and feature fusion on the multi-modal image data to generate a three-dimensional lesion probability distribution map and a lesion category confidence matrix; According to the three-dimensional lesion probability distribution map, constructing an adaptive annotation threshold model to generate a multi-modal annotation instruction set, the instruction set including an annotation area topology network and an annotation level dynamic allocation strategy; Based on a pre-trained lesion development chain model, simulating the cascade effect of lesion development within a preset future time window, and optimizing the collaborative decision-making parameters in the annotation instruction set; Iteratively optimizing the collaborative decision-making parameters through a distributed reinforcement learning framework, and outputting a lesion annotation action sequence to an ophthalmic diagnosis platform.

2. The intelligent annotation method for fundus lesions based on three-dimensional reconstruction according to claim 1, wherein The construction steps of the lesion feature fusion model include: Collecting a fundus lesion case library, and constructing a multi-dimensional feature training set including image abnormal patterns, physiological structure data, and lesion triggering conditions; Performing latent variable decoupling on the multi-dimensional feature training set through a deep variational autoencoder to extract independent representations of dominant and secondary lesion features; Combining a lesion evolution dynamics equation to construct a differential constraint condition for non-linear coupling relationships between features; Embedding the differential constraint condition into a graph convolutional network to generate the lesion feature fusion model supporting incremental learning.

3. The intelligent annotation method for fundus lesions based on three-dimensional reconstruction according to claim 1, wherein The adaptive annotation threshold model includes: Dynamically dividing the lesion probability density levels according to the gradient change of the three-dimensional lesion probability distribution map; Calculating a lesion impact sensitivity score based on the complexity of the patient's eye physiological structure and the risk index of past medical history; Non-linearly mapping the sensitivity score and the probability density level through a logistic function to generate an individual-specific annotation threshold; Triggering the activation condition of a multi-level annotation response protocol according to the threshold.

4. The intelligent labeling method for fundus lesions based on three-dimensional reconstruction according to claim 1, wherein The construction steps of the lesion development chain model include: Collecting secondary lesion association data in historical fundus lesion events to construct a lesion causal graph data set; Extracting the transfer probability and delay time parameters between lesion events through a causal inference algorithm; Combining complex network theory to construct a directed acyclic graph of lesion propagation, and quantifying the vulnerability dependence strength between nodes; Inputting the dependence strength and real-time physiological perturbation factors into a spatio-temporal capsule network to generate the lesion development chain model.

5. The intelligent annotation method for fundus lesions based on three-dimensional reconstruction according to claim 4, wherein Also including: Identifying key cascade interruption nodes according to the simulation results of the lesion development chain model; Configuring a blocking intervention annotation strategy for the nodes in the annotation instruction set; Automatically generating a cross-modal collaborative annotation scheme based on the blocking intervention annotation strategy, including the annotation priority and annotation fusion rules of different image modalities.

6. The intelligent labeling method for fundus lesions based on three-dimensional reconstruction according to claim 3, wherein The calculation of the lesion impact sensitivity score includes: Obtaining a real-time eye physiological function index matrix and a genetic risk factor heat map, and constructing an individual anti-lesion resilience assessment cube; Calculating the high-order association weights between multi-dimensional features through a hypergraph attention network; Performing a tensor contraction operation on the association weights and the assessment cube to obtain a comprehensive sensitivity score; Wherein, the calculation formula of the comprehensive sensitivity score is: Where S represents the comprehensive sensitivity score value, α i represents the vulnerability index of the i-th type of physiological structure, β i represents the attention priority weight of the i-th type of physiological structure, γ c represents the individual's basic disease resistance ability constant, represents the tensor Kronecker product, and n represents the total number of physiological structure classifications.

7. The method for intelligent annotation of fundus lesions based on three-dimensional reconstruction according to claim 2, wherein The embedding of the differential constraint conditions includes: Conduct Lagrangian stability analysis on the decoupling result of the latent variables to screen physically realizable coupling modes; Generate characteristic evolution trajectories that conform to dynamic constraints through Markov chain Monte Carlo sampling; Regularize and train the edge weights of the graph convolutional network using the trajectory data to ensure that the model output conforms to the physical laws of lesion development.

8. The method for intelligent annotation of fundus lesions based on three-dimensional reconstruction according to claim 1, characterized in that, The execution of the distributed reinforcement learning framework includes: Define a reward function for multi-agent collaborative decision-making, which includes the dual objectives of lesion annotation accuracy and resource utilization efficiency; Train the policy network of each modality annotation agent through a hierarchical meta-learning strategy; In each round of training, dynamically adjust the credit assignment weights between agents according to the degree of goal conflict; Output the lesion annotation action sequence that satisfies the Pareto optimality condition; Among them, the reward function is: R = ω·C a +(1 - ω)·η r where R represents the reward value, and C a represents the accurate scoring of lesion annotation, and η r represents the scoring of resource utilization efficiency, and ω is the dynamic balance factor; The quantification of the resource utilization efficiency target includes: Establish a spatio-temporal utility decay model for multi-type annotation resources and define a resource idle penalty function; Calculate the marginal utility value of resource scheduling for each time period according to the lesion evolution phase diagram; Use the marginal utility value as the dynamic gain coefficient of the reward function; Among them, the calculation formula of the decay model is: Where, K j represents the initial utility value of the j-th type of labeled resource, τ j represents the time-dependent attenuation coefficient, x j (t) represents the labeled resource activation status indicator function at time t, and m represents the total number of labeled resource types.

9. The intelligent annotation method for fundus lesions based on three-dimensional reconstruction according to claim 5, wherein It also includes: After configuring the blocking intervention annotation strategy, monitor the state transition probability of the cascading interruption node in real time; If the transition probability exceeds the preset critical value, trigger the simulated annealing optimization mechanism to re-plan the spatio-temporal collaboration plan for cross-modal annotation.

10. An intelligent fundus lesion diagnosis system based on three-dimensional reconstruction, characterized in that, It includes: Data reception module: used to receive the multi-modal image data stream of ophthalmic patients, and the data stream includes fundus color photos, optical coherence tomography (OCT) images, and fluorescein fundus angiography (FFA) images; Feature fusion and analysis module: Based on a pre-trained lesion feature fusion model, perform spatial registration and feature fusion on the multi-modal image data to generate a three-dimensional lesion probability distribution map and a lesion category confidence matrix; Adaptive annotation module: According to the three-dimensional lesion probability distribution map, construct an adaptive annotation threshold model to generate a multi-modal annotation instruction set, and the instruction set includes an annotation area topology network and an annotation level dynamic allocation strategy; Lesion development simulation module: Based on a pre-trained lesion development chain model, simulate the cascading effect of lesion development within a preset future time window and optimize the collaborative decision-making parameters in the annotation instruction set; Decision optimization and output module: Iteratively optimize the collaborative decision-making parameters through a distributed reinforcement learning framework and output the lesion annotation action sequence to the ophthalmic diagnosis platform.

Images