Glaucoma progress risk prediction method and system fused with space-time attention mechanism

Abstract

The invention discloses a glaucoma progress risk prediction method and system fused with a space-time attention mechanism. Belongs to the technical field of artificial intelligence medical treatment. The method comprises the following steps: acquiring a structural optical coherence tomography (OCT) image sequence and a functional visual field (VF) data sequence of a patient; coding the multi-modal data respectively and fusing the multi-modal data into a joint latent space representation sequence, and generating a time mask to process irregular follow-up visit; modeling the historical joint latent space representation sequence based on a space-time attention mechanism, and inputting the historical joint latent space representation sequence as a condition into a conditional diffusion model to generate a target latent vector of a future time point; and finally, analyzing the target latent vector into a prediction OCT image and a glaucoma progress risk score through a multi-task decoder. Through deep fusion of structure and function information and effective processing of irregular time sequences, the dynamic evolution law of diseases can be accurately captured, and the accuracy and clinical applicability of glaucoma progress prediction are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence medical technology, and more specifically, to a method and system for predicting the risk of glaucoma progression by incorporating a spatiotemporal attention mechanism. Background Technology

[0002] Currently, glaucoma is a chronic, progressive disease characterized by optic nerve damage and visual field defects, and is one of the leading causes of irreversible blindness worldwide. Due to its slow and insidious progression, effectively predicting the future progression of glaucoma is crucial for developing precise clinical intervention strategies and protecting patients' visual function.

[0003] Currently, deep learning-based artificial intelligence algorithms have achieved some application in glaucoma diagnosis, but mainstream methods typically use single-point-of-time fundus images or optical coherence tomography (OCT) images as input for static diagnosis. This approach lacks modeling of the dynamic progression of the disease, cannot provide clinicians with effective predictions of future progression trends, and fails to meet the needs of long-term dynamic disease management.

[0004] Furthermore, the clinical diagnosis and treatment of glaucoma involves structural imaging (such as OCT) and functional examinations (such as visual field, Visual Field, VF). However, current technologies rarely effectively fuse and jointly model data from these two modalities, resulting in the underutilization of the potential clinical value of the data and limiting the accuracy of predictions. At the same time, clinical follow-up data generally exhibits irregularities such as uneven collection times and missing data, posing a significant challenge to existing algorithms for effective sequential modeling. Simple interpolation or truncation can lead to data distortion, severely impacting the predictive performance and clinical applicability of the model.

[0005] Therefore, how to provide a method that can effectively integrate structural and functional multimodal data, process irregular follow-up sequences, and accurately predict the future progression risk of glaucoma has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] This invention aims to solve the technical problems in existing glaucoma prediction models, such as difficulty in integrating multimodal data, inability to handle irregular follow-up sequences, and insufficient prediction accuracy. Therefore, it provides a method and system for predicting the risk of glaucoma progression by integrating spatiotemporal attention mechanisms.

[0007] To achieve the above objectives, this invention provides a method for predicting glaucoma progression risk by incorporating a spatiotemporal attention mechanism. The method includes: acquiring structural optical coherence tomography (OCT) image sequences and functional visual field (VF) data sequences from multiple follow-up visits of patients; performing multimodal joint encoding on the OCT image sequences and VF data sequences to generate a joint latent space representation sequence, and generating a temporal mask based on follow-up time points to identify data gaps; based on the historical joint latent space representation sequence and temporal mask, performing context dependency modeling through a spatiotemporal attention mechanism to obtain a conditional context sequence; inputting the conditional context sequence into a conditional diffusion model to generate a target latent vector for a future target time point; wherein the spatiotemporal attention module is embedded within a diffusion denoising network, and injects the spatiotemporal context into the intermediate layer (or cross-layer features) of the U-Net through stepwise attention at each backdiffusion step t, thereby continuously applying temporal and spatial constraints during the stepwise denoising process from T to 0. Finally, the target latent vector is input into a multi-task output module to simultaneously generate a predicted OCT image and a glaucoma progression risk score for the future target time point.

[0008] A method for predicting the progression risk of glaucoma by incorporating spatiotemporal attention mechanisms, the method is as follows: Step S1: Obtain structural optical coherence tomography (OCT) image sequences and functional field of view (VF) data sequences from multiple follow-up visits of the patient; Step S2: Perform multimodal joint encoding on the OCT image sequence and VF data sequence to generate a joint latent space representation sequence, and generate a time mask based on the follow-up time point to identify data missing situations; Step S3: Based on the historical joint latent space representation sequence and the temporal mask, perform context dependency modeling through a spatiotemporal attention mechanism to obtain a conditional context sequence; Step S4: Input the conditional context sequence into the conditional diffusion model to generate the target latent vector at future target time points; Step S5: Input the target latent vector into the multi-task output module to simultaneously generate the predicted OCT image and glaucoma progression risk score for the future target time point.

[0009] Preferably, in step S2, the multimodal joint coding includes: Each frame of OCT image is encoded into a structural latent vector using a variational autoencoder (VAE). A fully connected mapper is used to encode the VF data of each frame into a functional latent vector; The structural latent vector and the functional latent vector are concatenated along the channel dimension to form a joint latent space representation.

[0010] Preferably, after step S2, the method further includes: During the model training phase, a frame is randomly selected from the valid data frames according to the time mask as the target prediction frame, and an occlusion mask is generated to block the target prediction frame in order to construct training samples.

[0011] Preferably, in step S3, the spatiotemporal attention mechanism is based on a modified Transformer structure and introduces temporal and spatial encoding to capture temporal dependencies and spatial feature responses.

[0012] Preferably, in step S5, the multi-task output module includes: An image decoder is used to restore the target latent vector into the predicted OCT image; A risk prediction classifier is used to calculate the glaucoma progression risk score based on the target latent vector.

[0013] Preferably, the method further includes: The model is trained based on a joint loss function, which includes an image reconstruction loss used to guide the training of the image decoder and a risk classification loss used to guide the training of the risk prediction classifier. Specifically, through an adaptive weight learning mechanism, the weights of the image reconstruction loss and the risk classification loss in the joint loss function are dynamically adjusted according to the task uncertainty.

[0014] Preferably, the method further includes: Obtain the patient's individual characteristic information and encode it into an individual condition vector; The individual conditional vectors are integrated into the spatiotemporal attention mechanism and / or the multi-task output module to achieve individualized modeling of different patients' pathological evolution patterns.

[0015] A second aspect of this invention discloses a glaucoma progression risk prediction system that integrates spatiotemporal attention mechanisms; the system includes: The data acquisition module is configured to acquire structural optical coherence tomography (OCT) image sequences and functional field of view (VF) data sequences from multiple follow-up visits of patients. The joint coding module is configured to perform multimodal joint coding on the OCT image sequence and VF data sequence to generate a joint latent space representation sequence, and generate a time mask based on the follow-up time point to identify data missing situations. The context modeling module is configured to model context dependencies based on the historical joint latent space representation sequence and the temporal mask, and obtain a conditional context sequence through a spatiotemporal attention mechanism. The prediction generation module is configured to input the conditional context sequence into the conditional diffusion model to generate the target latent vector for future target time points; The multi-task output module is configured to input the target latent vector to simultaneously generate a predicted OCT image and a glaucoma progression risk score for the future target time point.

[0016] A third aspect of this invention discloses an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the glaucoma progression risk prediction method that integrates a spatiotemporal attention mechanism, as described in any of the first aspects of this disclosure.

[0017] A fourth aspect of this invention discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a glaucoma progression risk prediction method incorporating a spatiotemporal attention mechanism, as described in any of the first aspects of this disclosure.

[0018] According to the technical content disclosed in this invention, the following beneficial effects are achieved: The method proposed in this invention integrates OCT images reflecting ocular structure with VF data reflecting changes in visual function into a unified latent space by constructing a multimodal joint coding mechanism. This achieves deep integration of structural and functional information, enabling a more comprehensive capture of the intrinsic correlations in glaucoma progression. Simultaneously, the method innovatively introduces temporal masking and spatiotemporal attention mechanisms, specifically designed to handle irregular and incomplete follow-up time series commonly encountered in clinical practice. This allows the model to accurately learn the nonlinear dynamic patterns of disease evolution over time, significantly enhancing its clinical adaptability and robustness. Finally, through a conditional diffusion model and a multi-task output architecture, this invention not only generates high-quality, clinically valuable future OCT images but also provides quantitative progression risk scores, offering clinicians a forward-looking and visualized basis for decision-making, significantly improving the accuracy and clinical application value of glaucoma progression prediction.

[0019] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0021] Figure 1 This is a flowchart of a glaucoma progression risk prediction method that integrates spatiotemporal attention mechanisms, according to an embodiment. Figure 2This is a schematic diagram of irregular time masking and random occlusion mechanism in an embodiment of the present invention; Figure 3 This is a schematic diagram of the conditional diffusion prediction architecture that integrates spatiotemporal attention mechanism in an embodiment of the present invention; Figure 4 This is a structural diagram of a glaucoma progression risk prediction system that integrates a spatiotemporal attention mechanism according to an embodiment of the present invention; Figure 5 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0022] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0023] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0024] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0025] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0026] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0027] Example 1: To achieve the above objectives, this invention provides a method for predicting glaucoma progression risk by incorporating a spatiotemporal attention mechanism. The method includes: acquiring structural optical coherence tomography (OCT) image sequences and functional visual field (VF) data sequences from multiple follow-up visits of patients; performing multimodal joint encoding on the OCT image sequences and VF data sequences to generate a joint latent space representation sequence, and generating a time mask based on follow-up time points to identify data gaps; based on the historical joint latent space representation sequence and time mask, performing context dependency modeling through a spatiotemporal attention mechanism to obtain a conditional context sequence; inputting the conditional context sequence into a conditional diffusion model to generate a target latent vector for a future target time point; and finally, inputting the target latent vector into a multi-task output module to simultaneously generate a predicted OCT image and a glaucoma progression risk score for the future target time point.

[0028] This invention discloses a method for predicting the risk of glaucoma progression that integrates spatiotemporal attention mechanisms. Figure 1 Here is a flowchart of a glaucoma progression risk prediction method incorporating spatiotemporal attention mechanisms according to an embodiment of the present invention, with reference to... Figure 1 This invention provides a method for predicting the progression risk of glaucoma by incorporating a spatiotemporal attention mechanism. The core idea is to construct a deep learning framework capable of processing multimodal, irregular time-series data. This framework jointly encodes structured OCT images and functional VF data, utilizes a spatiotemporal attention mechanism to capture the dynamic evolution pattern of the disease, and leverages a conditional diffusion model to generate predictions of future disease states, including future optic nerve structure images and quantified progression risk. The specific method is as follows: Step S1, Data Acquisition and Preprocessing: Acquire structural optical coherence tomography (OCT) image sequences and functional field of view (VF) data sequences from multiple follow-up visits of the patient; the data acquisition module 101 is responsible for collecting and processing the patient's longitudinal follow-up data. These data mainly include two core modalities: 1) Structural OCT image sequence: OCT images acquired during T follow-up visits, reflecting the changes over time in key structures such as the optic nerve head and retinal nerve fiber layer. Before being input into the model, all OCT images need to be resized (e.g., scaled to 256×256 pixels) and numerically normalized (e.g., z-score normalization).

[0029] 2) Functional Visual Function (VF) Data Sequence: Visual field examination data synchronized with OCT images, typically consisting of a functional feature vector composed of key indicators such as mean deviation (MD) and pattern standard deviation (PSD). Similarly, all VF indicators are standardized (e.g., zero-mean unit variance standardization).

[0030] Step S2, Multimodal Joint Encoding and Irregular Temporal Processing: Multimodal joint encoding is performed on the OCT image sequence and VF data sequence to generate a joint latent space representation sequence, and a time mask is generated according to the follow-up time point to identify data missing situations; The multimodal joint coding includes: A variational autoencoder (VAE) is used to encode each frame of OCT image into a structural latent vector. A fully connected mapper is used to encode the VF data of each frame into a functional latent vector; The structural latent vector and the functional latent vector are concatenated along the channel dimension to form a joint latent space representation.

[0031] After step S2, the method further includes: during the model training phase, randomly selecting a frame from the valid data frames as the target prediction frame according to the time mask, and generating an occlusion mask to mask the target prediction frame in order to construct training samples.

[0032] To uniformly process heterogeneous multimodal data and handle irregular follow-up, the joint coding module 102 performs the following operations: 1. Structured processing of irregular time series: Refer to Figure 2 Considering the uneven time intervals and missing data in clinical follow-up data, this invention introduces a time-aligned mask mechanism. For a follow-up sequence of length T, a time mask vector is defined, where an element value of 1 indicates that the data exists at that moment, and a value of 255 (or other preset value) indicates that the data is missing. This mask will guide subsequent models to selectively process valid data.

[0033] During the model training phase, a random frame occlusion mechanism was also designed. A frame is randomly selected from the valid time frames as the "prediction target frame" and explicitly masked using an occlusion mask. This mechanism forces the model to learn to predict the future from historical information, enhancing the model's generalization ability.

[0034] 2. Multimodal joint coding: This invention designs a multimodal joint coding strategy that maps OCT image sequences and VF data sequences to a shared latent representation space.

[0035] Structural modality coding: The encoder part of a variational autoencoder (VAE) is used to encode each preprocessed OCT image frame into a low-dimensional structural latent vector. This preserves the high-level structural semantics of the image while reducing computational complexity.

[0036] Functional modality coding: Design a lightweight fully connected mapper (which can be regarded as a multilayer perceptron) to map the normalized VF data vector of each frame to a functional latent vector with the same dimension as the structural latent vector, so as to achieve cross-modal semantic and spatial alignment.

[0037] Fusion: The structural latent vector and functional latent vector at each time point are concatenated along the channel dimension to form a joint latent space representation. For a sequence with T follow-ups, a final joint latent space representation sequence is obtained. For missing or occluded frames, their corresponding latent vectors can be filled with all-zero tensors, and their role in subsequent calculations is controlled by a temporal mask.

[0038] Step S3: Context modeling based on spatiotemporal attention mechanism: Based on the historical joint latent space representation sequence and the temporal mask, context dependency modeling is performed through a spatiotemporal attention mechanism to obtain a conditional context sequence; the spatiotemporal attention mechanism is based on a modified Transformer structure and introduces temporal encoding and spatial encoding to capture temporal dependencies and spatial feature responses.

[0039] The context modeling module 103 is responsible for extracting key information from historical sequences for predicting future states.

[0040] Reference Figure 3 This invention introduces a spatiotemporal attention mechanism, which is based on a modified Transformer structure to adapt to the input of image tensor sequences. First, each tensor in the joint latent space representation sequence is flattened into a series of patchtokens, and then temporal and spatial information is added to them through positional encoding.

[0041] The attention layer incorporates a dual location encoding mechanism, encoding both temporal and spatial locations separately. Through self-attention computation, the model learns complex dependencies between features at different time points and spatial locations. For example, subtle structural changes in a region in its early stages may foreshadow significant visual defects later. After processing by multiple layers of spatiotemporal attention mechanisms, the final output is a highly condensed sequence of conditional contexts that encapsulates historical evolutionary patterns.

[0042] Step S4: Prediction generation based on conditional diffusion model: Input the conditional context sequence into the conditional diffusion model to generate the target latent vector at the future target time point; The prediction generation module 104 is the core of the method of this invention, responsible for generating predictions of future states. This invention employs a conditional diffusion model to perform the generation task in a joint latent space. The diffusion model generates data through a process of progressively adding noise (forward process) and denoising (backward process).

[0043] During the training phase, the real latent vectors of randomly occluded target frames are denoised in multiple steps, and then a denoiser (usually a U-Net structured network) is trained so that it can gradually recover the original target latent vectors from the noise based on the conditional context sequence obtained in step S3.

[0044] During the inference phase, the complete historical follow-up sequence is input, and the context modeling module 103 generates a conditional context sequence. Then, starting from a standard Gaussian noise tensor, a trained denoiser is used to perform multi-step iterative denoising under the guidance of the conditional context sequence, ultimately generating a high-quality target latent vector for future target time points that conforms to the historical evolution pattern.

[0045] Step S5, Multi-task Output: The multi-task output module includes: an image decoder for restoring the target latent vector to the predicted OCT image; and a risk prediction classifier for calculating the glaucoma progression risk score based on the target latent vector. The multi-task output module 105 is responsible for parsing the generated target latent vector into clinically meaningful output.

[0046] This invention designs a multi-task prediction architecture with dual output paths: Image generation path: An image decoder (e.g., the decoder part of a VAE) receives the target latent vector and reconstructs it into a predicted OCT image. This provides doctors with a visual reference of the future state of the optic nerve structure.

[0047] Risk prediction path: A risk prediction classifier (e.g., a multilayer perceptron, MLP) receives the target latent vector and outputs a probability value between [0,1], i.e., a glaucoma progression risk score. This score can be used to quantify the likelihood of the disease worsening in the future.

[0048] Model training and optimization: The model is trained based on a joint loss function, which includes an image reconstruction loss to guide the training of the image decoder and a risk classification loss to guide the training of the risk prediction classifier. Specifically, an adaptive weight learning mechanism is used to dynamically adjust the weights of the image reconstruction loss and the risk classification loss in the joint loss function based on task uncertainty.

[0049] To achieve collaborative learning with dual objectives, this invention employs a multi-task joint training strategy. The total loss function consists of two parts: 1. Image reconstruction loss: used to measure the difference between the generated OCT image and the real image. A combination of L1 loss and structural similarity index (SSIM) loss can be used to balance pixel-level accuracy and structural fidelity.

[0050] 2. Risk classification loss: used to measure the difference between the predicted risk score and the actual progress label, usually using binary cross-entropy loss.

[0051] To balance the learning processes of the two tasks, an adaptive weight learning mechanism based on task uncertainty is introduced to dynamically adjust the weights of the two loss terms in the total loss function. Furthermore, gradient coordination strategies (such as GradNorm or PCGrad) can be employed to mitigate gradient conflicts in multi-task learning, ensuring stable model convergence.

[0052] Personalized modeling and credibility assessment To enhance the model's adaptability to different patients, this invention can also introduce an individual difference modeling mechanism. Patient demographic information, baseline clinical indicators, etc., are encoded into an individual conditional vector, which is then integrated into the model's spatiotemporal attention mechanism and risk prediction path to achieve personalized prediction.

[0053] The individual characteristics of the patient are acquired and encoded into an individual condition vector; the individual condition vector is then integrated into the spatiotemporal attention mechanism and / or the multi-task output module to achieve individualized modeling of the pathological evolution patterns of different patients.

[0054] To enhance the clinical interpretability and safety of the prediction results, this invention introduces a confidence estimation mechanism. For example, using Monte Carlo Dropout (MCDropout) technology, the standard deviation of the results is calculated multiple times during inference to quantify the uncertainty of the risk score. For the generated image, an uncertainty heatmap can be generated by generating and calculating pixel-level standard deviations multiple times, highlighting areas with low confidence in the model's generated results.

[0055] Example 2: This embodiment provides a specific case to illustrate a method for predicting the progression risk of glaucoma by incorporating a spatiotemporal attention mechanism. The method includes: Step S1: Obtain structural optical coherence tomography (OCT) image sequences and functional field of view (VF) data sequences from multiple follow-up visits of the patient; Specifically, the glaucoma progression prediction method proposed in this invention is based on multiple follow-up information of patients, involving two core data modalities: one is optical coherence tomography (OCT) images reflecting ocular structure, and the other is visual field examination data (Visual Field, VF index) reflecting changes in visual function. Together, they constitute a longitudinal, heterogeneous, and multimodal input sequence. Because clinical follow-up of glaucoma is characterized by irregular intervals and uneven recording frequency, the raw data needs to undergo unified structuring and feature alignment before deep modeling to ensure effective learning of spatiotemporal information in subsequent modeling tasks.

[0056] OCT image input is in the form of a two-dimensional or three-dimensional image sequence, representing the changes over time in key structures such as the optic nerve head and the retinal nerve fiber layer. The patient's condition is recorded... T The OCT image sequence collected during the follow-up was as follows:

[0057] in Indicates the first i OCT image frames from the second follow-up visit. C This represents the number of channels (e.g., 1 for a grayscale image). H ,W These represent the height and width of the image, respectively. Since the actual images come from different devices or sampling schemes, the original images need to be uniformly sized (e.g., scaled to 256×256 pixels) and standardized (e.g., z-score or min-max normalization) before being input into the model to eliminate systematic biases in the image's grayscale scale.

[0058] Unlike structural imaging, visual field testing data are quantitative numerical records at each follow-up visit, used to reflect the patient's visual function status. This invention uses key indicators such as Mean Deviation (MD) and Pattern Standard Deviation (PSD) as functional feature vectors to construct a time series synchronized with OCT.

[0059] in Indicates the first i The visual index vector corresponding to each follow-up visit, with dimensions... d It can be expanded according to the selected functional feature types. To ensure comparability between numerical features, all vision indices will be standardized with zero mean and unit variance, and will be aligned with the image coding results at a uniform scale in the latent space.

[0060] Considering the common occurrence of uneven time intervals and missing time points in clinical follow-up data, this invention introduces a time-aligned mask mechanism to identify which time points have valid observations. Specifically, a length of [missing information] is defined. T The time mask vector is:

[0061] in =1 indicates that the data at that moment actually exists. =255 indicates that data is missing or unavailable at that moment. This mask is not only used to guide the model to selectively process valid data frames during the input phase, but also participates in the random occlusion of training samples and the construction of reverse prediction paths during the diffusion modeling phase.

[0062] To further enhance the model's ability to model temporal structures, this invention designs a random frame occlusion mechanism during the training phase. A frame is randomly selected from the actual existing time frames as the "prediction target frame," and explicitly masked using an occlusion mask. The occlusion mask is denoted as:

[0063] Only one position is 0 (indicating that the frame is occluded and needs to be recovered by model prediction), while the rest of the frames remain 1. The occlusion operation is only enabled during training; during inference, the complete historical sequence is used to predict future states.

[0064] The data preprocessing process also includes operations such as image channel arrangement, dimension rearrangement, tensor normalization, missing field index filling and synchronization alignment. All processing steps are completed in a unified data interface layer to ensure that the image and functional information are strictly matched in time and to work with the masking mechanism to build a standardized model input.

[0065] Through the aforementioned structured modeling and masking mechanism design, the original clinical follow-up data is effectively transformed into a model input format with a unified temporal structure, explicit data validity identification, and standardized numerical range, forming an organic combination of multiple input signals such as OCT image sequences, visual field index sequences, temporal masks, and occlusion masks. This input design provides a fundamental guarantee for the subsequent multimodal modeling, spatiotemporal feature learning, and future prediction tasks of this invention.

[0066] To achieve unified modeling of structural and functional information, this invention, based on the preprocessing of input data, designs a multimodal joint coding strategy. This strategy aims to map OCT image sequences and visual field functional index sequences to a shared latent representation space, capturing their co-evolutionary relationship in both temporal and semantic dimensions. This latent space not only serves as the core carrier for subsequent diffusion modeling and prediction tasks but also provides effective support for integrated structure-function modeling.

[0067] Step S2: Perform multimodal joint encoding on the OCT image sequence and VF data sequence to generate a joint latent space representation sequence, and generate a time mask based on the follow-up time point to identify data missing situations; In step S2, the multimodal joint coding includes: A variational autoencoder (VAE) is used to encode each frame of OCT image into a structural latent vector. A fully connected mapper is used to encode the VF data of each frame into a functional latent vector; The structural latent vector and the functional latent vector are concatenated along the channel dimension to form a joint latent space representation.

[0068] After step S2, the method further includes: During the model training phase, a frame is randomly selected from the valid data frames according to the time mask as the target prediction frame, and an occlusion mask is generated to block the target prediction frame in order to construct training samples.

[0069] In terms of structural modality, this invention employs a variational autoencoder (VAE) as the image coding backbone. This is applied to each frame of the normalized and size-normalized OCT image. Using an image encoder Map it to a latent space representation:

[0070] in C′ < C , h < H , w < W Typically, a convolutional encoder compresses an image into lower-dimensional latent space features, preserving its high-level structural semantics while significantly reducing the computational burden on subsequent models. This encoding process can be initialized using a pre-trained image compression model to accelerate convergence and improve encoding stability.

[0071] In terms of functional modalities, the visual field index sequence Standardized to during the preprocessing stage d A real-valued vector. To align and fuse with the OCT image encoding results in the latent space, this invention designs a lightweight fully connected mapper. The view vector of each frame Mapped to a functional latent vector of the same dimension as the image encoding:

[0072] This mapper can be viewed as a parameter-learnable expansion process that embeds abstract numerical metrics into the same tensor structure as the image latent space, thereby achieving cross-modal semantic and spatial alignment.

[0073] After encoding the two modalities separately, this invention concatenates the structural latent vector and the functional latent vector along the channel dimension to form a joint latent space representation:

[0074] For length of T The follow-up sequence ultimately yields the joint representation sequence. This sequence will be used as input for subsequent spatiotemporal modeling processes.

[0075] Step S3: Based on the historical joint latent space representation sequence and the temporal mask, a spatiotemporal attention mechanism is used to model the context dependency relationship to obtain the conditional context sequence; In step S3, the spatiotemporal attention mechanism is based on a modified Transformer structure and introduces temporal and spatial encoding to capture temporal dependencies and spatial feature responses.

[0076] In practice, due to missing data at certain time points or the need to construct training samples by occlusion, it is necessary to combine time masks. M A and masking M The latent space sequence is further processed. For frames with missing or occluded images, their corresponding latent vectors are replaced with all-zero tensors; for frames with missing functional indicators, whether to use their embedded representation is controlled by a mask. In this way, the validity states at different time points can be explicitly modeled during the encoding stage, providing a more accurate structure-function sequence representation for downstream spatiotemporal modeling.

[0077] The construction of the joint latent space not only achieves dimensional unification and semantic synergy of structural and functional information, but also provides a standardized data representation foundation for subsequent diffusion modeling. The weight parameters among the multimodal encoders can be jointly trained or initialized using pre-trained weights according to different task objectives, ensuring flexible adaptability and good generalization performance in different application scenarios. Through the unified encoded multimodal sequences, the deep correlation between structural changes and functional impairments in the course of glaucoma can be further explored, thereby improving the overall model's performance in predicting future states.

[0078] The core of the glaucoma progression prediction method proposed in this invention lies in achieving the generation of optic nerve structure images and the quantitative prediction of progression risk at future time points through the fusion of structure-function temporal information modeling. To this end, based on the construction of a multimodal latent space, a generative architecture combining diffusion processes and spatiotemporal attention mechanisms is introduced. This architecture can capture temporal evolution patterns while preserving structural feature details and fully utilize historical information to conditionally model future states. The overall design uses the latent space sequence as input, models its contextual dependencies through a spatiotemporal attention mechanism, and gradually recovers the target frame representation based on a backdiffusion model to achieve the conditional generation task.

[0079] Step S4: Input the conditional context sequence into a conditional diffusion model to generate the target latent vector at future target time points; Diffusion models are a class of generative models based on a progressive noise addition and denoising process, exhibiting good stability and generalization ability in image generation and sequence modeling. Their core idea is to use a multi-step Markov process to gradually add noise to the original data until it reaches a standard Gaussian distribution, and then train an inverse model to progressively reconstruct the original data. In this invention, the joint latent vector corresponding to the OCT image at the target time point is considered as the target state to be generated, and the target frame is recovered in the joint latent space using a conditional diffusion mechanism.

[0080] set up This represents the encoded multimodal joint latent space sequence. Z target If the target prediction frame (i.e., the occluded frame) is to be first trained during the training phase... Z target Adding Gaussian noise creates a forward process:

[0081] in α t This is the forward diffusion coefficient sequence. T d This represents the number of diffusion steps. Then, a denoiser is trained. D θ ， At every step t According to the historical sequence Conditional generation （t-1） Gradually approaching the true latent vector Z target The ultimate goal is to minimize the denoising error.

[0082] in For conditional context sequences.

[0083] In the aforementioned diffusion modeling process, to enhance the representational ability of the context sequence for the target prediction frame, a spatiotemporal attention mechanism is introduced to extract key temporal dependencies and spatial feature responses from historical frames of variable length. This mechanism is based on a modified Transformer structure to adapt to the sequential input of image tensor structures. First, the latent space tensor... The sequence is flattened into a patchtoken sequence, and then temporal and spatial information is added through positional encoding to form the attention input matrix. Each token combines its original spatial location with a timestamp.

[0084] Within the attention layer, a dual positional encoding mechanism is introduced. Temporal encoding uses sine and cosine functions:

[0085] Spatial coding is based on direct embedding of two-dimensional location information from the image. The final attention weight calculation form is as follows:

[0086] in , These are the time and space bias matrices, respectively, used to guide the model to learn patterns that evolve over time in dynamic sequences.

[0087] Attention mechanisms can be stacked in multiple layers to enhance the ability to capture long-range dependencies through multi-head mechanisms, and to maintain training stability through residual connections and layer normalization. The final contextual representation is embedded as a conditional input into each step of the diffusion inverse process to help the denoising network recover the latent vector of the target frame.

[0088] Furthermore, in the design of the denoising process, latent space recovery is mainly achieved through a U-Net-type network. The input is a noisy frame. The spatiotemporal representation, along with the one derived from the contextual attention mechanism, is fused and fed into a multi-scale convolutional decoder to progressively reconstruct the latent representation from shallow to deep. This design (or processing method) is particularly well-suited for preserving spatial details and structural edges, enhancing the interpretability and clinical usability of the generated images.

[0089] During training, the location of the sampled target frame needs to match the occlusion mask. M Maintain consistency and ensure that the input context frame is a valid frame (i.e. m i =1). This conditional random occlusion training mechanism improves the robustness and generalization ability of the model, enabling it to adapt to real-world application scenarios with different follow-up densities and time spans.

[0090] Step S5: Input the target latent vector into a multi-task output module to simultaneously generate the predicted OCT image and glaucoma progression risk score for the future target time point.

[0091] In step S5, the multi-task output module includes: An image decoder is used to restore the target latent vector into the predicted OCT image; A risk prediction classifier is used to calculate the glaucoma progression risk score based on the target latent vector.

[0092] During the inference phase, this invention uses complete historical frames as context input and randomly initialized Gaussian noise tensors as the initial state of the target latent vector, sequentially passing through... Td After several denoising iterations, the latent space representation of the target time point is finally restored, and then the image decoder restores it into a structural image, realizing the visualization and prediction of the future state of the optic nerve.

[0093] This fusion architecture combines the generative advantages of diffusion models with the spatiotemporal modeling capabilities of Transformers. This not only improves the spatial fidelity of structural image predictions but also enhances the model's ability to capture glaucoma progression trends, making it particularly suitable for prediction tasks in clinical settings with complex situations such as sparse follow-up and inconsistent intervals. By performing the generation process in the latent space and introducing an explicit spatiotemporal modeling mechanism, it effectively alleviates the problem of deep models easily falling into overfitting or misprediction in the context of scarce samples, ensuring that the generated images possess high credibility and clinical interpretability.

[0094] This invention constructs a multi-task prediction architecture with dual output paths based on latent space modeling, modeling both the structural optic nerve image generation task and the functional glaucoma progression risk prediction task. By performing image decoding and risk score calculation in parallel on the same latent space representation, this invention achieves simultaneous reasoning for both structural and functional results, satisfying clinicians' needs for visualizing future images and providing quantitative support for disease progression trends at the decision-making level.

[0095] The image generation task uses the latent vectors output by the diffusion model as input to reconstruct the OCT image at the target time point. To ensure that the generated image possesses realistic structural features and spatial discriminability, the decoding step of this invention employs... The latent vectors of the target frame are restored to a two-dimensional image. The decoder adopts a top-down U-Net architecture, which has the ability to jointly restore shallow spatial edges and deep semantic information. The decoding process is as follows:

[0096] in This represents the latent vector of the target frame after the diffusion model has completed its iterations. This is the final generated grayscale OCT image. To guide the image decoder in learning high-quality structure reconstruction, a combined loss function is introduced during training, comprising two parts: pixel-level reconstruction error and structure-aware loss.

[0097] The first term is the L1 norm reconstruction error, reflecting the overall gray-level overlap of the image; the second term is the structural similarity index (SSIM), used to measure the consistency between the image's edge structure and texture distribution; λ1 and λ2 are weighting coefficients. This combined design ensures that the generated image maintains both macroscopic contours and detail resolution, meeting doctors' needs for visual identification of optic nerve atrophy areas.

[0098] The risk prediction task takes a structure-function joint representation sequence as input and outputs the probability of glaucoma progression at a target time point. The modeling objective is to infer whether future visual function will deteriorate based on existing historical frames; therefore, it is constructed as a binary classification task to predict the target label. This indicates whether the current time point is in a progress phase. Progress labels can be constructed based on the rate of change of MD values, custom clinical thresholds, or expert annotations, and have good generalization potential.

[0099] To adapt to latent space representation, the risk prediction path is designed based on a multilayer perceptron (MLP), with the risk classification step consisting of stacked fully connected layers. Receive the latent vector input of the target frame and output a progress risk score:

[0100] in This represents the Sigmoid activation function. This represents the predicted progress probability. The training process uses the binary cross-entropy loss function:

[0101] The design aims to minimize the probability deviation between predicted values ​​and true labels. It also features direct linkage to clinical thresholds, supporting flexible definition of "high-risk" criteria to aid in individualized follow-up and intervention timing assessment.

[0102] Considering the differences in target type and loss scale between the two types of tasks, this invention adopts a weight-balanced joint loss strategy for overall training optimization:

[0103] in α , β This is an adjustable coefficient used to control the training ratio between image reconstruction and risk classification. By introducing dual-task supervision signals during the training phase, the model can simultaneously consider the quality of structural image reconstruction and functional risk identification ability when learning latent space representations, thereby avoiding representational bias that is excessively biased towards one modality.

[0104] In addition, the method also includes: The model is trained based on a joint loss function, which includes an image reconstruction loss to guide the training of the image decoder and a risk classification loss to guide the training of the risk prediction classifier. Specifically, an adaptive weight learning mechanism is used to dynamically adjust the weights of the image reconstruction loss and the risk classification loss in the joint loss function based on task uncertainty.

[0105] Furthermore, to enhance the interpretability and visualization of the prediction results, this invention supports saliency labeling of progression regions in the generated images. Heatmap tools such as Grad-weighted Class Activation Mapping (Grad-CAM) are used to map activation regions in the risk prediction path back to the input image, assisting doctors in understanding the basis of the model's judgment. Simultaneously, it can also output auxiliary visualization results such as inter-frame difference maps and latent vector trajectory change trend maps, further improving the clinical user experience.

[0106] The entire multi-task output process possesses end-to-end inference capabilities. The input is a historical follow-up sequence, and the output includes both structural-level images and functional-level risk scores, meeting the requirements of an integrated prediction process design. Its dual-path structure also facilitates model pruning or functional customization according to actual use cases. For example, only the risk scoring branch can be retained on low-resource platforms, while image generation can be retained for progress simulation demonstrations in teaching scenarios, demonstrating good engineering adaptability and deployment flexibility.

[0107] Clinical follow-up of glaucoma patients exhibits significant irregularities, specifically manifested in inconsistent numbers of follow-up time points, uneven intervals, and missing time points for each individual. This data distribution is common in real-world electronic medical records. Using traditional equal-interval time series modeling methods may lead to misinterpretation of temporal dynamics or even loss of key trends. Therefore, this invention introduces a modeling mechanism and enhancement method designed for irregular time series in the input encoding and training strategies to improve the model's adaptability to real-world follow-up patterns and its time-awareness.

[0108] First, addressing the issue of inconsistent follow-up time intervals, this invention guides the model to focus on potential structure-function evolution shifts that occur over time by explicitly modeling timestamp difference vectors. Let the first... i The follow-up period is [time period]. t i Construct time difference sequences:

[0109] After log transformation and normalization, the sequence is used as a time-coded signal and embedded into the conditional channels of the attention mechanism and denoising network to achieve structural modeling of the differences in follow-up intervals. During the training phase, the time offset is used as a learnable bias in the calculation of attention weights, helping the model to determine that frames with similar times are more context-dependent.

[0110] Secondly, to alleviate the problem of incomplete input caused by missing frames in actual data, this invention employs a unified time masking mechanism for explicit identification. For any frame... Z iIf there is no corresponding observation data, then set it to the time mask. M A The corresponding term is 255, and the latent vector is padded with all-zero tensors during the input phase to ensure that the model automatically skips this position during encoding rather than incorrectly learning spurious features. This mechanism ensures that the model has robust missing value tolerance when facing real clinical samples.

[0111] To address the target frame occlusion construction problem, while randomly selecting time points as targets and occluding them, a dynamic occlusion enhancement strategy is introduced. This strategy adjusts the occlusion probability distribution of the target frame based on factors such as historical frame density and a custom time window width, enhancing the model's balance between near-term and long-term predictions. Specifically, when sampling occluded frames, a normalized Gaussian kernel function is used to construct the temporal sampling distribution:

[0112] in Indicates the center point of the current follow-up period. Controlling the dispersion of sampling locations. This strategy can adjust the model's focus on early predictions or long-term trends, adapting it to different clinical prediction scenarios.

[0113] In terms of temporal enhancement, this invention introduces frame order perturbation and local window resampling techniques. Frame order perturbation simulates follow-up recording errors or temporal drift, i.e., randomly swapping the order of two adjacent frames to improve the model's robustness to perturbations of neighboring frames. Local window resampling selects a time segment from the original sequence and reconstructs a continuous subsequence using a known mask, simulating partial sampling or truncation scenarios. It is often used for sub-model training in scenarios with insufficient data or lightweight deployment.

[0114] Furthermore, explicit temporal embedding is introduced during the joint latent space construction and subsequent modeling. This embedding converts each frame's timestamp into a fixed-length vector using sine and cosine encoding and concatenates it with the latent space tensor to provide a consistent temporal reference frame across individuals. Its form is:

[0115] in τ k This is a set of periodic scale hyperparameters. The temporal embedding vector is concatenated with the image latent vector and visual features during the multimodal joint encoding stage, improving the consistency of the model's expression across different time scales.

[0116] To achieve the dual objectives of structural image generation and glaucoma progression risk prediction, this invention constructs a multi-task joint modeling and training optimization framework. The core training tasks include: a structural reconstruction task using future OCT images as supervised targets, and a risk prediction task labeled by whether significant visual function deterioration occurs at a specific time point. These two tasks respectively characterize the anatomical structural changes and functional impairment risk in glaucoma progression, and their joint modeling is of great significance for achieving comprehensive disease progression prediction.

[0117] The image reconstruction task aims to recover the structure of OCT images at a target time point, with the supervision signal derived from real patient follow-up images. To improve the quality and clinical usability of the generated images, this invention designs a combined image loss function, encompassing both pixel-level error and structural similarity measurement. The pixel error component employs the L1 loss function to enhance the robustness and fidelity of edge features. For the structural similarity component, this invention introduces the Structural Similarity Index (SSIM) to perform holistic perceptual modeling of the image's brightness, contrast, and texture structure. The two are jointly defined as follows:

[0118] in Indicates the generation of an image. Represents a real image. and The weights of the two types of losses are controlled, initially set empirically to λ1=1.0 and λ2=0.5, and can be dynamically adjusted through validation set performance. This design balances grayscale error with the requirement of structure preservation, improving the model's ability to reproduce fine structures (such as the edge of the optic nerve head and the interlaminar interface of the macula).

[0119] The risk prediction task uses visual field function data to construct labels and define whether functional degradation criteria (such as a decrease in visual field performance (MD) exceeding a certain threshold or a significant decrease in response to fixed-point stimuli) will be reached at a future target time point. The prediction output is a risk probability value. [0,1], label The loss function uses the standard binary cross-entropy form:

[0120] It is used to measure the degree of match between the model output and the true risk label. In the training set, if the ratio of positive to negative samples is unbalanced, it can also be extended to focal loss or class-weighted cross-entropy to balance the learning process.

[0121] To achieve synchronous optimization between the two tasks, this invention constructs a unified multi-task total loss function:

[0122] in α and β This represents the task loss weights. Directly setting fixed weights often fails to adapt to the dynamic changes in task difficulty during training, easily leading to one task dominating the gradient direction and suppressing the learning of another task. To address this, this invention introduces an adaptive weight learning mechanism based on task uncertainty modeling.

[0123] This mechanism introduces a learnable variance term for each task. The relative importance of different task items in the loss function is dynamically adjusted. The reconstructed total loss function is:

[0124] in The variance term participates in model training through backpropagation, allowing the training process to adjust its learning rate according to the stability and information density of the task itself. For example, when the image reconstruction loss fluctuates greatly, its corresponding weight is automatically reduced to prevent its gradient from dominating global parameter updates; when the risk prediction loss decreases steadily, its weight is increased to enhance risk assessment capabilities.

[0125] In actual training, the two tasks share the encoder and part of the bridging layer structure, thus requiring special handling of gradient conflicts between tasks. If the gradient directions of different tasks are inconsistent, it can easily lead to oscillating updates of shared layer parameters or even convergence failure. To alleviate this problem, this invention introduces an optimization strategy based on gradient direction control.

[0126] On the one hand, the GradNorm method can be used to forcibly control the gradient norm of each task, ensuring that their rate of change remains consistent. Specifically, in each training round, the task loss gradient is calculated separately. norm and set a target rate of change. r i By adding a loss penalty term to adjust the task update intensity, the gradient growth rate is made more consistent, thus ensuring that different tasks do not become "unbalanced" due to differences in learning rate.

[0127] On the other hand, the Projected Gradient Descent (PCGrad) method can also be used to solve the gradient direction orthogonality problem. When the angle between the gradient directions of two tasks is too large (i.e., the negative projection is obvious), the conflicting gradient vectors are projected along the non-conflicting direction to reduce directional interference. Mathematically, this can be expressed as:

[0128] This ensures that the main gradient update direction follows the direction of minimum conflict. This strategy has been proven effective in multi-task learning such as semantic segmentation and image translation, and in this invention, it significantly improves training convergence speed and stability.

[0129] To further improve the efficiency of inter-task interaction, this invention employs an asynchronous gradient backpropagation mechanism, setting independent batch sizes and learning rate buffers in different sub-tasks. This ensures that structural modeling and functional classification tasks maintain independent convergence rhythms within the shared network, avoiding overfitting or gradient drift. Furthermore, to ensure that the supervision information for each task fully covers diverse case combinations, a batch sampling strategy based on label coverage is introduced during training. This ensures that each batch contains samples with multiple progression patterns and visual field impairment degrees, enriching the feature interaction capabilities between tasks.

[0130] In some specific embodiments, the method further includes: Obtain the patient's individual characteristic information and encode it into an individual condition vector; The individual conditional vectors are integrated into the spatiotemporal attention mechanism and / or the multi-task output module to achieve individualized modeling of different patients' pathological evolution patterns.

[0131] In clinical practice, significant differences exist among patients in terms of glaucoma progression rate, anatomical structure, and functional compensation ability. These individual differences not only affect the model's generalization performance but also directly limit its predictive accuracy in specific cases. Therefore, this invention introduces an individual difference modeling mechanism, constructing learnable conditional representation vectors to enhance the model's ability to perceive diverse pathological evolution patterns.

[0132] Specifically, for each patient p Construct an individual feature vector e p This includes demographic information (such as age and sex), structural indicators at the diagnostic stage (such as the vertical cup-to-disc ratio of the optic nerve head and the mean retinal thickness in the macular region), functional status (such as the initial MD value), and the frequency of historical treatment interventions. This vector is generated through an independent feature encoding network. Transform into low-dimensional conditional embedding z p This is then combined with the latent space representation output by the visual encoder. z OCT and functional characteristics representation z VF It is fused and used as a regulatory factor for spatiotemporal attention tasks to guide the shift of attention weights at the individual level, thereby adapting to the rate of structural evolution and damage pathways of different patients.

[0133] This conditional vector also participates in the multilayer perceptron (MLP) network in the risk prediction branch, injecting individual conditional information after the top-level feature concatenation. This allows the model to generate differentiated risk assessments for the same structural state in different populations, thus reflecting the clinical reality of "same structure, different risks".

[0134] Furthermore, to address the issues of sparse training samples and pattern deviations in a small number of special cases (such as high myopia combined with glaucoma, juvenile open-angle glaucoma, etc.), this invention designs a lightweight personalized fine-tuning mechanism. While keeping the core parameters of the model frozen, only the weights of the later layers in the risk prediction branch are locally optimized. The optimization method is gradient-freeze training, i.e., freezing all shared layers and only unlocking the last two layers of the individual task head, using a small set of individual samples (such as the patient's historical follow-up sequence) for minor updates. This method can significantly improve the model's prediction accuracy and clinical interpretability on marginal individuals without affecting overall convergence.

[0135] In medical image prediction, the reliability of the generated results is far more important than numerical accuracy, especially in progressive chronic diseases such as glaucoma, where excessively uncertain model outputs may mislead clinical judgment. Therefore, this invention introduces a confidence estimation mechanism in both the risk prediction path and the image generation path to improve the clinical interpretability and stability of the model output.

[0136] For the risk prediction part, the Monte Carlo Dropout (MCDropout) strategy is used for uncertainty modeling. Specifically, during the inference phase, the Dropout activation mechanism is retained (e.g., setting the Dropout probability to 0.2), and the algorithm is executed for each input sample. K The second forward propagation (typically K=30) yields the risk prediction distribution. Calculate its mean μ with standard deviation σ :

[0137] Standard deviation σ This refers to the model's uncertainty measure of the prediction result, with a threshold that can be set between 0.2 and 0.25. When σ is higher than the threshold, the model can output a low confidence level, suggesting that the doctor make a comprehensive judgment based on other test results or conduct a follow-up examination.

[0138] In the image generation path, to identify ambiguous regions in the model's anatomical structure generation, this invention employs an uncertainty heatmap generation mechanism. The specific process is as follows: generating images under the same input conditions... K OCT images For all images at each pixel ( x , y Calculate the standard deviation of grayscale:

[0139] in for K The pixel mean and standard deviation of the images. S ( x , y This can be mapped into a heatmap to show the model's generation confidence in areas such as the edge of the optic disc and the fovea of ​​the macula, helping doctors to determine whether there is unpredictability in the presence of key structures.

[0140] Regarding the optimizer settings, this invention employs the AdamW optimizer (an Adam optimizer with weight decay regularization) to improve convergence stability and anti-overfitting capabilities. The learning rate strategy uses cosine annealing, with an initial learning rate of 1×10⁻⁶. -4 The minimum learning rate is 1×10. -6 Before training T A linear warm-up process is introduced for 0=10 epochs. The total number of training epochs is controlled at 120–150 epochs. An early stopping strategy is introduced based on the performance on the validation set, and training is stopped if the loss does not decrease for 10 consecutive epochs.

[0141] To enhance the robustness of the model in real-world clinical scenarios, such as inconsistent follow-up time points, inconsistent image quality, and loss of functional indicators, this invention introduces several data augmentation strategies: ① Temporal perturbation enhancement: During training, the timestamps in the follow-up sequence are perturbed, such as by adding a ±Δt perturbation term to the original time points, to simulate the modeling uncertainty caused by time shifts in real clinical follow-ups. ② Structural occlusion enhancement: 5–15% of the original OCT images are randomly occluded, including the fovea, optic nerve area, and vascular projection area. This masking simulates problems such as image loss, occlusion, and interference artifacts, enhancing the model's structural reconstruction capability. ③ Modal interference enhancement: Gaussian noise or random missing points are introduced into the input visual field indicators to train the model to be robust to scenarios with incomplete functional information and improve its feature redundancy learning ability. Through these enhancement strategies, the model can more effectively adapt to the distribution of clinical data and noise interference in the real world, ensuring its stability and security after deployment.

[0142] This invention addresses the need for predicting the progression of glaucoma by proposing a multimodal spatiotemporal modeling path that integrates OCT images and visual field data. It achieves deep fusion of structural and functional information through joint encoding and latent space alignment mechanisms, and combines spatiotemporal attention structures and a diffusion generation model to realize high-quality prediction and risk probability output of future optic nerve images. The method has the ability to model irregular follow-up times and incorporates task-adaptive weighting, multi-task gradient coordination, and masking enhancement strategies during training to improve the overall model's convergence stability and predictive performance.

[0143] Furthermore, the method introduces individualized conditional vectors and fine-tuning mechanisms to enhance its adaptability to borderline cases. It also incorporates Monte Carlo Dropout and structural heatmaps to quantify prediction reliability, aiding clinical interpretation and decision-making. The overall scheme comprises a closed-loop technology system consisting of input modeling, encoding fusion, task optimization, and reliable output. This overcomes the limitations of traditional static image classification, providing methodological innovation and a basis for patent protection for the dynamic tracking and intervention timing assessment of chronic optic neuropathy.

[0144] This invention proposes a method for predicting glaucoma progression risk by integrating a spatiotemporal attention mechanism. It aims to address the technical challenges in current glaucoma diagnosis and treatment, such as insufficient predictive ability, fragmentation of structural and functional data, and irregular follow-up data. The method constructs a deep learning architecture capable of processing multimodal medical image sequences and supporting dynamic progression trend modeling and future state prediction. Using structural OCT image sequences and functional visual field examination data obtained from multiple patient follow-ups as core inputs, the method leverages an improved diffusion generation mechanism combined with a multidimensional spatiotemporal attention mechanism to achieve prospective prediction of glaucoma status at specific future time points. This includes the generation of future optic nerve structure images and quantitative estimation of progression risk probability, providing clinicians with more timely and valuable auxiliary decision support for intervention guidance.

[0145] In summary, the present invention is capable of: 1. High prediction accuracy: By adopting joint structural and functional modeling and integrating spatiotemporal masking mechanism, it effectively captures the dynamic progression pattern of the disease, and has higher accuracy in predicting the progression of glaucoma in the next 12 months compared with existing technologies; 2. Significant clinical application value: The model quantitatively outputs the probability of future disease progression and the corresponding predicted OCT images, enabling clinicians to proactively formulate or adjust treatment plans, such as optimizing medication strategies or the timing of surgery.

[0146] 3. High data utilization: By integrating OCT image sequences with visual field (VF) functional indicators into the same diffusion model framework, a comprehensive integration of structure, function, and time is achieved, effectively improving the overall utilization efficiency of medical data.

[0147] 4. Strong clinical adaptability: The spatiotemporal masking mechanism can effectively handle the problem of irregular collection time of clinical follow-up data, which is more in line with the actual needs of real clinical environment and enhances the generalization and applicability of the algorithm.

[0148] 5. Strong ability to model individual differences: This invention introduces conditional vectors and personalized parameter tuning mechanisms to adapt to the pathological evolution rhythm of different patients, improve the prediction accuracy under marginal samples, and help to realize more individualized disease management strategies.

[0149] 6. High credibility and strong interpretability: By combining Monte Carlo Dropout and structural heatmaps, the uncertainty of the model output results is quantitatively displayed, which improves the clinical interpretability and safety of the prediction results.

[0150] Example 3: This invention discloses a glaucoma progression risk prediction system that integrates spatiotemporal attention mechanisms. Figure 4 This is a structural diagram of a glaucoma progression risk prediction system incorporating a spatiotemporal attention mechanism according to an embodiment of the present invention; as shown. Figure 4 As shown, the system 100 includes: The data acquisition module 101 is configured to acquire structural optical coherence tomography (OCT) image sequences and functional field of view (VF) data sequences from multiple follow-up visits of the patient. The joint coding module 102 is configured to perform multimodal joint coding on the OCT image sequence and VF data sequence to generate a joint latent space representation sequence, and generate a time mask based on the follow-up time point to identify data missing situations. The context modeling module 103 is configured to model context dependencies based on the historical joint latent space representation sequence and the temporal mask, and obtain a conditional context sequence through a spatiotemporal attention mechanism. The prediction generation module 104 is configured to input the conditional context sequence into the conditional diffusion model to generate the target latent vector at a future target time point; The multi-task output module 105 is configured to input the target latent vector to simultaneously generate a predicted OCT image and a glaucoma progression risk score for the future target time point.

[0151] According to a system of a second aspect of the present invention, the joint coding module 102 is specifically configured such that the multimodal joint coding includes: Each frame of OCT image is encoded into a structural latent vector using a variational autoencoder (VAE). A fully connected mapper is used to encode the VF data of each frame into a functional latent vector; The structural latent vector and the functional latent vector are concatenated along the channel dimension to form a joint latent space representation.

[0152] According to the system of the second aspect of the present invention, the joint coding module 102 is specifically configured to, during the model training phase, randomly select a frame from the valid data frames as the target prediction frame according to the time mask, and generate an occlusion mask to mask the target prediction frame in order to construct training samples.

[0153] According to the system of the second aspect of the present invention, the context modeling module 103 is specifically configured such that the spatiotemporal attention mechanism is based on a modified Transformer structure and introduces temporal encoding and spatial encoding to capture temporal dependencies and spatial feature responses.

[0154] According to the system of the second aspect of the present invention, the multi-task output module 105 is specifically configured to include: an image decoder for restoring the target latent vector to the predicted OCT image; and a risk prediction classifier for calculating the glaucoma progression risk score based on the target latent vector.

[0155] According to the system of the second aspect of the present invention, a training module is further included, configured to train the model based on a joint loss function, the joint loss function including an image reconstruction loss for guiding the training of the image decoder and a risk classification loss for guiding the training of the risk prediction classifier; Specifically, through an adaptive weight learning mechanism, the weights of the image reconstruction loss and the risk classification loss in the joint loss function are dynamically adjusted according to the task uncertainty.

[0156] The system according to a second aspect of the invention further includes an individualization module configured to acquire individual characteristic information of the patient and encode it into an individual condition vector; The individual conditional vectors are integrated into the spatiotemporal attention mechanism and / or the multi-task output module to achieve individualized modeling of different patients' pathological evolution patterns.

[0157] Example 4: This invention discloses an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps in the glaucoma progression risk prediction method that integrates a spatiotemporal attention mechanism, as disclosed in Embodiment 1 of this invention.

[0158] Figure 5 This is a structural diagram of an electronic device according to an embodiment of the present invention, such as... Figure 5As shown, the electronic device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, Near Field Communication (NFC), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0159] Those skilled in the art will understand that Figure 5 The structure shown is merely a structural diagram of the part related to the technical solution of this disclosure and does not constitute a limitation on the electronic device to which the solution of this application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0160] Example 5: This invention discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a glaucoma progression risk prediction method in any of Embodiment 1 of this invention, which integrates a spatiotemporal attention mechanism.

[0161] Please note that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. The above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

[0162] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0163] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0164] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0165] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0166] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0167] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0168] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0169] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0170] While specific embodiments of the invention have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.

Claims

1. A method for predicting the risk of glaucoma progression by integrating spatiotemporal attention mechanisms, characterized in that, The method includes: Step S1: Obtain structural optical coherence tomography (OCT) image sequences and functional field of view (VF) data sequences from multiple follow-up visits of the patient; Step S2: Perform multimodal joint encoding on the OCT image sequence and VF data sequence to generate a joint latent space representation sequence, and generate a time mask based on the follow-up time point to identify data missing situations; Step S3: Based on the historical joint latent space representation sequence and the temporal mask, perform context dependency modeling through a spatiotemporal attention mechanism to obtain a conditional context sequence; Step S4: Input the conditional context sequence into the conditional diffusion model to generate the target latent vector at future target time points; Step S5: Input the target latent vector into the multi-task output module to simultaneously generate the predicted OCT image and glaucoma progression risk score for the future target time point.

2. The method for predicting glaucoma progression risk by incorporating spatiotemporal attention mechanisms according to claim 1, characterized in that, In step S2, the multimodal joint coding includes: Each frame of OCT image is encoded into a structural latent vector using a variational autoencoder (VAE). A fully connected mapper is used to encode the VF data of each frame into a functional latent vector; The structural latent vector and the functional latent vector are concatenated along the channel dimension to form a joint latent space representation.

3. The method for predicting the risk of glaucoma progression by incorporating spatiotemporal attention mechanisms according to claim 1, characterized in that, After step S2, the method further includes: During the model training phase, a frame is randomly selected from the valid data frames according to the time mask as the target prediction frame, and an occlusion mask is generated to block the target prediction frame in order to construct training samples.

4. The method for predicting the risk of glaucoma progression by incorporating spatiotemporal attention mechanisms according to claim 1, characterized in that, In step S3, the spatiotemporal attention mechanism is based on a modified Transformer structure and introduces temporal and spatial encoding to capture temporal dependencies and spatial feature responses.

5. The method for predicting the risk of glaucoma progression by incorporating spatiotemporal attention mechanisms according to claim 1, characterized in that, In step S5, the multi-task output module includes: An image decoder is used to restore the target latent vector into the predicted OCT image; A risk prediction classifier is used to calculate the glaucoma progression risk score based on the target latent vector.

6. The method for predicting the risk of glaucoma progression by incorporating spatiotemporal attention mechanisms according to claim 5, characterized in that, The method further includes: The model is trained based on a joint loss function, which includes an image reconstruction loss used to guide the training of the image decoder and a risk classification loss used to guide the training of the risk prediction classifier. Specifically, through an adaptive weight learning mechanism, the weights of the image reconstruction loss and the risk classification loss in the joint loss function are dynamically adjusted according to the task uncertainty.

7. The method for predicting the risk of glaucoma progression by incorporating spatiotemporal attention mechanisms according to claim 1, characterized in that, The method further includes: Obtain the patient's individual characteristic information and encode it into an individual condition vector; The individual conditional vectors are integrated into the spatiotemporal attention mechanism and / or the multi-task output module to achieve individualized modeling of different patients' pathological evolution patterns.

8. A glaucoma progression risk prediction system integrating spatiotemporal attention mechanisms, characterized in that, The system includes: The data acquisition module is configured to acquire structural optical coherence tomography (OCT) image sequences and functional field of view (VF) data sequences from multiple follow-up visits of patients. The joint coding module is configured to perform multimodal joint coding on the OCT image sequence and VF data sequence to generate a joint latent space representation sequence, and generate a time mask based on the follow-up time point to identify data missing situations. The context modeling module is configured to model context dependencies based on the historical joint latent space representation sequence and the temporal mask, and obtain a conditional context sequence through a spatiotemporal attention mechanism. The prediction generation module is configured to input the conditional context sequence into the conditional diffusion model to generate the target latent vector for future target time points; The multi-task output module is configured to input the target latent vector to simultaneously generate a predicted OCT image and a glaucoma progression risk score for the future target time point.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it implements the steps in the method for predicting the risk of glaucoma progression by incorporating a spatiotemporal attention mechanism as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the glaucoma progression risk prediction method that integrates a spatiotemporal attention mechanism as described in any one of claims 1 to 7.

Images