An eye fundus lesion evolution prediction method based on time-series multi-modal data

Abstract

This invention belongs to the field of deep learning and ocular lesion evolution prediction technology, and relates to a method for predicting the evolution of fundus lesions based on temporal multimodal data. The method involves inputting a dual-timepoint OCT slice sequence into an image feature extraction module, which outputs dual-timepoint slice-level visual features and dual-timepoint image-enhanced slice-level image features. The dual-timepoint slice-level visual features are fused to obtain a first fused feature. The dual-timepoint slice-level visual features and image biomarkers are input into an image biomarker-image cascade fusion module to obtain a second fused feature. The dual-timepoint image-enhanced slice-level image features and dual-timepoint clinical information are input into a clinical information-image progression fusion module to obtain a clinical information-image temporal progression representation. Pooling is performed on the first fused feature, the second fused feature, and the clinical information-image temporal progression representation. The three pooled features are then concatenated to obtain the fundus lesion evolution features, thereby predicting the evolution direction of fundus lesions.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and ocular lesion evolution prediction technology, and in particular to a method for predicting the evolution of fundus lesions based on temporal multimodal data. Background Technology

[0002] Macular-related fundus diseases, such as diabetic macular edema, wet age-related macular degeneration, and other diseases accompanied by retinal exudates, edema, or neovascularization, are a significant cause of vision loss in patients. These diseases typically have a long course, are characterized by recurrent activity, significant individual variability, and the need for long-term follow-up management. Anti-vascular endothelial growth factor (VEGF) injections are an important treatment for these fundus diseases, inhibiting abnormal angiogenesis and vascular leakage, reducing macular edema, and improving or maintaining visual function to some extent. However, the response to anti-VEGF treatment varies significantly among patients: some patients experience rapid control of lesion activity after treatment, allowing for extended follow-up visits or injection intervals, while others still experience residual fluid, disease fluctuations, or recurrence risk, requiring continued treatment. Therefore, making early predictions about potential recurrence based on previous examination results has become a crucial issue in the long-term management of fundus diseases.

[0003] Based on the above needs, the multi-source examination information obtained during follow-up provides an important basis for assessing subsequent treatment needs. Among them, optical coherence tomography (OCT) is one of the most important imaging examination methods in follow-up assessment, reflecting information such as retinal layer structure, fluid distribution, lesion morphology, and tissue abnormalities. A single OCT examination typically consists of multiple tomographic slices acquired in a certain order, often referred to as B-scans. Different slices collectively reflect the structural state of the macular region at different locations. Furthermore, the imaging biomarkers extracted or summarized from the OCT images can provide a more concentrated characterization of fluid status, interlaminar structural abnormalities, pigment epithelial changes, and other lesion signs, thus more effectively depicting the lesion status. Simultaneously, clinical information such as visual acuity (VA), central subfield thickness (CST), disease diagnosis, and previous treatment can provide auxiliary judgment from the perspectives of visual function, disease activity, and treatment response.

[0004] Predicting lesion evolution after anti-VEGF treatment is typically not determined by a single examination result, but rather relies more on dynamic information about lesion changes during continuous patient follow-up. Early prediction methods, relying on single examination results, manual experience, or a limited number of structured variables, often resulted in limited accuracy and difficulty in reflecting the temporal dynamics of the disease. Subsequently, data-driven methods for time-series follow-up scenarios have gradually emerged, encompassing analysis based on single-time-point images, temporal modeling based on multi-time-point image changes, and multimodal joint analysis integrating imaging biomarkers and clinical information. Overall, the key challenge in predicting retinal lesion evolution is how to simultaneously utilize temporal information from multiple follow-ups and multi-source modal information to accurately determine the future direction of lesion evolution in patients.

[0005] Existing predictive methods for the evolution of fundus lesions fall into three categories: The first is a prediction method based on structured clinical variables, which mainly uses structured information such as visual acuity (VA), visual streak (CST), disease diagnosis, and previous treatment as input. It employs traditional machine learning methods such as logistic regression, support vector machines, and random forests to model and analyze treatment response, changes in visual function, or recurrence risk. The second is an analysis method based on medical imaging and multimodal information fusion. This method uses convolutional neural networks to encode OCT images obtained from a single visit, extracting image features such as network hierarchical structure, fluid distribution, and lesion morphology to complete disease classification. Tasks such as lesion identification or treatment response prediction have been developed. To overcome the limitations of single image information, existing technologies have also emerged that combine OCT image features with image biomarkers, clinical variables, and other information for joint modeling. Comprehensive analysis is performed through feature stitching, unified mapping, weighted fusion, or attention mechanisms to improve the ability to represent lesion status and treatment response. The third type is a time-series modeling method for time-series follow-up data. This method extracts features from image data or multimodal data at each follow-up time point, constructs a feature sequence according to time order, inputs it into the model for encoding, and finally outputs the prediction result.

[0006] However, while prediction methods based on structured clinical variables can analyze lesion evolution using information such as VA, CST, and previous treatment, they are insufficient in utilizing fine-grained information contained in OCT images, such as changes in retinal structure, distribution of effusion, and morphology of local lesions. Analysis methods based on single-time-point OCT images, while able to identify the current lesion state, are insufficient in reflecting trends over time, thus failing to adequately support predictions in multi-time-point follow-up scenarios. Furthermore, although existing time-series modeling methods incorporate sequence models such as RNN, LSTM, and Transformer, they still lack the ability to accurately capture key changes in lesions. The utilization of information and the internal structural correlation of a single examination is still insufficient. That is, most follow-up time points are regarded as input sequences arranged in time. The temporal feature expression of patients is mainly obtained through feature stacking, sequence encoding or overall temporal modeling. It often focuses more on forming an overall sequence expression, while lacking more targeted modeling of key changes between visits. The above-mentioned technical defects of insufficient utilization of multiple information, insufficient accuracy of feature modeling and lack of dynamic trend capture ability directly lead to insufficient fit in the characterization of the evolution of fundus lesions, and poor accuracy and robustness in predicting the risk of lesion progression, regression and recurrence.

[0007] In summary, existing methods for predicting the evolution of fundus lesions suffer from insufficient accuracy in characterizing the evolutionary patterns of fundus lesions and low prediction precision. Summary of the Invention

[0008] Therefore, the technical problem to be solved by the present invention is to overcome the problems of insufficient characterization and low prediction accuracy of fundus lesion evolution in existing fundus lesion evolution prediction methods.

[0009] To address the aforementioned technical problems, this invention provides a method for predicting the evolution of fundus lesions based on temporal multimodal data, comprising: Acquire dual-timepoint OCT slice sequences, imaging biomarkers, and dual-timepoint clinical information of the subjects to be tested; The dual-time point OCT slice sequence is input into the image feature extraction module of the fundus lesion evolution prediction model, which outputs dual-time point slice-level visual features and dual-time point image-enhanced slice-level image features; the dual-time point slice-level visual features are fused to obtain the first fused feature; The dual-time point slice-level visual features and image biomarkers are input into the image biomarker-image cascade fusion module in the fundus lesion evolution prediction model. The dual-time point slice-level visual features and image biomarkers are cascaded and fused in chronological order to obtain the second fused feature. The enhanced slice-level image features of the dual time point map and the dual time point clinical information are input into the clinical information-image progression fusion module in the fundus lesion evolution prediction model. The enhanced slice-level image features of the dual time point map and the dual time point clinical information are aligned and fused at the node level in chronological order to obtain the clinical information-image time point progression representation. Pooling is performed on the first fusion feature, the second fusion feature, and the clinical information-image time-point progression representation. The three pooled features are then concatenated to obtain the fundus lesion evolution features. The fundus lesion evolution features are then classified to obtain the evolution direction of the fundus lesions in the subject to be tested.

[0010] Preferably, the dual-timepoint OCT slice sequence is input into the image feature extraction module, which outputs dual-timepoint slice-level visual features and dual-timepoint image-enhanced slice-level image features, including: The OCT slice sequences at each time point are input into the segmentation network, which outputs a binary segmentation mask of the retinal region. The binary segmentation mask is used as the region of interest to extract two-dimensional radiomics features from each slice of the OCT slice sequence at each time point, thereby obtaining the slice-by-slice radiomics features at each time point. The slice-by-slice image omics features at each time point are input into a gating network composed of a multilayer perceptron and a sigmoid activation function, and the slice-level gating weights at each time point are output. The OCT slice sequences at each time point are input into the parallel CNN branch and the visual Transformer branch respectively. The spatial feature maps of each time point output by the CNN branch are processed by generalized mean pooling and linear projection to obtain the slice-level CNN features at each time point. The output of the visual Transformer branch is processed by the projection layer to obtain the slice-level ViT features at each time point. By using slice-level gating weights at each time point, slice-level CNN features and slice-level ViT features at each time point are weighted and fused to obtain slice-level visual features at each time point; Using the slice-level visual features at each time point as the initial node representation, and based on the spatial continuity and spatial correlation between slices in the OCT slice sequence at each time point, a slice relationship graph at each time point is constructed. The slice relationship graph at each time point and the initial node representation are input into the graph attention network, and the graph-enhanced slice-level image features at each time point are output.

[0011] Preferably, the dual-time-point slice-level visual features and image biomarkers are input into the image biomarker-image cascade fusion module, and the dual-time-point slice-level visual features and image biomarkers are cascaded and fused in chronological order to obtain a second fused feature, including: Image biomarkers are input into the projection layer for mapping to obtain the initial image biomarker state; The initial image biomarker status and the slice-level visual features at the first time point are input into the orthogonal perception image biomarker-visual fusion module, and the initial fusion features are output. The initial fusion features and the slice-level visual features at the second time point are input into the orthogonal sensing image biomarker-visual fusion module, and the second fusion features are output.

[0012] Preferably, the orthogonal perception image biomarker-visual fusion module fuses the input slice-level visual features with the initial image biomarker state or initial fusion features, including the following steps: Layer normalization is performed on the slice-level visual features and the initial image biomarker state or initial fusion features respectively to obtain the target slice-level visual features and the target initial image biomarker state or target initial fusion features. After splicing the target slice-level visual features with the target's initial image biomarker state or the target's initial fused features along the feature dimension, the context features are obtained through linear mapping. The target initial image biomarker state or target initial fusion feature and context features are input into the multi-head attention branch. The target initial image biomarker state or target initial fusion feature is used as the query input, and the context features are used as the key input and value input to calculate the first relevance alignment result. Input the target initial image biomarker state or target initial fusion features and context features into the orthogonal modality fusion branch, use the target initial image biomarker state or target initial fusion features as query input, and use context features as key input and value input to calculate the second correlation alignment result; The first and second correlation alignment results are weighted and fused to obtain the comprehensive attention result. The comprehensive attention result, the target slice-level visual features and the target initial image biomarker state or the target initial fusion features are input into the gating mapping network to construct the first gating coefficient. The first gating coefficient is applied to the comprehensive attention result, and the initial image biomarker state or initial fusion feature is updated through residual method to obtain intermediate result; After layer normalization, the intermediate results are input into the feedforward mapping network. The intermediate results are enhanced based on the output of the feedforward mapping network to obtain the output of the orthogonal perception image biomarker-visual fusion module.

[0013] Preferably, the orthogonal modal fusion branch calculates the second relevance alignment result based on the query input, key input, and value input, including: Linear mapping is performed on the query input, key input, and value input to obtain the query representation, key representation, and value representation, respectively; wherein, the query representation is composed of multiple query node features, and the key representation is composed of multiple reference node features; Based on the inner product similarity between the features of each query node and the features of each reference node, the orthogonal weights of the features of each query node and the features of each reference node are obtained; the orthogonal weights are normalized along the dimension of the number of reference nodes to obtain the normalized weights of the features of each query node and the features of each reference node, thus obtaining the normalized weight matrix. The second correlation alignment result is obtained by weighting and summing the value representations using a normalized weight matrix.

[0014] Preferably, the enhanced slice-level image features of the dual-timepoint map and the dual-timepoint clinical information are input into the clinical information-image progression fusion module. The enhanced slice-level image features of the dual-timepoint map and the dual-timepoint clinical information are then aligned and fused at the node level according to chronological order to obtain a clinical information-image time-point progression representation, including: Clinical information at each time point is input into the clinical information node projection module for projection and expansion to obtain the clinical node features at each time point. The image enhancement slice-level image features and clinical node features at the same time point are input into the clinically-led image fusion module, which outputs clinical information-image collaborative features at each time point. The clinical information and image collaborative features at each time point are input into the time point progress perception and representation aggregation module, which outputs a clinical information-image time point progress representation.

[0015] Preferably, clinical information at each time point is input into the clinical information node projection module for projection and expansion to obtain clinical node features at each time point, including: Based on the mean and standard deviation of continuous variables in clinical information at two time points, the continuous variables in clinical information at each time point are normalized to obtain the normalized continuous variables in clinical information at each time point. Normalized continuous variables and discrete variables in clinical information at each time point are concatenated to obtain the clinical representation at each time point. The clinical representation at each time point is then input into a clinical information projection mapping network to obtain the projected clinical representation at each time point. The clinical representations at each time point after projection are copied and expanded in the row direction and added to the learnable slice position embedding matrix to obtain the clinical node features at each time point.

[0016] Preferably, the image enhancement slice-level features and clinical node features at the same time point are input into the clinically-led image fusion module, and the clinical information-image collaborative features at each time point are output, including: After performing layer normalization on the image enhancement slice-level image features and clinical node features at the same time point, they are spliced ​​together in the feature dimension. The spliced ​​features are then linearly mapped to obtain the visual reference context features at that time point. The visual reference context features and the layer-normalized clinical node features at each time point are input into the multi-head attention branch. The layer-normalized clinical node features are used as query input, and the visual reference context features are used as key input and value input to calculate the cross-attention coefficient at each time point. The cross-attention coefficients at each time point, the clinical node features after layer normalization, and the graph-enhanced slice-level image features after layer normalization are input into the gating mapping network to construct the second gating coefficients at each time point. The second gating coefficient at each time point is applied to the cross-attention coefficient, and the clinical node features at each time point are updated through residuals to obtain the intermediate features at each time point. The intermediate features at each time point are normalized by layers and then input into the feedforward mapping network. The output of the feedforward mapping network is used to enhance the intermediate features at each time point, thus obtaining the clinical information-image collaborative features at each time point.

[0017] Preferably, the clinical information-image collaborative features at each time point are jointly input into the time-point progression perception representation aggregation module, and the output clinical information-image time-point progression representation includes: Calculate the difference features of the clinical information-image co-features at two time points, and perform layer normalization on the difference features to obtain node-level change features; By stacking the clinical information-image co-features and differential features from two time points in chronological order, a short sequence representation is obtained; The short sequence representation is input into the multi-head attention branch to obtain the attention output; the attention output and the short sequence representation are then connected by a residual connection and then normalized by a layer to obtain the target short sequence representation. Nonlinear transformation, residual connection, and layer normalization are applied to the target short sequence representation to obtain time-enhanced short temporal features. The time-enhanced features at each sequence position in the time-enhanced short temporal features are linearly scored and then normalized along the short temporal dimension to obtain the feature weights at each sequence position. By using the feature weights of each sequence position, the temporal enhancement features of each sequence position in the temporal enhancement short temporal feature are weighted and aggregated to obtain the clinical information-image temporal progression representation.

[0018] Preferably, when training the fundus lesion evolution prediction model, the steps for constructing the fundus lesion evolution loss function include: The dual-time-point spatial feature maps of the training samples output by the image feature extraction module are rearranged to obtain a dual-time-point spatial feature sequence; based on the similarity between each spatial location in the spatial feature sequence of the two time points, a spatial similarity matrix is ​​obtained. Normalize each similarity in the spatial similarity matrix to obtain the similarity weight of each spatial location; use the similarity weight of each spatial location to reconstruct the spatial feature sequence of the previous time point in a weighted manner to obtain the historical reference features. After normalizing the spatial feature sequences of historical reference features and subsequent time points in the channel dimension, the temporal visual consistency response of each spatial location at the two time points is calculated; the temporal visual consistency responses of all spatial locations at the two time points are aggregated and mapped to obtain the visual progress score of the training samples. A visual progression loss function is constructed based on the true label of fundus lesion evolution direction and the smoothed L1 loss of visual progression score of the training samples. Based on the focus loss between the predicted evolution direction of fundus lesions in the training samples and the true labels, a main classification loss function is constructed. The loss function for fundus lesion evolution is obtained by weighted summation of the visual progression loss function and the main classification loss.

[0019] The method for predicting the evolution of fundus lesions based on temporal multimodal data provided in this application has the following beneficial effects: Simultaneously, by introducing dual-timepoint OCT slice sequences, imaging biomarkers, and dual-timepoint clinical information of the subjects to be tested, a multimodal, dual-timepoint targeted fusion modeling system was constructed. This effectively overcomes the problems of existing fundus lesion evolution prediction methods, such as insufficient utilization of fine-grained lesion information, one-sided capture of temporal evolution trends, and lack of targeted modeling of key changes, resulting in poor fit of evolutionary pattern characterization and low prediction accuracy. First, the image feature extraction module simultaneously outputs dual-timepoint slice-level visual features and image-enhanced slice-level image features, and based on the dual-timepoint slice-level visual features... The first fusion feature is obtained by fusing images to fully extract fine-grained microscopic pathological information such as retinal structure, lesion morphology, and fluid distribution in OCT slice images. This compensates for the loss of image details in traditional structured clinical variable methods. Based on a dual-time-point input architecture, a basic representation of the temporal changes of lesions is constructed, solving the problem that single-time-point imaging methods cannot reflect the dynamic evolution trend of lesions. Secondly, through an image biomarker-image cascade fusion module, the dual-time-point visual features and image biomarkers are cascaded and fused in chronological order to generate a second fusion feature, which reflects the appearance of the lesions. The organic combination of morphological changes and quantitative pathological index changes improves the expression of the underlying evolutionary features of lesions and further enhances the ability to depict the true evolutionary state of lesions. Simultaneously, using the clinical information-image progression fusion module, the enhanced image features of dual-time-point maps are aligned and fused with dual-time-point clinical information at the time-dimensional node level to obtain the clinical information-image time-point progression representation. This accurately captures the dynamic correlation between image lesion evolution and clinical diagnosis and treatment variables, overcoming the modeling shortcomings of traditional time-series models that only stack global features and ignore key decision-related changes and the correlation between single-frame internal structures. Finally, the first fusion feature, the second fusion feature, and the clinical information-image time-point progression representation are pooled and then spliced ​​together to construct a complete fundus lesion evolution feature that takes into account microscopic image details, quantitative pathological features, and clinical time-series evolution. Relying on multi-dimensional complementary feature information, it achieves accurate classification and prediction of the evolutionary direction of fundus lesions, solving the problems of insufficient characterization of lesion evolution patterns and insufficient prediction accuracy and robustness in existing methods. It can accurately adapt to multi-time-point follow-up scenarios, providing effective and reliable technical support for follow-up evaluation and subsequent needs assessment. Attached Figure Description

[0020] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 Flowchart of the fundus lesion evolution prediction method based on temporal multimodal data provided in this application; Figure 2 This is a diagram of the architecture of the fundus lesion evolution prediction model provided in this application; Figure 3 The image feature extraction module architecture diagram provided in this application; Figure 4 The image biomarker-image cascade fusion module architecture diagram provided in this application; wherein, Figure 4 (a) in the diagram is a cascaded fusion architecture diagram. Figure 4 (b) in the diagram is the architecture diagram of the orthogonal perception image biomarker-visual fusion module. Figure 4 (c) in the diagram represents the orthogonal modal fusion branch architecture. Figure 5 The clinical information-image progression fusion module architecture diagram provided for this application; wherein, Figure 5 (a) in the diagram is the alignment and fusion architecture diagram. Figure 5 (b) in the diagram is a structural diagram of the clinical information node projection module. Figure 5 (c) in the diagram is the architecture of the clinical-leading image fusion module. Figure 5 (d) in the diagram represents the architecture of the time-point progress perception and representation aggregation module. Figure 6 A schematic diagram illustrating the principle of calculating cross-temporal spatial alignment and visual progress score provided in this application. Detailed Implementation

[0021] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0022] Please refer to the following: Figure 1 and Figure 2 , Figure 1 The diagram shown is a flowchart of the fundus lesion evolution prediction method based on temporal multimodal data provided in this application. Figure 2 The diagram shown is an architecture diagram of the fundus lesion evolution prediction model provided in this application.

[0023] Specifically, such as Figure 1 As shown, the method includes S10~S50.

[0024] S10: Obtain the two-timepoint OCT slice sequences, imaging biomarkers, and two-timepoint clinical information of the subject to be tested.

[0025] S20: Input the dual-time point OCT slice sequence into the image feature extraction module of the fundus lesion evolution prediction model, and output the dual-time point slice-level visual features and the dual-time point image-enhanced slice-level image features; fuse the dual-time point slice-level visual features to obtain the first fused feature.

[0026] S30: Input the dual-time point slice-level visual features and image biomarkers into the image biomarker-image cascade fusion module in the fundus lesion evolution prediction model, and cascade the dual-time point slice-level visual features and image biomarkers in chronological order to obtain the second fused feature.

[0027] S40: Input the enhanced slice-level image features of the dual-time point map and the dual-time point clinical information into the clinical information-image progression fusion module in the fundus lesion evolution prediction model. Perform node-level alignment and fusion of the enhanced slice-level image features of the dual-time point map and the dual-time point clinical information in chronological order to obtain the clinical information-image time point progression representation.

[0028] S50: Pooling is performed on the first fusion feature, the second fusion feature, and the clinical information-image time-point progression representation respectively. The three pooled features are then concatenated to obtain the fundus lesion evolution features. The fundus lesion evolution features are then classified to obtain the evolution direction of the fundus lesions in the subject to be tested.

[0029] like Figure 2 As shown, the method provided in this application takes as input the OCT slice sequences at two time points before and after each target object, the image biomarkers corresponding to each OCT slice, and the clinical information at the two time points before and after each target object. The fundus lesion evolution prediction model extracts image features from the OCT slice sequences at the two time points to obtain slice-level visual features. Subsequently, the fundus lesion evolution prediction model introduces two fusion branches: one is the image biomarker-image cascade fusion branch, which is used to transmit and update image biomarker information along the two time points, and the other is the clinical information-image progression fusion branch, which is used to establish the correspondence between clinical information and image information at the two time points and to explicitly model the evolution of fundus lesions. Finally, attention convergence is performed on the image features, image biomarker-image fusion features, and clinical information-image progression fusion features, and the obtained features are spliced ​​to form the final representation, which is then output by the classifier to show the evolution direction of fundus lesions.

[0030] Specifically, dual-timepoint OCT slice sequences , This represents the OCT slice sequence at the previous time point. This represents the OCT slice sequence at the next time point, for any given time point. ,have ,in, The number of slices, For the number of channels, and These represent the height and width of the image, respectively.

[0031] Furthermore, the image feature extraction module in step S20 is implemented using a graph-enhanced dual-path radiomics-gated image encoder, which is used to perform slice-level visual encoding on the OCT slice sequences at two different time points and adaptively fuse them with radiomics features to obtain image features that contain both local details and global structural information. On this basis, the spatial continuity characteristics between multiple slices at the same time point are further utilized to construct a slice relationship graph to enhance the information interaction and structural relationship representation capabilities between slices.

[0032] Specifically, such as Figure 3 The diagram shown is an architecture diagram of the image feature extraction module provided in this application. In step S20, the dual-time point OCT slice sequence is input into the image feature extraction module, and the dual-time point slice-level visual features and dual-time point image-enhanced slice-level image features are output, including S200~S204.

[0033] S200: Input the OCT slice sequences at each time point into the segmentation network, output a binary segmentation mask of the retinal region, and use the binary segmentation mask as the region of interest to extract two-dimensional radiomics features from each slice of the OCT slice sequence at each time point, thereby obtaining the slice-by-slice radiomics features at each time point.

[0034] Specifically, to introduce interpretable structural prior information, this application first obtains a binary segmentation mask of the retinal-related region in the OCT image through a segmentation network. Then, using the binary segmentation mask as the region of interest, it extracts two-dimensional radiomics features such as intensity, texture, and shape from each OCT slice in the OCT slice sequence at each time point. The extracted features are then further processed to remove non-numerical terms, features with a high proportion of missing values, features that are approximately constant, and highly correlated redundant features. Standardization is then performed to obtain the standardized slice-by-slice radiomics features of the OCT slice sequence at each time point. ,in, This indicates the dimension of the image omics features after cleaning.

[0035] S201: Input the slice-by-slice image omics features at each time point into a gating network composed of a multilayer perceptron and a sigmoid activation function, and output the slice-level gating weights at each time point.

[0036] Specifically, using slice-by-slice image omics features As input, slice-level gating weights are generated via a multilayer perceptron (MLP) and a sigmoid activation function. ,in, This represents a gated network consisting of two fully connected layers. This represents the Sigmoid activation function, used to constrain the gate value to the range of 0 to 1. This represents the gating weight corresponding to each OCT slice.

[0037] S202: Input the OCT slice sequences at each time point into the parallel CNN branch and the visual Transformer branch respectively. Let the spatial feature maps of each time point output by the CNN branch be processed by generalized mean pooling and linear projection to obtain the slice-level CNN features at each time point. Let the output of the visual Transformer branch be processed by the projection layer to obtain the slice-level ViT features at each time point.

[0038] Specifically, for the OCT slice sequence at time point t, it is input into the CNN branch and the Transformer (ViT) branch respectively to extract local detail features and global structural features.

[0039] More specifically, the CNN branch first outputs the spatial feature map before global pooling. ,in, This represents the number of channels in the convolutional spatial feature map. , These represent the height and width of the convolutional spatial feature map, respectively; the final spatial feature map... After generalized mean pooling (GeM) and linear projection, slice-level CNN features are obtained. ,in, This represents a unified feature dimension. Simultaneously, the ViT branch performs global structural modeling on the OCT slice sequences at the same time point, and obtains slice-level ViT features through a subsequent projection layer. .

[0040] In a specific example of this application, the CNN branch parameters at the two time points are trained independently. The CNN branch adopts the ResNet-18 backbone structure, which includes 4 residual convolutional stages and a total of 8 residual blocks. The ViT branch adopts the ViT-Base / 16 structure, which includes an image patch embedding layer and 12 Transformer coding blocks, loads the OCT pre-trained weights from the VisionFM base model and keeps them frozen.

[0041] S203: Using the slice-level gating weights at each time point, the slice-level CNN features and slice-level ViT features at each time point are weighted and fused to obtain the slice-level visual features at each time point.

[0042] Specifically, gating weights are used to perform weighted fusion of the two feature streams to obtain slice-level visual features at each time point. ,in, , For element-wise multiplication, when A larger value indicates that the OCT slice relies more on the global structural information extracted by the ViT branch. When the value is smaller, it indicates that the OCT slice relies more on the local detail information extracted by the CNN branch. Therefore, different OCT slices can adaptively select and fuse between the two types of visual features based on their own radiomics features.

[0043] S204: Using the slice-level visual features at each time point as the initial node representation, construct the slice relationship graph at each time point based on the spatial continuity and spatial correlation between slices in the OCT slice sequence at each time point; input the slice relationship graph at each time point and the initial node representation into the graph attention network, and output the graph-enhanced slice-level image features at each time point.

[0044] Specifically, this application obtains slice-level visual features at various time points. Subsequently, considering the spatial continuity and correlation of multiple slices from the same OCT examination, a slice relationship graph was constructed for each time point. The features of the N OCT slices at time point t were considered as N nodes in the slice relationship graph. As the initial representation of a node, and connecting adjacent OCT slices according to the original order of the OCT slices, while preserving the self-connections of each node, its adjacency relationship can be represented as: , in, Representing the slice relationship diagram The adjacency matrix is ​​obtained. Based on this slice relationship graph, a Graph Attention Network (GAT) is used to update the node features, thus obtaining graph-enhanced slice-level image features. ,in, , This represents the attention update operation in the graph.

[0045] Slice-level visual features In comparison, graph-enhanced slice-level image features This not only preserves the visual representation of each OCT slice but also integrates contextual information from adjacent slices at the same time point, making it more effective in depicting the overall spatial distribution of lesions within the OCT slice sequence. Afterwards, , Image feature inputs are used as the clinical information-image progression fusion module. , The image features are used as input to the image biomarker-image cascade fusion module to maintain a one-to-one correspondence between slice-by-slice features and corresponding image biomarkers.

[0046] Furthermore, the image biomarker-image cascade fusion module in step S30 is used to cascade and fuse slice-by-slice image biomarker information with dual-time point image features, so that the image biomarker information not only participates in modeling statically at a single time point, but can be updated step by step along the time dimension and passed on to the next. The input of this module includes image biomarkers and dual-time point slice-level visual features, and the output is image biomarker fusion features that have been enhanced by image information and completed cross-time point transmission.

[0047] Specifically, Figure 4 The image biomarker-image cascade fusion module architecture diagram provided in this application includes step S300~S302.

[0048] S300: Input the image biomarkers into the projection layer for mapping to obtain the initial image biomarker state.

[0049] Specifically, imaging biomarkers , This represents the original image biomarker dimension. Since image biomarker features and slice-level visual features are inconsistent in dimensional space, direct cross-modal interaction is difficult. Therefore, this application first maps the image biomarkers to a feature space consistent with the slice-level visual features through a projection layer, obtaining the initial image biomarker state. ,in, This represents a biomarker projection network composed of MLPs. , This represents a unified feature dimension.

[0050] S301: Input the initial image biomarker status and the slice-level visual features at the first time point into the orthogonal perception image biomarker-visual fusion module, and output the initial fusion features.

[0051] S302: Input the initial fusion features and the slice-level visual features at the second time point into the orthogonal sensing image biomarker-visual fusion module, and output the second fusion features.

[0052] like Figure 4 As shown in (a), the image biomarker-image cascade fusion module adopts a cascaded approach, so that the fusion result of the previous time point continues to participate in the fusion of the next time point, thereby enabling the image biomarker information to be updated step by step along the time dimension and passed on to the next.

[0053] Specifically, the initial biological image biomarker status Slice-level visual features compared to the previous time point Input the Orthogonality-aware Biomarker-VisionFusion Block (OBVF) to obtain the initial fused features. Then the initial fusion features Slice-level visual features at a later time point Input the second orthogonal perception image biomarker-visual fusion module to obtain the second fusion feature. The resulting second fusion feature contains lesion status information at both time points, which is more conducive to characterizing the evolution process of lesions at two time points.

[0054] like Figure 4 As shown in (b) in the figure, the steps of the orthogonal perception image biomarker-visual fusion module to fuse the input slice-level visual features with the initial image biomarker state or initial fusion features include steps 1-1 to 1-7.

[0055] Specifically, given the slice-level visual features at a certain point in time. and initial image biomarker status or initial fusion features The input is a standard multi-head attention (MHA) branch and an orthogonal modality fusion (OMF) branch, processed in parallel. It should be noted that the input for the first fusion is the slice-level visual features from the previous time step. and initial image biomarker status The input for the second fusion is the slice-level visual features from the next time point. and initial fusion features .

[0056] Step 1-1: Perform layer normalization on the slice-level visual features and the initial image biomarker state or initial fusion features respectively to obtain the target slice-level visual features and the target initial image biomarker state or target initial fusion features.

[0057] Specifically, respectively and Perform layer normalization, and denote the normalized result as the initial biomarker state of the target image or the initial fusion feature of the target. and target slice-level visual features .

[0058] Steps 1-2: After splicing the target slice-level visual features with the target's initial image biomarker state or the target's initial fusion features along the feature dimension, the context features are obtained through linear mapping.

[0059] Specifically, context features ,in, This indicates a concatenation operation along the feature dimension. Represents the linear mapping parameters. .

[0060] Steps 1-3: Input the target initial image biomarker state or target initial fusion features and context features into the multi-head attention branch, use the target initial image biomarker state or target initial fusion features as query input, and use the context features as key input and value input to calculate the first relevance alignment result.

[0061] Specifically, in the MHA branch, with As input for the query, As key and value inputs, the first relevance alignment result is obtained. .

[0062] The MHA branch is mainly used to model shared information that is consistent in direction and highly correlated between two modalities, thereby achieving effective alignment between image biomarkers and image features. However, when using only the MHA branch, the model is more likely to focus on image information that is highly consistent with the current image biomarker representation, while underutilizing heterogeneous information that is significantly different but has complementary significance. Based on this, in order to simultaneously meet the fusion requirements of "correlation alignment" and "complementary supplementation", this application further introduces the OMF branch.

[0063] Steps 1-4: Input the target initial image biomarker state or target initial fusion feature and context features into the orthogonal modality fusion branch, use the target initial image biomarker state or target initial fusion feature as query input, and use the context features as key input and value input to calculate the second correlation alignment result.

[0064] Specifically, the OMF branch also uses As input for the query, As key and value inputs, the second relevance alignment result is obtained. .

[0065] Steps 1-5: Weightedly fuse the first correlation alignment result and the second correlation alignment result to obtain the comprehensive attention result; input the comprehensive attention result, the target slice-level visual features and the target initial image biomarker state or the target initial fusion features into the gating mapping network to construct the first gating coefficient.

[0066] Specifically, the overall attention results ,in, This represents the learnable balance coefficient, used to adjust the proportion of relevance alignment information and complementary supplementary information in the final fusion result.

[0067] To avoid all nodes receiving fusion information with the same intensity, further adjustments are made based on the target's initial image biomarker status or the target's initial fusion features. Target slice-level visual features and comprehensive attention results Constructing the first gating coefficient ,in, This represents a gated mapping network composed of MLPs. This represents the gating coefficient for each node and each channel.

[0068] Steps 1-6: Apply the first gating coefficient to the integrated attention result, and update the initial image biomarker state or initial fusion features through residual method to obtain intermediate results.

[0069] Specifically, intermediate results Different nodes and different channels can adaptively adjust the intensity of received fused information, thereby avoiding the indiscriminate introduction of visual information.

[0070] Steps 1-7: After layer normalization of the intermediate results, input them into the feedforward mapping network. The intermediate results are enhanced based on the output of the feedforward mapping network to obtain the output of the orthogonal perception image biomarker-visual fusion module.

[0071] Specifically, intermediate results are enhanced through feedforward mapping and residual refinement to obtain the output of the single-time-point orthogonal sensing image biomarker-visual fusion module. ,in, This represents a feedforward mapping network composed of MLPs. Representation layer normalization, This indicates the output state at that point in time after image biomarker-visual fusion.

[0072] Furthermore, such as Figure 4 As shown in (c), the orthogonal modality fusion branch calculates the second relevance alignment result based on the query input, key input, and value input, including steps 2-1 to 2-3.

[0073] Specifically, the query input is , key input is The input value is ,in, , These represent the number of nodes for query input and key input, respectively. Depend on Composed of features of a D-dimensional query node, key input Sum value input respectively by It consists of D-dimensional reference nodes.

[0074] Step 2-1: Perform linear mapping on the query input, key input, and value input respectively to obtain the query representation, key representation, and value representation; wherein, the query representation is derived from... The query node features are composed of keys representing the key. It consists of the features of each reference node.

[0075] Specifically, the query represents , key represents , value represents ,in, , , These represent the linear mapping parameters that map the relevant inputs to the query space, key space, and value space, respectively. Indicates proceeding along the feature dimension Normalization is performed to improve the stability of subsequent similarity calculations.

[0076] Step 2-2: Based on the inner product similarity between the features of each query node and the features of each reference node, obtain the orthogonal weights of the features of each query node and the features of each reference node; normalize the orthogonal weights along the dimension of the number of reference nodes to obtain the normalized weights of the features of each query node and the features of each reference node, thus obtaining the normalized weight matrix.

[0077] Specifically, set For query representation The feature vector of the i-th query node. Key representation The feature vector of the j-th reference node, for the i-th query node and the j-th reference node, its orthogonal weight can be expressed as: , This represents the normalized feature vector of the query node. With reference node feature vector The inner product similarity between the two is such that the larger the absolute value of the inner product and the smaller the corresponding weight, the greater the difference between the two and the closer they are to orthogonality, the smaller the absolute value of the inner product and the larger the corresponding weight. Therefore, this mechanism tends to introduce information that is supplementary to the current query representation from the node features corresponding to the key input and value input.

[0078] To ensure the comparability of the reference weights corresponding to each query node, the orthogonal weights are further normalized along the dimension of the number of reference nodes. ,in, To prevent extremely small constants with a denominator of zero.

[0079] Steps 2-3: Use the normalized weight matrix to perform a weighted summation on the value representation to obtain the second correlation alignment result.

[0080] Specifically, the second correlation alignment result ,in, This represents the normalized orthogonal weight matrix.

[0081] In this application, the OMF branch is used for interactive fusion between image biomarker features and image features. The image biomarker features to be updated are used as query input, and the image features at the same time point are used as reference input. Information that is supplementary to the image biomarker is extracted from the image features through orthogonal weights to obtain the enhanced image biomarker representation, which is then used for cross-time point cascaded updates and transmission.

[0082] Furthermore, the clinical information-image progression fusion module in step S40 is used to perform node-level alignment and fusion of dual-timepoint clinical information and dual-timepoint enhanced slice-level image features at each time point, and on this basis, further model the progression process of the lesion before and after. For example... Figure 5 The diagram shown is an architecture diagram of the clinical information-image progression fusion module provided in this application. Figure 5 As shown in (a), this application first projects and expands the clinical information at each time point into clinical node features that correspond one-to-one with the image slice nodes, then fuses them with the image enhancement slice-level image features at two time points respectively, and finally uses the time point progression perception representation aggregation module to jointly model the features of the previous time point, the features of the next time point and their difference features, thereby obtaining a clinical information-image progression representation that is more in line with the evolution of lesions.

[0083] Specifically, step S40 includes S400~S402.

[0084] Specifically, two-timepoint clinical information ,in, Let be the clinical information vector at time point t. This indicates the clinical characteristic dimension.

[0085] S400: Input clinical information at each time point into the clinical information node projection module for projection and expansion to obtain the clinical node features at each time point.

[0086] S401: Input the image enhancement slice-level image features and clinical node features at the same time point into the clinical-dominant image fusion module, and output the clinical information-image collaborative features at each time point.

[0087] S402: Input the clinical information and image collaborative features at each time point into the time point progress perception and representation aggregation module, and output the clinical information-image time point progress representation.

[0088] Furthermore, such as Figure 5 As shown in (b) of the diagram, step S400 includes steps 3-1 to 3-3.

[0089] Step 3-1: Based on the mean and standard deviation of the continuous variables in the clinical information at two time points, normalize the continuous variables in the clinical information at each time point to obtain the normalized continuous variables in the clinical information at each time point.

[0090] Specifically, clinical information vectors typically contain both continuous and discrete variables. Since the absolute values ​​of continuous variables differ significantly between different subjects, and this application focuses more on the trend of change of the same subject between two tests, the continuous variables of the same subject are normalized along the time dimension to highlight the relative magnitude of variables within an individual.

[0091] For example, for the k-th continuous variable of the b-th object to be detected, its normalized result at time t. ,in, , Let represent the mean and standard deviation of the k-th continuous variable of the b-th test object at two time points, respectively.

[0092] Step 3-2: Concatenate the normalized continuous variables and discrete variables in the clinical information at each time point to obtain the clinical representation at each time point. Input the clinical representation at each time point into the clinical information projection mapping network to obtain the projected clinical representation at each time point.

[0093] Specifically, the discrete variables are kept unchanged, and the normalized continuous variables are reassembled with the discrete variables to obtain the clinical representations at each time point. .

[0094] Step 3-3: After projection, the clinical representations at each time point are copied and expanded in the row direction and added to the learnable slice position embedding matrix to obtain the clinical node features at each time point.

[0095] Specifically, the clinical representations at each time point are mapped to a unified feature space consistent with the image features: ,in, This represents a clinical information projection mapping network composed of MLPs. , This represents a unified feature dimension.

[0096] Furthermore, since clinical information is originally presented in units of time points, while image features are presented in units of slice nodes, in order to enable the two modalities to align and interact at the node level, the clinical representation needs to be extended to N slice nodes, and distinguishable positional information needs to be introduced for different nodes. Specifically, the clinical representation needs to be extended to N slice nodes. Copy to Dimension, and embedding matrix with learnable slice positions. Adding them together yields the node-based clinical node features. ,in, , This indicates that copying and expansion are performed in the row direction.

[0097] Through the above steps, the original time-point-level clinical information is transformed into clinical node features that are consistent with the number and feature dimensions of image slice nodes. This provides a foundation for subsequent node-level fusion of clinical information and image features. It should be noted that step 3-3 obtains the clinical node features of the previous and next time points. and .

[0098] Furthermore, such as Figure 5 As shown in (c), step S401 includes steps 4-1 to 4-5.

[0099] Specifically, the clinical-dominant image fusion module uses clinical node features For the main branch to be updated, use graph enhancement to enhance slice-level image features. As a visual reference branch, slice structure information related to the current clinical state is introduced at the node level, thereby forming clinical information-image collaborative features within a time point.

[0100] Step 4-1: After performing layer normalization on the image enhancement slice-level image features and clinical node features at the same time point, the features are concatenated along the feature dimension. The concatenated features are then linearly mapped to obtain the visual reference context features at that time point.

[0101] Specifically, the image enhancement slice-level image features after layer normalization. Clinical node features after hierarchical normalization Visual reference context features ,in, Represents the linear mapping parameters. The visual reference context features contain image information of both the current clinical state and the corresponding slice location, which are used as key and value inputs in subsequent cross-attention.

[0102] Step 4-2: Input the visual reference context features and the layer-normalized clinical node features at each time point into the multi-head attention branch. Use the layer-normalized clinical node features as the query input and the visual reference context features as the key and value inputs to calculate the cross-attention coefficients at each time point.

[0103] Specifically, clinical node features are used as query input, and visual reference context features are used as key and value inputs, with cross-attention scores... The query input corresponds to the clinical node state, while the key and value inputs represent visual reference information constrained by the clinical node state. This design enables the model to selectively absorb image structural information related to the current slice node while maintaining the dominance of clinical information.

[0104] Step 4-3: Input the cross-attention coefficients, layer-normalized clinical node features, and layer-normalized graph-enhanced slice-level image features at each time point into the gating mapping network to construct the second gating coefficients at each time point.

[0105] Specifically, considering that different nodes have different needs for image information, in order to avoid visual information indiscriminately covering clinical representations, a node-by-node, channel-by-channel gating update mechanism is further introduced, with a second gating coefficient. ,in, This represents a gated mapping network composed of MLPs.

[0106] Step 4-4: Apply the second gating coefficient at each time point to the cross-attention coefficient, and update the clinical node features at each time point through residual method to obtain the intermediate features at each time point.

[0107] Specifically, intermediate features .

[0108] Steps 4-5: After layer normalization of the intermediate features at each time point, input them into the feedforward mapping network. Based on the output of the feedforward mapping network, enhance the intermediate features at each time point to obtain the clinical information-image co-features at each time point.

[0109] Specifically, clinical information-image co-features at various time points ,in, This represents the clinical node representation within time point t after image information enhancement. This representation primarily uses clinical information while also incorporating image structural features of the corresponding slice location. It can describe the clinical state and imaging manifestations at the node level. It should be noted that steps 4-5 obtain the clinical information-image collaborative features of the previous and next time points. and This provides input for subsequent cross-time point progress modeling.

[0110] Furthermore, such as Figure 5 As shown in (d), step S402 includes steps 5-1 to 5-5.

[0111] Step 5-1: Calculate the differential features of the clinical information-image co-features at two time points, and perform layer normalization on the differential features to obtain node-level change features.

[0112] Specifically, node-level change characteristics , It represents the node-level changes between two time points, which can directly reflect the direction and magnitude of changes between the two time points, and is an important basis for characterizing the progression of lesions.

[0113] Step 5-2: Stack the clinical information-image co-features and differential features from the two time points in chronological order to obtain a short sequence representation.

[0114] Specifically, for each slice node, the clinical information and image features from the previous and next time points are combined. and Node-level change characteristics Stacking them in chronological order yields a short sequence representation of length 3. , This indicates a stacking operation.

[0115] Step 5-3: Input the short sequence representation into the multi-head attention branch to obtain the attention output; perform residual connection between the attention output and the short sequence representation, and then perform layer normalization to obtain the target short sequence representation.

[0116] Specifically, a multi-head self-attention operation is applied to the short sequence representation to model the internal correlations between features from the previous time step, features from the next time step, and difference features, thereby obtaining the attention output. ; Continue using residual connections and layer normalization to obtain the target short sequence representation. .

[0117] Step 5-4: Perform nonlinear transformation, residual connection, and layer normalization on the target short sequence representation to obtain time-enhanced short time-series features; perform linear scoring on the time-enhanced features of each sequence position in the time-enhanced short time-series features and then normalize along the short time-series dimension to obtain the feature weights of each sequence position.

[0118] Furthermore, temporal enhancement of short-time-series features To adaptively measure the contribution of the three types of progress-related features to the final progress representation, the following measures are taken: Linear scoring is performed on each sequence position, and the results are normalized along a short temporal dimension of length 3 to obtain the feature weights for each sequence position. and feature weight matrix ,in, This represents the state scoring mapping parameters. This represents the adaptive weights corresponding to the collaborative features at the previous time point, the collaborative features at the next time point, and the difference features.

[0119] Step 5-5: Utilize the feature weights of each sequence position to perform weighted aggregation of the temporal enhancement features at each sequence position in the temporal enhancement short temporal features, and obtain the clinical information-image temporal progression representation.

[0120] Specifically, clinical information-image time-point progression representation , ,in, This indicates point-by-point multiplication. The feature weights represent the m-th sequence position. This represents the temporal enhancement feature at the m-th sequence position.

[0121] Furthermore, in some embodiments of this application, a Temporal Visual Progression (TVP) mechanism is introduced when training the fundus lesion evolution prediction model. This mechanism calculates the temporal visual consistency between the spatial feature maps of two consecutive time points and jointly optimizes it with the classification objective to enhance the network's ability to represent the evolution patterns of lesions. Specifically, TVP is used to explicitly constrain the structural consistency and change patterns between visual features at two time points during the training phase. This allows the network to further obtain temporal visual representations related to lesion evolution when learning the final classification prediction task. This mechanism primarily operates on the two-time-point spatial feature maps obtained by the image feature extraction module. and By establishing the correspondence between spatial locations at different time points, the TVP score is calculated and used as an auxiliary supervision signal for joint optimization with the classification objective.

[0122] Specifically, when training the fundus lesion evolution prediction model, the steps for constructing the fundus lesion evolution loss function include steps 6-1 to 6-6.

[0123] Step 6-1: Rearrange the dual-time point spatial feature maps of the training samples output by the image feature extraction module to obtain a dual-time point spatial feature sequence; based on the similarity between each spatial location in the spatial feature sequence of the two time points, obtain the spatial similarity matrix.

[0124] Specifically, dual-time point spatial feature map Since the dual-time point feature map contains both slice dimension and spatial dimension, in order to measure the correspondence between visual structures at different time points in a unified space, the spatial feature map is first rearranged to obtain the dual-time point spatial feature sequence. ,in, Through this step, the visual features that were originally distributed across multiple slices and two-dimensional space are uniformly represented as a positional sequence of uniform length, which facilitates subsequent calculation of positional correspondences across time points.

[0125] Based on this, the similarity relationship between each spatial location at the later time point and each spatial location at the previous time point is calculated to obtain the spatial similarity matrix. ,in, Each matrix element This indicates the degree of similarity between the k-th spatial location at a later time point and the v-th spatial location at a previous time point.

[0126] Step 6-2: Normalize each similarity in the spatial similarity matrix to obtain the similarity weight of each spatial location; use the similarity weight of each spatial location to reconstruct the spatial feature sequence of the previous time point in a weighted manner to obtain the historical reference features.

[0127] Specifically, since the same lesion region may undergo morphological changes, positional shifts, or boundary adjustments between two time points, corresponding regions at different time points may not necessarily fall strictly at the same index position. Therefore, direct position-by-position comparisons are not feasible. Instead, a soft alignment relationship across time points needs to be established based on similarity. To this end, the similarity matrix is ​​softmax normalized along the spatial position of the previous time point to obtain the similarity weights for each spatial position. ,in, This represents the summation index along the spatial location dimension at the previous time point, used to traverse all P spatial locations at the previous time point; This represents the weighted allocation result of each spatial location at the later time point to all spatial locations at the previous time point. This can be understood as follows: for each location at a later point in time, the model will search for the most relevant historical reference area among all spatial locations at the previous point in time, and aggregate them in a weighted manner.

[0128] Historical reference features ,thus, It no longer represents the features at the original spatial location at the previous time point, but rather the features after cross-location weighted reconstruction, compared to the features at the subsequent time point. Alignment historical reference features.

[0129] Step 6-3: After normalizing the historical reference features and the spatial feature sequences at the next time point in the channel dimension, calculate the temporal visual consistency response of each spatial location at the two time points; aggregate and map the temporal visual consistency responses of all spatial locations at the two time points to obtain the visual progress score of the training samples.

[0130] Specifically, the temporal visual consistency response of each spatial location at two points in time. ,in, This represents the feature vector of the k-th spatial location at a later time point. This represents the historical reference feature vector that it is aligned with. The L2 norm of a vector is used to measure the length of its eigenvectors. Represents the dot product of vectors; The larger the value, the more consistent the current spatial location is with its historical reference area. The smaller the value, the more significant the structural change at that location between the two time points.

[0131] Visual progress score ,in, The TVP score for each sample is represented by the logarithmic summation exponential aggregation method used in this application. This method can have a greater impact on key areas with strong consistency responses while integrating all spatial location responses, thereby making the obtained score more stable in reflecting the consistency of the overall visual structure at both time points.

[0132] like Figure 6 The diagram shown illustrates the principle of visual progress score calculation provided in this application.

[0133] Specifically, TVP is used as an auxiliary supervisory branch during the training phase. Its purpose is to introduce constraints related to visual changes at two time points during the image feature extraction process, so that the network can simultaneously focus on the lesion evolution label and the continuous changes of OCT image structure between the previous and next time points.

[0134] To ensure that the TVP score can serve as a basis for final evolutionary prediction, this application constructs a visually persistent supervision target based on the true labels of the evolutionary direction of fundus lesions in the training samples. The rationale is as follows: For samples where the evolution direction of fundus lesions is reversed, it indicates that the lesion activity still exists or is not adequately alleviated, and the related abnormal structures in the dual-time point OCT images may have high persistence, so a higher TVP score is expected; for samples where the evolution direction of fundus lesions is forward, it indicates that the lesion has been alleviated or the structure has improved compared to the previous time point, and the visual difference between the dual-time point OCT images is relatively more obvious, so a lower TVP score is expected.

[0135] In a binary classification prediction task, a positive evolution direction of fundus lesions is denoted as a positive class label, and a negative evolution direction is denoted as a negative class label. Therefore, the visual persistence supervision target can be defined as... ,in, Indicates the direction label of fundus lesion evolution in the training samples, when When, it indicates that the training sample corresponds to a high visual persistence target, when When the time is right, it indicates that the training sample corresponds to a low visual persistence target.

[0136] Step 6-4: Construct a visual progression loss function based on the true labels of fundus lesion evolution direction and visual progression scores of the training samples using smoothed L1 loss.

[0137] Specifically, due to It is calculated from persistent visual features, and since the evolutionary degree itself is not a strictly binary state, this application will... and The difference between them is modeled as a continuous value regression problem, and the visual progress loss function is defined. ,in, Indicates smoothness Loss function.

[0138] Step 6-5: Based on the focus loss between the predicted evolution direction of fundus lesions in the training samples and the true labels, construct the main classification loss function.

[0139] Specifically, in the main classification task, to alleviate the class imbalance problem and enhance the model's attention to hard-to-classify samples, this application adopts Focal Loss with class weights as the main classification loss. ,in, The model represents the true category. The predicted probability, Represents the true category The corresponding class weights are calculated as an inverse ratio to the number of samples in each class in the training set; that is, the fewer the samples, the larger the corresponding weight, and the more the samples, the smaller the corresponding weight. This represents a moderating factor used to reduce the contribution of easily classified samples to the loss. In a specific instance, .

[0140] Step 6-6: Weighted summation of the visual progression loss function and the main classification loss to obtain the fundus lesion evolution loss function.

[0141] Specifically, the loss function for the evolution of fundus lesions , The weighting coefficients of the visual progress loss function are used to adjust the proportion of the auxiliary supervision branch in the overall optimization objective. In a specific example, .

[0142] Through the aforementioned joint optimization method, the main classification loss directly constrains the model's ability to discriminate the evolution direction of fundus lesions, while the TVP loss provides auxiliary constraints on the image feature extraction process from the perspective of the persistence of visual structures at both time points. Together, these two losses enable the network to learn category discrimination boundaries while paying more attention to key visual patterns related to the persistence, mitigation, or change of lesions, thereby improving the accuracy of subsequent predictions and training stability.

[0143] To verify the effectiveness of the above-mentioned method for predicting the evolution of fundus lesions based on temporal multimodal data, this application embodiment also uses a publicly available dataset to verify the method: 1. Dataset: In this embodiment, a publicly available OCT follow-up dataset, such as the APTOS2021-stage1 dataset, is used to experimentally validate the proposed method. This type of dataset contains OCT image sequences, imaging biomarkers, relevant clinical information, and subsequent lesion evolution direction labels collected from patients with fundus lesions at different time points. It can be used to construct a dual-time-point multimodal lesion evolution prediction task. It should be noted that in this embodiment, only the imaging biomarkers at the first time point are used as the baseline lesion prior, and the dual-time-point OCT images and dual-time-point clinical information are used as temporal inputs for subsequent prediction.

[0144] To adapt to the dual-time-point prediction task of this application, the raw data was first processed by eye. For each sample, OCT B-scan image sequences from two visit time points were extracted. Six OCT B-scan slices were selected at each time point, and the image size of each slice was 650×400 pixels.

[0145] Imaging biomarkers include binary labels summarizing the presence of intraretinal fluid (IRF), subretinal fluid (SRF), pigment epithelial detachment (PED), and hyperreflective foci (HRF) in the entire OCT sequence. These labels characterize key information related to lesion structures in OCT images. Clinical information includes gender, age, diagnosis type, type of anti-VEGF medication, VA, and CST.

[0146] During data preprocessing, the OCT image sequences at each time point were sliced ​​and quality controlled to remove samples with poor image quality, missing key clinical information, or incomplete labels. Finally, each sample was organized into a dual-timepoint multimodal input format, including dual-timepoint OCT images, corresponding imaging biomarkers, clinical information, and binary labels indicating the direction of lesion evolution.

[0147] It should be noted that the dataset, the number of OCT slices selected at each time point, and the specific screening rules used in this embodiment are only for illustrating the experimental verification process of the method provided in this application, and do not constitute a limitation on the scope of application of this application. In practical applications, the number of time points, the number of slices, and the modality information can be adjusted accordingly based on the data acquisition methods, OCT scanning protocols, and the completeness of clinical records of different medical institutions.

[0148] 2. The area under the receiver operating characteristic curve (AUC), accuracy, sensitivity, specificity, precision, and F1 score were used as evaluation metrics. They are defined as follows: , , , , , In binary classification tasks, relevant metrics are calculated based on the confusion matrix, where TP represents the number of samples correctly predicted as positive, TN represents the number of samples correctly predicted as negative, FP represents the number of samples incorrectly predicted as positive, and FN represents the number of samples incorrectly predicted as negative.

[0149] Considering that the F1-score can comprehensively reflect the model's sensitivity and accuracy, and is more representative in cases of imbalanced class distribution, this embodiment uses the F1-score as the main evaluation metric to prioritize the comparison of the performance of different methods.

[0150] 3. Experimental Results: To verify the effectiveness of the method provided in this application, experiments were conducted under the same data partitioning, input preprocessing, and evaluation metrics. Each sample was input into the model in a dual-timepoint multimodal format, including OCT image sequences from the previous and subsequent time points, image biomarkers, clinical information, and subsequent lesion evolution direction labels. In the experiment, OCT images were first resized, and the original order of multiple B-scan slices within the same time point was preserved. To ensure the reliability of radiomics feature extraction, U-Net was used to segment the retinal-related regions in the OCT images, and slice-by-slice radiomics features were extracted based on the segmentation mask. The Dice coefficient of the segmentation network on the labeled data was [value missing]. This indicates that it can accurately locate the ROI region, providing a stable foundation for subsequent radiomics feature extraction.

[0151] Table 1 shows the ablation test results for the key components: Table 1

[0152] The experiment, based on an image feature extractor, removed or retained the image biomarker-image cascade fusion module, the clinical information-image progression fusion module, and the TVP mechanism to verify the contribution of each key design to predictive performance. Results showed that with only the feature extractor and TVP retained, the model's F1-score was 83.71; after adding the image biomarker-image cascade fusion module, the F1-score improved to 84.58, indicating that cascade updates enable continuous interaction between biomarker information and image structural changes during the two-time-point process. Adding the clinical information-image progression fusion module resulted in an F1-score of 84.01, demonstrating that the nodalization of clinical information and its corresponding fusion with image features can provide supplementary evidence for lesion evolution prediction. When both the cascaded fusion module and the progress fusion module are retained, but TVP is removed, the F1-score is 84.74; the complete model further improves to 85.69, achieving an AUC of 96.39, an Accuracy of 92.25, a Specificity of 94.10, and a Precision of 84.33. These results demonstrate that the TVP mechanism can further constrain the model to focus on dual-timepoint visual progress features based on multimodal fusion, thereby improving overall prediction stability.

[0153] Table 2 summarizes the comparative experimental results of different methods: Table 2

[0154] Experimental results show that the method provided in this application achieves the best results in AUC, Accuracy, Specificity, Precision and F1-score, reaching 96.39%, 92.25%, 94.10%, 84.33% and 85.69% respectively, indicating that the method provided in this application can more effectively integrate dual-time point OCT images, imaging biomarkers and clinical information, thereby improving subsequent prediction performance.

[0155] In summary, ablation experiments show that the graph-enhanced dual-path radiomics-gated image encoder, the image biomarker-image cascade fusion module, the clinical information-image progression fusion module, and the TVP progression constraint all make practical contributions to the model performance. Comparative experiments further demonstrate that the method provided in this application can make fuller use of structural changes, lesion signs, and clinical status changes in dual-time point follow-up data, thereby improving the predictive performance of fundus lesions.

[0156] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0157] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0158] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0159] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0160] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for predicting the evolution of fundus lesions based on temporal multimodal data, characterized in that, include: Acquire dual-timepoint OCT slice sequences, imaging biomarkers, and dual-timepoint clinical information of the subjects to be tested; The dual-time point OCT slice sequence is input into the image feature extraction module of the fundus lesion evolution prediction model, and the output is dual-time point slice-level visual features and dual-time point map-enhanced slice-level image features; The visual features at the two time points slice level are fused to obtain the first fused feature; The dual-time point slice-level visual features and image biomarkers are input into the image biomarker-image cascade fusion module in the fundus lesion evolution prediction model. The dual-time point slice-level visual features and image biomarkers are cascaded and fused in chronological order to obtain the second fused feature. The enhanced slice-level image features of the dual time point map and the dual time point clinical information are input into the clinical information-image progression fusion module in the fundus lesion evolution prediction model. The enhanced slice-level image features of the dual time point map and the dual time point clinical information are aligned and fused at the node level in chronological order to obtain the clinical information-image time point progression representation. Pooling is performed on the first fusion feature, the second fusion feature, and the clinical information-image time-point progression representation. The three pooled features are then concatenated to obtain the fundus lesion evolution features. The fundus lesion evolution features are then classified to obtain the evolution direction of the fundus lesions in the subject to be tested.

2. The method for predicting the evolution of fundus lesions based on temporal multimodal data according to claim 1, characterized in that, The dual-timepoint OCT slice sequence is input into the image feature extraction module, which outputs dual-timepoint slice-level visual features and dual-timepoint image-enhanced slice-level image features, including: The OCT slice sequences at each time point are input into the segmentation network, which outputs a binary segmentation mask of the retinal region. The binary segmentation mask is used as the region of interest to extract two-dimensional radiomics features from each slice of the OCT slice sequence at each time point, thereby obtaining the slice-by-slice radiomics features at each time point. The slice-by-slice image omics features at each time point are input into a gating network composed of a multilayer perceptron and a sigmoid activation function, and the slice-level gating weights at each time point are output. The OCT slice sequences at each time point are input into the parallel CNN branch and the visual Transformer branch respectively. The spatial feature maps of each time point output by the CNN branch are processed by generalized mean pooling and linear projection to obtain the slice-level CNN features at each time point. The output of the visual Transformer branch is processed by the projection layer to obtain the slice-level ViT features at each time point. By using slice-level gating weights at each time point, slice-level CNN features and slice-level ViT features at each time point are weighted and fused to obtain slice-level visual features at each time point; Using the slice-level visual features at each time point as the initial node representation, and based on the spatial continuity and spatial correlation between slices in the OCT slice sequence at each time point, a slice relationship graph at each time point is constructed. The slice relationship graph at each time point and the initial node representation are input into the graph attention network, and the graph-enhanced slice-level image features at each time point are output.

3. The method for predicting the evolution of fundus lesions based on temporal multimodal data according to claim 1, characterized in that, The dual-time-point slice-level visual features and image biomarkers are input into the image biomarker-image cascade fusion module. The dual-time-point slice-level visual features and image biomarkers are cascaded and fused in chronological order to obtain the second fused feature, which includes: Image biomarkers are input into the projection layer for mapping to obtain the initial image biomarker state; The initial image biomarker status and the slice-level visual features at the first time point are input into the orthogonal perception image biomarker-visual fusion module, and the initial fusion features are output. The initial fusion features and the slice-level visual features at the second time point are input into the orthogonal sensing image biomarker-visual fusion module, and the second fusion features are output.

4. The method for predicting the evolution of fundus lesions based on temporal multimodal data according to claim 3, characterized in that, The orthogonal perception image biomarker-visual fusion module fuses the input slice-level visual features with the initial image biomarker state or initial fusion features through the following steps: Layer normalization is performed on the slice-level visual features and the initial image biomarker state or initial fusion features respectively to obtain the target slice-level visual features and the target initial image biomarker state or target initial fusion features. After splicing the target slice-level visual features with the target's initial image biomarker state or the target's initial fused features along the feature dimension, the context features are obtained through linear mapping. The target initial image biomarker state or target initial fusion feature and context features are input into the multi-head attention branch. The target initial image biomarker state or target initial fusion feature is used as the query input, and the context features are used as the key input and value input to calculate the first relevance alignment result. Input the target initial image biomarker state or target initial fusion features and context features into the orthogonal modality fusion branch, use the target initial image biomarker state or target initial fusion features as query input, and use context features as key input and value input to calculate the second correlation alignment result; The first and second correlation alignment results are weighted and fused to obtain the comprehensive attention result. The comprehensive attention result, the target slice-level visual features and the target initial image biomarker state or the target initial fusion features are input into the gating mapping network to construct the first gating coefficient. The first gating coefficient is applied to the comprehensive attention result, and the initial image biomarker state or initial fusion feature is updated through residual method to obtain intermediate result; After layer normalization, the intermediate results are input into the feedforward mapping network. The intermediate results are enhanced based on the output of the feedforward mapping network to obtain the output of the orthogonal perception image biomarker-visual fusion module.

5. The method for predicting the evolution of fundus lesions based on temporal multimodal data according to claim 4, characterized in that, The orthogonal modal fusion branch computes a second relevance alignment result based on the query input, key input, and value input, including: Linear mapping is performed on the query input, key input, and value input to obtain the query representation, key representation, and value representation, respectively; wherein, the query representation is composed of multiple query node features, and the key representation is composed of multiple reference node features; Based on the inner product similarity between the features of each query node and the features of each reference node, the orthogonal weights of the features of each query node and the features of each reference node are obtained; the orthogonal weights are normalized along the dimension of the number of reference nodes to obtain the normalized weights of the features of each query node and the features of each reference node, thus obtaining the normalized weight matrix. The second correlation alignment result is obtained by weighting and summing the value representations using a normalized weight matrix.

6. The method for predicting the evolution of fundus lesions based on temporal multimodal data according to claim 1, characterized in that, The enhanced slice-level image features and clinical information from both time points are input into the clinical information-image progression fusion module. The module performs node-level alignment and fusion of the enhanced slice-level image features and clinical information from both time points in chronological order to obtain a clinical information-image time-point progression representation, including: Clinical information at each time point is input into the clinical information node projection module for projection and expansion to obtain the clinical node features at each time point. The image enhancement slice-level image features and clinical node features at the same time point are input into the clinically-led image fusion module, which outputs clinical information-image collaborative features at each time point. The clinical information and image collaborative features at each time point are input into the time point progress perception and representation aggregation module, which outputs a clinical information-image time point progress representation.

7. The method for predicting the evolution of fundus lesions based on temporal multimodal data according to claim 6, characterized in that, Clinical information at each time point is input into the clinical information node projection module for projection and expansion to obtain the clinical node features at each time point, including: Based on the mean and standard deviation of continuous variables in clinical information at two time points, the continuous variables in clinical information at each time point are normalized to obtain the normalized continuous variables in clinical information at each time point. Normalized continuous variables and discrete variables in clinical information at each time point are concatenated to obtain the clinical representation at each time point. The clinical representation at each time point is then input into a clinical information projection mapping network to obtain the projected clinical representation at each time point. The clinical representations at each time point after projection are copied and expanded in the row direction and added to the learnable slice position embedding matrix to obtain the clinical node features at each time point.

8. The method for predicting the evolution of fundus lesions based on temporal multimodal data according to claim 6, characterized in that, Image enhancement slice-level features and clinical node features from the same time point are input into the clinically-led image fusion module, which outputs clinical information-image collaborative features from each time point, including: After performing layer normalization on the image enhancement slice-level image features and clinical node features at the same time point, they are spliced ​​together in the feature dimension. The spliced ​​features are then linearly mapped to obtain the visual reference context features at that time point. The visual reference context features and the layer-normalized clinical node features at each time point are input into the multi-head attention branch. The layer-normalized clinical node features are used as query input, and the visual reference context features are used as key input and value input to calculate the cross-attention coefficient at each time point. The cross-attention coefficients at each time point, the clinical node features after layer normalization, and the graph-enhanced slice-level image features after layer normalization are input into the gating mapping network to construct the second gating coefficients at each time point. The second gating coefficient at each time point is applied to the cross-attention coefficient, and the clinical node features at each time point are updated through residuals to obtain the intermediate features at each time point. The intermediate features at each time point are normalized by layers and then input into the feedforward mapping network. The output of the feedforward mapping network is used to enhance the intermediate features at each time point, thus obtaining the clinical information-image collaborative features at each time point.

9. The method for predicting the evolution of fundus lesions based on temporal multimodal data according to claim 6, characterized in that, The clinical information and image-coordinated features at various time points are jointly input into the time-point progression perception and representation aggregation module, which outputs a clinical information-image time-point progression representation, including: Calculate the difference features of the clinical information-image co-features at two time points, and perform layer normalization on the difference features to obtain node-level change features; By stacking the clinical information-image co-features and differential features from two time points in chronological order, a short sequence representation is obtained; The short sequence representation is input into the multi-head attention branch to obtain the attention output; the attention output and the short sequence representation are then connected by a residual connection and then normalized by a layer to obtain the target short sequence representation. Nonlinear transformation, residual connection, and layer normalization are applied to the target short sequence representation to obtain time-enhanced short temporal features. The time-enhanced features at each sequence position in the time-enhanced short temporal features are linearly scored and then normalized along the short temporal dimension to obtain the feature weights at each sequence position. By using the feature weights of each sequence position, the temporal enhancement features of each sequence position in the temporal enhancement short temporal feature are weighted and aggregated to obtain the clinical information-image temporal progression representation.

10. The method for predicting the evolution of fundus lesions based on temporal multimodal data according to claim 1, characterized in that, When training a model for predicting the evolution of fundus lesions, the steps for constructing the loss function for fundus lesion evolution include: The dual-time-point spatial feature maps of the training samples output by the image feature extraction module are rearranged to obtain a dual-time-point spatial feature sequence; based on the similarity between each spatial location in the spatial feature sequence of the two time points, a spatial similarity matrix is ​​obtained. Normalize each similarity in the spatial similarity matrix to obtain the similarity weight of each spatial location; use the similarity weight of each spatial location to reconstruct the spatial feature sequence of the previous time point in a weighted manner to obtain the historical reference features. After normalizing the spatial feature sequences of historical reference features and subsequent time points in the channel dimension, the temporal visual consistency response of each spatial location at the two time points is calculated; the temporal visual consistency responses of all spatial locations at the two time points are aggregated and mapped to obtain the visual progress score of the training samples. A visual progression loss function is constructed based on the true label of fundus lesion evolution direction and the smoothed L1 loss of visual progression score of the training samples. Based on the focus loss between the predicted evolution direction of fundus lesions in the training samples and the true labels, a main classification loss function is constructed. The loss function for fundus lesion evolution is obtained by weighted summation of the visual progression loss function and the main classification loss.

Images