A deep learning-based irregularly sampled fundus image prediction system and method

Abstract

The application relates to a deep learning-based irregular sampling fundus image prediction system and method, and belongs to the technical field of image processing.The application comprises the following steps: 1, initializing a fundus image dataset; 2, constructing a style transfer model; 3, training the style transfer model to obtain a trained style transfer model; 4, training a space-time prediction model by using time sequence data; 5, jointly analyzing a predicted image and a registered test set to generate a disease classification curve and a lesion segmentation curve for fundus image analysis; the application extracts a fundus image to perform content feature and style feature decomposition, uses a convolutional recurrent unit and a motion estimation unit to obtain predicted content features, fuses the predicted content features and style features to generate a predicted fundus image, and generates a curve in a disease classification and lesion segmentation mode, so that the evolution result of fundus diseases and lesions can be obtained according to the trend of the curve.

Description

TECHNICAL FIELD

[0001] The present application relates to a deep learning-based irregularly sampled fundus image prediction system and method, and belongs to the technical field of image processing. BACKGROUND

[0002] A fundus image is a detailed view of the back of the eye, including the retina, optic nerve, blood vessels, and macula, obtained through a non-invasive examination. According to the fundus image, specific lesions or features of diseases such as diabetic retinopathy, glaucoma, pathological myopia, and age-related macular degeneration can be analyzed in depth, thereby assisting doctors in clinical research. The development of these diseases is a gradual process. If the progression of the disease can be monitored based on follow-up images for fundus image prediction analysis, doctors can provide personalized diagnosis and treatment recommendations for patients, aiming to effectively slow down or even prevent the progression of the disease and improve the quality of life of patients.

[0003] Existing research on fundus images mainly focuses on disease classification or lesion segmentation. These studies are immediate analyses of the current input fundus condition. In pixel-level fundus image prediction, existing methods often use a single time point image and a time interval to perform fundus image prediction analysis, without fully utilizing the time dimension information in sequence data. Although there are some methods that use sequence fundus images for prediction tasks, these methods are essentially classification operations that do not achieve fine-grained prediction at the pixel level. When processing sequence data, these methods mainly extract parameters or features at each time point in sequence and learn the temporal correlation between sequence features, ignoring spatial information. In addition, existing prediction tasks based on sequence images usually require input sequence images with uniform sampling intervals, which does not match the irregularly sampled follow-up data obtained in actual clinical applications. If a method can be developed to perform image prediction based on irregularly sampled sequence fundus data, it will help doctors understand the progression trend of diseases or lesions, thereby providing more comprehensive diagnosis and treatment plans, which is of great significance for improving disease prevention and control.

[0004] Therefore, how to use irregularly sampled fundus image data for pixel-level fundus image prediction has become a problem to be solved. SUMMARY

[0005] The present application aims to solve the technical problem that existing fundus image prediction analysis cannot use irregularly sampled fundus image data for pixel-level fundus image prediction analysis, and proposes a deep learning-based irregularly sampled fundus image prediction system and method.

[0006] The working principle of the application is that: by embedding a neural differential-based motion estimation unit between the convolutional recurrent units of the recursive convolutional neural network stack, the continuous modeling of the sequence image is realized, thereby effectively solving the problem of irregular eye fundus image data input. At the same time, the style transfer method is introduced in the overall prediction structure to avoid the influence of the difference between the styles of images at different times on the prediction analysis of the eye fundus image. The application realizes the prediction analysis of the eye fundus image using irregularly sampled eye fundus data by using the convolutional recurrent neural network method.

[0007] The purpose of the application is realized by the following technical scheme:

[0008] The application discloses an irregularly sampled eye fundus image prediction method based on deep learning, which comprises the following steps:

[0009] Step 1: initialize the eye fundus image dataset, and divide the dataset into a training set and a test set by using a random allocation method;

[0010] Step 2: construct a style transfer model;

[0011] Step 2.1: initialize the feature encoder, feature decoder, content adaptive module, style adaptive module and adaptive interaction module in the style transfer model;

[0012] Step 2.2: input the image into the feature encoder of the VGG19 network to obtain image features;

[0013] Step 2.3: normalize the input image features, and adaptively obtain content features by using the pixel point self-attention of the content adaptive module;

[0014] Step 2.4: style representation of the input image features is performed by using the channel self-attention of the style adaptive module, so as to obtain style features;

[0015] Step 2.5: the content features and style features obtained in steps 2.3 and 2.4 are fused by using the cross-attention of the adaptive interaction module, so as to obtain fused features;

[0016] Step 2.6: the feature decoder is constructed by cross-stacking the convolutional layer and the activation layer, and the fused features are output by the feature decoder;

[0017] Step 3: train the style transfer model to obtain a trained style transfer model;

[0018] Step 3.1: image features are extracted from the images in the training set by using the feature encoder;

[0019] Step 3.2: enhancing the content features and style features of the extracted features of the image respectively by using the content adaptive module and the style adaptive module of the style transfer model;

[0020] Step 3.3: inputting the enhanced content features and style features into the adaptive interaction module to obtain enhanced fusion features;

[0021] Step 3.4: performing feature decoding on the enhanced fusion features by the feature decoder;

[0022] Step 4: training the spatio-temporal prediction model by using the time series data;

[0023] Step 4.1: construction of the spatio-temporal prediction model;

[0024] Step 4.1.1: reconstructing the style transfer model completed in step 3 to obtain a content encoding module, a style encoding module and a target decoding module; further, the feature encoder is cascaded with the content adaptive module to form the content encoding module, the feature encoder is cascaded with the style adaptive module to form the style encoding module, and the adaptive interaction module is cascaded with the feature decoder to form the target decoding module;

[0025] Step 4.1.2: reconstructing the convolutional recurrent unit and the motion estimation unit in the spatio-temporal prediction model in the manner shown in formula (1);

[0026]

[0027] wherein, represents the hidden state output by the lth layer convolutional recurrent unit at the tth moment, Enc and Dec represent encoding and decoding operations respectively, H' is the feature after encoding, E t represents the time interval between the (t-1)th moment and the tth moment, and respectively represent the motion offset and the motion trend of the lth layer at the (t-1)th moment, is the calculation result of H' with the time interval E t , * and W g respectively represent convolution operation and convolution kernel, σ represents sigmoid activation operation, ⊙ represents Hadamard product, g t is the calculated gating variable, is the hidden state obtained by the motion estimation unit using neural differential.

[0028] Step 4.1.3: optimizing the motion estimation unit in step 4.1.2 in the manner shown in formula (2);

[0029]

[0030] wherein the ODESolver utilizes equation (2) to model the continuity in a cumulative manner; Δt represents the time interval of the current cycle, u t for the update gate, r t for the reset gate, represents the instantaneous motion velocity, F′ t-1+Δt represents the instantaneous change, a is a preset hyperparameter (between 0 and 1), tanh represents a tanh activation operation, and Warp is a bilinear interpolation deformation operation;

[0031] Step 4.2: obtaining a predicted image by training the spatio-temporal prediction model;

[0032] Step 4.2.1: registering the training set and the test set by using the key point detection method on the fundus images of the historical samples and the first-year samples in the dataset, to obtain a registered training set and a registered test set;

[0033] Step 4.2.1: extracting the content features and the style features of the registered training set by using the content encoding module and the style encoding module;

[0034] Step 4.2.2: inputting the content features and the time interval information of the registered training set into the stacked convolutional recurrent unit and the motion estimation unit in sequence to perform content feature prediction;

[0035] Step 4.2.3: generating a predicted image set by fusing the style features of the registered training set and the content features output by the spatio-temporal prediction model through the target decoding module;

[0036] Step 4.2.4: obtaining the trained spatio-temporal prediction model by using the loss function shown in equation (3) through a cyclic iteration manner;

[0037]

[0038] L = L L1 + βL mse + L feature + L perceptual

[0039] wherein n represents the batch size of the training iteration, I i,2:T and respectively represent the original input image and the output predicted image of the i-th registered training sample at the 2nd to T-th moment; f i,2:T and respectively represent the content features and the output predicted content features of the i-th registered training sample at the 2nd to T-th moment; φ j represents the jth layer of the VGG19 network pre-trained by using the ImageNet. β is a preset hyperparameter.

[0040] Step 4.2.5: input the registered test set into the trained spatio-temporal prediction model to obtain a predicted image;

[0041] Step 5: jointly analyze the predicted image and the registered test set to generate a disease classification curve and a lesion segmentation curve for fundus image analysis;

[0042] Step 5.1: respectively perform disease classification on the predicted image and the registered test set according to disease types;

[0043] Step 5.2: respectively perform lesion segmentation on the predicted image and the registered test set according to lesion states;

[0044] Step 5.3: respectively perform joint analysis on the predicted image and the registered test set through the disease classification of step 5.1 and the lesion segmentation of step 5.2;

[0045] Step 5.4: respectively generate a disease classification curve and a lesion segmentation curve through the joint analysis of step 5.3 by using a statistical method;

[0046] The irregular sampling fundus image prediction system based on deep learning disclosed in the application is used to realize the above method. The irregular sampling fundus image prediction system based on deep learning disclosed in the application comprises a feature decomposition module, a feature library module, a prediction module, a target decoding module and an application module.

[0047] The feature decomposition module is used to extract content features and style features from sample fundus images; further, the sample fundus images input into the feature decomposition module are used to respectively obtain the content features and the style features by using pixel point self-attention and channel self-attention, and the content features and the style features are placed in the feature library module;

[0048] The feature library module is used to store the content features and the style features of the sample from the feature decomposition module, input the content features of the sample into the prediction module, and input the style features of the sample into the target decoding module;

[0049] The prediction module is used to convert the content features into predicted content features, and then predict the content features corresponding to the fundus images; the prediction module comprises a stacked convolutional recurrent unit and a motion estimation unit based on ordinary differential, the convolutional recurrent unit obtains updated content features by capturing dynamic features of sequence data from the content features from the feature library module, and inputs the updated content features into the motion estimation unit; the motion estimation unit combines the content features with time interval information, outputs predicted content features after correcting the motion features, and inputs the predicted content features as the input of the target decoding module;

[0050] The target decoding module is used for fusing the predicted content feature and the style feature of the sample and outputting a predicted fundus image; further, the predicted content feature and the style feature of the sample are input into the target decoding module, the target decoding module fuses the content feature and the style feature of the sample by using cross attention, and decodes the fused content feature and the style feature of the sample by using a feature decoder to output the predicted fundus image, and the predicted fundus image is taken as an input of the application module;

[0051] The application module is used for performing prediction analysis on the fundus image, jointly analyzing the predicted fundus image and the sample image according to disease classification and lesion segmentation, generating a disease classification curve and a lesion segmentation curve of the irregularly sampled fundus image, and obtaining an evolution result of fundus diseases and lesions according to the trend of the disease classification curve and the lesion segmentation curve.

[0052] Beneficial effects:

[0053] Compared with the existing fundus image prediction network, the present application has the following beneficial effects:

[0054] 1. The present application does not need to perform style normalization operation on the sequence fundus image in advance, introduces a style transfer mechanism in the prediction network, decouples the style feature and the content feature of the image, only predicts and learns the content feature, so that the difference in image style between different input moments can be avoided to interfere with the prediction, and finally an arbitrary specified style feature is used to fuse and decode the predicted content feature to generate a visual fundus image;

[0055] 2. The present application stacks a convolutional recurrent unit and a neural differential-based motion estimation unit in the spatio-temporal prediction module to model the changes of the sequence fundus image in a continuous method, so as to realize the spatio-temporal prediction when the input data is an irregularly sampled image;

[0056] 3. The present application can realize the prediction of future fundus images, the images contain more information, and through the analysis of disease classification or lesion segmentation of the existing sequence images and the predicted images, the development trend of various fundus diseases can be evaluated to assist doctors in providing more targeted and personalized prevention or diagnosis suggestions. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 is a flowchart of the present application;

[0058] Figure 2 is a structure diagram of the style transfer model of the present application;

[0059] Figure 3 is a structure diagram of the spatio-temporal prediction model of the present application;

[0060] Figure 4is a result figure of the joint analysis of the present application. DETAILED DESCRIPTION

[0061] For better illustrating the purposes and advantages of the present application, the summary will be further explained in combination with the drawings and examples. It should be noted that the implementation of the present application is not limited to the following examples, and any form of variation or change made to the present application will fall within the scope of protection of the present application.

[0062] EMBODIMENT

[0063] As shown in the drawings, a deep learning-based irregular sampling fundus image prediction method of the present application has the following specific implementation steps: Figure 1 Step 1: initialize the fundus image dataset, and divide the dataset into a training set and a test set by using a random allocation method;

[0064] In the embodiment, 342 samples from 179 patients are clinically collected, each sample containing 4 fundus images collected between 2011 and 2016; 292 samples are used as the training set, and 50 samples are used as the test set; there is no overlap between the patients' data in the training set and the test set when the dataset is divided; and all image data is scaled to 512*512*3;

[0065] Step 2: build a style transfer model;

[0066] Step 2.1: initialize the feature encoder, feature decoder, content adaptive module, style adaptive module and adaptive interaction module in the style transfer model;

[0067] Step 2.2: input the image into the feature encoder of the VGG19 network to obtain the image feature;

[0068] In the embodiment, the parameters of the VGG19 network are obtained by pre-training using the ImageNet dataset, and the network parameters in this part are frozen during subsequent style transfer model training.

[0069] Step 2.3: normalize the input image feature, and use the pixel point self-attention of the content adaptive module for self-adaptation to obtain the content feature;

[0070] Step 2.4: use the channel self-attention of the style adaptive module to perform style representation on the input image feature to obtain the style feature;

[0071] Step 2.5: use the cross-attention of the adaptive interaction module to perform feature fusion on the content feature and the style feature described in steps 2.3 and 2.4 to obtain the fusion feature;

[0072] Step 2.5: use the cross-attention of the adaptive interaction module to perform feature fusion on the content feature and the style feature described in steps 2.3 and 2.4 to obtain the fusion feature;

[0073] Step 2.6: build the feature decoder by cross-stacking of convolution layers and activation layers, and output the fusion features through the feature decoder;

[0074] Step 3: train the style transfer model to obtain a trained style transfer model;

[0075] Step 3.1: extract image features of images in the training set through the feature encoder;

[0076] Step 3.2: enhance the content features and style features of the extracted features of the images respectively by using the content adaptive module and the style adaptive module of the style transfer model;

[0077] Step 3.3: input the enhanced content features and style features into the adaptive interaction module to obtain enhanced fusion features;

[0078] Step 3.4: decode the enhanced fusion features through the feature decoder;

[0079] In the embodiment, the specific network structure is as shown in Figure 2 The input of the network includes two images, one of which is a reference image and the other is a source image. The purpose of the network is to ensure that the generated image has consistent content information with the source image and the same style information as the reference image. First, the two input images are respectively input into the feature extractor to obtain the features of the two images. The features of the source image are input into the content adaptive module to obtain the content features, and the features of the reference image are input into the style adaptive module to obtain the style features. The obtained content features and style features are fused by using the adaptive interaction module to obtain the fusion features. Finally, the fusion features are output through the feature decoder to obtain the target image. The content of the target image is consistent with the source image, and the style is consistent with the reference image.

[0080] Step 4: train the spatio-temporal prediction model using time series data;

[0081] Step 4.1: construction of the spatio-temporal prediction model;

[0082] Step 4.1.1: reconstruct the style transfer model completed in step 3 to obtain a content encoding module, a style encoding module and a target decoding module; further, the feature encoder and the content adaptive module are cascaded to form the content encoding module, the feature encoder and the style adaptive module are cascaded to form the style encoding module, and the adaptive interaction module and the feature decoder are cascaded to form the target decoding module;

[0083] Step 4.1.2: reconstruct the convolutional recurrent unit and the motion estimation unit in the spatio-temporal prediction model in the manner shown in formula (1);

[0084]

[0085] in, This represents the hidden state output by the l-th layer convolutional recurrent unit at time t, where Enc and Dec represent the encoding and decoding operations, respectively, H′ is the encoded feature, and E t This represents the time interval between time t-1 and time t. and These represent the motion offset and motion trend of the l-th layer at time t-1, respectively. To H′ at time interval E t The result of continuous integration, * and W g They represent the convolution operation and the convolution kernel, respectively; σ represents the sigmoid activation operation; ⊙ represents the Hadamard product; g t For the calculated gated variables, The hidden state is obtained by using the motion estimation unit with frequent differentials.

[0086] Step 4.1.3: Optimize the motion estimation unit in step 4.1.2 using the method shown in equation (2);

[0087]

[0088] In this model, ODESolver uses equation (2) to perform continuous modeling in a cumulative manner; Δt represents the time interval of the current cycle, u t To update the gate, r t To reset the door, F′ represents the instantaneous velocity. t-1+Δt Indicates instantaneous change, α is a preset hyperparameter (between 0 and 1), tanh indicates tanh activation operation, and Warp is a transformation operation of bilinear interpolation;

[0089] In the embodiment, the reconstructed content encoding module, style encoding module, and target decoding module are structured as follows: Figure 2 The following section illustrates the process. After training the style transfer model, the feature encoder is cascaded with the content adaptation module to form the content encoding module, the feature encoder is cascaded with the style adaptation module to form the style encoding module, and the adaptive interaction module is cascaded with the feature decoder to form the target decoding module. During subsequent training of the spatiotemporal prediction network, the parameters of the content encoding module, style encoding module, and target decoding module are frozen. The convolutional recurrent unit uses a Convolutional Long Short-Term Memory (ConvLSTM) recursive unit; α is set to 0.5, and Δt equals E. t / 3;

[0090] Step 4.2: Obtain the predicted image by training the spatiotemporal prediction model;

[0091] Step 4.2.1: The historical samples in the data set and the first-year sample fundus image are respectively registered with the key point detection method to obtain the registered training set and the registered test set;

[0092] Step 4.2.1: The content features and style features of the registered training set are extracted by using the content encoding module and the style encoding module;

[0093] Step 4.2.2: The content features and time interval information of the registered training set are sequentially input into the stacked convolutional recurrent unit and the motion estimation unit for content feature prediction;

[0094] In the embodiment, the stacked manner of the convolutional recurrent unit and the motion estimation unit is to insert two constant differential based motion estimation units in the three ConvLSTM units, as shown by the dashed box in Figure 3 . Among them, the ConvLSTM is used to update the content features, and the updated content features are combined with the time interval information to input the motion estimation module for motion correction of the content features, and finally the predicted content features are output.

[0095] Step 4.2.3: The style features of the registered training set and the content features output by the spatiotemporal prediction model are fused by the target decoding module to generate a predicted image set;

[0096] Step 4.2.4: The loss function shown in formula (3) is used to obtain the trained spatiotemporal prediction model by means of iterative loop;

[0097]

[0098] L=L L1 +βL mse +L feature +L perceptual

[0099] Where n represents the batch size of training iteration, I i,2:T and respectively represent the original input image and the output predicted image of the i-th registered training sample at the 2nd to Tth moment; f i,2:T and respectively represent the content features and the output predicted content features of the i-th registered training sample at the 2nd to Tth moment; φ j represents the jth layer of the VGG19 network pre-trained by ImageNet. β is a preset hyperparameter.

[0100] In the embodiment, n is set to 2, T is equal to 4, and β is set to 10. The fundus images at 3 time points are used to predict the fundus image at the 4th time point.

[0101] Step 4.2.5: input the registered test set into the trained spatio-temporal prediction model to obtain a predicted image;

[0102] Step 5: jointly analyze the predicted image and the registered test set to generate a disease classification curve and a lesion segmentation curve for fundus image analysis;

[0103] Step 5.1: respectively perform disease classification on the predicted image and the registered test set according to disease types;

[0104] Step 5.2: respectively perform lesion segmentation on the predicted image and the registered test set according to lesion states;

[0105] In the embodiment, the registered follow-up fundus image and the predicted image of a sample are segmented by using a pre-trained atrophy arc segmentation network to obtain the area of the atrophy arc at each time point.

[0106] Step 5.3: jointly analyze the predicted image and the registered test set by the disease classification of step 5.1 and the lesion segmentation of step 5.2;

[0107] In the embodiment, the registered follow-up fundus image and the predicted image of a sample are segmented by using a pre-trained atrophy arc segmentation network to obtain the area of the atrophy arc at each time point.

[0108] Step 5.4: generate a disease classification curve and a lesion segmentation curve by the joint analysis of step 5.3 using a statistical method;

[0109] In the embodiment, the area of the atrophy arc at each time point calculated in step 5.3 is plotted into a change curve, as shown in FIG. 5. Figure 4 The atrophy arc of the sample presents a trend of continuous expansion, and since the area growth of the atrophy arc is positively correlated with the development of myopia disease, the patient should be reminded to pay attention to eye use and avoid the development of myopia.

[0110] The irregular sampling fundus image prediction system based on deep learning disclosed in the present application is used to realize the above method. The irregular sampling fundus image prediction system based on deep learning disclosed in the present application comprises a feature decomposition module, a feature library module, a prediction module, a target decoding module and an application module.

[0111] The feature decomposition module is used to extract content features and style features from sample fundus images; further, the sample fundus images input into the feature decomposition module are used to obtain content features and style features by using pixel point self-attention and channel self-attention respectively, and the content features and the style features are placed in the feature library module;

[0112] The feature library module is configured to store the content features and style features of the sample from the feature decomposition module, input the content features of the sample to the prediction module, and input the style features of the sample to the target decoding module.

[0113] The prediction module is configured to convert the content features into predicted content features, and further predict the content features corresponding to the fundus image.

[0114] The target decoding module is configured to fuse the predicted content features and the style features of the sample and output a predicted fundus image.

[0115] The application module is configured to perform prediction analysis on the fundus image, jointly analyze the predicted fundus image and the sample image according to disease classification and lesion segmentation, generate a disease classification curve and a lesion segmentation curve of the irregularly sampled fundus image, and obtain the evolution result of the fundus disease and lesion according to the trend of the disease classification curve and the lesion segmentation curve.

[0116] In the embodiment, the content features and the style features of the sample fundus image are decomposed, the stacked convolution recurrent unit and the motion estimation unit based on ordinary differential equation in the prediction module are used to obtain the predicted content features, the target decoding module is used to fuse the predicted content features and the style features of the sample to generate the predicted fundus image, and in the application module, the disease classification curve and the lesion segmentation curve of the image are generated in the manner of disease classification and lesion segmentation, so that the evolution result of the fundus disease and lesion is obtained according to the trend of the curve.

[0117] In order to further illustrate the superiority of the present application, an ablation experiment is used for illustration.

[0118] In the ablation experiment, the effectiveness of the reconstructed ordinary differential motion estimation unit and the introduction of the style transfer strategy are verified respectively. The evaluation indicators of the model are peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and mean square error (MSE). The larger the PSNR and SSIM, the better the quality of the predicted image generated by the model, and the closer to the true value. The smaller the MSE, the better the performance of the model.

[0119] In order to verify the effectiveness of the ordinary differential motion estimation unit, it is compared with the original motion estimation unit. Since the original motion estimation unit cannot process irregularly sampled data, the prediction output of the prediction network is used to replace the missing input.

[0120] In the experiment of verifying the effectiveness of the style transfer strategy, the input and output features of the stacked convolutional recurrent unit and the motion estimation unit are no longer content features, but fusion features obtained by fusing content features and style features. The final predicted image is obtained by decoding the output prediction fusion features using a feature decoder.

[0121] Table 1 shows the results of the ablation experiment.

[0122]

[0123] By comparing the results in the first row and the third row in Table 1, the effectiveness of the reconstructed ordinary differential motion estimation unit can be shown. By comparing the last two rows in Table 1, the necessity of introducing the style transfer strategy can be shown.

[0124] In summary, the specific description of the above is further explained in detail, and the purpose, technical scheme and beneficial effects of the application are further explained in detail. It should be understood that the above is only a specific embodiment of the application and does not limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the application should be included in the protection scope of the application.

Claims

1. A deep learning-based irregularly sampled fundus image prediction method, characterized by: Comprising the following steps, Step 1: initialize the fundus image dataset, and split the dataset into a training set and a test set by random allocation; Step 2: build a style transfer model; Step 3: train the style transfer model to obtain a trained style transfer model; Step 4: train the spatio-temporal prediction model using time series data; Step 4.1: construction of the spatio-temporal prediction model; Step 4.1.1: reconstruct the style transfer model completed in step 3 to obtain a content encoding module, a style encoding module and a target decoding module; further, the feature encoder is cascaded with the content adaptive module to form the content encoding module, the feature encoder is cascaded with the style adaptive module to form the style encoding module, and the adaptive interaction module is cascaded with the feature decoder to form the target decoding module; Step 4.1.2: reconstruct the convolutional recurrent unit and the motion estimation unit in the spatio-temporal prediction model in the manner shown in formula (1); wherein, denotes the hidden state outputted by the l-th layer convolutional recurrent unit at the t-th time, Enc and Dec denote the encoding and decoding operations respectively, H' is the feature after encoding, E t denotes the time interval between the (t-1)-th time and the t-th time, and denote the motion offset and the motion trend of the l-th layer at the (t-1)-th time respectively, is the hidden state of the motion estimation unit, H' is integrated with the time interval E t , and W g denote the convolution operation and the convolution kernel respectively, σ denotes the sigmoid activation operation, ⊙ denotes the Hadamard product, g t is the calculated gating variable, is the hidden state obtained by the motion estimation unit using neural differential. Step 4.1.3: optimize the motion estimation unit in step 4.1.2 in the manner shown in formula (2); wherein ODESolver utilizes equation (2) to model continuously in a cumulative manner; Δt represents the time interval of the current loop, u t for the update gate, r t for the reset gate, represents the instantaneous motion velocity, F′ t-1+Δt represents the instantaneous change, a is a preset hyperparameter (between 0 and 1), tanh represents a tanh activation operation, and Warp is a deformation operation of bilinear interpolation; Step 4.2: obtain the predicted image by training the spatio-temporal prediction model; Step 4.2.1: register the training set and the test set by key point detection of the fundus images of the samples in each year and the samples in the first year in the dataset, to obtain a registered training set and a registered test set; Step 4.2.1: extract the content features and the style features of the registered training set by using the content encoding module and the style encoding module; Step 4.2.2: input the content features and the time interval information of the registered training set into the stacked convolutional recurrent unit and the motion estimation unit in sequence to predict the content features; Step 4.2.3: generate the predicted image set by fusing the content features output by the target decoding module after the content features and the style features of the registered training set are input into the target decoding module; Step 4.2.4: obtain the trained spatio-temporal prediction model by iterative training of the loss function shown in formula (3) in step 4.2.3; where n denotes the batch size of training iterations, I i,2:T and respectively represent the original input image and the output prediction image corresponding to the 2nd to Tth moment of the ith registered training sample; f i,2:T and respectively represent the content feature and the output prediction content feature corresponding to the 2nd to Tth moment of the ith registered training sample; φ j represents the jth layer of the VGG19 network pre-trained using ImageNet; β is a preset hyperparameter; Step 4.2.5: input the registered test set into the trained spatio-temporal prediction model to obtain the predicted image; Step 5: jointly analyze the predicted image and the registered test set to generate a disease classification curve and a lesion segmentation curve for fundus image analysis; Step 5.1: classify the predicted image and the registered test set according to the disease type; Step 5.2: segment the predicted image and the registered test set according to the lesion state; Step 5.3: jointly analyze the predicted image and the registered test set by the disease classification in step 5.1 and the lesion segmentation in step 5.2; Step 5.4: generate the disease classification curve and the lesion segmentation curve by the joint analysis in step 5.3 using statistical methods.

2. The irregularly sampled fundus image prediction method based on deep learning according to claim 1, wherein: The implementation method of step 2 is, Step 2.1: initialize the feature encoder, the feature decoder, the content adaptive module, the style adaptive module and the adaptive interaction module in the style transfer model; Step 2.2: input the image into the feature encoder of the VGG19 network to obtain the image features; Step 2.3: normalize the input image features, and adaptively obtain the content features by using the pixel self-attention of the content adaptive module; Step 2.4: use the channel self-attention of the style adaptive module to represent the input image features in style, thereby obtaining the style features; Step 2.5: use the cross-attention of the adaptive interaction module to fuse the content features and style features obtained in steps 2.3 and 2.4, thereby obtaining the fused features; Step 2.6: build a feature decoder by cross-stacking convolution layers and activation layers, and output the fused features through the feature decoder. 3.The irregularly sampled fundus image prediction method based on deep learning of claim 1, wherein: The implementation method of step 3 is as follows: Step 3.1: extract image features of images in the training set through the feature encoder; Step 3.2: use the content adaptive module and the style adaptive module of the style transfer model to enhance the content features and the style features extracted from the images, respectively; Step 3.3: input the enhanced content features and style features into the adaptive interaction module to obtain enhanced fused features; Step 3.4: decode the enhanced fused features through the feature decoder.

4. A deep learning-based irregularly sampled fundus image prediction system that implements the method of claim 1, wherein: The feature decomposition module, the feature library module, the prediction module, the target decoding module, and the application module are included. The feature decomposition module is configured to extract content features and style features from sample fundus images, and further configured to obtain the content features and the style features from the sample fundus images input into the feature decomposition module by using pixel self-attention and channel self-attention, respectively, and store the content features and the style features in the feature library module. The feature library module is configured to store the content features and the style features of the sample from the feature decomposition module, input the content features of the sample into the prediction module, and input the style features of the sample into the target decoding module. The prediction module is configured to convert the content features into predicted content features, and further configured to predict the content features corresponding to the fundus images. The prediction module includes stacked convolutional recurrent units and a constant differential-based motion estimation unit. The convolutional recurrent units obtain updated content features from the content features from the feature library module by capturing dynamic features of the sequence data, and input the updated content features into the motion estimation unit. The motion estimation unit combines the content features with time interval information, outputs predicted content features after correcting the motion features, and inputs the predicted content features into the target decoding module. The target decoding module is configured to fuse the predicted content features and the style features of the sample and output a predicted fundus image. Further, the predicted content features and the style features of the sample are input into the target decoding module, the target decoding module fuses the content features and the style features of the sample by using cross-attention, decodes the fused content features and the style features of the sample by using a feature decoder, and outputs a predicted fundus image. The predicted fundus image is input into the application module. The application module is used for the prediction analysis of the fundus image, combined analysis of the predicted fundus image and the sample image according to disease classification and lesion segmentation, generation of disease classification curves and lesion segmentation curves of the irregularly sampled fundus image, and obtaining of the evolution result of the fundus disease and lesion according to the trend of the disease classification curves and the lesion segmentation curves.

Images