Irregular sampling eye fundus image prediction system and method based on deep learning
Through deep learning technology, combined with recursive convolutional neural network and style transfer method, the prediction problem of irregular sampling of fundus image data is solved, pixel-level prediction and disease trend analysis are realized, more targeted diagnosis and treatment suggestions are provided, and more improved Disease prevention and control effect.
Patent Information
- Application Number
- CN202410999179.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-07-24
AI Technical Summary
Existing fundus image prediction methods cannot effectively use irregularly sampled sequence data for pixel-level prediction analysis, ignore the time dimension and spatial information, and cannot achieve accurate prediction of fundus disease progression trends.
Using an irregular sampling fundus image prediction system based on deep learning, the motion estimation unit embedded in the frequent differential of the gods is continuously modeled through recursive convolutional neural network and style transfer method, and the irregular sampling is processed using the convolutional loop unit and the motion estimation unit. Data, introduces a style transfer mechanism to decouple content features and style features to achieve spatial and temporal prediction.
The pixel-level prediction of irregularly sampled fundus images is achieved, which avoids interference with image styles, can more accurately predict the development trend of fundus diseases, provides more targeted diagnosis and treatment suggestions, and improves the effectiveness of disease prevention and control.
Smart Images

Figure CN119090706A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a system and method for predicting irregularly sampled fundus images based on deep learning, and belongs to the technical field of image processing. Background Art
[0002] Fundus images are obtained through a non-invasive examination to show a detailed view of the back of the eye, including the retina, optic nerve, blood vessels, and macula. Based on fundus images, specific lesions or features of diseases such as diabetic retinopathy, glaucoma, pathological myopia, and age-related macular degeneration can be deeply analyzed, thereby assisting doctors in clinical research. The development of these diseases is a gradual process. If the progression of the disease can be monitored based on follow-up images and predictive analysis of fundus images can be performed, it can assist doctors in providing patients with personalized diagnosis and treatment recommendations, aiming to effectively slow down or even prevent the progression of the disease and improve the quality of life of patients.
[0003] Existing research on fundus images mainly focuses on the classification of diseases or the segmentation of lesions. These studies are all about real-time analysis of the current input fundus condition. In pixel-level fundus image prediction, existing methods often perform fundus image prediction and analysis based on images and time intervals at a single moment, and do not fully utilize the time dimension information in the sequence data. Although there are some methods that use sequence fundus images for prediction tasks, these methods are all for predicting a single disease, which is essentially a classification operation, and do not achieve fine pixel-level prediction. When processing sequence data, these methods mainly extract parameters or features at each moment in turn, learn the temporal association between sequence features, and ignore spatial information. In addition, in existing prediction tasks based on sequence images, it is usually required that the input sequence images have a uniform sampling interval, which is inconsistent with the irregular sampling follow-up data obtained in actual clinical applications. If a method for image prediction based on irregularly sampled sequence fundus data can be developed, it will help doctors understand the progression trend of diseases or lesions, thereby providing more comprehensive diagnosis and treatment plans, which is of great significance for improving the prevention and control of diseases.
[0004] Therefore, how to use irregularly sampled fundus image data to perform pixel-level fundus image prediction has become an urgent problem to be solved. Summary of the invention
[0005] The purpose of the present invention is to solve the technical problem that irregularly sampled fundus image data cannot be used for pixel-level fundus image prediction and analysis in the existing fundus image prediction and analysis, and to propose an irregularly sampled fundus image prediction system and method based on deep learning.
[0006] The working principle of the present invention is: by embedding a motion estimation unit based on neural constant differential between the convolution cycle units of the recursive convolutional neural network stack to achieve continuous modeling of the sequence image, the problem of irregular fundus image data input is effectively solved. At the same time, the present invention introduces a style transfer method into the overall prediction structure to avoid the influence of the difference between the image styles at different moments on the prediction and analysis of the fundus image. The present invention adopts a convolutional recursive neural network method to realize the prediction and analysis of the fundus image using irregularly sampled fundus data.
[0007] The objective of the present invention is achieved through the following technical solutions:
[0008] The present invention discloses a method for predicting irregularly sampled fundus images based on deep learning, comprising the following steps:
[0009] Step 1: Initialize the fundus image dataset and divide the dataset into training set and test set by random allocation;
[0010] Step 2: Build a style transfer model;
[0011] Step 2.1: Initialize the feature encoder, feature decoder, content adaptation module, style adaptation module and adaptive interaction module in the style transfer model;
[0012] Step 2.2: Input the image into the feature encoder of the VGG19 network to obtain image features;
[0013] Step 2.3: Normalize the input image features and use the pixel self-attention of the content adaptation module to adapt to obtain content features;
[0014] Step 2.4: Use the channel self-attention of the style adaptation module to represent the style of the input image features, thereby obtaining the style features;
[0015] Step 2.5: The content features and style features described in step 2.3 and step 2.4 are fused using the cross attention of the adaptive interaction module to obtain fused features;
[0016] Step 2.6: Construct a feature decoder by cross-superimposing the convolution layer and the activation layer, and output the fused features through the feature decoder;
[0017] Step 3: Train the style transfer model to obtain a trained style transfer model;
[0018] Step 3.1: Extract image features from the images in the training set through a feature encoder;
[0019] Step 3.2: Use the content adaptation module and style adaptation module of the style transfer model to enhance the content features and style features of the features extracted from the image respectively;
[0020] Step 3.3: Input the enhanced content features and style features into the adaptive interaction module to obtain enhanced fusion features;
[0021] Step 3.4: The enhanced fusion features are decoded through a feature decoder;
[0022] Step 4: Use time series data to train the spatiotemporal prediction model;
[0023] Step 4.1: Construction of spatiotemporal prediction model;
[0024] Step 4.1.1: Reconstruct the style transfer model completed in step 3 to obtain a content encoding module, a style encoding module and a target decoding module; further cascade the feature encoder and the content adaptation module to form a content encoding module, cascade the feature encoder and the style adaptation module to form a style encoding module, and cascade the adaptive interaction module and the feature decoder to form a target decoding module;
[0025] Step 4.1.2: Reconstruct the convolutional cycle unit and motion estimation unit in the spatiotemporal prediction model using the method shown in formula (1);
[0026]
[0027] in, represents the hidden state of the output of the convolutional recurrent unit at the lth layer at the tth time, Enc and Dec represent the encoding and decoding operations respectively, H′ is the feature after encoding, and E t represents the time interval between the t-1th moment and the tth moment, and They represent the motion offset and motion trend of the lth layer at time t-1, respectively. For H′ is measured at the time interval E t The result after continuous integration, * and W g They represent convolution operation and convolution kernel respectively, σ represents sigmoid activation operation, ⊙ represents Hadamard product, g t is the calculated gating variable, is the hidden state obtained by the motion estimation unit using neural constant differentiation.
[0028] Step 4.1.3: Optimize the motion estimation unit in step 4.1.2 using the method shown in formula (2);
[0029]
[0030] ODESolver uses formula (2) to perform continuous modeling in a cumulative manner; Δt represents the time interval of the current cycle, u t is the update gate, r t To reset the gate, Indicates the instantaneous velocity, F′ t-1+Δt represents instantaneous change, α is a preset hyperparameter (between 0 and 1), tanh represents the tanh activation operation, and Warp is the deformation operation of bilinear interpolation;
[0031] Step 4.2: Obtain the predicted image by training the spatiotemporal prediction model;
[0032] Step 4.2.1: Use key point detection to register the training set and test set respectively with the fundus images of the previous year samples and the first year samples in the data set, and obtain the registered training set and the registered test set;
[0033] Step 4.2.1: Use the content encoding module and the style encoding module to extract the content features and style features of the registered training set;
[0034] Step 4.2.2: Input the content features and time interval information of the registered training set into the stacked convolutional loop unit and motion estimation unit in sequence for content feature prediction;
[0035] Step 4.2.3: The style features of the registered training set and the content features output by the spatiotemporal prediction model are fused through the target decoding module to generate a predicted image set;
[0036] Step 4.2.4: Apply the loss function shown in formula (3) in step 4.2.3 to obtain the trained spatiotemporal prediction model through a loop iteration method;
[0037]
[0038] L=L L1 +βL mse +L feature +L perceptual
[0039] Where n is the batch size of the training iteration, I i,2:T and They represent the original input image and output prediction image from time 2 to time T corresponding to the i-th registered training sample; f i,2:T and Respectively represent the content features and output prediction content features of the 2nd to Tth moments corresponding to the i-th registered training sample; φ j represents the jth layer of the VGG19 network pre-trained with ImageNet. β is a preset hyperparameter.
[0040] Step 4.2.5: Input the registered test set into the trained spatiotemporal prediction model to obtain the predicted image;
[0041] Step 5: Jointly analyze the predicted image and the registered test set to generate disease classification curves and lesion segmentation curves for fundus image analysis;
[0042] Step 5.1: Classify the predicted image and the registered test set according to the disease type;
[0043] Step 5.2: Perform lesion segmentation on the predicted image and the registered test set according to the lesion status;
[0044] Step 5.3: Jointly analyze the predicted image and the registered test set by the disease classification described in step 5.1 and the lesion segmentation described in step 5.2;
[0045] Step 5.4: Generate disease classification curve and lesion segmentation curve respectively by using statistical methods through the joint analysis described in step 5.3;
[0046] The present invention discloses an irregularly sampled fundus image prediction system based on deep learning, which is used to implement the above method. The present invention discloses an irregularly sampled fundus image prediction system based on deep learning, which includes a feature decomposition module, a feature library module, a prediction module, a target decoding module, and an application module.
[0047] The feature decomposition module is used to extract content features and style features from the sample fundus image; further, the sample fundus image input to the feature decomposition module is used to obtain content features and style features using pixel self-attention and channel self-attention, and the content features and style features are placed in the feature library module;
[0048] The feature library module is used to store the content features and style features of the samples from the feature decomposition module, input the content features of the samples to the prediction module, and input the style features of the samples to the target decoding module;
[0049] The prediction module is used to convert the content features into predicted content features, and then predict the content features corresponding to the fundus image; the prediction module includes a stacked convolution recursive unit and a motion estimation unit based on ordinary differentials, the convolution recursive unit obtains updated content features from the feature library module by capturing the dynamic features of the sequence data, and inputs the updated content features to the motion estimation unit; the motion estimation unit combines the content features with the time interval information and outputs the predicted content features after correcting the motion features, and uses the predicted content features as the input of the target decoding module;
[0050] The target decoding module is used to fuse the predicted content features and the style features of the sample and output a predicted fundus image; further, the predicted content features and the style features of the sample are input into the target decoding module, the target decoding module fuses the content features and the style features of the sample using cross attention, and uses a feature decoder to decode the fused content features and the style features of the sample and output a predicted fundus image, and the predicted fundus image is used as the input of the application module;
[0051] The application module is used for predictive analysis of fundus images, jointly analyzing the predicted fundus images and sample images according to disease classification and lesion segmentation, generating disease classification curves and lesion segmentation curves for irregularly sampled fundus images, and obtaining the evolution results of fundus diseases and lesions according to the trends of the disease classification curves and lesion segmentation curves.
[0052] Beneficial effects:
[0053] Compared with the existing fundus image prediction network, it has the following beneficial effects:
[0054] 1. The present invention does not need to perform style normalization operation on the sequence fundus images in advance. By introducing the style transfer mechanism in the prediction network, the style features of the image are decoupled from the content features, and only the content features are predicted and learned. This can avoid the interference of the image style differences between different input moments on the prediction. Finally, in order to generate a visualized fundus image, any specified style features are used to fuse and decode with the predicted content features;
[0055] 2. The present invention models the changes of the sequence fundus images in a continuous manner by stacking convolution cycle units and motion estimation units based on neural constant differentials in the spatiotemporal prediction module, thereby realizing spatiotemporal prediction when the input data is an irregularly sampled image;
[0056] 3. The present invention can realize the prediction of future fundus images. The images contain more information. By performing disease classification or lesion segmentation analysis on existing sequence images and predicted images, it can assist doctors in evaluating the development trends of various fundus diseases and help doctors provide more targeted and personalized prevention, control or treatment recommendations. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] Figure 1 It is a schematic diagram of the process of the present invention;
[0058] Figure 2 It is a schematic diagram of the structure of the style transfer model of the present invention;
[0059] Figure 3 It is a schematic diagram of the structure of the spatiotemporal prediction model of the present invention;
[0060] Figure 4It is a result diagram of the joint analysis of the present invention. DETAILED DESCRIPTION
[0061] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below in conjunction with the accompanying drawings and examples. It should be noted that the implementation of the present invention is not limited to the following embodiments, and any form of modification or change made to the present invention will fall within the protection scope of the present invention.
[0062] Example
[0063] like Figure 1 As shown, the irregular sampling fundus image prediction method based on deep learning of the present invention is specifically implemented in the following steps:
[0064] Step 1: Initialize the fundus image dataset and divide the dataset into training set and test set by random allocation;
[0065] In the embodiment, a total of 342 samples from 179 patients were collected clinically, each sample contained 4 fundus images collected between 2011 and 2016; 292 samples were used as training sets and 50 samples were used as test sets; when dividing the data set, there was no overlap between the patient data in the training set and the test set; and all image data were scaled to 512*512*3;
[0066] Step 2: Build a style transfer model;
[0067] Step 2.1: Initialize the feature encoder, feature decoder, content adaptation module, style adaptation module and adaptive interaction module in the style transfer model;
[0068] Step 2.2: Input the image into the feature encoder of the VGG19 network to obtain image features;
[0069] In the embodiment, the parameters of the VGG19 network are obtained by pre-training using the ImageNet dataset, and these network parameters are frozen during subsequent style transfer model training.
[0070] Step 2.3: Normalize the input image features and use the pixel self-attention of the content adaptation module to adapt to obtain content features;
[0071] Step 2.4: Use the channel self-attention of the style adaptation module to represent the style of the input image features, thereby obtaining the style features;
[0072] Step 2.5: The content features and style features described in step 2.3 and step 2.4 are fused using the cross attention of the adaptive interaction module to obtain fused features;
[0073] Step 2.6: Construct a feature decoder by cross-superimposing the convolution layer and the activation layer, and output the fused features through the feature decoder;
[0074] Step 3: Train the style transfer model to obtain a trained style transfer model;
[0075] Step 3.1: Extract image features from the images in the training set through a feature encoder;
[0076] Step 3.2: Use the content adaptation module and style adaptation module of the style transfer model to enhance the content features and style features of the features extracted from the image respectively;
[0077] Step 3.3: Input the enhanced content features and style features into the adaptive interaction module to obtain enhanced fusion features;
[0078] Step 3.4: The enhanced fusion features are decoded through a feature decoder;
[0079] In the embodiment, the specific network structure is as follows Figure 2 As shown. The input of the network contains two images, one of which is the reference image and the other is the source image. The purpose of this network is to ensure that the content information of the generated image is consistent with the source image and the style information is the same as the reference image. First, the two input images are respectively passed through the feature extractor to obtain the features of the two images. The features of the source image are input into the content adaptation module to obtain the content features, and the features of the reference image are input into the style adaptation module to obtain the style features. The obtained content features and style features are fused using the adaptive interaction module to obtain the fused features. Finally, the fused features are output as the target image through the feature decoder. The content of the target image is consistent with the source image, and the style is consistent with the reference image.
[0080] Step 4: Use time series data to train the spatiotemporal prediction model;
[0081] Step 4.1: Construction of spatiotemporal prediction model;
[0082] Step 4.1.1: Reconstruct the style transfer model completed in step 3 to obtain a content encoding module, a style encoding module and a target decoding module; further cascade the feature encoder and the content adaptation module to form a content encoding module, cascade the feature encoder and the style adaptation module to form a style encoding module, and cascade the adaptive interaction module and the feature decoder to form a target decoding module;
[0083] Step 4.1.2: Reconstruct the convolutional cycle unit and motion estimation unit in the spatiotemporal prediction model using the method shown in formula (1);
[0084]
[0085] in, represents the hidden state of the output of the convolutional recurrent unit at the lth layer at the tth time, Enc and Dec represent the encoding and decoding operations respectively, H′ is the feature after encoding, and E t represents the time interval between the t-1th moment and the tth moment, and They represent the motion offset and motion trend of the lth layer at time t-1, respectively. For H′ is measured at the time interval E t The result after continuous integration, * and W g They represent convolution operation and convolution kernel respectively, σ represents sigmoid activation operation, ⊙ represents Hadamard product, g t is the calculated gating variable, is the hidden state obtained by the motion estimation unit using neural constant differentiation.
[0086] Step 4.1.3: Optimize the motion estimation unit in step 4.1.2 using the method shown in formula (2);
[0087]
[0088] ODESolver uses formula (2) to perform continuous modeling in a cumulative manner; Δt represents the time interval of the current cycle, u t is the update gate, r t To reset the gate, Indicates the instantaneous velocity, F′ t-1+Δt represents instantaneous change, α is a preset hyperparameter (between 0 and 1), tanh represents the tanh activation operation, and Warp is the deformation operation of bilinear interpolation;
[0089] In the embodiment, the reconstructed content encoding module, style encoding module and target decoding module structures are as follows: Figure 2 As shown in the following part. The feature encoder obtained after the style transfer model training is cascaded with the content adaptation module to form a content encoding module, the feature encoder is cascaded with the style adaptation module to form a style encoding module, and the adaptive interaction module is cascaded with the feature decoder to form a target decoding module. The parameters of the content encoding module, style encoding module and target decoding module are frozen during the subsequent spatiotemporal prediction network training. The convolutional recurrent unit uses the convolutional long short-term memory recursive unit ConvLSTM; α is set to 0.5, and Δt is equal to E t / 3;
[0090] Step 4.2: Obtain the predicted image by training the spatiotemporal prediction model;
[0091] Step 4.2.1: Use key point detection to register the training set and test set respectively with the fundus images of the previous year samples and the first year samples in the data set, and obtain the registered training set and the registered test set;
[0092] Step 4.2.1: Use the content encoding module and the style encoding module to extract the content features and style features of the registered training set;
[0093] Step 4.2.2: Input the content features and time interval information of the registered training set into the stacked convolutional loop unit and motion estimation unit in sequence for content feature prediction;
[0094] In the embodiment, the convolutional loop unit and the motion estimation unit are stacked in such a way that two ordinary differential based motion estimation units are inserted into three ConvLSTM units, such as Figure 3 As shown in the dotted box in . ConvLSTM is used to update the content features, and the updated content features are combined with the time interval information and input into the motion estimation module to perform motion correction of the content features, and finally the predicted content features are output.
[0095] Step 4.2.3: The style features of the registered training set and the content features output by the spatiotemporal prediction model are fused through the target decoding module to generate a predicted image set;
[0096] Step 4.2.4: Apply the loss function shown in formula (3) in step 4.2.3 to obtain the trained spatiotemporal prediction model through a loop iteration method;
[0097]
[0098] L=L L1 +βL mse +L feature +L perceptual
[0099] Where n is the batch size of the training iteration, I i,2:T and They represent the original input image and output prediction image from time 2 to time T corresponding to the i-th registered training sample; f i,2:T and Respectively represent the content features and output prediction content features of the 2nd to Tth moments corresponding to the i-th registered training sample; φ j represents the jth layer of the VGG19 network pre-trained with ImageNet. β is a preset hyperparameter.
[0100] In the embodiment, n is set to 2, T is equal to 4, and β is set to 10. The fundus images at three moments are used to predict the fundus image at the fourth moment.
[0101] Step 4.2.5: Input the registered test set into the trained spatiotemporal prediction model to obtain the predicted image;
[0102] Step 5: Jointly analyze the predicted image and the registered test set to generate disease classification curves and lesion segmentation curves for fundus image analysis;
[0103] Step 5.1: Classify the predicted image and the registered test set according to the disease type;
[0104] Step 5.2: Perform lesion segmentation on the predicted image and the registered test set according to the lesion status;
[0105] In the embodiment, a pre-trained atrophic arc segmentation network is used to segment the atrophic arc region of a follow-up fundus image and a predicted image after registration of a sample, and the area of the atrophic arc at each moment is calculated.
[0106] Step 5.3: Jointly analyze the predicted image and the registered test set by the disease classification described in step 5.1 and the lesion segmentation described in step 5.2;
[0107] In the embodiment, a pre-trained atrophic arc segmentation network is used to segment the atrophic arc region of a follow-up fundus image and a predicted image after registration of a sample, and the area of the atrophic arc at each moment is calculated.
[0108] Step 5.4: Generate disease classification curve and lesion segmentation curve respectively by using statistical methods through the joint analysis described in step 5.3;
[0109] In the embodiment, the shrinkage arc area at each moment calculated in step 5.3 is plotted as a change curve, such as Figure 4 The atrophy arc of this sample shows a trend of continuous expansion. Since the growth of the atrophy arc area is positively correlated with the development of myopia, patients should be reminded to pay attention to their eye use to avoid the development of myopia.
[0110] The present invention discloses an irregularly sampled fundus image prediction system based on deep learning, which is used to implement the above method. The present invention discloses an irregularly sampled fundus image prediction system based on deep learning, which includes a feature decomposition module, a feature library module, a prediction module, a target decoding module, and an application module.
[0111] The feature decomposition module is used to extract content features and style features from the sample fundus image; further, the sample fundus image input to the feature decomposition module is used to obtain content features and style features using pixel self-attention and channel self-attention, and the content features and style features are placed in the feature library module;
[0112] The feature library module is used to store the content features and style features of the samples from the feature decomposition module, input the content features of the samples into the prediction module, and input the style features of the samples into the target decoding module;
[0113] The prediction module is used to convert the content features into predicted content features, and then predict the content features corresponding to the fundus image; the prediction module includes a stacked convolution recursive unit and a motion estimation unit based on ordinary differentials, the convolution recursive unit obtains updated content features from the feature library module by capturing the dynamic features of the sequence data, and inputs the updated content features to the motion estimation unit; the motion estimation unit combines the content features with the time interval information and outputs the predicted content features after correcting the motion features, and uses the predicted content features as the input of the target decoding module;
[0114] The target decoding module is used to fuse the predicted content features and the style features of the sample and output a predicted fundus image; further, the predicted content features and the style features of the sample are input into the target decoding module, the target decoding module fuses the content features and the style features of the sample using cross attention, and uses a feature decoder to decode the fused content features and the style features of the sample and output a predicted fundus image, and the predicted fundus image is used as the input of the application module;
[0115] The application module is used for predictive analysis of fundus images, jointly analyzing the predicted fundus images and sample images according to disease classification and lesion segmentation, generating disease classification curves and lesion segmentation curves for irregularly sampled fundus images, and obtaining the evolution results of fundus diseases and lesions according to the trends of the disease classification curves and lesion segmentation curves.
[0116] In the embodiment, a sample fundus image is extracted to perform content feature and sample style feature decomposition, and the predicted content feature is obtained using the stacked convolution recursive unit and the ordinary differential-based motion estimation unit under the prediction module; the predicted content feature and the sample style feature are fused using the target decoding module to generate a predicted fundus image; and the predicted fundus image and the sample image are subjected to disease classification and lesion segmentation in the application module to generate a disease classification curve and a lesion segmentation curve for the image, thereby obtaining the evolution results of fundus diseases and lesions based on the trend of the curves.
[0117] In order to further illustrate the advantages of the present invention, an ablation experiment is used for illustration.
[0118] In the ablation experiment, the present invention verifies the effectiveness of the reconstructed ordinary differential motion estimation unit and the introduction of style transfer strategy. The evaluation indicators of the model adopt peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and mean square error (MSE). The larger the PSNR and SSIM, the better the quality of the predicted image generated by the model, which is close to the true value. The smaller the MSE, the better the performance of the model.
[0119] In order to verify the effectiveness of the ordinary differential motion estimation unit, it is compared with the original motion estimation unit. Since the original motion estimation unit cannot handle irregularly sampled data, the predicted output of the prediction network is used to replace the missing input.
[0120] When verifying the effectiveness of the style transfer strategy, the input and output features of the stacked convolutional loop unit and motion estimation unit in the comparative experiment are no longer content features, but fused features obtained by fusing content features and style features. The final predicted image is the image obtained by decoding the output predicted fused features using a feature decoder.
[0121] Table 1 Ablation experiment result index table
[0122]
[0123] By comparing the results in the first row with those in the third row in Table 1, it can be shown that the effectiveness of the reconstructed ordinary differential motion estimation unit is effective. By comparing the last two in Table 1, it can be shown that it is necessary to introduce a style transfer strategy.
[0124] The specific description above further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for predicting irregularly sampled fundus images based on deep learning, characterized in that: The following steps are included: Step 1: Initialize the fundus image dataset and divide the dataset into training set and test set by random allocation; Step 2: Build a style transfer model; Step 3: Train the style transfer model to obtain a trained style transfer model; Step 4: Use time series data to train the spatiotemporal prediction model; Step 4.1: Construction of spatiotemporal prediction model; Step 4.1.1: Reconstruct the style transfer model completed in step 3 to obtain a content encoding module, a style encoding module and a target decoding module; further cascade the feature encoder and the content adaptation module to form a content encoding module, cascade the feature encoder and the style adaptation module to form a style encoding module, and cascade the adaptive interaction module and the feature decoder to form a target decoding module; Step 4.1.2: Reconstruct the convolutional cycle unit and motion estimation unit in the spatiotemporal prediction model using the method shown in formula (1); in, represents the hidden state of the output of the convolutional recurrent unit at the lth layer at the tth time, Enc and Dec represent the encoding and decoding operations respectively, H′ is the feature after encoding, and E t represents the time interval between the t-1th moment and the tth moment, and They represent the motion offset and motion trend of the lth layer at time t-1, respectively. For H′ is measured at the time interval E t The result after continuous integration, * and W g They represent convolution operation and convolution kernel respectively, σ represents sigmoid activation operation, ⊙ represents Hadamard product, g t is the calculated gating variable, is the hidden state obtained by the motion estimation unit using neural constant differentiation; Step 4.1.3: Optimize the motion estimation unit in step 4.1.2 using the method shown in formula (2); ODESolver uses formula (2) to perform continuous modeling in a cumulative manner; Δt represents the time interval of the current cycle, u t is the update gate, r t To reset the gate, Indicates the instantaneous velocity, F′ t-1+Δt represents instantaneous change, α is a preset hyperparameter (between 0 and 1), tanh represents the tanh activation operation, and Warp is the deformation operation of bilinear interpolation; Step 4.2: Obtain the predicted image by training the spatiotemporal prediction model; Step 4.2.1: Use key point detection to register the training set and test set respectively with the fundus images of the previous year samples and the first year samples in the data set, and obtain the registered training set and the registered test set; Step 4.2.1: Use the content encoding module and the style encoding module to extract the content features and style features of the registered training set; Step 4.2.2: Input the content features and time interval information of the registered training set into the stacked convolutional loop unit and motion estimation unit in sequence for content feature prediction; Step 4.2.3: The style features of the registered training set and the content features output by the spatiotemporal prediction model are fused through the target decoding module to generate a predicted image set; Step 4.2.4: Apply the loss function shown in formula (3) in step 4.2.3 to obtain the trained spatiotemporal prediction model through a loop iteration method; Where n is the batch size of the training iteration, I i,2:T and They represent the original input image and output prediction image from time 2 to time T corresponding to the i-th registered training sample; f i,2:T and Respectively represent the content features and output prediction content features of the 2nd to Tth moments corresponding to the i-th registered training sample; φ j represents the jth layer of the VGG19 network pre-trained with ImageNet; β is a preset hyperparameter; Step 4.2.5: Input the registered test set into the trained spatiotemporal prediction model to obtain the predicted image; Step 5: Jointly analyze the predicted image and the registered test set to generate disease classification curves and lesion segmentation curves for fundus image analysis; Step 5.1: Classify the predicted image and the registered test set according to the disease type; Step 5.2: Perform lesion segmentation on the predicted image and the registered test set according to the lesion status; Step 5.3: Jointly analyze the predicted image and the registered test set by the disease classification described in step 5.1 and the lesion segmentation described in step 5.2; Step 5.4: Generate disease classification curves and lesion segmentation curves respectively using statistical methods through the joint analysis described in step 5.
3.
2. The irregularly sampled fundus image prediction method based on deep learning as claimed in claim 1, characterized in that: Step 2 is implemented as follows: Step 2.1: Initialize the feature encoder, feature decoder, content adaptation module, style adaptation module and adaptive interaction module in the style transfer model; Step 2.2: Input the image into the feature encoder of the VGG19 network to obtain image features; Step 2.3: Normalize the input image features and use the pixel self-attention of the content adaptation module to adapt to obtain content features; Step 2.4: Use the channel self-attention of the style adaptation module to represent the style of the input image features, thereby obtaining the style features; Step 2.5: The content features and style features described in step 2.3 and step 2.4 are fused using the cross attention of the adaptive interaction module to obtain fused features; Step 2.6: Construct a feature decoder by cross-stacking the convolution layer and the activation layer, and output the fused features through the feature decoder.
3. The irregularly sampled fundus image prediction method based on deep learning as claimed in claim 1, characterized in that: Step 3 is implemented as follows: Step 3.1: Extract image features from the images in the training set through a feature encoder; Step 3.2: Use the content adaptation module and style adaptation module of the style transfer model to enhance the content features and style features of the features extracted from the image respectively; Step 3.3: Input the enhanced content features and style features into the adaptive interaction module to obtain enhanced fusion features; Step 3.4: The enhanced fusion features are decoded through the feature decoder.
4. A deep learning-based irregular sampling fundus image prediction system implementing the method of claim 1, characterized in that: It includes feature decomposition module, feature library module, prediction module, target decoding module and application module; The feature decomposition module is used to extract content features and style features from the sample fundus image; further, the sample fundus image input to the feature decomposition module is used to obtain content features and style features using pixel self-attention and channel self-attention, and the content features and style features are placed in the feature library module; The feature library module is used to store the content features and style features of the samples from the feature decomposition module, input the content features of the samples to the prediction module, and input the style features of the samples to the target decoding module; The prediction module is used to convert the content features into predicted content features, and then predict the content features corresponding to the fundus image; The prediction module includes a stacked convolutional recursive unit and a motion estimation unit based on ordinary differentials. The convolutional recursive unit obtains updated content features from the feature library module by capturing the dynamic features of the sequence data, and inputs them into the motion estimation unit; the motion estimation unit combines the content features with the time interval information and outputs the predicted content features after correcting the motion features, and uses the predicted content features as the input of the target decoding module; The target decoding module is used to fuse the predicted content features and the style features of the sample and output a predicted fundus image; further, the predicted content features and the style features of the sample are input into the target decoding module, the target decoding module fuses the content features and the style features of the sample using cross attention, and uses a feature decoder to decode the fused content features and the style features of the sample and output a predicted fundus image, and the predicted fundus image is used as the input of the application module; The application module is used for predictive analysis of fundus images, jointly analyzing the predicted fundus images and sample images according to disease classification and lesion segmentation, generating disease classification curves and lesion segmentation curves for irregularly sampled fundus images, and obtaining the evolution results of fundus diseases and lesions according to the trends of the disease classification curves and lesion segmentation curves.
Citation Information
Patent Citations
Fundus image classification method based on self-adaption during testing
CN116343315A
Diabetic retinopathy detection algorithm based on adversarial learning
CN117455829A
Disease prediction method based on unbalanced fundus image data
CN117576012A
Method and system for analysing images of a retina
US20210319556A1
