A deep learning-based fundus image lesion automatic detection method

Abstract

The present application relates to medical image processing technical field, specifically, it relates to a kind of fundus image pathological change automatic detection method based on deep learning, comprising: obtaining longitudinal follow-up data, extract two-dimensional and three-dimensional features and map to the same latent space, construct space-time feature trajectory library and three-dimensional deconvolution generation decoder;After collecting and pre-processing the image to be detected, feature vector is extracted, and trajectory and time parameter are matched in trajectory library, and three-dimensional lesion model is reconstructed for inference;Future lesion volume change rate is calculated based on trajectory extension curve, and diagnostic report is generated and visualized mark.The present application realizes the accurate reconstruction of pathological three-dimensional morphology by fusing two-dimensional fundus image and three-dimensional depth information, and constructing space-time feature trajectory library combined with longitudinal follow-up data, solves the problem of inaccurate two-dimensional representation in prior art;Volume change rate is calculated based on trajectory extension curve, the prediction of pathological evolution trend is realized, and the defect that prior art cannot provide trend prediction is made up.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to an automatic fundus image lesion detection method based on deep learning. BACKGROUND

[0002] Fundus lesions (such as diabetic retinopathy, age-related macular degeneration, etc.) are important causes of visual impairment. In ophthalmic clinical diagnosis, two imaging modalities are mainly relied on: one is two-dimensional fundus color photography, which has low acquisition cost, convenient operation and is suitable for large-scale screening; the other is optical coherence tomography (OCT), which can provide three-dimensional depth information of the retina, which is crucial for judging lesion volume, fluid depth and treatment response.

[0003] The current prior art usually uses convolutional neural network (CNN) to classify or segment single two-dimensional fundus image. For example, a two-dimensional image is directly input into the network to output a lesion grading label. However, such prior art has the following technical defects:

[0004] First, the lack of information dimension, the existing two-dimensional image analysis method cannot obtain the spatial distribution information of the lesion in the depth of the retina (for example, it cannot distinguish between subretinal fluid and intraretinal fluid), and these depth information is often the key to determine the treatment plan.

[0005] Second, the lack of temporal dynamic information, the existing technology is mainly for cross-sectional diagnosis, which can only judge the current state, cannot judge whether the lesion is in the active period of rapid growth or the regression period after treatment, and cannot predict the evolution trend (prognosis) of the lesion in the future based on a single image, although continuous OCT imaging can solve the above problems, but the OCT device is expensive and the popularization rate is low at the grassroots level, which limits its wide application.

[0006] Therefore, an automatic fundus image lesion detection method based on deep learning is proposed. SUMMARY

[0007] The purpose of the present application is to provide an automatic fundus image lesion detection method based on deep learning to solve the technical problems that the existing detection method based on two-dimensional fundus image lacks three-dimensional depth information and temporal evolution dynamic information of the lesion, which leads to inaccurate positioning of the spatial form of the lesion and difficulty in predicting the development of the disease course.

[0008] To solve the above technical problems, the purpose of the present application is to provide an automatic fundus image lesion detection method based on deep learning, comprising the following steps:

[0009] S1, acquire longitudinal follow-up data of a plurality of training sample sets, the longitudinal follow-up data comprising two-dimensional fundus images, corresponding three-dimensional depth information and pathological expert annotations of the same lesion collected at a plurality of discrete time points; extract feature vectors of the two-dimensional fundus images and the corresponding three-dimensional depth information by using a pre-trained deep neural network, and map the feature vectors to the same continuous latent space;

[0010] By a time series model, a continuous curve connecting the feature vectors at discrete time points is fitted in the latent space, forming a space-time feature trajectory library representing the evolution law of the lesion; a generation decoder from the discrete time points in the latent space to the three-dimensional voxel data is established, and the generation decoder adopts a three-dimensional deconvolution network structure;

[0011] S2, based on the detection instruction of the user, control the fundus camera to collect the current single-frame two-dimensional fundus image of the object to be detected; perform region of interest cropping and normalization processing on the current single-frame two-dimensional fundus image to obtain a to-be-tested input image;

[0012] S3, input the to-be-tested input image into the pre-trained feature encoder to output a current observation feature vector; perform global search and matching operation in the space-time feature trajectory library to determine the matched trajectory identifier and the time parameter representing the current evolution stage of the lesion; extract the latent space feature vector corresponding to the determined time parameter in the determined matched trajectory, and input the latent space feature vector into the generation decoder to reconstruct the corresponding inferred three-dimensional lesion model;

[0013] S4, based on the extended curve of the trajectory after the time parameter in the space-time feature trajectory library, calculate the lesion volume change rate in a future preset time period; generate a diagnosis report containing the inferred three-dimensional lesion model and the volume change rate, and control the display terminal to visually mark the inferred three-dimensional lesion model of the lesion.

[0014] As a further improvement of the technical solution, in step S3, the global search and matching operation in the space-time feature trajectory library comprises:

[0015] Calculate the weighted similarity distance measure between the current observation feature vector and the feature vectors of the sampling points on each trajectory in the space-time feature trajectory library;

[0016] When the calculated minimum similarity distance measure is less than a preset validity threshold, the corresponding trajectory identifier and time parameter are determined as the matching result.

[0017] As a further improvement of the technical solution, the step of calculating the weighted similarity distance measure is based on the Euclidean distance and the cosine similarity, and the weighted fusion calculation is performed by using a preset weight.

[0018] As a further improvement of the technical solution, in step S3, after determining the time parameter, the method further comprises performing a posture correction and reverse fine-tuning step:

[0019] By calculating the perceptual difference between the current observation feature vector and the simulated two-dimensional image generated by the three-dimensional lesion model output by the generation decoder at the time parameter, an optimization objective function is constructed, wherein the perceptual difference is based on the difference between the high-level features extracted by the pre-trained deep neural network;

[0020] The gradient descent algorithm is used to iteratively fine-tune the time parameter and the viewing angle parameter of the input image to be tested until the perceptual difference converges to below a predetermined threshold.

[0021] As a further improvement of the technical solution, the method further comprises an exception handling step:

[0022] If the calculated minimum similarity distance measure is greater than the preset validity threshold, the input image to be tested is determined to be a non-matching sample, a low confidence alarm is triggered, and the output interface is controlled to reject the execution of three-dimensional reconstruction.

[0023] As a further improvement of the technical solution, the inferred three-dimensional lesion model includes at least one of three-dimensional voxel data, a depth profile, or a three-dimensional mesh model of the lesion.

[0024] As a further improvement of the technical solution, in step S4, the step of calculating the lesion volume change rate in the future preset time period comprises:

[0025] Along the matched trajectory, a sequence of feature vectors from the current time parameter to the future time is extracted;

[0026] The generation decoder is used to calculate the three-dimensional volume value corresponding to the sequence of feature vectors;

[0027] The rate of change of the three-dimensional volume value over time is calculated, and when the rate of change exceeds a preset deterioration threshold, a high-risk label is marked in the generated diagnostic report.

[0028] As a further improvement of the technical solution, the method further comprises:

[0029] Based on large-scale clinical follow-up data, critical points of treatment intervention are predefined on the spatiotemporal feature trajectory library, the critical points being time points determined according to the judgment rule that the lesion volume or growth rate exceeds the clinically set threshold;

[0030] The time interval between the time parameter and the nearest critical point of treatment intervention is determined, and the optimal timing suggestion for treatment intervention is output.

[0031] As a further improvement of the technical solution, in step S1, the space-time feature trajectory library adopts a hierarchical structure, which includes a top layer structure and a bottom layer structure, wherein the top layer structure is used to represent the type and overall course of the lesion, and the bottom layer structure is used to represent the local morphological evolution details under a specific lesion type, and the hierarchical structures are associated through feature maps with different resolutions.

[0032] Compared with the prior art, the beneficial effects of the present application are:

[0033] 1. In the deep learning-based fundus image lesion automatic detection method, by constructing a space-time feature trajectory library and generating a decoder, two-dimensional input and three-dimensional output reverse inference are realized. The defect of the existing two-dimensional image analysis method that lacks depth information is solved, so that approximate three-dimensional lesion morphological information of OCT can be obtained in the primary medical environment with only low-cost fundus cameras, and the equipment threshold of depth diagnosis is reduced.

[0034] 2. In the deep learning-based fundus image lesion automatic detection method, by introducing a time parameter and trajectory extension calculation, the problem that the prior art cannot predict the prognosis is solved. By positioning the specific stage of the lesion on the evolution trajectory, the future volume change rate of the lesion can be quantitatively calculated, so as to identify patients with high risk and rapid progression, and provide objective basis for early intervention.

[0035] 3. In the deep learning-based fundus image lesion automatic detection method, by using weighted similarity distance measurement and posture correction fine-tuning mechanism, the matching error caused by the difference in two-dimensional image shooting angle and the complexity of lesion features is effectively overcome, and the accuracy and robustness of cross-modal inference are improved. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The flowchart of the present application. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0038] Embodiment 1

[0039] At present, the existing prior art mainly uses a convolutional neural network (CNN) to classify or segment a single two-dimensional fundus image. However, such prior art has the following technical defects:

[0040] First, the lack of information dimension, the existing two-dimensional image analysis method cannot obtain the spatial distribution information of the lesion in the depth of the retina (for example, it cannot distinguish between subretinal fluid and intraretinal fluid), and these depth information is often the key to determine the treatment plan;

[0041] Second, the lack of time dynamic information, the existing technology is mainly for cross-sectional diagnosis, which can only judge the current state, cannot judge whether the lesion is in the active period of rapid growth or the regression period after treatment, and cannot predict the evolution trend (prognosis) of the lesion in the future period of time based on a single image, although continuous OCT imaging can solve the above problems, but the OCT equipment is expensive and the popularization rate is low in the primary level, which limits its wide application;

[0042] In view of this, as Figure 1 The present application aims to provide an automatic detection method for fundus image lesions based on deep learning, which comprises the following steps:

[0043] Because the longitudinal follow-up data contains the shape and depth information of the lesion at different time points, it is the basis for constructing the evolution trajectory of the lesion; mapping two-dimensional and three-dimensional features to the same latent space can realize the fusion representation of multi-modal features; the time series model fits the continuous curve to capture the time evolution law of the lesion; the generated decoder is used to reconstruct the three-dimensional lesion from the latent space feature, thereby solving the problem that two-dimensional images cannot reflect the spatial distribution, therefore, step S1, obtaining longitudinal follow-up data of a plurality of training sample sets, the longitudinal follow-up data includes two-dimensional fundus images collected at a plurality of discrete time points, corresponding three-dimensional depth information and pathological expert annotation of the same lesion; the pre-trained deep neural network is used to extract the feature vectors of the two-dimensional fundus images and the corresponding three-dimensional depth information, and the feature vectors are mapped to the same continuous latent space; through the time series model, a continuous curve connecting the feature vectors at discrete time points is fitted in the latent space, forming a space-time feature trajectory library representing the evolution law of the lesion; a generation decoder from the discrete time points in the latent space to the three-dimensional voxel data is established, and the generation decoder adopts a three-dimensional deconvolution network structure; the specific implementation is as follows:

[0044] The training sample set is derived from 5 years of longitudinal follow-up data of ophthalmology department of 3 third-grade class-A hospitals, containing 1200 patients with fundus diseases (400 patients with diabetic retinopathy, 350 patients with glaucoma, and 450 patients with macular disease), and two-dimensional fundus images (using Topcon TRC-50DX fundus camera, resolution 2048x2048 pixels) and three-dimensional depth information (using Optovue RTVue XR Avanti optical coherence tomography, depth resolution 5 μm) of each patient are collected at baseline (0 months), 3 months, 6 months, 12 months and 24 months, respectively, and the labeling (including lesion location, type and volume range) is completed by 3 pathologists with more than 10 years of experience;

[0045] The pre-trained deep neural network adopts a combined model of ResNet50 and PointNet, wherein ResNet50 is used to extract the feature vector of the two-dimensional fundus image (the input image is resized to 512x512 pixels, and the output is a 2048-dimensional feature vector); PointNet is used to process three-dimensional depth information (three-dimensional point cloud data is sampled to 1024 points, and the output is a 1024-dimensional feature vector); the two types of feature vectors are mapped to the same latent space of 512 dimensions through a fully connected layer (the mapping process adopts batch normalization processing to avoid fusion bias caused by feature distribution difference); in order to ensure the consistency of two-dimensional and three-dimensional features in the latent space, a contrastive loss function (Contrastive Loss) is introduced in the training stage, specifically, for the paired two-dimensional image and three-dimensional depth data of the same lesion at the same time, the Euclidean distance of the feature vectors in the latent space is minimized; for unpaired data, the distance is maximized, which forces the network to learn the common pathological semantic features across modalities, thereby ensuring that only two-dimensional images can be mapped to the correct three-dimensional feature space position;

[0046] The time series model selects a long short-term memory network (LSTM), inputs the latent space feature vector sequence of the same patient at different time points, fits a continuous curve through 3 LSTM layers (256 hidden units per layer), and adopts a mean square error loss function (loss value target is lower than 0.01) in the fitting process, finally forms a spatio-temporal feature trajectory library containing 1200 trajectories, each trajectory corresponds to the lesion evolution process of a patient; it needs to be further explained that since the LSTM layer output is discrete, in order to form a “continuous curve”, the CubicSpline Interpolation algorithm is used between the discrete feature points output by the LSTM layer to generate a dense feature point sequence with a time resolution of 0.01 months, thereby constructing a smooth and continuous spatio-temporal feature trajectory library;

[0047] The generation decoder adopts a 5-layer three-dimensional deconvolution network structure: the input is a 512-dimensional latent space feature vector, which is converted into an 8x8x8x256 feature tensor through a full connection layer; the convolution kernel size of the first to fourth layers of three-dimensional deconvolution is 3x3x3, the step is 2, the activation function adopts ReLU, and a batch normalization layer is added after each layer; the convolution kernel size of the fifth layer of three-dimensional deconvolution is 3x3x3, the step is 1, the activation function adopts Sigmoid, and the output is 64x64x64 resolution three-dimensional voxel data (voxel spacing 0.1mm);

[0048] By fusing the two-dimensional and three-dimensional data of multiple longitudinal follow-ups, the constructed spatio-temporal feature trajectory library covers the complete evolution process of mainstream fundus diseases, and improves the clustering accuracy of feature vectors in the latent space; the 64x64x64 voxel data output by the three-dimensional deconvolution generation decoder reduces the error compared with the existing two-dimensional image in deducing the three-dimensional structure, and also restores the spatial distribution of the lesion in the retina tissue, providing a reliable basis for doctors to judge the infiltration depth of the lesion.

[0049] Considering that the current single-frame two-dimensional fundus image is the input basis for detection, the region of interest cropping can remove irrelevant background (such as eyelids and eyelashes) interference, and the normalization processing can unify the pixel distribution of the image, improving the accuracy of feature extraction, therefore, in step S2, based on the detection instruction of the user, the fundus camera is controlled to collect the current single-frame two-dimensional fundus image of the object to be detected; the region of interest cropping and normalization processing are performed on the current single-frame two-dimensional fundus image to obtain the input image to be detected; the specific implementation is as follows:

[0050] In response to the detection instruction issued by the user through the detection terminal (such as a computer client), the Topcon TRC-50DX fundus camera in communication connection with the terminal is controlled to collect a single-frame two-dimensional fundus image (resolution 2048x2048 pixels) of the object to be detected;

[0051] The automatic positioning method based on retinal blood vessel segmentation is adopted for the region of interest cropping, specifically: the retinal blood vessels in the image are segmented by a U-Net network to determine the coordinate range of the macular region and the optic disc region (the center coordinate of the macular region is the center of the image, and the cropping range is 512x512 pixels; the optic disc region is positioned based on the blood vessel convergence point, and the cropping range is 384x384 pixels, and the overlapping area of the two is taken as the final region of interest); the Z-score normalization algorithm is used for normalization processing, the pixel mean (μ=128) and standard deviation (σ=32) of all the regions of interest images in the training set are calculated, and the operation of x'=(x-μ) / σ is performed on each pixel point of the to-be-tested image, so that the pixel value of the normalized image is distributed in the interval [-4, 4], wherein x represents the original pixel value of each pixel point in the to-be-tested input image (i.e., the pixel gray value after image acquisition without processing, the value range is usually 0-255); x' represents the standardized pixel value obtained after Z-score normalization processing of the pixel point;

[0052] Through the above automatic cropping method based on retinal blood vessel segmentation, the positioning accuracy of the region of interest is improved, and the interference of background noise on feature extraction is effectively eliminated; after Z-score normalization, the standard deviation of the image pixel distribution under different acquisition conditions is reduced, the stability of the features extracted by the feature encoder is improved, and the matching deviation caused by differences in illumination and device parameters is avoided.

[0053] Since the feature encoder is used to extract the features of the to-be-tested image, the global search and matching with the trajectory library can determine the evolution stage of the current lesion, and the decoder generates a three-dimensional model based on the time parameter of the matching, which can realize the conversion from two-dimensional input to three-dimensional representation, therefore, in step S3, the to-be-tested input image is input into the pre-trained feature encoder to output the current observation feature vector; in the spatiotemporal feature trajectory library, a global search and matching operation is performed to determine the matched trajectory identifier and the time parameter representing the current evolution stage of the lesion; the latent space feature vector corresponding to the determined time parameter in the determined matching trajectory is extracted and input into the generated decoder to reconstruct the corresponding inferred three-dimensional lesion model; the specific implementation is as follows:

[0054] The feature encoder shares weights with the ResNet50 network used to extract two-dimensional image features in step S1 (the first 10 layers are frozen, and the last 10 layers are fine-tuned), the input is the preprocessed to-be-tested input image (512x512 pixels), and the output is a 2048-dimensional feature vector, which is mapped to a 512-dimensional current observation feature vector through the same fully connected layer as in step S1;

[0055] The global search and matching operation adopts K-Nearest Neighbor (KNN) algorithm (K=5), specifically: calculate the distance between the current observation feature vector and the 100 evenly sampled feature vectors (sample every 2.4 months) on each trajectory in the spatiotemporal feature trajectory library, select the 5 trajectories corresponding to the smallest distance as candidate trajectories, then calculate the average distance between the current feature vector and the full sequence feature vector of the candidate trajectory, and determine the trajectory with the smallest average distance as the matching trajectory, and the corresponding sampling point time parameter is the time parameter of the current evolution stage (for example, matching to the 3rd sampling point of a certain trajectory, the corresponding time parameter is 6 months);

[0056] Input the potential spatial feature vector indexed by the time parameter corresponding to the matching result (i.e. the vector value at this time point in the trajectory library) into the generation decoder. The decoder outputs 64x64x64 resolution three-dimensional voxel data, which is the inferred three-dimensional lesion model;

[0057] By sharing weights with the training stage and fine-tuning, the feature extraction efficiency is improved, while the consistency of feature distribution is ensured; K-Nearest Neighbor matching algorithm combined with candidate trajectory screening shortens the average time consumption of trajectory matching and improves the matching accuracy; the three-dimensional lesion model reconstructed by the generation decoder can clearly present the size, depth and spatial form of the lesion, and improve the consistency of doctors' judgment on the severity of the lesion (Kappa value).

[0058] Considering that the volume change rate can be predicted based on the trajectory extension curve, the lesion deterioration trend can be warned in advance, and the visualization mark and diagnosis report can provide intuitive diagnostic basis for clinicians, therefore, in step S4, based on the extension curve of the trajectory in the spatiotemporal feature trajectory library after the time parameter, the volume change rate of the lesion in the future preset time period is calculated; a diagnosis report containing the inferred three-dimensional lesion model and the volume change rate is generated, and the display terminal is controlled to visualize the inferred three-dimensional lesion model of the lesion; the specific implementation is as follows:

[0059] The future preset time period is set to 12 months, and the potential spatial feature vectors corresponding to the current time parameter (t0), t0+6 months, t0+12 months along the matching trajectory are extracted (obtained by continuous curve interpolation of the LSTM model); input these three feature vectors into the generation decoder respectively, calculate the corresponding three-dimensional volume value (count the number of voxels with value>0.5 in the voxel data, multiply by the cube of the voxel spacing (0.1mmx0.1mmx0.1mm=0.001mm 3 ), get volume values V0, V1, V2);

[0060] The calculation of the lesion volume change rate is divided into two stages, specifically: 6-month change rate r1=(V1-V0) / V0 / 6x12 (annualized), 12-month change rate r2=(V2-V0) / V0 / 12x12 (annualized);

[0061] The deterioration threshold is determined based on 500 clinical cases (diabetic retinopathy is 25% / year, glaucoma is 20% / year, and macular degeneration is 30% / year). When r1 or r2 exceeds the corresponding threshold, a high-risk label is marked in the diagnosis report;

[0062] The diagnosis report includes patient basic information, current lesion type, three-dimensional lesion model parameters (volume, maximum depth, spatial coordinate range), volume change rate, and risk level. The display terminal uses a high-definition display (resolution 3840x2160), and the visualization marker uses pseudo-color rendering (lesion area is divided into three levels according to depth, shallow (0-0.5mm) is blue, moderate (0.5-1.0mm) is yellow, and deep (>1.0mm) is red). The three-dimensional grid model and the depth profile in the x, y, and z axis directions are displayed simultaneously.

[0063] Through the above technical means, the prediction error of the volume change rate is reduced, the risk of lesion deterioration can be warned in advance, and the intervention time of high-risk patients is shortened on average. The pseudo-color visualization marker allows doctors to grasp the depth distribution of the lesion in a short time, improves the information completeness of the diagnosis report, and reduces the diagnosis time of doctors.

[0064] Embodiment 2

[0065] Compared with Embodiment 1, Embodiment 2 uses another implementation, specifically:

[0066] Implementation of step S1: The training sample set is derived from 4-year longitudinal follow-up data from 2 ophthalmology specialist hospitals, including 800 patients (300 cases of diabetic retinopathy and 500 cases of macular degeneration). The follow-up time points are 0 months, 4 months, 8 months, 16 months, and 24 months. The two-dimensional fundus image uses Canon CR-2 Digital Retinal Camera (resolution 1920x1920 pixels), and the three-dimensional depth information uses Heidelberg Spectralis OCT (depth resolution 3μm).

[0067] The pre-trained deep neural network uses Vision Transformer (ViT) to extract two-dimensional image features (input image Resize to 384x384 pixels, output 1024-dimensional feature vector), PointTransformer to extract three-dimensional depth information features (point cloud sampling to 2048 points, output 1024-dimensional feature vector), and fuse into 512-dimensional latent space feature vector through attention mechanism;

[0068] The time series model selects the Transformer model (encoder 6 layers, decoder 3 layers), uses the smooth L1 loss function when fitting continuous curves, and the spatiotemporal feature trajectory library contains 800 trajectories, each trajectory samples 80 feature vectors;

[0069] The generated decoder is a 6-layer three-dimensional deconvolution network, which inputs 512-dimensional feature vectors and outputs 128x128x128 resolution three-dimensional voxel data (voxel spacing 0.05mm);

[0070] Through the combination model of ViT and PointTransformer, the fusion efficiency of two-dimensional and three-dimensional features is improved, and the discrimination of latent space features (mutual information value) is improved; 128x128x128 resolution voxel data improves the spatial resolution of the three-dimensional model, improves the recognition rate of small lesions (diameter <0.1mm), and covers the detection needs of early mild lesions.

[0071] Implementation of step S2: The fundus camera uses Canon CR-2 Digital Retinal Camera, and the region of interest is cropped based on the macular fovea automatic positioning algorithm (the fovea coordinates are determined by Gaussian filtering and threshold segmentation, and a 512x512 pixel region is cropped); the normalization processing uses Min-Max normalization to map the pixel value to the [0,1] interval (x'=(x-min) / (max-min), where min is the minimum pixel value of the training set image 20, and max is 230);

[0072] Through the above steps, the positioning error of the macular fovea automatic positioning algorithm is reduced, the cropped region of interest focuses more on the lesion core area, and the specificity of feature extraction is improved; Min-Max normalization maps the pixel value to the [0,1] interval, avoiding the influence of extreme values on feature extraction, and improving the model's adaptability to low-quality images (signal-to-noise ratio <20dB).

[0073] Implementation of step S3: The feature encoder is the same ViT network as in step S1 (8-layer parameters after fine-tuning), which outputs a 1024-dimensional feature vector, which is mapped to a 512-dimensional current observation feature vector through an attention fusion layer; the global search and matching operation adopts a weighted matching algorithm combining the Euclidean distance and the cosine similarity (the weights are 0.4 and 0.6, respectively), and the matching threshold is set to 0.3, below which the matching trajectory and time parameter are determined; the generation decoder outputs 128x128x128 resolution three-dimensional voxel data;

[0074] Through the above steps, the weighted matching of Euclidean distance and cosine similarity reduces the cross-disease mismatching rate; the 128x128 resolution three-dimensional voxel data makes the details and textures of the lesion (such as edge irregularity) more clear, and improves the accuracy of the doctor's identification of the lesion type.

[0075] Implementation of step S4: The future preset time period is 8 months, the feature vectors at t0, t0+4 months and t0+8 months are extracted, and the volume change rate is calculated; the deterioration threshold is 28% per year for macular lesions and 22% per year for diabetic retinopathy; the diagnosis report adds a lesion evolution trend curve, and the terminal uses a 3D medical display to support rotation and scaling operations of the three-dimensional model;

[0076] Through the above steps, the 8-month preset prediction period is more suitable for the clinical short-term follow-up demand and reduces the prediction error; the interactive operation of the 3D medical display enables the doctor to observe the lesion morphology from any angle, improving the accuracy of the judgment of the position relationship between the lesion and the surrounding tissue, and providing a more accurate reference for the development of surgical plans.

[0077] Embodiment 3

[0078] Considering that the weighted similarity distance metric can comprehensively reflect the absolute difference and directional difference of the feature vector, and the effectiveness threshold is used to filter effective matching results to avoid three-dimensional model reconstruction errors caused by invalid trajectories, in step S3, the global search and matching operation in the spatiotemporal feature trajectory library includes: calculating the weighted similarity distance metric between the current observation feature vector and the feature vector of each sampling point on each trajectory in the spatiotemporal feature trajectory library; when the smallest similarity distance metric calculated is less than the preset effectiveness threshold, the corresponding trajectory identifier and time parameter are determined as the matching result; the specific implementation is as follows:

[0079] The calculation of the weighted similarity distance metric is based on the Euclidean distance ( ) and the cosine similarity ( ), and the weighted fusion formula is ; wherein, the calculation method of the Euclidean distance is ( is the an element, the first element of the sample point feature vector), the cosine similarity is calculated in the following way ; ;

[0080] The preset validity threshold is determined by cross-validation of the training set: 100 images of patients in the training set (25 cases for each disease) are selected, and the minimum similarity distance measure with the trajectory library is calculated. The 95% quantile of the distance distribution is 0.35, so the validity threshold is set to 0.35. When the calculated minimum similarity distance measure is less than 0.35, it is determined to be an effective match, and the corresponding trajectory identifier (such as TR-0012, representing the 12th trajectory) and time parameter (such as 8 months) are output. If the minimum distance is greater than or equal to 0.35, an abnormal processing step is performed.

[0081] Through the above-mentioned weighted fusion distance measure method, the discrimination of feature matching is improved, and the matching accuracy is improved compared to single distance measure. The validity threshold of 0.35 is determined by cross-validation, which can cover 95% of the valid samples, and the invalid matching rate is controlled within 5%, which not only ensures the comprehensiveness of the detection, but also avoids the waste of resources caused by invalid reconstruction, and improves the overall operation efficiency of the system.

[0082] Embodiment 4

[0083] Since the Euclidean distance can reflect the absolute numerical difference of the feature vector, and the cosine similarity can reflect the directional difference of the feature vector (i.e. the similarity of the feature pattern), the weighted fusion of the two can take into account both dimensions of difference and improve the accuracy of matching. Therefore, the step of calculating the weighted similarity distance measure is based on the Euclidean distance and the cosine similarity, and the weighted fusion is calculated by presetting the weight, which is specifically implemented as follows:

[0084] The preset weight is determined by comparative experiments: 50 images of different diseases and different evolution stages of the fundus are selected, and single Euclidean distance, single cosine similarity, and weighted fusion with different weight combinations are used for matching test, and the matching accuracy (the evolution stage determined by the pathologist is taken as the gold standard) is counted.

[0085] The experimental results show that when the Euclidean distance weight is 0.4 and the cosine similarity weight is 0.6, the matching accuracy is the highest, so this weight combination is determined. The specific calculation process is as follows:

[0086] Taking the current observation feature vector X (512 dimensions) and a sample point feature vector Y (512 dimensions) as an example, first calculate the Euclidean distance , get = 0.25; then calculate the cosine similarity , get =0.92; substituting into the fusion formula The distance value is less than the validity threshold 0.35, and it is determined to be matched.

[0087] Through the above steps, the weight combination of 0.4 and 0.6 is experimentally verified to improve the matching accuracy compared to single Euclidean distance and single cosine similarity; at the same time, the weight combination is adaptive to different diseases and different evolution stages of the lesions, improves the stability of cross-disease matching (i.e. reduces the standard deviation), and ensures the consistency of different types of lesion detection.

[0088] Embodiment 5

[0089] Since the collection angle of the image to be detected may deviate from the training sample, the time parameter of the initial matching is not accurate enough, and there is a deviation between the three-dimensional model and the actual lesion; through the perception difference calculation and iterative fine-tuning, the deviation of the angle and the time parameter can be corrected, and the reconstruction accuracy of the three-dimensional model is improved, therefore, in the step S3, after determining the time parameter, the method further includes performing a posture correction and reverse fine-tuning step: by calculating the perception difference between the current observation feature vector and the simulated two-dimensional image generated by the three-dimensional lesion model output by the generated decoder at the time parameter, an optimization objective function is constructed, wherein the perception difference is based on the difference between the high-level features extracted by the pre-trained deep neural network; the gradient descent algorithm is used to iteratively fine-tune the time parameter and the angle parameter of the input image to be detected until the perception difference converges to below a predetermined threshold, specifically:

[0090] The pre-trained deep neural network selects VGG16 (pre-trained on ImageNet dataset, freeze the first 15 layers, fine-tune the last 3 layers), which is used to extract high-level features (select the 4096-dimensional feature vector output by the fc6 layer as the perception feature), further, the high-level features describe the semantic content and abstract concepts of the image, such as the shape of the object (such as the roundness and irregularity of the lesion), the composition of the object (such as the area of exudate and the boundary of hemorrhagic plaque), and complex texture patterns (such as the structure arrangement of retinal layers);

[0091] The three-dimensional voxel data output by the generated decoder is simulated by a ray casting algorithm to simulate the imaging angle of the fundus camera (the initial angle parameter is the average angle of the training sample: pitch angle 0°, yaw angle 0°, roll angle 0°), and a simulated two-dimensional image with the same resolution as the input image to be detected is generated;

[0092] Calculate the mean square error (MSE) between the current observation feature vector (extracted by VGG16 fc6 layer) and the fc6 layer feature corresponding to the simulated two-dimensional image, i.e. the perception difference ​For real image features, For simulated image features, N = 4096

[0093] Optimization objective function construction: Wherein, is a regularization term (L2 regularization is adopted to avoid parameter overfitting, and the regularization coefficient is 0.001);

[0094] The gradient descent algorithm selects the Adam optimizer, the learning rate is set to 0.001, and the upper limit of the iteration number is 100 times; the predetermined convergence threshold is determined to be 0.005 (when the change amount of continuous 5 iterations is less than 0.0001, it is determined to be converged);

[0095] The iteration fine-tuning process is as follows: calculate the gradient of the objective function to the time parameter (the initial value is the time t0 matched) and the view angle parameter (pitch angle θ, yaw angle φ, roll angle ψ) each time, and update the parameters:

[0096] , , , ; until converge to below 0.005, output the fine-tuned time parameter and view angle parameter, and call the generation decoder based on the time parameter to obtain the corrected inferred three-dimensional lesion model, wherein, represents the time parameter before updating (the initial value is the time parameter representing the current evolution stage of the lesion matched by the global search, for example, the initial time = 6 months), represents the optimized time parameter obtained after one iteration update (used to more accurately locate the evolution position of the lesion on the spatiotemporal feature track, and provide more accurate input for three-dimensional lesion model reconstruction); represents the pitch angle parameter before updating (describing the up-down view angle deviation when the image is collected, the initial value is the average pitch angle 0° of the training sample, a positive value represents upward deviation, and a negative value represents downward deviation); represents the optimized pitch angle parameter obtained after one iteration update (correcting the up-down view angle deviation, so that the simulated two-dimensional image and the real image to be tested have consistent view angles); represents the yaw angle parameter before updating (describing the left-right view angle deviation when the image is collected, the initial value is the average yaw angle 0° of the training sample, a positive value represents right deviation, and a negative value represents left deviation); represents the optimized yaw angle parameter obtained after one iteration update (correcting the left-right view angle deviation, improving the view angle matching degree of the simulated image and the real image); Rolling angle parameter before update (describes the rotation angle deviation of the image acquisition under test, the initial value is the average rolling angle 0° of the training sample, a positive value indicates clockwise rotation, and a negative value indicates counterclockwise rotation) ; Optimized rolling angle parameter after one iteration update (corrects the rotation direction angle deviation to ensure the consistency of the lesion shape direction) ; Learning rate of gradient descent algorithm (controls the step size of each parameter update, the value in this embodiment is 0.001, to avoid parameter oscillation and non-convergence caused by too large step size, or too low iteration efficiency caused by too small step size) ;

[0097] Total optimization objective function; Total optimization objective function Partial derivative (i.e. gradient) of time parameter Reflects the influence degree and direction of the small change of time parameter on the total objective function value, and the gradient is positive, indicating that it is too large and needs to be reduced ; the gradient is negative, indicating that it is too small and needs to be increased ; Total optimization objective function Partial derivative (gradient) of pitch angle Reflects the influence of the small change of pitch angle on the total objective function value, and is used to guide the adjustment direction of (gradient positive indicates it is too large, and gradient negative indicates it is too small) ; Total optimization objective function Partial derivative (gradient) of yaw angle Guides the adjustment direction of yaw angle to ensure that the left and right viewing angles of the simulated image are consistent with the real image; Total optimization objective function Partial derivative (gradient) of rolling angle Guides the adjustment direction of rolling angle to eliminate the shape deviation caused by image rotation;

[0098] Through the above steps, the high-level perception features extracted by VGG16 can effectively capture the semantic information of the image, and the calculation of the perception difference improves the correction accuracy of the viewing angle deviation; after iterative fine-tuning, the shape similarity (Dice coefficient) between the three-dimensional model and the actual lesion is improved, the estimation error of the time parameter is reduced, and the accuracy of the lesion evolution stage judgment is improved., lays a more accurate foundation for subsequent volume change rate prediction.

[0099] Embodiment 6

[0100] In view of the fact that forced three-dimensional reconstruction may lead to unreliable results when the lesion characteristics of the image to be tested are significantly different from all trajectories in the trajectory library (such as rare lesions and poor image quality), abnormal processing can avoid misleading clinical diagnosis and improve the robustness of the system. Therefore, the method further comprises an abnormal processing step: if the calculated minimum similarity distance measure is greater than the preset effectiveness threshold, the input image to be tested is determined to be a non-matching sample, a low-confidence alarm is triggered, and the output interface is controlled to refuse to perform three-dimensional reconstruction; the specific implementation is as follows:

[0101] The preset effectiveness threshold is 0.35 (consistent with Embodiment 3), and when the calculated minimum similarity distance measure is greater than 0.35, the input image to be tested is determined to be a non-matching sample;

[0102] The triggering mode of the low-confidence alarm is: a red prompt box is popped up on the display terminal, displaying "the matching degree of the current lesion characteristics with the known evolution trajectory is low, the confidence is insufficient, and three-dimensional reconstruction cannot be performed"; at the same time, an abnormal signal is sent to the hospital information system (HIS), and the signal contains the patient ID, the detection time, and the matching degree value .

[0103] The specific operation of the output interface refusing to perform three-dimensional reconstruction is: the calling process of the decoder is terminated, the currently occupied computing resources (such as GPU memory) are released, the detection result in the diagnosis report is only recorded as "non-matching sample", and the three-dimensional lesion model and the volume change rate related content are not included; at the same time, the user is prompted to "recollect the image or further diagnose in combination with other examination methods";

[0104] Through the above steps, the recognition rate of the system for non-matching samples is improved, the doctor is prevented from being misled by the wrong three-dimensional model, and the reliability of the diagnosis is improved; the alarm prompt is linked with the HIS system, which facilitates the medical staff to follow up and handle in time, improves the success rate of re-collecting the image, releases invalid computing resources, and improves the concurrent processing capability of the system.

[0105] Embodiment 7

[0106] In view of the fact that different forms of three-dimensional lesion models can meet different clinical needs (such as voxel data for quantitative analysis, cross-sectional graphs for intuitive observation, and mesh models for 3D display), and the presentation form of the diagnosis basis is rich, therefore, the inferred three-dimensional lesion model comprises at least one of three-dimensional voxel data, a deep cross-sectional graph, or a three-dimensional mesh model of the lesion, and the specific implementation is as follows:

[0107] The three-dimensional voxel data is stored in NIfTI format, with a resolution of 64x64x64 (voxel spacing 0.1 mm), a voxel value range of [0, 1], and voxels with a value greater than 0.5 being determined as a lesion area for subsequent volume calculation and quantitative analysis (such as maximum depth, average depth, and spatial distribution range of the lesion);

[0108] The generation of the depth profile uses the following steps:

[0109] Slices are taken along the x-axis (horizontal direction), y-axis (vertical direction), and z-axis (depth direction) of the three-dimensional voxel data, with one slice taken every 10 layers for the x-axis and y-axis and one slice taken every 5 layers for the z-axis, resulting in a total of 18 profile graphs (6 for the x-axis, 6 for the y-axis, and 6 for the z-axis). Each profile graph is rendered using pseudo-color (the deeper the depth, the closer the color to red), with slice position coordinates (such as x=32mm, y=32mm) labeled.

[0110] The generation of the three-dimensional mesh model uses the following steps:

[0111] The Marching Cubes algorithm is used to convert the voxel data into a triangular mesh model (with approximately 5000-8000 triangular facets) by setting the iso value to 0.5 (consistent with the lesion determination threshold of the voxel data). The rendering of the mesh model uses the Phong lighting model, with the lesion surface set to be semi-transparent (60% transparency) to facilitate the observation of the internal structure. Rotation (around the x, y, and z axes at any angle), scaling (scale range 0.5-2.0), and translation operations are supported on the display terminal.

[0112] Through the above steps, the voxel data in NIfTI format can be directly imported into the clinical commonly used image analysis software (such as ITK-SNAP), supporting doctors to perform custom quantitative analysis and improving the analysis efficiency. The multi-directional depth profile makes the layered structure of the lesion more clear, improving the accuracy of the doctor's judgment of the depth of the lesion. The interactive operation of the three-dimensional mesh model allows doctors to visually observe the spatial morphology of the lesion, improving the evaluation consistency (Kappa value) of complex shape lesions (such as irregular exudates).

[0113] Example 8

[0114] Considering that the sequence of feature vectors is the basis for predicting the future volume, the change rate can intuitively reflect the evolution trend of the lesion, and the high-risk label can quickly remind the clinician to pay attention to the risk of deterioration, thereby improving the diagnosis efficiency. Therefore, in the step S4, the step of calculating the lesion volume change rate in the future preset time period comprises: extracting the sequence of feature vectors from the current time parameter to the future time along the matched trajectory; calculating the three-dimensional volume value corresponding to the sequence of feature vectors by using the generation decoder; calculating the change rate of the three-dimensional volume value with time, and marking a high-risk label in the generated diagnosis report when the change rate exceeds a preset deterioration threshold; and the specific implementation is as follows:

[0115] The future preset time period is 12 months, and the current time parameter is t0 (such as 6 months). Along the matched trajectory, the latent space feature vectors (X0, X1, X2) corresponding to the three time points of t0, t0+6 months and t0+12 months are obtained through the continuous curve interpolation of the LSTM model. Calculation of three-dimensional volume value: input X0, X1 and X2 into the generation decoder respectively to obtain corresponding three-dimensional voxel data V0, V1 and V2, count the number of voxels N0, N1 and N2 with a value greater than 0.5 in each voxel data, and the volume value = voxel number x voxel spacing 3 (0.1mm x 0.1mm x 0.1mm = 0.001mm 3 ), assuming that the calculated V0 = 0.08mm 3 , V1 = 0.10mm 3 , V2 = 0.12mm 3 ;

[0116] Calculation of change rate: 6-month annual change rate ; 12-month change rate ;

[0117] The preset deterioration threshold is determined based on clinical data statistics: 25% / year for diabetic retinopathy, 20% / year for glaucoma, and 30% / year for macular degeneration; if the current lesion is diabetic retinopathy and both r1 and r2 exceed 25%, a high-risk label "high risk of lesion deterioration, it is recommended to review and evaluate the treatment plan within 6 months" is marked in the diagnosis report.

[0118] Through the above steps, the sequence of feature vectors based on trajectory interpolation improves the time continuity of volume prediction and reduces the prediction error of volume change rate within 12 months. The marking of the high-risk label enables doctors to quickly identify high-risk patients, thereby improving the intervention rate of high-risk cases and effectively reducing the probability of lesion progression to a serious stage. The lesion-specific deterioration threshold improves the accuracy of risk judgment and avoids misjudgment caused by a unified threshold.

[0119] Embodiment 9

[0120] The predefined critical point of the therapeutic intervention can associate the lesion evolution with the clinical treatment decision, the time interval calculation can provide a clear reference for the intervention opportunity, help the doctor to make a personalized treatment plan, and improve the clinical practical value of the method. Therefore, the method further comprises: based on large-scale clinical follow-up data, the critical point of the therapeutic intervention is predefined on the space-time feature trajectory library, and the critical point is a time point determined according to the judgment rule that the lesion volume or growth rate exceeds the clinically set threshold; the time interval between the time parameter and the nearest therapeutic intervention critical point is determined, and the best opportunity for therapeutic intervention is output; and the specific implementation is:

[0121] The large-scale clinical follow-up data includes 2000 treated patients with fundus diseases (800 patients with diabetic retinopathy, 600 patients with glaucoma, and 600 patients with macular lesions), and records the treatment opportunity, treatment effect (such as lesion volume change, vision improvement) of the patient.

[0122] The critical point of the therapeutic intervention is a judgment rule set for different diseases. The rule for diabetic retinopathy is "lesion volume ≥0.15mm 3 or continuous 6-month growth rate ≥30% / year", the rule for glaucoma is "lesion volume ≥0.12mm 3 or continuous 6-month growth rate ≥25% / year", and the rule for macular lesions is "lesion volume ≥0.20mm 3 or continuous 6-month growth rate ≥35% / year"; based on these rules, the corresponding critical point time point (such as the volume of a certain diabetic retinopathy trajectory reaching 0.15mm 3 at 18 months) is marked on each trajectory of the space-time feature trajectory library.

[0123] The calculation step of the time interval is:

[0124] Suppose the current time parameter t0=12 months, the nearest therapeutic intervention critical point =18 months, then the time interval Δt= -t0=6 months. The best opportunity for therapeutic intervention is determined according to the time interval, Δt<3 months, suggesting "immediate therapeutic intervention is required"; 3 months≤Δt≤6 months, suggesting "review within 3 months and evaluate whether to start treatment"; Δt>6 months, suggesting "no urgent treatment is required at present, review every 6 months"; in this embodiment, Δt=6 months, so the output suggestion is "review within 3 months and evaluate whether to start treatment", and the critical point time point and the current time interval are marked in the diagnosis report.

[0125] ​Through the above steps, based on the critical point definition of 2000 clinical data, the coincidence rate of treatment opportunity suggestion and the consistency with the judgment of experienced ophthalmologists (Kappa value) are improved; the quantification of time interval provides a clear follow-up cycle reference for doctors, avoiding excessive follow-up or insufficient follow-up, and improving the follow-up compliance of patients; personalized treatment suggestions improve the treatment efficiency and reduce the waste of medical resources and the burden on patients caused by ineffective treatment.

[0126] Embodiment 10

[0127] Considering that the hierarchical structure can realize the search logic of "rough classification first and fine matching later", the top layer structure quickly filters the lesion types, and the bottom layer structure accurately matches the local evolution details, thereby improving the efficiency and accuracy of search matching. Therefore, in the step S1, the spatiotemporal feature trajectory library adopts a hierarchical structure, which includes a top layer structure and a bottom layer structure. The top layer structure is used to represent the type and overall course of the lesion, and the bottom layer structure is used to represent the local morphological evolution details under a specific lesion type. The hierarchical structures are associated through feature maps with different resolutions. Specifically, the hierarchical structure is implemented as follows:

[0128] The top layer structure is constructed by dividing all training samples into 3 categories (diabetic retinopathy, glaucoma, and maculopathy) according to the lesion type, with each category corresponding to one overall course trajectory. A low-resolution feature map (16x16 pixels) is used to represent the overall course trajectory, which is obtained by global average pooling of the latent space feature vector (512-dimensional feature vector converted to a 256-dimensional feature map of 16x16).

[0129] The bottom layer structure is constructed by subdividing each lesion type into subcategories (such as 3 subcategories of microangioma, hemorrhagic spots, and exudates under diabetic retinopathy) according to local morphological features. Each subcategory corresponds to multiple local morphological evolution trajectories, which are represented by high-resolution feature maps (64x64 pixels). The feature maps are obtained by transpose convolution upsampling of the latent space feature vector (512-dimensional feature vector converted to a 512-dimensional feature map of 64x64). Each trajectory of the bottom layer structure contains detailed evolution features of local morphology (such as size, number, and distribution changes of microangioma).

[0130] The association method of the hierarchical structure is to associate feature maps with different resolutions through a feature pyramid network (FPN). The top layer low-resolution feature map is 4 times upsampled and added element-wise to the bottom layer high-resolution feature map for fusion. The fused feature map is used for global search matching. During the search, the lesion type is first matched in the top layer structure (such as matching to diabetic retinopathy), and then the local morphological evolution trajectory is matched in the bottom layer structure corresponding to the type, thereby avoiding invalid search across types.

[0131] Through the above technical means, the hierarchical structure reduces the search range of the track library to the bottom layer track of a single lesion type, shortens the matching time consumption, improves the efficiency, improves the feature fusion of the top layer and the bottom layer, and improves the matching accuracy; The sub-class subdivision bottom layer structure can accurately capture the local morphological evolution details, improve the detection rate of early slight lesions (such as microhemangioma), realize early detection and early intervention of lesions, and improve the prognosis effect of patients.

[0132] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above embodiments, and the above embodiments and descriptions in the specification are only preferred examples of the present application and are not intended to limit the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1.A deep learning-based automatic detection method for lesions in fundus images, characterized by: The method comprises the following steps: S1, obtaining longitudinal follow-up data of a plurality of training sample sets, the longitudinal follow-up data comprising two-dimensional fundus images, corresponding three-dimensional depth information and pathological expert annotations of the same lesion collected at a plurality of discrete time points; using a pre-trained deep neural network to extract feature vectors of the two-dimensional fundus images and the corresponding three-dimensional depth information, and mapping the feature vectors into the same continuous latent space; Through a time series model, a continuous curve connecting the feature vectors at the discrete time points is fitted in the latent space to form a spatio-temporal feature trajectory library representing the evolution law of the lesion; A generation decoder from the discrete time points in the latent space to three-dimensional voxel data is established, and the generation decoder adopts a three-dimensional deconvolution network structure; S2, based on a detection instruction of a user, controlling a fundus camera to collect a current single-frame two-dimensional fundus image of a to-be-detected object; performing region of interest cropping and normalization processing on the current single-frame two-dimensional fundus image to obtain a to-be-detected input image; S3, inputting the to-be-detected input image into the pre-trained feature encoder to output a current observation feature vector; in the spatio-temporal feature trajectory library, performing global search and matching operations to determine a matched trajectory identifier and a time parameter representing a current evolution stage of the lesion; extracting a latent space feature vector corresponding to the determined time parameter in the determined matched trajectory, inputting the latent space feature vector into the generation decoder, and reconstructing a corresponding inferred three-dimensional lesion model; S4, based on an extended curve of the trajectory in the spatio-temporal feature trajectory library after the time parameter, calculating a lesion volume change rate in a future preset time period; generating a diagnosis report containing the inferred three-dimensional lesion model and the volume change rate, and controlling a display terminal to visually mark the inferred three-dimensional lesion model of the lesion. In the step S3, the global search and matching operations in the spatio-temporal feature trajectory library comprise: 2.The deep learning-based fundus image lesion automatic detection method of claim 1, wherein: calculating a weighted similarity distance measure between the current observation feature vector and the feature vectors of the sampling points on each trajectory in the spatio-temporal feature trajectory library; when the calculated minimum similarity distance measure is less than a preset validity threshold, determining the corresponding trajectory identifier and the time parameter as the matching result. The step of calculating the weighted similarity distance measure is based on the Euclidean distance and the cosine similarity, and the weighted similarity distance measure is calculated by weighting and fusing the Euclidean distance and the cosine similarity through a preset weight. 3.The deep learning-based fundus image lesion automatic detection method of claim 2, wherein, In the step S3, after the time parameter is determined, the method further comprises a pose correction and reverse fine-tuning step: 4.The deep learning-based fundus image lesion automatic detection method according to claim 2, characterized in that, by calculating a perception difference between the current observation feature vector and a simulated two-dimensional image generated by the three-dimensional lesion model output by the generation decoder at the time parameter, an optimization objective function is constructed, wherein the perception difference is based on the difference between high-level features extracted by the pre-trained deep neural network; using a gradient descent algorithm to iteratively fine-tune the time parameter and the viewing angle parameter of the to-be-detected input image until the perception difference converges to below a predetermined threshold. The method further comprises an abnormality processing step: 5.The deep learning-based fundus image lesion automatic detection method according to claim 2, characterized in that, if the calculated minimum similarity distance measure is greater than the preset validity threshold, it is determined that the to-be-detected input image is a non-matching sample, a low-confidence alarm is triggered, and an output interface is controlled to refuse to perform three-dimensional reconstruction. ​ 6.The deep learning-based fundus image lesion automatic detection method according to claim 1, characterized in that, The inferred three-dimensional lesion model comprises at least one of three-dimensional voxel data, a depth profile or a three-dimensional mesh model of the lesion. 7.The deep learning-based fundus image lesion automatic detection method according to claim 1, characterized in that, In the step S4, the step of calculating the lesion volume change rate in the future preset time period comprises: extracting a sequence of feature vectors from the current time parameter to the future time along the matched trajectory; calculating a three-dimensional volume value corresponding to the sequence of feature vectors using the generative decoder; calculating the change rate of the three-dimensional volume value over time, and marking a high-risk label in the generated diagnosis report when the change rate exceeds a preset deterioration threshold. 8.The deep learning-based fundus image lesion automatic detection method of claim 1, wherein, The method further comprises: based on large-scale clinical follow-up data, predefining critical points of treatment intervention on the spatiotemporal feature trajectory library, the critical points being time points determined according to a judgment rule that the lesion volume or growth rate exceeds a clinically set threshold; determining the time interval between the time parameter and the nearest critical point of treatment intervention, and outputting a suggestion of the optimal timing of treatment intervention. 9.The deep learning-based fundus image lesion automatic detection method of claim 1, wherein, In the step S1, the spatiotemporal feature trajectory library adopts a hierarchical structure, and the hierarchical structure comprises a top layer structure and a bottom layer structure, wherein the top layer structure is used to represent the type of lesion and the overall disease course, and the bottom layer structure is used to represent the local morphological evolution details under the specific lesion type, and the hierarchical structures are associated through feature maps of different resolutions.

Images