Retina thickness prediction method and system based on multi-modal image

Abstract

The invention discloses a retina thickness prediction method and system based on a multi-modal image, and the method and system achieve the effective estimation of the retina thickness under a low-cost condition through feature alignment and fusion modeling, and improve the basic screening and follow-up visit capability. According to the invention, through fusion of the multi-mode retina image data, the structure and function information of the optic nerve can be more comprehensively obtained, and the prediction accuracy of the thickness of the retina nerve fiber layer (RNFL) is improved. The OCT high-resolution hierarchical structure and the wide-view texture features of the eye fundus image are combined, so that anatomy and pathological states of optic nerves can be truly restored. The method can be used as an auxiliary method for early screening and early warning of optic neurodegenerative diseases such as glaucoma, and provides support for low-cost and high-efficiency primary screening and clinical auxiliary decision making.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing and artificial intelligence-assisted diagnosis technology, specifically involving a method and system for predicting retinal nerve fiber layer thickness based on multimodal medical image fusion, which is applicable to early screening and auxiliary assessment of fundus diseases such as glaucoma. Background Technology

[0002] Retinal nerve fiber layer (RNFL) thickness is an important biomarker for neurodegenerative diseases such as glaucoma, and early thinning of the RNFL is often accompanied by visual impairment. The current gold standard for detection is optical coherence tomography (OCT), but its expensive equipment and specialized operation limit its widespread use in primary healthcare institutions.

[0003] In contrast, fundus imaging has advantages such as low cost and easy access, and has been widely used in fields such as diabetic retinopathy in recent years. However, traditional fundus imaging is difficult to directly measure RNFL thickness and can only assess morphological features such as blood vessels or hemorrhages, which is insufficient to replace OCT.

[0004] Existing research has shown that there may be a correlation between grayscale and texture distribution in Fundus images and RNFL thickness. Some deep learning methods have attempted to predict OCT indicators such as macular thickness from Fundus images, but most of them focus on a single modality and have failed to effectively solve problems such as feature dimension differences and registration errors between multiple modalities.

[0005] Therefore, there is an urgent need for a low-cost, high-precision cross-modal fusion prediction method to achieve OCT-level structural index estimation and serve primary-level visual function health screening. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method and system for predicting retinal thickness based on multimodal images. Through feature alignment and fusion modeling, it achieves effective estimation of retinal thickness under low-cost conditions, thereby improving the screening and follow-up capabilities at the grassroots level.

[0007] To achieve the above objectives, this application provides the following technical solution:

[0008] In a first aspect, embodiments of this application provide a method for predicting retinal thickness based on multimodal images, comprising the following steps:

[0009] S1. Acquire multimodal image data: Acquire paired fundus images, optical coherence tomography (OCT) images, and near-infrared reference images, all of which cover the same optic disc region.

[0010] The data collected by S1 covers normal eyes and types of retinal and optic nerve lesions, ensuring disease diversity and clinical applicability during model training and evaluation. Inclusion criteria: no history of glaucoma, high myopia with intraocular pressure <21 mmHg, and normal optic disc morphology.

[0011] S2. Construct a thickness map based on structural annotations.

[0012] The OCT image was segmented using a U-Net neural network, and the upper and lower boundaries of the RNFL layer were automatically extracted. An interpolation method was used to smooth the boundary region and a thickness map was constructed.

[0013] S3. Region of Interest (ROI) Extraction and Gray-Scale Feature Construction.

[0014] Multiple regions of interest (ROIs) were selected around the optic disc in the Fundus image, avoiding the main blood vessel region; local statistical features were extracted from the R / G / B channel images respectively, and R, G, and B grayscale images were generated.

[0015] S4. Construct cross-modal joint feature vectors.

[0016] S41. Based on the spatial correspondence between the near-infrared reference image and the Fundus image, complete the geometric registration between the two to confirm the precise location of the OCT scan area in the Fundus image.

[0017] S42. The thickness map is precisely mapped to the corresponding OCT scan area in the Fundus image through spatial interpolation, thereby achieving scale alignment and pixel-level correspondence between images.

[0018] S43. Simultaneously extract the color features of the Fundus image and the thickness information of the corresponding region in each ROI region obtained in S3, and construct a joint feature vector.

[0019] S5. Construction and prediction of regression prediction model:

[0020] A multimodal feature fusion and retinal nerve fiber layer thickness prediction model based on backpropagation neural network was constructed, and the model was used for regression prediction. The model adopts a three-layer design, including a feature-level fusion network, a semantic-level fusion network, and an output layer. The input of the model is the cross-modal joint feature vector of each ROI, and the output is the predicted RNFL thickness value of that ROI.

[0021] S51, Feature-Level Fusion Network: This network uses the constructed joint feature vector as input to achieve pairwise fusion of features from different modalities. This fusion is accomplished through a fully connected network structure with 8 neurons and ReLU activation function.

[0022] S52. The three fusion feature results are weighted and averaged, with the weights dynamically adjusted according to the channel signal-to-noise ratio to obtain the first comprehensive feature.

[0023] S53, Semantic-level Fusion Network: Receives the first comprehensive features and further performs high-level semantic expression and feature abstraction. This network consists of a fully connected hidden layer with four neurons, using ReLU activation function to extract high-order semantic representations of the fused features as input for subsequent prediction outputs.

[0024] S54, Output Layer: Receives the semantic feature output of the semantic-level fusion network and completes the numerical prediction of the RNFL thickness of the ROI region through a linear output neuron activated by an activation function.

[0025] S55. Input the predicted RNFL thickness values ​​of all ROI regions into the Softmax layer to generate an RNFL thickness probability distribution map.

[0026] S56. Extract clinical indicators based on the RNFL thickness probability distribution map.

[0027] S6. Training and Optimization.

[0028] The training process combines a main loss function and an auxiliary loss function to ensure structural consistency and prediction accuracy. The main loss function uses mean squared error (MSE), and the auxiliary loss function uses contrastive loss. An image enhancement strategy is employed to simulate real-world acquisition conditions: Gaussian blur and brightness perturbation are applied to Fundus images, and noise interference is added to OCT images to enhance model robustness. Registration keypoints remain unchanged during the enhancement process.

[0029] S7. Model Output and Evaluation.

[0030] Secondly, this application provides a retinal nerve fiber layer thickness prediction system based on multimodal imaging, characterized in that it includes the following modules:

[0031] Multimodal image data acquisition module: Acquires paired fundus images, optical coherence tomography (OCT) images, and near-infrared reference images, all of which cover the same optic disc region.

[0032] The thickness map construction module uses a U-Net neural network to segment the OCT image, automatically extracts the upper and lower boundaries of the RNFL layer, smooths the boundary regions using interpolation methods, and constructs a thickness map.

[0033] ROI extraction module: Select multiple regions of interest (ROIs) around the optic disc of the Fundus image, avoiding the main blood vessel region; extract local statistical features on the R / G / B channel images respectively, and generate R, G, and B grayscale images.

[0034] Joint Feature Vector Construction Module: Constructs joint feature vectors for thickness prediction through spatial registration and scale alignment of multimodal images.

[0035] The regression prediction module performs regression predictions using a multimodal feature fusion and retinal nerve fiber layer thickness prediction model based on a backpropagation neural network. The model employs a three-layer design, including a feature-level fusion network, a semantic-level fusion network, and an output layer. The input to the model is the joint feature vector of each ROI region, and the output is the predicted RNFL thickness value for that ROI.

[0036] The training and optimization module trains the prediction model of the regression prediction module. The training process uses mean squared error (MSE) and contrast loss to ensure structural consistency and prediction accuracy. An image enhancement strategy is employed to simulate real-world acquisition conditions: Gaussian blur and brightness perturbation are applied to Fundus images, and noise interference is added to OCT images to enhance model robustness. Registration keypoints remain unchanged during the enhancement process.

[0037] Through the collaborative design of the above modules, this invention not only improves the model's adaptability and prediction accuracy among multimodal images, but also has good system scalability, making it easy to deploy and apply in grassroots screening terminals or cloud-based auxiliary platforms.

[0038] The beneficial effects of this invention are as follows:

[0039] This invention, through the fusion of multimodal retinal image data, can more comprehensively acquire information on the structure and function of the optic nerve, improving the accuracy of predicting the thickness of the retinal nerve fiber layer (RNFL). Combining high-resolution OCT hierarchical structures with the wide-field texture features of fundus images helps to realistically reconstruct the anatomical and pathological state of the optic nerve. It can serve as an auxiliary method for early screening and warning of optic nerve degenerative diseases such as glaucoma, providing support for low-cost, high-efficiency primary care screening and clinical decision support. Attached Figure Description

[0040] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.

[0041] Figure 2 , Figure 3 The results of predicting the thickness of the retinal nerve fiber layer (RNFL) on a test set are shown based on the BP neural network model designed according to the embodiments of the present invention. Detailed Implementation

[0042] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments. However, the present invention is not limited to the following embodiments. Without departing from the essential spirit of the present invention, those skilled in the art can make various modifications and variations, all of which fall within the protection scope of the present invention.

[0043] A method for predicting retinal thickness based on multimodal images includes the following steps:

[0044] S1. Acquire multimodal image data: Acquire paired fundus images, optical coherence tomography (OCT) images, and near-infrared reference images (reference images used for scanning and positioning in the fundus OCT device, which are acquired by the SLO or IR camera in the OCT system), all of which cover the same optic disc area.

[0045] S2. Construct a thickness map based on structural annotation: The OCT image is segmented using a U-Net neural network, and the upper and lower boundaries of the RNFL layer are automatically extracted; the boundary region is smoothed using an interpolation method, and a thickness map is constructed.

[0046] S3. Region of Interest (ROI) Extraction and Grayscale Feature Construction: Select multiple regions of interest (ROI) around the optic disc in the Fundus image, avoiding the main blood vessel region; extract local statistical features on the R / G / B channel images respectively, and generate R, G, and B grayscale images.

[0047] S4. Construct a joint feature vector across modalities: Construct a joint feature vector for subsequent thickness prediction tasks through spatial registration and scale alignment of multimodal images.

[0048] S5. Construction and prediction of regression prediction model:

[0049] A multimodal feature fusion and retinal nerve fiber layer (RNFL) thickness prediction model based on backpropagation neural network (BP neural network) was constructed and used for regression prediction. The model adopts a three-layer design, including a feature-level fusion network, a semantic-level fusion network, and an output layer. The input of the model is the mean gray value of the RGB channels of the Fundus image within each ROI region and the mean local thickness in the corresponding OCT thickness map. The output is the predicted RNFL thickness value of that ROI. The first layer, the feature-level fusion network, consists of a fully connected layer with 8 neurons (ReLU activation) to fuse RGB features and thickness features. The second layer, the semantic-level fusion network, contains a hidden layer with 4 neurons and the activation function is ReLU, used to extract the high-order semantic representation of the fused features. The third layer is the output layer, consisting of a single output neuron with the activation function linear, used for regression prediction of the RNFL thickness value of the ROI.

[0050] S6. Training and Optimization: The training process uses the main loss function (mean squared error, MSE) and the auxiliary loss function (contrast loss) to ensure structural consistency and prediction accuracy. Image enhancement strategies are employed to simulate real acquisition conditions: Gaussian blur and brightness perturbation are applied to Fundus images, and noise interference is added to OCT images to enhance model robustness. Registration keypoints remain unchanged during the enhancement process.

[0051] S7. Model Output and Evaluation:

[0052] Input the Fundus image of the new sample to be predicted into the trained model, and output the RNFL thickness prediction value for each ROI; performance evaluation includes thickness prediction error (mean difference from manually interpolated thickness) and registration error (sum of squared differences in key point coordinates).

[0053] As a preferred embodiment, the data collected in S1 covers normal eyes and types of retinal and optic nerve lesions, ensuring disease diversity and clinical applicability during model training and evaluation. (Inclusion criteria: no history of glaucoma, normal intraocular pressure (<21 mmHg) in high myopia, and normal optic disc morphology). The types of retinal and optic nerve lesions include non-proliferative diabetic retinopathy (NPDR), diabetic patients without diabetic retinopathy manifestations (No-DR), optic atrophy, pregnancy-related hypertension ophthalmopathy, focal telangiectasia (microaneurysm), and macular cystoid edema.

[0054] As a preferred embodiment, step S2 is specifically performed as follows.

[0055] S21. Manual annotation and training data preparation: Select a portion of the acquired OCT image for manual annotation to obtain the upper and lower boundary contours of the RNFL. Use this and the corresponding original OCT image as training samples to input into the U-Net neural network model.

[0056] S22, U-Net Neural Network Training: The U-Net neural network receives the original OCT image and the upper and lower boundary contours of the RNFL. It learns the features of the RNFL structure in the OCT image through an encoder-decoder structure and optimizes the model weights to improve segmentation accuracy.

[0057] The U-Net neural network uses binary cross-entropy loss and Dice loss as its loss functions, and the total loss formula is expressed as follows:

[0058]

[0059] in, For the total loss function, , The two are the binary cross-entropy loss. and Dice loss The weighting coefficients satisfy This is used to adjust the relative importance of the binary cross-entropy loss and the Dice loss.

[0060] S23. Segmentation Inference: After training is completed, the trained U-Net neural network is used to infer the new OCT image to be segmented to obtain the segmentation results of the upper and lower boundaries of RNFL.

[0061] S24. RNFL Thickness Calculation: To improve the continuity and robustness of the thickness vector, the upper and lower boundary curves are first smoothed (e.g., using Gaussian filtering or spline interpolation) to reduce local discontinuities caused by segmentation errors. Subsequently, based on the upper boundary of the RNFL extracted from the segmentation results... and lower boundary For each horizontal pixel position Calculate vertical thickness:

[0062]

[0063] A one-dimensional RNFL thickness vector is obtained from the vertical thickness. This one-dimensional RNFL thickness vector is then expanded into a two-dimensional thickness map for thickness distribution visualization and subsequent model training.

[0064] As a preferred embodiment, S3 includes:

[0065] S31. Vessel Segmentation Model Construction: To extract and identify major vascular regions from Fundus images to aid in the selection of subsequent Regions of Interest (ROIs), a neural network model specifically for vessel segmentation is constructed. This model is independent of the RNFL segmentation network for OCT images, employing a U-Net architecture and trained separately to achieve accurate segmentation of the retinal vascular network. Its specific operation is as follows:

[0066] a) Manual annotation and training data preparation: Select Fundus images with blood vessel labels provided in the public dataset, and further filter a subset that has the same style as the actual acquired images in terms of resolution, field of view, brightness contrast, tone distribution and optic disc position, as the training dataset.

[0067] b) U-Net Neural Network Training: The blood vessel segmentation model uses the classic U-Net architecture as the backbone network. The input is the original Fundus image from the public dataset, and the output is a binary mask image of the blood vessel region. Image enhancement (such as rotation, brightness perturbation, and affine transformation) is introduced during training to improve the model's generalization ability. The loss function is a weighted combination of Dice loss and binary cross-entropy loss (same as the S2 method) to improve the detection capability of fine blood vessel structures and alleviate the foreground / background imbalance problem. The total loss function is defined as:

[0068]

[0069] in, For the total loss function, , The two are the binary cross-entropy loss. and Dice loss The weighting coefficients satisfy This is used to adjust the relative importance of the binary cross-entropy loss and the Dice loss.

[0070] c) Segmentation Inference: After training, the trained blood vessel segmentation model is used to infer the segmentation of new Fundus images to be segmented, generating a binary segmentation mask for the retinal blood vessel structure. Post-processing (such as morphological operations and connected component analysis) preserves the main blood vessel region, providing a masking reference for ROI selection and ensuring that interference regions of the main blood vessel are avoided in subsequent grayscale feature extraction.

[0071] S32. ROI Dynamic Avoidance Strategy: When selecting ROIs around the optic disc, avoid vascular areas by following these steps:

[0072] a) Calculate the overlap rate between the ROI candidate region and the blood vessel mask (the proportion of blood vessel pixels within the ROI region).

[0073] b) Set an overlap rate threshold (e.g., <10%). If the candidate region exceeds the threshold, adjust the ROI position or reduce the radius.

[0074] c) Iterative optimization until a set of ROI regions that meets the non-vascular proportion requirement is obtained.

[0075] S33. Multi-channel feature extraction: Within the final determined ROI region, perform the following operations respectively:

[0076] a) Separate the R / G / B channels of the original Fundus image;

[0077] b) Use MATLAB software to extract the local statistical features (mean) of each channel;

[0078] c) Concatenate the local statistical features by channel to form a feature vector (R channel feature || G channel feature || B channel feature). Here, R channel feature, G channel feature, and B channel feature represent the mean features of the R, G, and B grayscale images within the ROI region, respectively; the symbol "||" indicates a vector concatenation operation, which concatenates the features extracted from each channel sequentially into a multidimensional vector.

[0079] d) Gray-scale interpolation processing of vascular regions: In order to avoid interference of the vascular occlusion area on the gray-scale feature statistics, the part covered by the vascular mask does not directly participate in the mean calculation, but uses a linear interpolation method to fill the gray-scale value.

[0080] As a preferred embodiment, S4 includes:

[0081] S41. Based on the spatial correspondence between the near-infrared reference image and the Fundus image, complete the geometric registration between the two to confirm the precise location of the OCT scan area in the Fundus image.

[0082] S42. The thickness map is precisely mapped to the corresponding OCT scan area in the Fundus image through spatial interpolation, thereby achieving scale alignment and pixel-level correspondence between images.

[0083] S43. Simultaneously extract the color features of the Fundus image and the thickness information of the corresponding region in each ROI region obtained in S3, and construct a joint feature vector as the input of the regression model in S5.

[0084] As a preferred embodiment, the extracted multimodal features are fused and predicted using a backpropagation neural network (BP neural network):

[0085] S51, First Layer Network (Feature-Level Fusion Network): This layer takes the constructed joint feature vector as input and achieves pairwise fusion between different modal features (color statistical features and structural thickness features). This fusion is accomplished through a fully connected network structure with 8 neurons and ReLU activation function.

[0086] S52. Perform a weighted average of the three fusion feature results (the weights are dynamically adjusted according to the channel signal-to-noise ratio) to obtain the first comprehensive feature F_fused.

[0087] S53. Second Layer Network (Semantic Fusion Network): This layer receives the first comprehensive feature F_fused from the output of the first layer network and further performs high-level semantic expression and feature abstraction. This network consists of a fully connected hidden layer with 4 neurons and the activation function is ReLU, which is used to extract the high-order semantic representation of the fused features as input for subsequent prediction output.

[0088] S54, Third layer network (output layer): Receives the semantic feature output of the second layer and completes the numerical prediction of the RNFL thickness of the ROI region through a linear output neuron activated by an activation function.

[0089] S55. Input the predicted RNFL thickness values ​​of all ROI regions into the Softmax layer to generate an RNFL thickness probability distribution map.

[0090] S56. Extract the following clinical indicators based on the RNFL thickness probability distribution map.

[0091] a) Structural assessment: mean thickness around the optic disc, differences in quadrant thickness (nasal / temporal / upper / lower).

[0092] b) Functional correlation: Thickness gradient change rate (reflects the trend of nerve fiber layer damage).

[0093] c) Abnormal areas: areas of localized thinning (related to early glaucoma lesions).

[0094] To further enhance the modularity and system integration capabilities of the prediction process, this invention also provides a retinal nerve fiber layer thickness prediction system based on multimodal imaging. The overall structure of this system consists of four functional modules, each with the following functions:

[0095] Multimodal image data acquisition module: Acquires paired fundus images, optical coherence tomography (OCT) images, and near-infrared reference images (reference images used for scanning and positioning in fundus OCT equipment, which are acquired by the SLO or IR camera in the OCT system), all of which cover the same optic disc area.

[0096] The thickness map construction module uses a U-Net neural network to segment the OCT image, automatically extracts the upper and lower boundaries of the RNFL layer, smooths the boundary regions using interpolation methods, and constructs a thickness map.

[0097] ROI extraction module: Select multiple regions of interest (ROIs) around the optic disc of the Fundus image, avoiding the main blood vessel region; extract local statistical features on the R / G / B channel images respectively, and generate R, G, and B grayscale images.

[0098] Joint Feature Vector Construction Module: Constructs joint feature vectors for thickness prediction through spatial registration and scale alignment of multimodal images.

[0099] The regression prediction module performs regression prediction using a multimodal feature fusion and retinal nerve fiber layer (RNFL) thickness prediction model based on a backpropagation neural network (BP neural network). The model employs a three-layer design, including a feature-level fusion network, a semantic-level fusion network, and an output layer. The input to the model is the mean grayscale features of the RGB channels of the Fundus image within each ROI region and the mean local thickness in the corresponding OCT thickness map. The output is the predicted RNFL thickness value for that ROI. The first layer, the feature-level fusion network, consists of a fully connected layer with 8 neurons (ReLU activation) to fuse RGB and thickness features. The second layer, the semantic-level fusion network, contains a hidden layer with 4 neurons and a ReLU activation function, used to extract a high-order semantic representation of the fused features. The third layer is the output layer, consisting of a single output neuron with a linear activation function, used for regression prediction of the RNFL thickness value for that ROI.

[0100] The training and optimization module trains the prediction model of the regression prediction module. The training process uses the main loss function (mean squared error, MSE) and the auxiliary loss function (contrast loss) to ensure structural consistency and prediction accuracy. An image enhancement strategy is employed to simulate real-world acquisition conditions: Gaussian blur and brightness perturbation are applied to the Fundus image, and noise interference is added to the OCT image to enhance model robustness. Registration keypoints remain unchanged during the enhancement process.

[0101] like Figure 1 As shown, this invention provides a method for predicting retinal nerve fiber layer (RNFL) thickness based on multimodal medical imaging, mainly including the following steps:

[0102] S1. Acquire multimodal image data:

[0103] Thirty subjects (60 eyes in total) were selected, including those with normal eyes, non-proliferative diabetic retinopathy (NPDR), diabetic patients without diabetic retinopathy (No-DR), optic nerve atrophy, pregnancy-induced hypertension-related eye disease, focal telangiectasia (microaneurysm), macular cystoid edema, and other common retinal and optic nerve diseases. The following image data were collected from each eye.

[0104] Fundus color fundus photographs, with a resolution of 3912×3910 pixels, covering the optic disc and macula.

[0105] The OCT scan image, with a resolution of 938×625 pixels, was obtained using the Cirrus HD-OCT system. The B-scan image contains the RNFL structure.

[0106] Near-infrared reference image (IR / SLO), acquired synchronously with OCT scan image, is used as a registration reference.

[0107] All images were acquired using the same system to ensure pairing and regional consistency. Subjects were excluded from RNFL assessment due to factors such as glaucoma and high myopia (axial length > 26 mm).

[0108] S2. Construct a thickness map:

[0109] S21. Manual annotation and training data preparation:

[0110] In the OCT image data of the above 60 eyes, 20% of the B-scan images (a total of 240 images) were randomly selected. The upper and lower boundaries of the RNFL layer were manually labeled using professional software (such as MATLAB). After the labeling results were tested for consistency (Kappa value > 0.85), they were included in the training set along with the corresponding original OCT images.

[0111] S22, U-Net neural network training:

[0112] A standard U-Net network is constructed, taking the original OCT image and manually annotated upper and lower boundary contours of the RNFL as input. The manually annotated RNFL upper and lower boundary contours are used to guide network training, ensuring that the model can accurately identify and segment the upper and lower boundaries of the RNFL. The output is a two-channel RNFL boundary segmentation map. The Adam optimizer is used with an initial learning rate of 0.001, a batch size of 8, and 100 training epochs. The loss function is a combination of weighted binary cross-entropy and Dice loss (with weights of 0.4 and 0.6, respectively).

[0113] S23, Segmentation Reasoning:

[0114] The trained model is used for segmentation of unlabeled OCT images, and the corresponding upper and lower boundaries of the RNFL are output.

[0115] S24. RNFL thickness calculation: In each OCT image, based on the upper and lower boundaries of the RNFL segmented by the U-Net neural network, the vertical distance between the two boundaries (unit: μm) is calculated pixel by pixel to obtain a one-dimensional RNFL thickness vector, which represents the local thickness of the RNFL corresponding to each column of images.

[0116] To improve the continuity and robustness to segmentation errors of the thickness vector, the extracted upper and lower boundary curves are first smoothed using methods such as spline interpolation to remove discontinuities caused by noise or local segmentation errors, ensuring the smoothness and accuracy of the thickness calculation. Subsequently, the one-dimensional thickness vector is expanded into a two-dimensional thickness map and used as the training input for the subsequent RNFL thickness prediction model. The specific operations are as follows:

[0117] a) Boundary smoothing: To avoid edge artifacts introduced by interpolation anomalies, a 2-pixel wide mirrored boundary extension is performed around the thickness map before interpolation. Subsequently, spline interpolation is used to reconstruct the boundary points, thereby generating a smoother and more continuous boundary curve.

[0118] b) Thickness vector calculation: Based on the upper boundary of the RNFL extracted from the segmentation results. and lower boundary For each horizontal pixel position Calculate vertical thickness:

[0119]

[0120] One-dimensional RNFL thickness vector is obtained from the vertical thickness. , where N represents the number of horizontal pixels.

[0121] c) Generation of 2D thickness map: Constructing a 1D permutation vector The thickness vector has the same length. With position index vector Combined, they form a two-dimensional thickness map. ,in:

[0122]

[0123] This two-dimensional thickness map not only preserves the thickness value corresponding to each pixel position, but also encodes its position information in the sequence, which is suitable for subsequent model training.

[0124] S3, ROI extraction and grayscale feature construction:

[0125] S31. Vessel Segmentation Model Construction: A dedicated U-Net model was trained to segment vessels in Fundus images. The DRIVE, CHASE_DB1, and STARE datasets were used for training, with 300 training images at a uniform resolution of 3910×3910. Enhancement strategies included affine transformation, brightness perturbation, and cropping / flipping. The loss function was a combination of weighted binary cross-entropy and Dice loss.

[0126] S32, ROI Dynamic Avoidance Strategy: 12 candidate ROI regions (with a radius of approximately 50 pixels) are generated at equal intervals with the center of the visual disc as the center. By calculating the proportion of blood vessel pixels in each ROI in the blood vessel mask, a threshold of <10% is set. If it exceeds this threshold, the ROI radius is reduced to 40 pixels or shifted outwards. This process is iteratively adjusted until the requirement is met.

[0127] S33. Multi-channel feature extraction: Perform the following operations on each ROI region:

[0128] a) Separate the R, G, and B channels of the Fundus graph.

[0129] b) Calculate the mean pixel grayscale value within the ROI for each channel to obtain the local statistical feature vector. || || ].

[0130] c) Gray-level interpolation processing in the vascular region: To avoid interference from vascular occlusion areas on gray-level feature statistics, the portion covered by the vascular mask is not directly included in the mean calculation. Instead, a linear interpolation method is used to fill in the gray-level values. The specific procedure is as follows:

[0131] For each pixel in a ROI covered by a blood vessel mask, extract the grayscale value along the boundary pixels of the surrounding non-blood vessel regions based on the nearest neighbor direction.

[0132] One-dimensional or two-dimensional linear interpolation (such as bilinear interpolation) is used to smoothly fill the region based on the grayscale information of surrounding non-vascular pixels.

[0133] The interpolated grayscale image maintains the overall texture continuity while avoiding feature shifts caused by blood vessel occlusion, thus improving the robustness of grayscale features.

[0134] S4. Construct cross-modal joint feature vectors:

[0135] S41. Based on the spatial correspondence between the near-infrared reference image and the Fundus image, MATLAB software is used to extract the corresponding feature points of blood vessel direction in the Fundus image and the near-infrared reference image to complete the geometric registration between the two, thereby confirming the precise location of the OCT scan area in the Fundus image.

[0136] S42. Reconstruct the RNFL thickness map using spline interpolation and map it to the actual anatomical region covered by the OCT scan in the Fundus image, so that the thickness map is consistent with the Fundus image in terms of spatial coordinate system and resolution. Figure 1 To.

[0137] S43. Twelve Regions of Interest (ROIs) are extracted from each Fundus image, corresponding to OCT image regions. Three feature vectors are constructed for each ROI:

[0138] X=[ || ], Y=[ || ], Z=[ || ]

[0139] in , , The mean feature of the RGB grayscale image within this ROI. This represents the average thickness of the RNFL within the corresponding ROI region.

[0140] S5. Construction and prediction of regression prediction model:

[0141] S51. Model Structure Design: This step aims to design a lightweight three-layer BP neural network regression prediction model based on the aforementioned multimodal joint features, to predict the retinal nerve fiber layer (RNFL) thickness of each ROI region. The input features are two-dimensional, including the mean grayscale features of the R, G, and B channels of the ROI region in the Fundus image (1-dimensional, input separately) and the mean thickness of the region in the OCT thickness map (1-dimensional), for a total of two feature dimensions.

[0142] The model uses a three-layer fully connected neural network structure, as follows:

[0143] a) First layer network (feature-level fusion network): The input is a 2-dimensional joint feature vector, which is mapped to an 8-dimensional intermediate representation through a fully connected layer, followed by a Batch Normalization layer to improve training stability, and the ReLU activation function is used to introduce non-linearity.

[0144] b) Weighted Fusion Features: To improve the stability and expressive power of feature fusion across different modalities, a signal-to-noise ratio (SNR) adjustment mechanism is introduced. This mechanism weights the fusion features output by the sub-networks corresponding to the three RGB channels (independent processing branches that pair and fuse the grayscale mean of each RGB channel with the average OCT thickness of the same ROI region. Each sub-network consists of a set of structurally identical fully connected layers used to extract the local fusion feature representation between the channel and thickness information). The specific weights are determined based on the average SNR of the extracted features from each channel in the training set.

[0145] The training set analysis yielded the following average SNR for each channel across the entire ROI region:

[0146] Red channel (R): SNR_R = 16.8dB.

[0147] Green channel (G): SNR_G = 14.2dB.

[0148] Blue channel (B): SNR_B = 13.1 dB.

[0149] The SNR value is normalized to a weighted coefficient, calculated as follows:

[0150]

[0151] ≈0.37 ≈0.31 ≈0.29

[0152] F_fused = 0.37X* + 0.31Y* + 0.29Z*, where * represents the output feature of the sub-network.

[0153] The second layer (semantic fusion network): After the Dropout layer (dropout probability p=0.3), a fully connected layer is set to compress the feature dimension from 8 dimensions to 4 dimensions, and the ReLU activation function is continued.

[0154] The third layer (output layer): The 4D features are input to the output layer (fully connected layer), and the output is a single scalar value, which represents the predicted RNFL thickness of the current ROI region, in micrometers (μm).

[0155] The entire neural network model has approximately 1200 parameters, exhibiting excellent lightweight characteristics, making it suitable for deployment and operation in edge computing devices or portable ophthalmic screening terminals.

[0156] S52. Loss Function Design: To improve prediction accuracy and enhance the physiological consistency of model output, a loss function design method combining the main loss and auxiliary structure regularization term is proposed, as follows:

[0157] a) The main loss term is the mean squared error (MSE), which measures the distance between the model's predicted value and the actual RNFL thickness value in the thickness map.

[0158] b) The auxiliary regularization term is a thickness gradient smoothing term, which encourages the model to output a smooth thickness distribution with continuous physiological structure in space, and prevents unnatural jumps or breaks in the prediction.

[0159] c) The total loss function is defined as follows:

[0160]

[0161] in, λ represents the predicted thickness of the i-th ROI region, and λ is the weighting coefficient of the smoothing term, with a value of 0.1.

[0162] S53. Thickness Map Restoration: For the RNFL thickness prediction values ​​of each ROI region obtained from the regression prediction (a total of 12 regions), this implementation uses bicubic interpolation to perform image-level continuity reconstruction on the sparse prediction results, generating a thickness prediction map with the same size as the original Fundus map. The specific process is as follows:

[0163] a) First, based on the projection position of the OCT scan path in the Fundus image, the 12 thickness prediction values ​​are mapped to the geometric center point of the corresponding region according to their spatial layout.

[0164] b) Then, the thickness prediction values ​​are interpolated in two dimensions using a bicubic interpolation algorithm to generate a continuous thickness map (floating-point map). Compared with bilinear interpolation, this method has a smoother and more natural response in the edge and transition regions, which helps to preserve the local details of the physiological structure.

[0165] c) To suppress high-frequency noise introduced by interpolation and enhance structural consistency, the thickness map is further processed using guided filtering or bilateral filtering:

[0166] Guided Filter: Using the grayscale channels of the original Fundus image as a guide map, this method performs structure-preserving smoothing on the thickness map. It effectively preserves edge details while eliminating high-frequency artifacts and improving structural consistency.

[0167] Bilateral Filter: Combines pixel spatial distance and intensity similarity for weighted smoothing, reducing noise and maintaining edge sharpness. It is suitable for regions with local abrupt changes in the distribution of thickness prediction values.

[0168] d) Finally, to facilitate clinicians' intuitive interpretation of structural change trends, the processed thickness map was visualized using a pseudo-color (Jet colormap) method. The predicted thickness values ​​were mapped to a color gradient based on physiological ranges (e.g., 40μm–120μm), and the color changes intuitively reflected the relative thickness of the local optic nerve fiber layer.

[0169] S54. Extraction of Clinical Indicators:

[0170] Average thickness: the average of the predicted values ​​of 12 ROIs around the viewing disc.

[0171] Quadrant difference: the difference in mean values ​​between the upper / lower and lower / left / right quadrants.

[0172] Thickness gradient: First derivative of thickness along the nasotemporal direction.

[0173] Anomaly detection: Regions of origin (ROI) less than 70% of the average thickness are marked as suspected lesions.

[0174] S6. Model Training and Optimization:

[0175] a) The model training uses the joint loss function structure designed by S52.

[0176] In each round of training, 80% of the samples are selected as the training set and 20% as the validation set, using five-fold cross-validation.

[0177] b) Add Gaussian blur (σ=2.0) and brightness perturbation (±10%) to the Fundus plot; add Poisson noise to the OCT plot.

[0178] c) During the enhancement process, the coordinates of the ROI and the registration point remain unchanged to ensure training consistency.

[0179] S7. Model Output and Evaluation:

[0180] The Fundus test set images are input into the trained model to predict the RNFL thickness for each ROI. The evaluation metrics are as follows:

[0181] a) Prediction error: Root mean square error (RMSE) of the difference between the interpolated thickness map and the prediction error;

[0182] b) Registration error: The sum of the squared differences in coordinates between the center point of the ROI and the thickness map;

[0183] c) Correlation coefficient R: The goodness of linear fit between predicted thickness and actual thickness.

[0184] Figure 2 The image presents a typical sample (94 data sets) from the test set, reflecting the comparison between the model's predicted thickness distribution and the actual OCT thickness map. The image shows that the model's predicted results have a high degree of consistency with the actual RNFL thickness in terms of spatial distribution, verifying the network's good ability to perceive image structural features.

[0185] Figure 3 The graph shows the regression analysis results for the entire test set. The horizontal axis represents the actual RNFL thickness value, and the vertical axis represents the model's predicted value. Linear fitting analysis shows a significant correlation between the two, with a correlation coefficient R = 0.7607, indicating that the model has strong predictive ability.

[0186] The above description, in conjunction with specific / preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. Those skilled in the art can make various substitutions or modifications to these described embodiments without departing from the inventive concept, and all such substitutions or modifications should be considered within the scope of protection of the present invention.

[0187] The parts of this invention not described in detail are well-known to those skilled in the art.

Claims

1. A method for predicting retinal thickness based on multimodal images, characterized in that, Includes the following steps: S1. Acquire multimodal image data: Acquire paired fundus images, optical coherence tomography (OCT) images, and near-infrared reference images, all of which cover the same optic disc region; S2. Construct a thickness map based on structural annotations; S3. Region of Interest (ROI) extraction and grayscale feature construction; S4. Construct cross-modal joint feature vectors; S5. Construction and prediction of regression prediction model: A multimodal feature fusion and retinal nerve fiber layer thickness prediction model based on backpropagation neural network was constructed, and the model was used for regression prediction. The model adopts a three-layer design, including a feature-level fusion network, a semantic-level fusion network, and an output layer. The input of the model is the cross-modal joint feature vector of each ROI, and the output is the predicted RNFL thickness of the ROI. S6. Training and Optimization.

2. The retinal thickness prediction method based on multimodal images according to claim 1, characterized in that, The data collected by S1 covers normal eyes and types of retinal and optic nerve lesions, ensuring disease diversity and actual clinical adaptability during model training and evaluation; inclusion criteria: no history of glaucoma, high myopia with intraocular pressure <21 mmHg, and normal optic disc morphology.

3. The method for predicting retinal thickness based on multimodal images according to claim 1, characterized in that, The specific operation of S2 is as follows. The OCT image was segmented using a U-Net neural network, and the upper and lower boundaries of the RNFL layer were automatically extracted. An interpolation method was used to smooth the boundary region and a thickness map was constructed.

4. The method for predicting retinal thickness based on multimodal images according to claim 1, characterized in that, S3 includes: Multiple regions of interest (ROIs) were selected around the optic disc in the Fundus image, avoiding the main blood vessel region; local statistical features were extracted from the R / G / B channel images respectively, and R, G, and B grayscale images were generated.

5. The method for predicting retinal thickness based on multimodal images according to claim 1, characterized in that, S4 includes: S41. Based on the spatial correspondence between the near-infrared reference image and the Fundus image, complete the geometric registration between the two to confirm the precise location of the OCT scan area in the Fundus image. S42. The thickness map is precisely mapped to the corresponding OCT scan area in the Fundus image through spatial interpolation, thereby achieving scale alignment and pixel-level correspondence between images. S43. Simultaneously extract the color features of the Fundus image and the thickness information of the corresponding region in each ROI region obtained in S3, and construct a joint feature vector.

6. The method for predicting retinal thickness based on multimodal images according to claim 1, characterized in that, S5 includes: S51, Feature-level Fusion Network: The constructed joint feature vector is used as input to achieve pairwise fusion between features of different modalities; this fusion is accomplished through a fully connected network structure with 8 neurons and ReLU activation function. S52. The three fusion feature results are weighted and averaged, with the weights dynamically adjusted according to the channel signal-to-noise ratio, to obtain the first comprehensive feature. S53, Semantic-level Fusion Network: Receives the first comprehensive features and further performs high-level semantic expression and feature abstraction; this network consists of a fully connected hidden layer with 4 neurons and the activation function is ReLU, used to extract the high-order semantic representation of the fused features as the input for subsequent prediction output; S54, Output Layer: Receives the semantic feature output of the semantic-level fusion network and completes the numerical prediction of the RNFL thickness of the ROI region through a linear output neuron activated by an activation function. S55. Input the predicted RNFL thickness values ​​of all ROI regions into the Softmax layer to generate an RNFL thickness probability distribution map. S56. Extract clinical indicators based on the RNFL thickness probability distribution map.

7. The method for predicting retinal thickness based on multimodal images according to claim 1, characterized in that, The training process combines the main loss function and the auxiliary loss function to ensure structural consistency and prediction accuracy. The main loss function uses mean squared error (MSE), and the auxiliary loss function uses contrastive loss. An image enhancement strategy is used to simulate real acquisition conditions: Gaussian blur and brightness perturbation are applied to Fundus images, and noise interference is added to OCT images to enhance the robustness of the model. Registration keypoints remain unchanged during enhancement.

8. A retinal nerve fiber layer thickness prediction system based on multimodal imaging, characterized in that, Includes the following modules: Multimodal image data acquisition module: Acquires paired fundus images, optical coherence tomography (OCT) images, and near-infrared reference images, all of which cover the same optic disc region; The thickness map construction module uses a U-Net neural network to segment OCT images, automatically extracts the upper and lower boundaries of the RNFL layer, smooths the boundary regions using interpolation methods, and constructs a thickness map. ROI extraction module: Selects multiple regions of interest (ROIs) around the optic disc in the Fundus image, avoiding the main blood vessel region; extracts local statistical features on the R / G / B channel images respectively, and generates R, G, and B grayscale images; Joint Feature Vector Construction Module: Constructs joint feature vectors for thickness prediction through spatial registration and scale alignment of multimodal images; The regression prediction module performs regression predictions using a multimodal feature fusion based on a backpropagation neural network and a retinal nerve fiber layer thickness prediction model. The model adopts a three-layer design, including a feature-level fusion network, a semantic-level fusion network, and an output layer. The input of the model is the joint feature vector of each ROI region, and the output is the predicted RNFL thickness of that ROI. The training and optimization module trains the prediction model of the regression prediction module. The training process uses mean squared error (MSE) and contrast loss to ensure structural consistency and prediction accuracy. An image enhancement strategy is adopted to simulate real acquisition conditions: Gaussian blur and brightness perturbation are applied to Fundus images, and noise interference is added to OCT images to enhance the robustness of the model. Registration keypoints remain unchanged during enhancement.

Images