Method, device and equipment for generating a three-dimensional model of an eyeball based on a single fundus photograph

Abstract

The application relates to a method, device and equipment for generating an eyeball three-dimensional model based on a single fundus photo. The method comprises the following steps: obtaining fundus photo data to be reconstructed, and preprocessing the fundus photo data to be reconstructed to obtain preprocessed fundus photo data; inputting the preprocessed fundus photo data into a preset depth estimation network model, and predicting depth map data by using the preset depth estimation network model, wherein the preset depth estimation network model comprises a ViT-Hybrid encoder and a multi-task decoder; converting the depth map data into point cloud data, and reconstructing an eyeball three-dimensional model by using a preset B-spline algorithm based on the point cloud data. Thus, the problems that the OCT equipment is high in cost and limited in scanning range in the prior art, and the depth map output by the existing AI model is insufficient in resolution and cannot meet the clinical diagnosis requirements are solved, the generation process of the eyeball three-dimensional model is greatly simplified, and the generation efficiency is improved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and image processing technology, and in particular to a method, apparatus and device for generating a three-dimensional model of the eyeball based on a single fundus photograph. Background Technology

[0002] The morphological features of the three-dimensional structures at the back of the eye (such as the macula and optic disc) are crucial for diagnosing diseases such as high myopia and diabetic retinopathy.

[0003] Commonly used methods in related technologies include: using OCT (Optical Coherence Tomography) to obtain three-dimensional data of the eyeball; or using monocular depth estimation models (such as MiDaS (Monocular Depth Sensing) and DPT (Dense Prediction Transformer)) to output depth maps.

[0004] However, OCT equipment is expensive, complex to operate, and has a limited scanning range, making it difficult to cover a wide field of view of the fundus. The depth maps output by existing monocular depth estimation models have insufficient resolution, which cannot meet the needs of clinical diagnosis and urgently needs to be addressed. Summary of the Invention

[0005] This application provides a method, apparatus, and device for generating a three-dimensional model of the eyeball based on a single fundus photograph, which solves the problems of high cost and limited scanning range of existing OCT equipment, as well as insufficient resolution of depth maps output by existing AI models, which cannot meet the needs of clinical diagnosis. It greatly simplifies the generation process of the three-dimensional model of the eyeball and improves the generation efficiency.

[0006] To achieve the above objectives, the first aspect of this application proposes a method for generating a three-dimensional model of the eyeball based on a single fundus photograph, comprising the following steps:

[0007] Acquire fundus image data to be reconstructed, and preprocess the fundus image data to be reconstructed to obtain preprocessed fundus image data;

[0008] The preprocessed fundus photograph data is input into a preset depth estimation network model, and the preset depth estimation network model is used to predict depth map data. The preset depth estimation network model includes a ViT-Hybrid encoder and a multi-task decoder, and the preset depth estimation network model is trained by a historical fundus photograph dataset and a historical three-dimensional eye scan dataset.

[0009] The depth map data is converted into point cloud data, and a three-dimensional model of the eyeball is reconstructed based on the point cloud data using a preset B-spline algorithm.

[0010] According to one embodiment of this application, before inputting the preprocessed fundus photograph data into the preset depth estimation network model, the method further includes:

[0011] Obtain the historical fundus photograph dataset and the historical three-dimensional eye scan dataset, wherein the data in the historical fundus photograph dataset corresponds one-to-one with the data in the historical three-dimensional eye scan dataset;

[0012] The data in the historical fundus photograph dataset and the historical eyeball 3D scan dataset are pixel-level aligned using a preset geometric transformation algorithm and a preset feature point matching algorithm. The data augmentation operation is then performed on the processing results to obtain the target multimodal dataset.

[0013] The target multimodal dataset is divided into a training set and a validation set. Based on a preset loss function, a preset neural network is trained using the training set to obtain an initial depth estimation network model. The initial depth estimation network model is then validated using the validation set. The iterative training of the initial depth estimation network model ends when the initial depth estimation network model meets preset conditions, thus obtaining the preset depth estimation network model. Otherwise, the weights of the preset loss function are dynamically adjusted before iterative training continues.

[0014] According to one embodiment of this application, the preset loss function is:

[0015]

[0016] Where α is the weight of the scale-invariant logarithmic loss, β is the weight of the edge-aware loss, and γ is the weight of the semantic segmentation loss. For scale-invariant logarithmic loss, For edge perception loss, This is the semantic segmentation loss.

[0017] According to one embodiment of this application, the step of inputting the preprocessed fundus photograph data into a preset depth estimation network model and using the preset depth estimation network model to predict depth map data includes:

[0018] The ViT-Hybrid encoder is used to extract multi-scale features from the preprocessed fundus image data;

[0019] Based on the multi-scale features, the multi-task decoder is used to perform cross-layer feature fusion to obtain the depth map data.

[0020] According to one embodiment of this application, after inputting the preprocessed fundus photograph data into the preset depth estimation network model, the method further includes:

[0021] Based on the preprocessed fundus photograph data, the feature distribution of the preset depth estimation network model is dynamically adjusted using a preset normalization strategy.

[0022] The method for generating a 3D eye model based on a single fundus photograph, as proposed in this application, involves preprocessing the fundus photograph data to be reconstructed. The preprocessed fundus photograph data is then input into a preset depth estimation network model, which predicts depth map data. The depth map data is then converted into point cloud data, and a preset B-spline algorithm is used to reconstruct the 3D eye model based on the point cloud data. This method solves the problems of high cost and limited scanning range of existing OCT equipment, as well as insufficient resolution of depth maps output by existing AI models, which cannot meet clinical diagnostic needs. It greatly simplifies the process of generating a 3D eye model and improves generation efficiency.

[0023] To achieve the above objectives, a second aspect of this application provides an apparatus for generating a three-dimensional model of the eyeball based on a single fundus photograph, comprising:

[0024] The preprocessing module is used to acquire fundus image data to be reconstructed and to preprocess the fundus image data to be reconstructed to obtain preprocessed fundus image data.

[0025] The prediction module is used to input the preprocessed fundus photograph data into a preset depth estimation network model and use the preset depth estimation network model to predict depth map data. The preset depth estimation network model includes a ViT-Hybrid encoder and a multi-task decoder, and the preset depth estimation network model is trained by a historical fundus photograph dataset and a historical three-dimensional eye scan dataset.

[0026] The reconstruction module is used to convert the depth map data into point cloud data, and based on the point cloud data, to reconstruct a three-dimensional model of the eyeball using a preset B-spline algorithm.

[0027] According to one embodiment of this application, before inputting the preprocessed fundus photograph data into the preset depth estimation network model, the prediction module is further configured to:

[0028] Obtain the historical fundus photograph dataset and the historical three-dimensional eye scan dataset, wherein the data in the historical fundus photograph dataset corresponds one-to-one with the data in the historical three-dimensional eye scan dataset;

[0029] The data in the historical fundus photograph dataset and the historical eyeball 3D scan dataset are pixel-level aligned using a preset geometric transformation algorithm and a preset feature point matching algorithm. The data augmentation operation is then performed on the processing results to obtain the target multimodal dataset.

[0030] The target multimodal dataset is divided into a training set and a validation set. Based on a preset loss function, a preset neural network is trained using the training set to obtain an initial depth estimation network model. The initial depth estimation network model is then validated using the validation set. The iterative training of the initial depth estimation network model ends when the initial depth estimation network model meets preset conditions, thus obtaining the preset depth estimation network model. Otherwise, the weights of the preset loss function are dynamically adjusted before iterative training continues.

[0031] According to one embodiment of this application, the preset loss function is:

[0032]

[0033] Where α is the weight of the scale-invariant logarithmic loss, β is the weight of the edge-aware loss, and γ is the weight of the semantic segmentation loss. For scale-invariant logarithmic loss, For edge perception loss, This is the semantic segmentation loss.

[0034] According to one embodiment of this application, the prediction module is specifically used for:

[0035] The ViT-Hybrid encoder is used to extract multi-scale features from the preprocessed fundus image data;

[0036] Based on the multi-scale features, the multi-task decoder is used to perform cross-layer feature fusion to obtain the depth map data.

[0037] According to one embodiment of this application, after inputting the preprocessed fundus photograph data into the preset depth estimation network model, the prediction module is further configured to:

[0038] Based on the preprocessed fundus photograph data, the feature distribution of the preset depth estimation network model is dynamically adjusted using a preset normalization strategy.

[0039] The apparatus for generating a 3D eye model based on a single fundus photograph, as proposed in this application, preprocesses the fundus photograph data to be reconstructed. The preprocessed fundus photograph data is then input into a preset depth estimation network model, which predicts depth map data. The depth map data is then converted into point cloud data, and a preset B-spline algorithm is used to reconstruct the 3D eye model based on the point cloud data. This solves the problems of high cost and limited scanning range of existing OCT equipment, and insufficient resolution of depth maps output by existing AI models, which cannot meet clinical diagnostic needs. It greatly simplifies the process of generating a 3D eye model and improves generation efficiency.

[0040] To achieve the above objectives, a third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for generating a three-dimensional model of the eyeball based on a single fundus photograph as described in the above embodiments.

[0041] To achieve the above objectives, a fourth aspect of this application provides a computer-readable storage medium storing a computer program that is executed by a processor to implement the method for generating a three-dimensional model of the eyeball based on a single fundus photograph as described in the above embodiments.

[0042] To achieve the above objectives, a fifth aspect of this application provides a computer program product comprising a computer program that, when executed by a processor, is used to implement the method for generating a three-dimensional model of the eyeball based on a single fundus photograph as described in the above embodiments.

[0043] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0044] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0045] Figure 1 This is a flowchart illustrating a method for generating a three-dimensional model of the eyeball based on a single fundus photograph, according to an embodiment of this application.

[0046] Figure 2 This is a schematic diagram illustrating the implementation process of a method for generating a three-dimensional model of the eyeball based on a single fundus photograph according to an embodiment of this application.

[0047] Figure 3 This is a schematic visualization of a three-dimensional model of an eyeball generated according to an embodiment of this application;

[0048] Figure 4 This is a block diagram of a device for generating a three-dimensional model of the eyeball based on a single fundus photograph, according to an embodiment of this application.

[0049] Figure 5 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0050] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0051] The method, apparatus, and device for generating a three-dimensional model of the eyeball based on a single fundus photograph, according to embodiments of this application, will now be described with reference to the accompanying drawings. First, the method for generating a three-dimensional model of the eyeball based on a single fundus photograph, according to embodiments of this application, will be described with reference to the accompanying drawings.

[0052] Figure 1 This is a flowchart of a method for generating a three-dimensional model of the eyeball based on a single fundus photograph, according to an embodiment of this application.

[0053] For example, such as Figure 1 As shown, the method for generating a 3D model of the eyeball based on a single fundus photograph includes the following steps:

[0054] In step S101, fundus image data to be reconstructed is obtained, and the fundus image data to be reconstructed is preprocessed to obtain preprocessed fundus image data.

[0055] It is understood that the fundus image data to be reconstructed refers to two-dimensional static or dynamic image records of the posterior part of the eyeball (including structures such as the retina, optic nerve, macula, and blood vessels) acquired through specialized imaging equipment (such as a fundus camera). Preprocessing refers to the initial processing of the fundus image data to be reconstructed, with the aim of improving data quality or adapting it to the needs of subsequent algorithms. Here, preprocessing may include techniques such as image enhancement and noise reduction.

[0056] Specifically, firstly, a single fundus image to be reconstructed can be acquired using a standard fundus camera. Then, a series of preprocessing operations are performed on this image to ensure that the data quality and format meet the requirements of subsequent processing. These preprocessing steps yield clearer and more accurate fundus image data, laying a solid foundation for the subsequent reconstruction work.

[0057] In step S102, the preprocessed fundus photograph data is input into a preset depth estimation network model, and the preset depth estimation network model is used to predict the depth map data. The preset depth estimation network model includes a ViT-Hybrid encoder and a multi-task decoder, and the preset depth estimation network model is trained by a historical fundus photograph dataset and a historical three-dimensional eye scan dataset.

[0058] The ViT-Hybrid encoder combines the advantages of the Vision Transformer (ViT) and ResNet (Residual Network)-50 to extract multi-scale features. ViT is a Transformer-based visual model that segments an image into multiple small patches and processes these patches like natural language, using a self-attention mechanism to capture relationships between them. The Hybrid architecture combines the advantages of Convolutional Neural Networks (CNNs) and Transformers. Typically, CNNs are used first to extract local features, which are then fed into a Transformer for global feature modeling. This hybrid architecture better captures both local details and global structural information of the image. The multi-task decoder, based on a multi-task learning framework, jointly optimizes depth estimation and semantic segmentation tasks, generating high-resolution depth maps through cross-layer feature fusion. Depth estimation refers to the model's ability to predict depth information from a single fundus image; semantic segmentation refers to identifying and segmenting different semantic regions, such as the optic disc and macula.

[0059] Specifically, after obtaining the preprocessed fundus photograph data, it can be input into a preset depth estimation network model. This preset depth estimation network model predicts and generates depth map data. A depth map is an image where the value of each pixel represents the depth information of that pixel in three-dimensional space. In fundus image processing, depth maps can represent information such as retinal thickness and choroidal depth. In other words, this embodiment uses fundus photographs as basic data and combines them with a deep learning model (i.e., the preset depth estimation network model) to achieve a two-dimensional to three-dimensional conversion, breaking through the limitations of traditional imaging and replacing the three-dimensional scanning of traditional OCT equipment.

[0060] Understandably, the preset depth estimation network model is obtained by training a large number of historical fundus photograph datasets and historical three-dimensional eye scan datasets. This training process enables the model to learn the complex features of the fundus structure and accurately predict depth information. The following section will explain in detail how the preset depth estimation network model is obtained.

[0061] To facilitate understanding, the following details how to use a pre-defined depth estimation network model to predict depth map data.

[0062] As one possible approach, in some embodiments, preprocessed fundus image data is input into a preset depth estimation network model, and depth map data is predicted using the preset depth estimation network model. This includes: using a ViT-Hybrid encoder to extract multi-scale features from the preprocessed fundus image data; and using a multi-task decoder to perform cross-layer feature fusion based on the multi-scale features to obtain depth map data.

[0063] Specifically, such as Figure 2 As shown, after inputting the preprocessed fundus image data into a preset depth estimation network model, the ViT-Hybrid encoder of the preset depth estimation network model can extract multi-scale features from the preprocessed fundus images. That is, features are extracted at different scales (or resolutions). These features capture the details and structural information of the fundus images at different levels, including the morphology of retinal vessels, changes in the thickness of the retinal layer, etc. Then, using the multi-task decoder of the preset depth estimation network model, the multi-scale features extracted by the ViT-Hybrid encoder can be recombined (i.e., cross-layer feature fusion) to obtain depth map data containing comprehensive information from the fundus images for further analysis and diagnosis.

[0064] Furthermore, in some embodiments, after inputting the preprocessed fundus photograph data into a preset depth estimation network model, the method further includes: dynamically adjusting the feature distribution of the preset depth estimation network model based on the preprocessed fundus photograph data using a preset normalization strategy.

[0065] It is understandable that in medical image processing, fundus cameras of different brands or models may generate fundus images with different feature distributions (such as brightness, contrast, noise patterns, etc.). Directly using a pre-defined depth estimation network model to process these images may lead to decreased accuracy because the model did not fully cover these feature differences during training. To address this issue, embodiments of this application employ a zero-shot transfer learning framework in the model inference stage. This framework solves the problem of depth prediction across devices and scenarios by introducing domain-adaptive distillation techniques. This framework combines the capabilities of zero-shot learning with the idea of ​​domain adaptation, enabling it to quickly adapt to new environments or hardware conditions without requiring labeled data in the target domain, thereby improving the model's generalization ability.

[0066] In other words, this application's embodiments introduce a preset normalization strategy, such as Adaptive Instance Normalization (AdaIN), during the model inference stage. AdaIN is a method for dynamically adjusting features, which can adjust the feature distribution within the model in real time based on the characteristics of the input image (such as mean and variance) without retraining the model, thereby improving the model's adaptability to different device inputs.

[0067] For example, when the input fundus image comes from an unseen fundus camera, AdaIN can adjust the feature representation of the preset depth estimation network model according to the imaging characteristics of that fundus camera, thereby making the depth prediction results more accurate.

[0068] In step S103, the depth map data is converted into point cloud data, and a three-dimensional model of the eyeball is reconstructed based on the point cloud data using a preset B-spline algorithm.

[0069] Specifically, after obtaining the depth map data, each pixel in the depth map can be mapped to a point in three-dimensional space based on its depth value and pixel position. Collecting all mapped pixels forms point cloud data, which contains three-dimensional information about the fundus structure. However, while point cloud data can represent three-dimensional geometric structures, it is essentially a discrete set of points and cannot directly form a continuous and smooth surface. To generate a more refined and realistic three-dimensional model of the eye, this embodiment can use a preset B-spline algorithm (such as NURBS (Non-Uniform Rational B-Splines) algorithm) to fit the point cloud data. Through this algorithm, the point cloud data can be transformed into a three-dimensional surface with geometric accuracy, i.e., a three-dimensional model of the eye.

[0070] Furthermore, such as Figure 3 As shown in the embodiments of this application, the reconstructed three-dimensional model of the eyeball can also be displayed in the form of dynamic video, allowing observers to view the three-dimensional morphology of the fundus structure from different angles and perspectives, thereby enhancing visual effects and comprehension. Specifically, computer graphics technologies (such as OpenGL (Open Graphics Library)) can be used to render the reconstructed three-dimensional model of the eyeball, thereby generating high-quality images and supporting multi-view rotation, zooming, and other operations to generate a series of continuous image frames, forming animation effects. It can even generate cross-sectional views to help observers more intuitively observe the three-dimensional structures at the back of the eyeball (such as the macula and optic disc).

[0071] Next, we will explain in detail how to obtain the preset depth estimation network model.

[0072] In some embodiments, before inputting the preprocessed fundus photograph data into the preset depth estimation network model, the method further includes: acquiring a historical fundus photograph dataset and a historical three-dimensional eye scan dataset, wherein the data in the historical fundus photograph dataset corresponds one-to-one with the data in the historical three-dimensional eye scan dataset; performing pixel-level alignment processing on the data in the historical fundus photograph dataset and the data in the historical three-dimensional eye scan dataset using a preset geometric transformation algorithm and a preset feature point matching algorithm, and performing data augmentation operations on the processing results to obtain a target multimodal dataset; dividing the target multimodal dataset into a training set and a validation set, training a preset neural network using the training set based on a preset loss function to obtain an initial depth estimation network model, and validating the initial depth estimation network model using the validation set until the initial depth estimation network model meets preset conditions, ending the iterative training of the initial depth estimation network model to obtain the preset depth estimation network model; otherwise, dynamically adjusting the weights of the preset loss function and continuing iterative training.

[0073] Specifically, in training the preset depth estimation network model, firstly, wide-view 3D scan data from different OCT devices (i.e., historical eye 3D scan datasets) and 2D images from fundus cameras (i.e., historical fundus photograph datasets) are acquired. The wide-view 3D scan data from different OCT devices are then paired with the corresponding 2D fundus camera images to obtain a training dataset with comprehensive coverage of pathological features. Next, preset geometric transformation algorithms (such as affine transformation, which is commonly used for image alignment or correction in image processing) and preset feature point matching algorithms (such as deep learning-based feature point matching, used to find corresponding feature points in two or more images) are used to perform pixel-level alignment processing on the data in the historical fundus photograph dataset and the historical eye 3D scan dataset. That is, the relative position and orientation of the OCT eye 3D scan data and fundus images are initially adjusted through affine transformation, and then further refined using deep learning-based feature point matching, ultimately achieving pixel-level alignment of the two images. Pixel-level alignment refers to the process of precisely aligning two or more images at the pixel level. The purpose of alignment is to make images from different sources spatially consistent for subsequent analysis or fusion. After alignment, data augmentation operations can be performed on the results, such as random rotation (±15°), brightness adjustment (±20%), contrast adjustment (±20%), and color-to-grayscale conversion, thereby increasing the diversity and quantity of the dataset and obtaining the target multimodal dataset.

[0074] Next, the target multimodal dataset is used for model training, combined with a multi-task loss function with dynamic weight allocation. First, the target multimodal dataset can be divided into a training set and a validation set. The training set is used to train a pre-defined neural network. In each iteration, the weights of the neural network are adjusted according to the pre-defined loss function, enabling the model to learn patterns and features in the data, resulting in an initial depth estimation network model. The validation set is used to evaluate the performance of the initial depth estimation network model on unseen data, preventing overfitting. If the initial depth estimation network model's performance on the validation set meets the pre-defined conditions (i.e., the model meets performance metrics, such as a loss function value below a certain threshold or an accuracy reaching a certain standard), then training ends, yielding the final depth estimation network model, i.e., the pre-defined depth estimation network model. Conversely, if the initial depth estimation network model's performance on the validation set does not meet the pre-defined conditions, the weights of the pre-defined loss function need to be adjusted. This can be achieved by increasing or decreasing the weight of a certain loss term to change the model's optimization objective. After adjusting the loss function weights, iterative training of the initial depth estimation network model continues until the model meets the pre-defined conditions, resulting in the pre-defined depth estimation network model.

[0075] Optionally, in some embodiments, the preset loss function is:

[0076]

[0077] Where α is the weight of the scale-invariant logarithmic loss, β is the weight of the edge-aware loss, and γ is the weight of the semantic segmentation loss. For scale-invariant logarithmic loss, For edge perception loss, This is the semantic segmentation loss.

[0078] Understandably, scale-invariant logarithmic loss is used to optimize the global consistency of depth prediction; edge-aware loss is used to enhance the structural details of the depth map; and semantic segmentation loss enhances the model's sensitivity to pathological regions through auxiliary tasks. During training, the weight allocation is dynamically adjusted based on the gradient magnitude and convergence speed of each loss term, which not only achieves a balance between global depth prediction and local feature enhancement, but also significantly improves the model's adaptability to the specific needs of medical images.

[0079] In summary, the method for generating a three-dimensional eyeball model based on a single fundus photograph proposed in this application has at least the following advantages:

[0080] Beneficial effects:

[0081] (1) High-precision three-dimensional reconstruction: It can not only provide the three-dimensional structure of fundus images, but also provide ophthalmologists with a more intuitive perspective for pathological analysis, which is of great value, especially in the diagnosis of high myopia.

[0082] (2) Reduce equipment costs and improve accessibility: Reduce reliance on high-cost equipment such as OCT, and three-dimensional reconstruction can be achieved with only standard fundus photos, which makes three-dimensional reconstruction and analysis of fundus images universally feasible.

[0083] (3) Improve diagnostic efficiency and accuracy: Rapidly process fundus images and generate three-dimensional morphology, improving diagnostic efficiency; through accurate depth estimation, the system can help doctors detect risk factors early, especially in the early stages of diseases such as high myopia.

[0084] (4) Wide clinical applicability: It can adapt to image input under different equipment and shooting conditions, and can be widely used in a variety of clinical scenarios to meet the needs of different hospitals and different equipment.

[0085] The method for generating a 3D eye model based on a single fundus photograph, as proposed in this application, involves preprocessing the fundus photograph data to be reconstructed. The preprocessed fundus photograph data is then input into a preset depth estimation network model, which predicts depth map data. The depth map data is then converted into point cloud data, and a preset B-spline algorithm is used to reconstruct the 3D eye model based on the point cloud data. This method solves the problems of high cost and limited scanning range of existing OCT equipment, as well as insufficient resolution of depth maps output by existing AI models, which cannot meet clinical diagnostic needs. It greatly simplifies the process of generating a 3D eye model and improves generation efficiency.

[0086] Next, referring to the accompanying drawings, an apparatus for generating a three-dimensional model of the eyeball based on a single fundus photograph, according to an embodiment of this application, is described.

[0087] Figure 4 This is a block diagram of an apparatus for generating a three-dimensional model of the eyeball based on a single fundus photograph, according to an embodiment of this application.

[0088] like Figure 4 As shown, the device 10 for generating a three-dimensional model of the eyeball based on a single fundus photograph includes: a preprocessing module 100, a prediction module 200, and a reconstruction module 300.

[0089] The preprocessing module 100 is used to acquire fundus image data to be reconstructed and to preprocess the fundus image data to be reconstructed to obtain preprocessed fundus image data.

[0090] The prediction module 200 is used to input the preprocessed fundus photograph data into a preset depth estimation network model and use the preset depth estimation network model to predict the depth map data. The preset depth estimation network model includes a ViT-Hybrid encoder and a multi-task decoder, and the preset depth estimation network model is trained by a historical fundus photograph dataset and a historical three-dimensional eye scan dataset.

[0091] The reconstruction module 300 is used to convert depth map data into point cloud data, and based on the point cloud data, to reconstruct a three-dimensional model of the eyeball using a preset B-spline algorithm.

[0092] Optionally, in some embodiments, before inputting the preprocessed fundus image data into a preset depth estimation network model, the prediction module 200 is further configured to:

[0093] Obtain historical fundus photograph datasets and historical 3D eye scan datasets, where the data in the historical fundus photograph dataset corresponds one-to-one with the data in the historical 3D eye scan dataset;

[0094] The data in the historical fundus photograph dataset and the historical eyeball 3D scan dataset are pixel-level aligned using a preset geometric transformation algorithm and a preset feature point matching algorithm. The data augmentation operation is then performed on the processing results to obtain the target multimodal dataset.

[0095] The target multimodal dataset is divided into a training set and a validation set. Based on a preset loss function, the preset neural network is trained using the training set to obtain an initial depth estimation network model. The initial depth estimation network model is then validated using the validation set. The iterative training of the initial depth estimation network model ends when it meets the preset conditions, and the preset depth estimation network model is obtained. Otherwise, the weights of the preset loss function are dynamically adjusted and iterative training continues.

[0096] Optionally, in some embodiments, the preset loss function is:

[0097]

[0098] Where α is the weight of the scale-invariant logarithmic loss, β is the weight of the edge-aware loss, and γ is the weight of the semantic segmentation loss. For scale-invariant logarithmic loss, For edge perception loss, This is the semantic segmentation loss.

[0099] Optionally, in some embodiments, the prediction module 200 is specifically used for:

[0100] Multiscale features of preprocessed fundus image data were extracted using the ViT-Hybrid encoder.

[0101] Based on multi-scale features, a multi-task decoder is used to perform cross-layer feature fusion to obtain depth map data.

[0102] Optionally, in some embodiments, after inputting the preprocessed fundus image data into a preset depth estimation network model, the prediction module 200 is further configured to:

[0103] Based on preprocessed fundus images, the feature distribution of a preset depth estimation network model is dynamically adjusted using a preset normalization strategy.

[0104] It should be noted that the foregoing explanation of the method embodiment for generating a three-dimensional model of the eyeball based on a single fundus photograph also applies to the apparatus for generating a three-dimensional model of the eyeball based on a single fundus photograph in this embodiment, and will not be repeated here.

[0105] The apparatus for generating a 3D eye model based on a single fundus photograph, as proposed in this application, preprocesses the fundus photograph data to be reconstructed. The preprocessed fundus photograph data is then input into a preset depth estimation network model, which predicts depth map data. The depth map data is then converted into point cloud data, and a preset B-spline algorithm is used to reconstruct the 3D eye model based on the point cloud data. This solves the problems of high cost and limited scanning range of existing OCT equipment, and insufficient resolution of depth maps output by existing AI models, which cannot meet clinical diagnostic needs. It greatly simplifies the process of generating a 3D eye model and improves generation efficiency.

[0106] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0107] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0108] When the processor 502 executes the program, it implements the method for generating a three-dimensional model of the eyeball based on a single fundus photograph provided in the above embodiments.

[0109] Furthermore, electronic devices also include:

[0110] Communication interface 503 is used for communication between memory 501 and processor 502.

[0111] The memory 501 is used to store computer programs that can run on the processor 502.

[0112] The memory 501 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0113] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0114] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0115] The processor 502 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0116] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described method for generating a three-dimensional model of the eyeball based on a single fundus photograph.

[0117] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method for generating a three-dimensional model of the eyeball based on a single fundus photograph.

[0118] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0119] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0120] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for generating a three-dimensional model of the eyeball based on a single fundus photograph, characterized in that, Includes the following steps: Acquire fundus image data to be reconstructed, and preprocess the fundus image data to be reconstructed to obtain preprocessed fundus image data; The preprocessed fundus photograph data is input into a preset depth estimation network model, and the preset depth estimation network model is used to predict depth map data. The preset depth estimation network model includes a ViT-Hybrid encoder and a multi-task decoder, and the preset depth estimation network model is trained by a historical fundus photograph dataset and a historical three-dimensional eye scan dataset. The depth map data is converted into point cloud data, and a three-dimensional model of the eyeball is reconstructed based on the point cloud data using a preset B-spline algorithm. The ViT-Hybrid encoder is a hybrid encoder that combines a visual converter and a residual network. The residual network extracts local features, and the visual converter performs global feature modeling to extract multi-scale features. The multi-task decoder is based on a multi-task learning framework, jointly optimizing the depth estimation task and the semantic segmentation task, and generating the depth map data through cross-layer feature fusion.

2. The method according to claim 1, characterized in that, Before inputting the preprocessed fundus image data into the preset depth estimation network model, the method further includes: Obtain the historical fundus photograph dataset and the historical three-dimensional eye scan dataset, wherein the data in the historical fundus photograph dataset corresponds one-to-one with the data in the historical three-dimensional eye scan dataset; The data in the historical fundus photograph dataset and the historical eyeball 3D scan dataset are pixel-level aligned using a preset geometric transformation algorithm and a preset feature point matching algorithm. The data augmentation operation is then performed on the processing results to obtain the target multimodal dataset. The target multimodal dataset is divided into a training set and a validation set. Based on a preset loss function, a preset neural network is trained using the training set to obtain an initial depth estimation network model. The initial depth estimation network model is then validated using the validation set. The iterative training of the initial depth estimation network model ends when the initial depth estimation network model meets preset conditions, thus obtaining the preset depth estimation network model. Otherwise, the weights of the preset loss function are dynamically adjusted before iterative training continues.

3. The method according to claim 2, characterized in that, The preset loss function is: in, The weights for the scale-invariant logarithmic loss are... The weights for edge-aware loss are... The weights are for semantic segmentation loss. For scale-invariant logarithmic loss, For edge perception loss, This is the semantic segmentation loss.

4. The method according to claim 1, characterized in that, The step of inputting the preprocessed fundus image data into a preset depth estimation network model and using the preset depth estimation network model to predict depth map data includes: The ViT-Hybrid encoder is used to extract multi-scale features from the preprocessed fundus image data; Based on the multi-scale features, the multi-task decoder is used to perform cross-layer feature fusion to obtain the depth map data.

5. The method according to claim 1, characterized in that, After inputting the preprocessed fundus image data into the preset depth estimation network model, the method further includes: Based on the preprocessed fundus photograph data, the feature distribution of the preset depth estimation network model is dynamically adjusted using a preset normalization strategy.

6. A device for generating a three-dimensional model of the eyeball based on a single fundus photograph, characterized in that, include: The preprocessing module is used to acquire fundus image data to be reconstructed and to preprocess the fundus image data to be reconstructed to obtain preprocessed fundus image data. The prediction module is used to input the preprocessed fundus photograph data into a preset depth estimation network model and use the preset depth estimation network model to predict depth map data. The preset depth estimation network model includes a ViT-Hybrid encoder and a multi-task decoder, and the preset depth estimation network model is trained by a historical fundus photograph dataset and a historical three-dimensional eye scan dataset. The reconstruction module is used to convert the depth map data into point cloud data, and reconstruct a three-dimensional model of the eyeball based on the point cloud data using a preset B-spline algorithm. The ViT-Hybrid encoder is a hybrid encoder that combines a visual converter and a residual network. The residual network extracts local features, and the visual converter performs global feature modeling to extract multi-scale features. The multi-task decoder is based on a multi-task learning framework, jointly optimizing the depth estimation task and the semantic segmentation task, and generating the depth map data through cross-layer feature fusion.

7. The apparatus according to claim 6, characterized in that, Before inputting the preprocessed fundus image data into the preset depth estimation network model, the prediction module is further configured to: Obtain the historical fundus photograph dataset and the historical three-dimensional eye scan dataset, wherein the data in the historical fundus photograph dataset corresponds one-to-one with the data in the historical three-dimensional eye scan dataset; The data in the historical fundus photograph dataset and the historical eyeball 3D scan dataset are pixel-level aligned using a preset geometric transformation algorithm and a preset feature point matching algorithm. The data augmentation operation is then performed on the processing results to obtain the target multimodal dataset. The target multimodal dataset is divided into a training set and a validation set. Based on a preset loss function, a preset neural network is trained using the training set to obtain an initial depth estimation network model. The initial depth estimation network model is then validated using the validation set. The iterative training of the initial depth estimation network model ends when the initial depth estimation network model meets preset conditions, thus obtaining the preset depth estimation network model. Otherwise, the weights of the preset loss function are dynamically adjusted before iterative training continues.

8. An electronic device, characterized in that, include: The method for generating a three-dimensional model of the eyeball based on a single fundus photograph, as described in any one of claims 1-5, includes a memory, a processor, and a computer program stored in the memory and executable on the processor.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method for generating a three-dimensional model of the eyeball based on a single fundus photograph as described in any one of claims 1-5.

10. A computer program product, characterized in that, The method includes a computer program, which, when executed by a processor, is used to implement the method for generating a three-dimensional model of the eyeball based on a single fundus photograph as described in any one of claims 1-5.

Images