Intelligent cataract vision prediction system and method based on slit lamp image

Abstract

The invention discloses an intelligent cataract vision prediction system and method based on slit lamp images, and relates to the field of medical image processing, the system comprises an information acquisition module, a data processing module, a model construction module and a vision prediction module; the information acquisition module is used for acquiring BCVA of a cataract patient before an operation and BCVA of the cataract patient after the operation for one month; meanwhile, the information acquisition module is also used for acquiring an anterior segment slit lamp image of the cataract patient; the data processing module is used for preprocessing the anterior segment slit lamp image to obtain processed data; the model construction module is used for constructing a vision prediction model based on the pre-operation BCVA and the BCVA and the anterior segment slit lamp graph of the post-operation month; and the vision prediction module is used for predicting the vision of the cataract patient before and after the operation by using the vision prediction model. According to the method, the BCVA and postoperative BCVA of the cataract patient under different crystal opacity degrees can be quickly and accurately predicted, and the working efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a cataract vision intelligent prediction system and method based on slit lamp images. Background Art

[0002] Currently, surgery is the only effective way to treat cataracts. Most patients recover their vision well after surgery if their visual function is not impaired, but some patients do not respond well due to other fundus diseases. However, in the case of severe lens opacity, the refractive medium is turbid, and the fundus is often unclear before surgery, so doctors cannot fully understand the fundus condition before surgery.

[0003] Preoperative visual assessment of cataract patients is of great significance, as it provides key information for doctors. Traditional assessment methods usually rely on the doctor's experience. Ophthalmologists use slit lamp microscopy and Lens Opacity Classification System III (LOCS III) to determine the degree of lens opacity, evaluate the consistency between the best-corrected visual acuity (BCVA) corresponding to the degree of opacity and the actual BCVA, determine whether cataract is the main cause of the patient's vision loss, identify potential fundus diseases, assist in assessing the patient's surgical risks, and develop personalized treatment plans for patients.

[0004] At the same time, due to the physician's clinical experience, the traditional method of visual acuity assessment is highly subjective. In addition, in patients with other fundus diseases, their postoperative visual acuity is usually limited by the fundus condition, which makes it difficult for doctors to make accurate predictions about the extent of postoperative visual recovery in such patients.

[0005] The advent of the era of artificial intelligence (AI) has made it possible to solve the above problems and contradictions. As a hot field of artificial intelligence, deep learning simulates the physiological functions of the human brain through multi-layer and multi-division artificial neural networks, and understands complex patterns from a wide range of data sets. Compared with traditional algorithms, it can better handle large-scale and high-dimensional data, and is widely used in computer vision, image processing, natural language processing and other fields. The diagnosis and treatment of ophthalmic diseases are highly dependent on imaging technology and visual data, which makes deep learning have great prospects in the field of ophthalmology. Through the study of large-scale medical image data, accurate segmentation of lesion areas, and extraction of image features, deep learning realizes the automated analysis of medical images and provides accurate and efficient medical services.

[0006] At present, the diagnosis and grading of cataracts are mainly based on narrow-bandwidth photography, diffuse light photography and retro-illumination of anterior segment slit lamp images, and evaluation with reference to LOCS III standard images, while making a rough prediction of visual acuity based on the different forms of the lens in the image; however, there is no artificial intelligence system that uses anterior segment slit lamp images to evaluate current BCVA and predict postoperative BCVA. Therefore, in order to address the above issues, it is of great significance to provide a method and system for quickly, efficiently, simply and accurately predicting the visual acuity of cataract patients before and after surgery. Summary of the invention

[0007] The present invention aims to overcome the deficiencies of the above-mentioned prior art and provide a cataract vision intelligent prediction system and method based on slit lamp images. According to the anterior segment slit lamp examination images, the preoperative BCVA and postoperative BCVA can be quickly and accurately evaluated. In the case of limited medical resources, important information can be efficiently provided to ophthalmologists, thereby formulating a reasonable treatment plan for patients.

[0008] To achieve the above-mentioned purpose, the present invention provides a cataract vision intelligent prediction system based on slit lamp images, comprising: an information acquisition module, a data processing module, a model building module and a vision prediction module;

[0009] The information acquisition module is used to obtain the BCVA of cataract patients before surgery and BCVA one month after surgery; at the same time, the information acquisition module is also used to obtain the slit lamp image of the anterior segment of the eye of the cataract patient, and the slit lamp image of the anterior segment of the eye includes: slit lamp narrow broadband illumination, slit lamp diffuse illumination and slit lamp back illumination;

[0010] The data processing module is used to pre-process the anterior segment slit lamp image to obtain the processed data, and the process includes: firstly, screening the anterior segment slit lamp image to obtain a screening image; then, constructing a ROI segmentation model and inputting the screening image into the ROI segmentation model and extracting the ROI image; adjusting the resolution of the ROI image to 224*224, and normalizing the pixel values ​​of the ROI image; converting the visual acuity data of the BCVA before surgery and the BCVA one month after surgery into the logMAR recording method, and performing normalization processing at the same time;

[0011] The model building module is used to build a vision prediction model based on the BCVA before surgery, the BCVA one month after surgery, and the anterior segment slit lamp image. The vision prediction model includes: a multimodal model and a multi-task model; wherein the multimodal model uses EfficientNetV2-M as a basic model, processes the images of each modality in the anterior segment slit lamp image, and extracts the corresponding high-dimensional feature vector; in the multi-task model, the preoperative predicted vision and the postoperative predicted vision are output simultaneously, and the loss functions of the two tasks are both mean square error MSELoss, and the total loss is the weighted sum of the losses of the two tasks;

[0012] The vision prediction module is used to predict the vision of cataract patients before and after surgery using the vision prediction model.

[0013] Preferably, the anterior segment of the eye slit lamp image is acquired using a narrow slit light illumination method or a rear reflective illumination method, and the requirements for shooting the anterior segment of the eye slit lamp image include: the patient's pupil is fully dilated before shooting, the pupil diameter is dilated to more than 5 mm, the pupil area of ​​the image is centered and unobstructed, and the focus is adjusted so that the inspection area is clearly visible.

[0014] Preferably, the data processing module uses ResNet34 as the front-end network for feature extraction and U-Net as the back-end network for feature fusion to construct the ROI segmentation model based on Mask R-CNN;

[0015] The screened image is input into the ROI segmentation model. First, it is passed through the ResNet34 front end, with Relu as the activation function, and residual learning between the two layers is performed during the feature extraction process. ResNet34 performs a total of 5 downsamplings, consisting of a 7×7 convolution operation and 4 convolution blocks. After the last convolution in the coding area, the obtained feature map will be used as the input of the back-end U-Net. In the decoder path, the 4 upsampling processes of the original U-Net are retained. Each decoding layer will splice the feature map of the current layer with the feature map of the corresponding coding layer to achieve feature fusion at the channel level. Finally, upsampling is performed again to restore the original image size to obtain the feature map;

[0016] The generated feature map is input into the RPN region proposal network of Mask R-CNN to generate candidate regions. The candidate regions are aligned and normalized through the ROI Align layer to ensure that the size pixels in each candidate region are consistent. Finally, classification and regression are performed through the fully connected layer, and mask prediction is generated through the fully convolutional network. In the prediction stage, when the confidence output by the Mask R-CNN model is greater than the threshold, it means that the ROI is detected.

[0017] Preferably, the EfficientNetV2-M convolutional neural network includes: a Fused-MBConv module and an MBConv module;

[0018] First, the input ROI image is preprocessed through a standard convolution layer with a stride of 2, a size of 3x3, and a channel number of 24 to quickly reduce the resolution of the image. Then, it is passed through 4 groups of Fused-MBConv modules and 4 groups of MBConv modules for feature learning and representation. The Fused-MBConv module is composed of a combination of 3x3 convolution, SE module, and 1x1 convolution, and the MBConv module is composed of a stack of 1x1 convolution, 3x3 depth convolution, SE module, and 1x1 convolution.

[0019] After that, the ROI image passes through a 1x1 convolution to increase the number of channels, and passes through a global average pooling layer to average the feature maps of each channel to obtain a 1x1 feature vector, which is input into the fully connected layer;

[0020] Finally, at the back end, the features of different modalities are fused by splicing fully connected layers and ReLU activation functions; the fused feature vector is passed as input to two multi-layer MLPs, one MLP is used to predict preoperative vision, and the other MLP is used to predict postoperative vision.

[0021] Preferably, in the multi-task model, the preoperative predicted visual acuity and the postoperative predicted visual acuity are output simultaneously, and the loss functions of the two tasks are both mean square error MSELoss, and the total loss is the weighted sum of the losses of the two tasks, that is:

[0022] Loss = 0.5*Loss1 + 0.5*Loss2,

[0023] In the formula, Loss1 is the loss function of task 1; Loss2 is the loss function of task 2; the coefficient 0.5 is the weighting coefficient; Loss is the total loss function, that is, the weighted sum of the losses of the two tasks. The total loss function is used to measure the overall error in the model training process. The model minimizes the loss function and improves model performance by adjusting parameters.

[0024] Preferably, the vision prediction model uses the Adam optimizer with a batch size of 16; under the Ubuntu 20 operating system, programming is based on the Pytorch framework, and the training set and validation set are input to complete the training and internal verification of the model.

[0025] The present invention also provides a method for intelligent prediction of cataract vision based on slit lamp images, which is applied to the above system and comprises the following steps:

[0026] S1. Obtain the BCVA of the cataract patient before surgery and the BCVA one month after surgery; and simultaneously obtain the anterior segment slit lamp image of the cataract patient, wherein the anterior segment slit lamp image includes: slit lamp narrow broadband illumination, slit lamp diffuse illumination, and slit lamp back illumination;

[0027] S2. Preprocessing the anterior segment slit lamp image to obtain the processed data, the process comprising: firstly screening the anterior segment slit lamp image to obtain a screening image; then, constructing a ROI segmentation model and inputting the screening image into the ROI segmentation model and extracting the ROI image; adjusting the resolution of the ROI image to 224*224, and normalizing the pixel values ​​of the ROI image; converting the visual acuity data of the BCVA before surgery and the BCVA one month after surgery into the logMAR recording method, and performing normalization processing at the same time;

[0028] S3. A vision prediction model is constructed based on the BCVA before surgery, the BCVA one month after surgery, and the anterior segment slit lamp image. The vision prediction model includes: a multimodal model and a multi-task model; wherein the multimodal model uses EfficientNetV2-M as a basic model, processes the images of each modality in the anterior segment slit lamp image, and extracts the corresponding high-dimensional feature vector; in the multi-task model, the loss functions of the two tasks are both mean square error MSELoss, and the total loss is the weighted sum of the two task losses, for simultaneously outputting the preoperative predicted vision and the postoperative predicted vision.

[0029] S4. Use the vision prediction model to predict the vision of cataract patients before and after surgery.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] (1) The present invention can quickly and accurately predict the BCVA and postoperative BCVA corresponding to different degrees of lens opacity based on preoperative anterior segment slit lamp images by using deep learning algorithms to train models on a large amount of data, thereby assisting clinicians to quickly and accurately evaluate the main causes and surgical efficacy.

[0032] (2) The present invention can realize intelligent prediction of postoperative vision of cataract patients with concurrent fundus diseases, better understand the different degrees of postoperative vision recovery under different fundus conditions, assist doctors in clinical decision-making, provide personalized diagnosis and treatment plans, help patients establish reasonable surgical expectations, and avoid unnecessary medical disputes. It has important scientific research value.

[0033] (3) By using computer vision technology, the present invention can quickly process a large amount of image data, significantly improve work efficiency, reduce the workload of ophthalmologists, reduce medical costs, and shorten patients' consultation time. Due to its strong universality and easy understanding and mastery, the present invention can be widely used in medical and health institutions at all levels, helping to narrow the differences in diagnosis and treatment levels between different medical institutions and improve the overall medical environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0035] Figure 1 A schematic diagram of the system structure of an embodiment of the present invention;

[0036] Figure 2 Schematic diagram of the ROI segmentation model framework according to an embodiment of the present invention;

[0037] Figure 3 Schematic diagram of a multi-model network framework according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] Embodiment 1

[0041] like Figure 1 As shown, it is a schematic diagram of the system structure of this embodiment, including: an information collection module, a data processing module, a model building module and a vision prediction module;

[0042] The following will describe in detail how the present invention solves the technical problems in practical work in conjunction with this embodiment.

[0043] First, the information acquisition module is used to obtain the BCVA of cataract patients before surgery and one month after surgery; at the same time, the information acquisition module is also used to obtain the slit lamp image of the anterior segment of the eye of the cataract patient. In this embodiment, the slit lamp image of the anterior segment of the eye includes: slit lamp narrow broadband illumination, slit lamp diffuse illumination and slit lamp back illumination.

[0044] Specifically, the anterior segment of the subject's eyes was examined using a Zeiss slit lamp microscope to obtain anterior segment slit lamp images, which included narrow broadband illumination of the slit lamp, diffuse illumination of the slit lamp, and back illumination of the slit lamp. Specifically, the chin rest of the slit lamp and the patient's head position were adjusted, the image position was adjusted so that the pupil area was located in the center of the examination interface, the focus point was adjusted so that the examination site was clearly visible, and one narrow broadband illumination of the slit lamp, one diffuse illumination of the slit lamp, and one back illumination of the slit lamp were obtained. Among them, the requirements for shooting the anterior segment slit lamp image are as follows: before taking the photo, the patient's pupil must be fully dilated, the pupil diameter must be dilated to more than 5mm, the pupil area of ​​the image must be centered and unobstructed, and the focus point must be adjusted so that the examination site is clearly visible. When using the narrow slit light illumination method, the slit width is set to 0.2-0.3mm, and the mirror arm angle is about 30°; when using rear reflective illumination, the slit is short rectangular or half-moon shaped, with a height of 2mm, a width of a narrow slit, about 1mm, and a mirror arm angle of about 90°. This embodiment includes 1603 samples with a total of 4809 images. Afterwards, the subjects' preoperative BCVA and BCVA one month after surgery were collected as preoperative and postoperative visual acuity data for subsequent processing.

[0045] Then the data processing module pre-processes the eye image data to obtain processed data.

[0046] First, the anterior segment slit lamp images were screened, requiring that the eye structure be clearly focused, the pupil area be centered and unobstructed, the pupil diameter be dilated to more than 5 mm, and the lens structure be clearly visible.

[0047] The degree of cataract lesions and the impact on vision in the anterior segment slit lamp image mainly depend on the lens in the pupil area, which accounts for a small proportion of the narrow-bandwidth slit lamp image. Therefore, it is necessary to first segment the meaningful area (i.e., region of interest, ROI) in the screened image (hereinafter referred to as the screened image). The specific steps include:

[0048] First, labelme software was used to annotate 100 ROIs of slit lamp narrow broadband illumination, slit lamp diffuse illumination and slit lamp post-illumination. The extracted ROI was treated as an instance segmentation problem, and ResNet34 was used as the front-end network for feature extraction, and U-Net was used as the back-end network for feature fusion to construct a ROI (Region of Interest) segmentation model based on Mask R-CNN.

[0049] The filtered image is input into the ROI segmentation model, first through the ResNet34 front end, with Relu as the activation function, and residual learning between the two layers is performed during the feature extraction process. ResNet34 performs a total of 5 downsamplings, consisting of a 7×7 convolution operation and 4 convolution blocks. After the last convolution in the encoding area, the obtained feature map will be used as the input of the back-end U-Net. In the decoder path, the original U-Net's 4 upsampling processes are retained. Each decoding layer will splice the feature map of the current layer with the feature map of the corresponding encoding layer to achieve feature fusion at the channel level. Finally, one upsampling is performed to restore the original image size to obtain the feature map.

[0050] The generated feature map is input into the RPN region proposal network of Mask R-CNN to generate candidate regions. The candidate regions are aligned and normalized through the ROI Align layer to ensure that the size pixels in each candidate region are consistent. Finally, classification and regression are performed through the fully connected layer, and mask prediction is generated through the fully convolutional network. In the prediction stage, if the confidence of the output of the Mask R-CNN model is greater than a certain threshold (set to 0.75 in this system), it means that the ROI is detected. Using the developed ROI segmentation model, the ROI image segmentation of all data sets is extracted as the input of the prediction model. The framework diagram of the ROI segmentation model is shown below. Figure 2 shown.

[0051] The resolution of the segmented ROI image was adjusted to 224*224, and the pixel values ​​of the image were normalized; then, the visual acuity of the predicted data (visual acuity data before and after surgery) was converted to the logMAR recording method and normalized. The data sets obtained in the above two steps were randomly divided into training set, validation set, and test set at the subject level, with a split ratio of 75:10:15.

[0052] Use the model building module to build a deep learning-based vision prediction model.

[0053] The vision prediction model includes a multimodal model and a multitask model. In this embodiment, three modal data, namely, slit lamp narrow broadband illumination, slit lamp diffuse illumination, and slit lamp post-illumination, are used as input, so a multimodal model needs to be constructed; at the same time, the output prediction data includes preoperative vision and postoperative vision, so a multitask model needs to be constructed. The specific process is as follows:

[0054] Select EfficientNetV2-M as the basic model, process the images of each modality, and extract the corresponding high-dimensional feature vectors. The EfficientNetV2-M convolutional neural network includes: Fused-MBConv module and MBConv module.

[0055] First, the input ROI image is subjected to feature preprocessing through a standard convolutional layer with a stride of 2, a size of 3x3, and a channel number of 24 to quickly reduce the resolution of the image, and then subjected to feature learning and representation through 4 groups of Fused-MBConv modules and 4 groups of MBConv modules as shown in the figure; the Fused-MBConv module is composed of a combination of 3x3 convolution, SE module, and 1x1 convolution, and the MBConv module is composed of a stack of 1x1 convolution, 3x3 depth convolution, SE module, and 1x1 convolution.

[0056] After that, the ROI image passes through a 1x1 convolution to increase the number of channels, and passes through a global average pooling layer to average the feature maps of each channel to obtain a 1x1 feature vector. The feature vector is input to the fully connected layer.

[0057] Finally, at the back end, the features of different modalities are fused by splicing the fully connected layer and the ReLU activation function; the fused feature vector is used as input and passed to two multi-layer perceptrons (MLPs), one MLP is used to predict preoperative vision and the other MLP is used to predict postoperative vision. Figure 3 shown.

[0058] In the multi-task model, the BCVA before surgery and the BCVA one month after surgery were input into the multi-task model for training.

[0059] The output of the multi-task model is the output of the preoperative predicted visual acuity and the postoperative predicted visual acuity. The loss functions of the two tasks are both mean square error MSELoss, and the total loss is the weighted sum of the losses of the two tasks, that is:

[0060] loss = 0.5*loss1 + 0.5*loss2,

[0061] In the formula, Loss1 is the loss function of task 1; Loss2 is the loss function of task 2; the coefficient 0.5 is the weighting coefficient; Loss is the total loss function, that is, the weighted sum of the losses of the two tasks. The total loss function is used to measure the overall error in the model training process. The model minimizes the loss function and improves model performance by adjusting parameters.

[0062] In this embodiment, the vision prediction model uses the Adam optimizer with a batch size of 16. Under the Ubuntu 20 operating system, programming is based on the Pytorch framework, the training set and the validation set are input, and 500 epochs are run on two GPUs with 24G of video memory to complete the training and internal verification of the model.

[0063] The performance of the model is evaluated using the test set. The evaluation indicators are the mean absolute error (MAE) and the coefficient of determination (R).2 , mean square error MSE.

[0064] Finally, the vision prediction module uses the vision prediction model to predict the vision of cataract patients before and after surgery.

[0065] Embodiment 2

[0066] This embodiment also provides a method for intelligently predicting cataract vision based on slit lamp images, the steps comprising:

[0067] S1. Obtain the BCVA of the cataract patient before surgery and the BCVA one month after surgery; and simultaneously obtain the anterior segment slit lamp image of the cataract patient, wherein the anterior segment slit lamp image includes: slit lamp narrow broadband illumination, slit lamp diffuse illumination, and slit lamp back illumination;

[0068] S2. Preprocessing the anterior segment slit lamp image to obtain the processed data, the process comprising: firstly screening the anterior segment slit lamp image to obtain a screening image; then, constructing a ROI segmentation model and inputting the screening image into the ROI segmentation model and extracting the ROI image; adjusting the resolution of the ROI image to 224*224, and normalizing the pixel values ​​of the ROI image; converting the visual acuity data of the BCVA before surgery and the BCVA one month after surgery into the logMAR recording method, and performing normalization processing at the same time;

[0069] S3. A vision prediction model is constructed based on the BCVA before surgery, the BCVA one month after surgery, and the anterior segment slit lamp image. The vision prediction model includes: a multimodal model and a multi-task model; wherein the multimodal model uses EfficientNetV2-M as a basic model, processes the images of each modality in the anterior segment slit lamp image, and extracts the corresponding high-dimensional feature vector; in the multi-task model, the loss functions of the two tasks are both mean square error MSELoss, and the total loss is the weighted sum of the two task losses, for simultaneously outputting the preoperative predicted vision and the postoperative predicted vision.

[0070] S4. Use the vision prediction model to predict the vision of cataract patients before and after surgery.

[0071] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A cataract vision intelligent prediction system based on slit lamp images, characterized in that: include: Information collection module, data processing module, model building module and vision prediction module; The information acquisition module is used to obtain the BCVA of cataract patients before surgery and BCVA one month after surgery; at the same time, the information acquisition module is also used to obtain the slit lamp image of the anterior segment of the eye of the cataract patient, and the slit lamp image of the anterior segment of the eye includes: slit lamp narrow broadband illumination, slit lamp diffuse illumination and slit lamp back illumination; The data processing module is used to pre-process the anterior segment slit lamp image to obtain the processed data, and the process includes: firstly, screening the anterior segment slit lamp image to obtain a screening image; then, constructing a ROI segmentation model and inputting the screening image into the ROI segmentation model and extracting the ROI image; adjusting the resolution of the ROI image to 224*224, and normalizing the pixel values ​​of the ROI image; converting the visual acuity data of the BCVA before surgery and the BCVA one month after surgery into the logMAR recording method, and performing normalization processing at the same time; The model building module is used to build a vision prediction model based on the BCVA before surgery, the BCVA one month after surgery, and the anterior segment slit lamp image. The vision prediction model includes: a multimodal model and a multi-task model; wherein the multimodal model uses EfficientNetV2-M as a basic model, processes the images of each modality in the anterior segment slit lamp image, and extracts the corresponding high-dimensional feature vector; in the multi-task model, the preoperative predicted vision and the postoperative predicted vision are output simultaneously, and the loss functions of the two tasks are both mean square error MSELoss, and the total loss is the weighted sum of the losses of the two tasks; The vision prediction module is used to predict the vision of cataract patients before and after surgery using the vision prediction model.

2. The cataract vision intelligent prediction system based on slit lamp images according to claim 1, characterized in that: The anterior segment of the eye slit lamp image is obtained using a narrow slit light illumination method or a rear reflective illumination method. The requirements for shooting the anterior segment of the eye slit lamp image include: the patient's pupil is fully dilated before shooting, the pupil diameter is dilated to more than 5 mm, the pupil area of ​​the image is centered and unobstructed, and the focus is adjusted so that the inspection area is clearly visible.

3. The cataract vision intelligent prediction system based on slit lamp images according to claim 1, characterized in that: The data processing module uses ResNet34 as the front-end network for feature extraction and U-Net as the back-end network for feature fusion to construct the ROI segmentation model based on Mask R-CNN; The screened image is input into the ROI segmentation model. First, it is passed through the ResNet34 front end, with Relu as the activation function, and residual learning between the two layers is performed during the feature extraction process. ResNet34 performs a total of 5 downsamplings, consisting of a 7×7 convolution operation and 4 convolution blocks. After the last convolution in the coding area, the obtained feature map will be used as the input of the back-end U-Net. In the decoder path, the 4 upsampling processes of the original U-Net are retained. Each decoding layer will splice the feature map of the current layer with the feature map of the corresponding coding layer to achieve feature fusion at the channel level. Finally, upsampling is performed again to restore the original image size to obtain the feature map; The generated feature map is input into the RPN region proposal network of Mask R-CNN to generate candidate regions; The candidate regions are aligned and normalized through the ROIAlign layer to ensure that the size pixels in each candidate region are consistent. Finally, classification and regression are performed through the fully connected layer, and mask prediction is generated through the fully convolutional network. In the prediction stage, when the confidence output by the Mask R-CNN model is greater than the threshold, it means that the ROI is detected.

4. The cataract vision intelligent prediction system based on slit lamp images according to claim 1, characterized in that: The EfficientNetV2-M convolutional neural network includes: Fused-MBConv module and MBConv module; First, the input ROI image is preprocessed through a standard convolution layer with a stride of 2, a size of 3x3, and a channel number of 24 to quickly reduce the resolution of the image. Then, it is passed through 4 groups of Fused-MBConv modules and 4 groups of MBConv modules for feature learning and representation. The Fused-MBConv module is composed of a combination of 3x3 convolution, SE module, and 1x1 convolution, and the MBConv module is composed of a stack of 1x1 convolution, 3x3 depth convolution, SE module, and 1x1 convolution. After that, the ROI image passes through a 1x1 convolution to increase the number of channels, and passes through a global average pooling layer to average the feature maps of each channel to obtain a 1x1 feature vector, which is input into the fully connected layer; Finally, at the back end, the features of different modalities are fused by splicing fully connected layers and ReLU activation functions; the fused feature vector is passed as input to two multi-layer MLPs, one MLP is used to predict preoperative vision, and the other MLP is used to predict postoperative vision.

5. The cataract vision intelligent prediction system based on slit lamp images according to claim 1, characterized in that: In the multi-task model, the preoperative predicted visual acuity and the postoperative predicted visual acuity are output simultaneously. The loss functions of the two tasks are both mean square error MSELoss, and the total loss is the weighted sum of the two task losses, that is: Loss = 0.5*Loss1 + 0.5*Loss2, In the formula, Loss1 is the loss function of task 1; Loss2 is the loss function of task 2; the coefficient 0.5 is the weighting coefficient; Loss is the total loss function, that is, the weighted sum of the losses of the two tasks. The total loss function is used to measure the overall error in the model training process. The model minimizes the loss function and improves model performance by adjusting parameters.

6. The cataract vision intelligent prediction system based on slit lamp images according to claim 1, characterized in that: The vision prediction model uses the Adam optimizer with a batch size of 16. Under the Ubuntu 20 operating system, it is programmed based on the Pytorch framework, and the training set and validation set are input to complete the training and internal validation of the model.

7. A method for intelligent prediction of cataract vision based on slit lamp images, the method being applied to the system according to any one of claims 1 to 6, characterized in that the steps include: S1. Obtain the BCVA of the cataract patient before surgery and the BCVA one month after surgery; and simultaneously obtain the anterior segment slit lamp image of the cataract patient, wherein the anterior segment slit lamp image includes: slit lamp narrow broadband illumination, slit lamp diffuse illumination, and slit lamp back illumination; S2. Preprocessing the anterior segment slit lamp image to obtain the processed data, the process comprising: firstly screening the anterior segment slit lamp image to obtain a screening image; then, constructing a ROI segmentation model and inputting the screening image into the ROI segmentation model and extracting the ROI image; adjusting the resolution of the ROI image to 224*224, and normalizing the pixel values ​​of the ROI image; converting the visual acuity data of the BCVA before surgery and the BCVA one month after surgery into the logMAR recording method, and performing normalization processing at the same time; S3. A vision prediction model is constructed based on the BCVA before surgery, the BCVA one month after surgery, and the anterior segment slit lamp image. The vision prediction model includes: a multimodal model and a multi-task model; wherein the multimodal model uses EfficientNetV2-M as a basic model, processes the images of each modality in the anterior segment slit lamp image, and extracts the corresponding high-dimensional feature vector; in the multi-task model, the loss functions of the two tasks are both mean square error MSELoss, and the total loss is the weighted sum of the two task losses, for simultaneously outputting the preoperative predicted vision and the postoperative predicted vision. S4. Use the vision prediction model to predict the vision of cataract patients before and after surgery.

Images