A cataract surgery vision prediction system based on multimodal fusion network
Through a multimodal fusion network based on Transformer, combined with a convolutional neural network and a multimodal cross attention network, the problem of difficulty in multimodal information fusion in the existing technology is solved, and high-precision for vision prediction after cataract surgery is achieved and low cost.
Patent Information
- Application Number
- CN202210491704.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-07
- Publication Date
- 2025-02-14
- Estimated Expiration
- 2042-05-07
AI Technical Summary
The existing post-cataract vision prediction technology is difficult to effectively integrate multimodal information, resulting in high computational cost and insufficient prediction accuracy.
A multimodal fusion network based on Transformer is adopted to extract image features through a convolutional neural network and fuse it with preoperative vision information. A multimodal cross attention network is used to perform deep information fusion to reduce computing costs.
Accurate prediction of vision after cataract surgery is achieved, calculation costs and landing costs are reduced, and auxiliary support is provided for doctors to evaluate the necessity of surgery.
Smart Images
Figure CN114782394B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of medical image processing, and in particular relates to a cataract surgery postoperative vision prediction system based on a multimodal fusion network. Background Art
[0002] Cataract is the most common blinding eye disease that affects the vision of the elderly, and its clinical manifestation is decreased vision. For this public health problem, cataract surgery is the only means of treatment. In clinical practice, patients may have other eye diseases that together lead to decreased vision, making cataract surgery not absolutely effective in restoring vision. Therefore, predicting whether the patient's postoperative vision can be effectively restored before surgery is helpful for doctors to evaluate the necessity of surgery.
[0003] In clinical practice, doctors mainly observe the fundus state based on OCT images to evaluate what eye diseases the patient has, whether it will affect the effect of cataract surgery, and predict whether the vision can be restored after surgery. However, due to the complexity of the fundus state, this morphological task is time-consuming and laborious for doctors, and the judgment is not necessarily accurate. In order to help doctors make a diagnosis, computer algorithms can provide auxiliary analysis. With the development of deep learning, its powerful representation learning ability has been widely used in medical image scenarios. By establishing a deep learning model, there is no need to manually select fixed statistical features. The model can efficiently and automatically extract the feature information of the image and make a morphological diagnosis. This deep learning method can effectively reduce the cost of clinical diagnosis and save doctors time and effort.
[0004] However, in medical application scenarios, data usually has multiple modalities, such as images, cases, clinical information, etc. The information contained in different modalities is different, and there is a potential dependency between modalities. Therefore, how to design a multimodal fusion model so that it can effectively fuse multimodal information is a key issue. Multimodal fusion is also an important research direction in deep learning. At present, most of the existing methods are based on brute force vectorization splicing. Through the fully connected layer, the multimodal data is weighted summed to establish a linear relationship. This method is easy to implement but lacks deep fusion. Transformer is a mainstream framework that has begun to rise. It can be independent of the input modality and dimension. It unifies the modality through serialization, that is, splicing the modal information as blocks. Through the self-attention mechanism, the three matrices of the science system are generated, namely Query, Key and Value. Not only does it add randomness, but it can also learn the dependency between modalities through the attention fusion between blocks. Therefore, the network framework based on Transformer can more effectively fuse information at a deeper level than brute force splicing. On the other hand, the quality of the fusion effect also depends to a certain extent on the specific fusion method, including early fusion, mid-term fusion, late fusion, etc.
[0005] Many studies have shown that mid-term fusion is the most effective method. However, no matter what fusion method is used, the current Transformer-based fusion will have the problem of computational redundancy. That is, the fusion process will generate a lot of matrix calculations, resulting in a high computational cost. Currently, few studies have focused on and solved this problem. Summary of the invention
[0006] The present invention discloses a cataract surgery vision prediction system based on a multimodal fusion network, which can automatically extract image modal features, fuse them with preoperative vision modal features, learn potential inter-modal relationships, and achieve accurate prediction of cataract surgery vision at a relatively low computational cost.
[0007] A cataract postoperative vision prediction system based on a multimodal fusion network, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, wherein a trained cataract postoperative vision prediction model is stored in the computer memory, and the cataract postoperative vision prediction model uses a convolutional neural network as a backbone network, is used to receive image data in input data, and extract features from a transverse scanning image and a longitudinal scanning image respectively; and then the learned features are serialized and input into a multimodal cross attention network;
[0008] The multimodal cross-attention network that inputs the image features of the two views is merged with the vision block encoded by the preoperative vision, the randomly generated regression block and the cross block, and deep feature extraction and information fusion within the range of view features are performed through the Transformer layer; the regression blocks of the two views are spliced into the fully connected layer to output the predicted postoperative vision;
[0009] When the computer processor executes the computer program, the following steps are implemented:
[0010] Obtain the patient's preoperative OCT images and preoperative visual acuity data, wherein the OCT images include two views: horizontal scanning and vertical scanning;
[0011] The patient's preoperative OCT images and preoperative vision data are input into the trained cataract postoperative vision prediction model for feature extraction and prediction, and finally the patient's postoperative vision value is output.
[0012] The system of the present invention effectively integrates multimodal information through the Transformer architecture, can accurately predict postoperative visual acuity for cataract surgery, and provide assistance to doctors in evaluating the necessity of cataract surgery. At the same time, it avoids redundant attention calculations through the exchange of cross blocks, thereby reducing computing costs and implementation costs.
[0013] Furthermore, the convolutional neural network includes 4 convolutional layers and 3 pooling layers. After each convolutional layer, the image is compressed to half the resolution. The feature map of the image is extracted through the final convolutional layer, mapped to blocks of uniform dimension through a fully connected layer, and then spliced to obtain a feature sequence.
[0014] Furthermore, the input of the multimodal cross-attention network includes two views, namely, horizontal scanning and vertical scanning. The image feature sequence of each view is regarded as an image block, which is spliced with the vision block after preoperative vision encoding, the randomly generated regression block and the cross block and input into the multimodal cross-attention network. The network includes 12 Transformer layers. In the first 6 layers, the modal features of each view are fused through attention. In the last 6 layers, the information between views is interacted through the exchange of cross blocks, thereby fusing the features between views. Finally, the regression blocks of the two views are averaged, and then a fully connected layer is used to predict postoperative vision.
[0015] Furthermore, the Transformer layer consists of 2 normalization layers, 1 attention layer and 1 feedforward neural network layer.
[0016] Furthermore, the training process of the cataract surgery postoperative vision prediction model is as follows:
[0017] (1) Obtain the patient's transverse and longitudinal scan images as sample data. After image screening, obtain the visual acuity annotations of the follow-up visit one month after surgery. The annotated sample data are divided into a training set, a validation set, and a test set in proportion;
[0018] (2) Build a cataract postoperative vision prediction model CTT-Net, input a group of samples, extract features from the sample images, and perform feature fusion and postoperative vision prediction together with the preoperative vision; the prediction value range is 0-3.0.
[0019] (3) Compare the output prediction results with the actual postoperative visual acuity annotations, use the MAE function as the loss of the current model, and pass it back to the model to update the parameters in the model until the model converges;
[0020] (4) When the set MAE threshold is reached or the specified number of iterations is reached, the model stops updating, the training is completed, and the results are output.
[0021] In step (1), the image screening rule is: based on whether there are missing values and whether the image is clear, samples with severe distortion that cannot be estimated by clinicians and samples with missing images, preoperative visual acuity or postoperative visual acuity are eliminated.
[0022] In step (3), the back propagation rule is used to calculate the gradient of the model, and the stochastic gradient descent method is used to update the model parameters.
[0023] During the model training process, the indicators of the validation set are calculated after a certain number of iterations. The indicators include mean square error (MSE), mean absolute error (MAE), and accuracy (ACC). The hyperparameters of the model are adjusted according to the evaluation results of the validation set. After the model training is completed, the model parameters saved from many iterations are selected based on the evaluation results. The selected model parameters are loaded and tested on the test set. The indicators are the same as those of the validation set to obtain the evaluation results of the final model.
[0024] Compared with the prior art, the present invention has the following beneficial effects:
[0025] 1. Different from the existing postoperative prediction technology of cataract surgery, which only predicts postoperative vision through images, the present invention introduces preoperative vision as additional information supervision, promotes the extraction of image features and inter-modal feature fusion, and realizes high-precision prediction of postoperative vision of cataract surgery.
[0026] 2. Different from the existing full-attention feature fusion technology through fully connected layers or Transformer, the present invention introduces a cross-attention mechanism to perform intra-view feature fusion based on a single cross-block, and achieves inter-view feature fusion by exchanging between views, while avoiding redundant attention and reducing computational costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 is a training flow chart of a post-cataract surgery vision prediction model in an embodiment of the present invention;
[0028] Figure 2 is a network structure diagram of a convolutional neural network in an embodiment of the present invention;
[0029] Figure 3 : is a network structure diagram of a multimodal cross attention network in an embodiment of the present invention;
[0030] Figure 4 Schematic diagram of a multimodal cross-attention mechanism for cross-block exchange in an embodiment of the present invention. DETAILED DESCRIPTION
[0031] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.
[0032] Step 1: OCT image screening
[0033] The sources of OCT images include Heidelberg, Colin and Zeiss machines. The image resolution of Heidelberg is 768*496 (pixel) and the scanning depth is 9600*2000 (um); the image resolution of Zeiss is 938*625 (pixel) and the scanning depth is 6000*2000 (um); the image resolution of Colin is 1020*960 (pixel) and the scanning depth is 10000*2000 (um). Due to the low quality of OCT images of some patients, or the lack of preoperative or postoperative vision, these samples were excluded.
[0034] Step 2: Data Division
[0035] The transverse OCT images, longitudinal OCT images and preoperative visual acuity were regarded as sample pairs, and all sample pairs were divided into training set, validation set and test set with a ratio of 6:2:2.
[0036] Step 3: Training of multimodal cataract surgery postoperative vision prediction model
[0037] Each set of sample pairs of the multimodal cataract postoperative visual acuity prediction model of the present invention is inputted as preoperative OCT images (including transverse scans and longitudinal scans) and preoperative visual acuity.
[0038] The overall training process is as follows Figure 1 As shown. Multiple groups (generally a batch = 16) of sample pairs are input into the multimodal cataract postoperative vision prediction model to obtain the final predicted postoperative vision. The multimodal cataract postoperative vision prediction model includes a convolutional neural network (feature extraction network) and a multimodal cross-attention network.
[0039] The OCT image is firstly extracted through the convolutional neural network Resnet-18. The network structure is as follows: Figure 2 As shown in the figure, it includes 4 convolutional layers and 3 pooling layers. After each convolutional layer, the image will be compressed to half the resolution. After the final convolutional layer, the feature map of the image is extracted. The feature map includes concentrated and extracted image features. It is encoded, that is, mapped to blocks of uniform dimension (set to 128 dimensions) through a fully connected layer and then spliced to obtain a feature sequence. The feature sequence is input into the multimodal cross attention network, as shown in Figure 3 As shown in the figure, the image feature block is concatenated with the preoperative vision encoding block, regression block and cross block to obtain the horizontal and vertical image feature sequences. Among them, the preoperative vision encoding block provides additional modality information, the regression block is used to extract global information for final vision prediction, and the cross block is used to fuse information from different modalities and views.
[0040] The horizontal and vertical image features are first fused within the view through the Transformer in the early stage. That is, in the first 6 layers of the Transformer encoder, the cross block remains unchanged so that the cross block can fully extract the information within the view. The last 6 layers fuse the information between the views by exchanging the cross block within the two views. The fusion steps are as follows: Figure 4 As shown, the cross-block positions in the feature sequences of the two views are exchanged repeatedly each time through a layer. The information between views is supplemented and fused through the exchange of cross-blocks to realize a multimodal cross-attention mechanism. Finally, the regression blocks of the two views are extracted and fused to finally obtain the predicted postoperative vision.
[0041] When training the model, the postoperative vision prediction results are made as close as possible to the actual postoperative vision. The loss function is used to generate the loss of the model's prediction results and the real results, and the model parameters are updated through the gradient backpropagation algorithm. Then the next set of samples is input into the segmentation model and the training of the above steps is continued. Finally, the training is terminated until the model converges, that is, when the loss function no longer decreases.
[0042] Step 4: Validation and testing of the multimodal cataract surgery visual acuity prediction model
[0043] The model will be verified on the validation set after every certain number of training steps. That is, the data in the validation set is input into the trained model, and the loss value, accuracy, recall rate, precision and other indicators are observed to adjust hyperparameters such as learning rate to make the model's generalization performance better.
[0044] Testing is to simulate the actual implementation scenario, that is, input the data in the test set into the trained model, select and load the model parameters that perform best on the validation set, and observe the final accuracy, recall rate, precision and other indicators to determine the model performance and actual effect.
[0045] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A cataract postoperative visual acuity prediction system based on a multimodal fusion network, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, characterized in that: The computer memory stores a trained cataract surgery vision prediction model, which uses a convolutional neural network as a backbone network to receive image data in input data, extract features from the horizontal scanning image and the vertical scanning image respectively; and then serializes the learned features and inputs them into a multimodal cross attention network; The multimodal cross-attention network includes 12 Transformer layers. The first 6 layers fuse the features of each modality in the view through attention, and the last 6 layers exchange cross blocks to make the information between views interactive, thereby fusing the features between views; the multimodal cross-attention network that inputs the image features of the two views is respectively merged with the vision blocks encoded by the preoperative vision, the randomly generated regression blocks and the cross blocks, and deep feature extraction and information fusion within the range of view features are performed through the Transformer layer; the regression blocks of the two views are spliced into the fully connected layer, and the predicted postoperative vision is output; wherein the regression block is used to extract global information for final vision prediction, and the cross block is used to fuse information of different modalities and views; When the computer processor executes the computer program, the following steps are implemented: Obtain the patient's preoperative OCT images and preoperative visual acuity data, wherein the OCT images include two views: horizontal scanning and vertical scanning; The patient's preoperative OCT images and preoperative vision data are input into the trained cataract postoperative vision prediction model for feature extraction and prediction, and finally the patient's postoperative vision value is output.
2. The cataract surgery vision prediction system based on multimodal fusion network according to claim 1 is characterized in that: The convolutional neural network includes 4 convolutional layers and 3 pooling layers. After each convolutional layer, the image is compressed to half the resolution. The feature map of the image is extracted through the final convolutional layer, mapped to blocks of uniform dimension through a fully connected layer, and then spliced to obtain a feature sequence.
3. The cataract surgery vision prediction system based on multimodal fusion network according to claim 1, characterized in that: The Transformer layer consists of 2 normalization layers, 1 attention layer and 1 feedforward neural network layer.
4. The cataract surgery vision prediction system based on multimodal fusion network according to claim 1, characterized in that: The training process of the post-cataract surgery vision prediction model is as follows: (1) Obtain the patient's transverse and longitudinal scan images as sample data. After image screening, obtain the visual acuity annotations of the follow-up visit one month after surgery. The annotated sample data are divided into a training set, a validation set, and a test set in proportion; (2) Build a cataract postoperative vision prediction model CTT-Net, input a group of samples, extract features from the sample images, and perform feature fusion and postoperative vision prediction together with preoperative vision; the prediction value range is 0-3.0; (3) Compare the output prediction results with the actual postoperative visual acuity annotations, use the MAE function as the loss of the current model, and pass it back to the model to update the parameters in the model until the model converges; (4) When the set MAE threshold is reached or the specified number of iterations is reached, the model stops updating, the training is completed, and the results are output.
5. The cataract surgery vision prediction system based on multimodal fusion network according to claim 4, characterized in that: In step (1), the image screening rule is: based on whether there are missing values and whether the image is clear, samples with severe distortion that cannot be estimated by clinicians and samples that lack images, preoperative visual acuity, or postoperative visual acuity are eliminated.
6. The cataract surgery vision prediction system based on multimodal fusion network according to claim 4, characterized in that: In step (3), the back propagation rule is used to calculate the gradient of the model, and the stochastic gradient descent method is used to update the model parameters.
7. The cataract surgery vision prediction system based on multimodal fusion network according to claim 4, characterized in that: During the model training process, the indicators of the validation set are calculated after a certain number of iterations. The indicators include mean square error (MSE), mean absolute error (MAE), and accuracy (ACC). The hyperparameters of the model are adjusted according to the evaluation results of the validation set. After the model training is completed, the model parameters saved from many iterations are selected based on the evaluation results. The selected model parameters are loaded and tested on the test set. The indicators are the same as those of the validation set to obtain the evaluation results of the final model.