VF value multi-modal prediction method and system based on deep learning

Abstract

The invention relates to the field of medical image processing, and discloses a VF value multi-modal prediction method and system based on deep learning, and the method comprises the steps: data preparation; the method comprises the following steps: constructing a basic ophthalmology model based on a Vision Transform model; performing first-stage pre-training on the basic ophthalmology model by using the first data set to obtain a first pre-trained ophthalmology model; performing second-stage pre-training on the first pre-training ophthalmology model by using the second data set to obtain a second pre-training ophthalmology model, and taking the second pre-training ophthalmology model as an image encoder; the method comprises the steps that a unified multi-modal framework is constructed, the unified multi-modal framework comprises an image encoder, a text encoder, a multi-modal unified fusion module and a prediction module, and the multi-modal unified fusion module fuses an image token output by the image encoder and a text token output by the text encoder to obtain a fusion token; training the prediction module by adopting the fusion token to obtain a trained unified multi-modal framework; and importing to-be-predicted data into the framework, and outputting a predicted VF value. The visual field value prediction efficiency and accuracy can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a VF value multimodal prediction method and system based on deep learning. Background Art

[0002] Visual Field (VF) refers to the range of space a person can see assessed through visual examination. VF testing is a commonly used ophthalmic test to detect visual field defects caused by diseases. The prediction of VF values ​​is also a key component of early detection and long-term management of glaucoma.

[0003] Conventional visual field examination methods mainly include static visual field examination, dynamic visual field examination, contrast sensitivity visual field examination, etc. These methods require the high cooperation of the subjects during the examination. The subjects need to respond to the signals of the detection instrument, such as clicking a button to mark the time when the signal appears. In the actual detection process, the subjects may not be focused or operate improperly, the subjects may be unable to cooperate due to factors such as age and intelligence, and the detection instrument may be inaccurately calibrated, which greatly affects the accuracy of the test results. These uncertain factors lead to limitations in conventional visual field examination methods, such as strong subjectivity, low accuracy, and long detection time. Summary of the invention

[0004] The purpose of the embodiment of the present invention is to provide a multimodal tracking and management method for glaucoma based on deep learning, aiming to improve the prediction efficiency and accuracy of VF values ​​through advanced deep learning technology. The method includes a two-stage pre-training model based on VisionTransformer, which can not only integrate multimodal data, but also use pre-training technology to improve the accuracy and generalization ability of the model, while building a unified multimodal framework, and predicting VF values ​​by fusing image and text data through a multimodal unified fusion module. Through this method, the present invention can significantly improve the prediction efficiency and accuracy of visual field values.

[0005] In order to achieve the above object, the present invention provides a VF value multimodal prediction method based on deep learning, comprising:

[0006] Data preparation, including the first data set and the second data set;

[0007] Build a basic ophthalmology model based on the Vision Transformer model;

[0008] Using the first data set to perform a first-stage pre-training on the basic ophthalmology model to obtain a first pre-trained ophthalmology model;

[0009] Using the second data set to perform a second stage of pre-training on the first pre-trained ophthalmological model to obtain a second pre-trained ophthalmological model as an image encoder;

[0010] Constructing a unified multimodal framework, including the image encoder, the text encoder, a multimodal unified fusion module and a prediction module, wherein the multimodal unified fusion module fuses the image token output by the image encoder and the text token output by the text encoder to obtain a fused token;

[0011] The prediction module is trained using fusion tokens to obtain a trained unified multimodal framework;

[0012] Import the data to be predicted into the trained unified multimodal framework and output the predicted VF value.

[0013] Optionally, the first data set includes:

[0014] Image and biometric information, including age, intraocular pressure, and visual field defects;

[0015] The second dataset includes:

[0016] Color fundus photography images and region of interest images, where the region of interest includes the optic cup and optic disc areas.

[0017] Optionally, the first data set is used to perform a first-stage pre-training on the basic ophthalmological model to obtain a first pre-trained ophthalmological model, including:

[0018] Initialize the parameters of the basic ophthalmology model, including the number of Transformer layers, the number of hidden units, and the number of attention heads;

[0019] Normalizing the first data set and inputting it into the basic ophthalmology model;

[0020] Using the Adam optimization algorithm to update the basic ophthalmology model parameters;

[0021] Calculate the loss function and verify whether it meets the preset criteria. If so, stop training.

[0022] If the preset criteria are not met, repeating the step of updating the basic ophthalmology model parameters using the Adam optimization algorithm;

[0023] Using the second data set to perform second-stage pre-training on the first pre-trained ophthalmology model includes:

[0024] dividing the color fundus photography image into a plurality of patches;

[0025] Set the mask strategy;

[0026] Pass the input containing the patch embedding to the transform encoder, which outputs a feature map;

[0027] The feature map is pooled, and the pooled feature map is passed to a classification head for classification.

[0028] Optionally, setting the mask policy includes:

[0029] We mask some parts of each region at a ratio of 50%-80% to guide the model to focus on key areas in the input.

[0030] Optionally, the text encoder includes:

[0031] The text encoder converts structured text data into tokens that can be processed by the Transformer network;

[0032] The multimodal unified fusion module includes:

[0033] Self-attention mechanism and cross-attention mechanism; the multimodal unified fusion module has a fusion function, and the text tokens output by the text encoder are fused with the image tokens output by the image encoder through the self-attention mechanism and the cross-attention mechanism to obtain fused tokens;

[0034] The prediction module includes:

[0035] Field of view reconstruction model,The prediction head in the field of view reconstruction model maps the fusion token to the final output through a convolutional layer to obtain a predicted VF value.

[0036] Optionally, the self-attention mechanism expression is as follows:

[0037]

[0038] Where Q is the query, K is the key, V is the value, T is the sequence length, and d k is the dimension of the word vector;

[0039] The cross attention mechanism is expressed as formula (2) and (3):

[0040]

[0041] Among them, Q image is the image query, K text is the text key, V text is the text value, T is the sequence length, d k is the dimension of the word vector;

[0042]

[0043] Among them, Q text is a text query, Kimage is the image key, V image is the image value, T is the sequence length, d k is the dimension of the word vector.

[0044] Optionally, the prediction module is trained using fusion tokens, including:

[0045] inputting the fused token into a model of a prediction module;

[0046] The model of the prediction module extracts features from the input fusion tokens and predicts the reconstruction result of the field of view;

[0047] Calculating a loss function between the predicted reconstruction result and the true value;

[0048] The model parameters of the prediction module are updated according to the loss function result.

[0049] Optionally, the VF value multimodal prediction method based on deep learning further includes training a unified multimodal framework, including:

[0050] Get image and text tokens;

[0051] The image and text tokens are input into the multimodal unified fusion module for fusion to obtain fused tokens.

[0052] Importing the fusion token into a prediction module and outputting a predicted VF value;

[0053] Iterate the unified multimodal framework n times.

[0054] Optionally, getting image and text tokens includes:

[0055] An image encoder is used to extract the average value of tokens from the input image features as image tokens, and a text encoder is used to convert the input structured text data into text tokens.

[0056] On the other hand, the present invention also provides a VF value multimodal prediction system based on deep learning, wherein the system comprises a processor configured to execute any of the methods described above.

[0057] Through the above technical solution, the embodiment of the present invention provides a VF value multimodal prediction method based on deep learning, which builds a pre-trained model based on Vision Transformer, integrates multimodal data, and uses two-stage pre-training technology to improve the accuracy and generalization ability of the model. At the same time, a unified multimodal framework is built, and the image and text data are fused through a multimodal unified fusion module to predict the VF value. Compared with the prior art, the method provided by the present invention can significantly improve the efficiency and accuracy of visual field value prediction.

[0058] Other features and advantages of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0060] Figure 1 is a flowchart of a VF value multimodal prediction method based on deep learning according to an embodiment of the present invention;

[0061] Figure 2 is a framework diagram of an image encoder obtained by first pre-training and second pre-training according to an embodiment of the present invention;

[0062] Figure 3 is a unified multimodal framework diagram according to an embodiment of the present invention;

[0063] Figure 4 4 is a diagram of predicted visual field performance evaluation results according to one embodiment of the present invention. DETAILED DESCRIPTION

[0064] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the embodiment of the present invention.

[0065] For the English terms appearing in this invention, the corresponding Chinese counterparts are given in the following table for reference:

[0066] original Chinese version VF (Visual Field) Field of view Adam Stochastic Optimization Method with Adaptive Momentum SPE Selective pre-training enhancement MLP Multilayer Perceptron OCT measurements Optical coherence tomography measurements CFP Color Fundus Photography

[0067] like Figure 1 FIG. 1 is a flowchart of a multimodal prediction method for VF value based on deep learning according to an embodiment of the present invention. Figure 1 In the method, the method may include the following steps:

[0068] In step S10, data is prepared, including a first data set and a second data set;

[0069] In step S11, a basic ophthalmology model based on the Vision Transformer model is constructed;

[0070] In step S12, the first data set is used to perform a first-stage pre-training on the basic ophthalmological model to obtain a first pre-trained ophthalmological model;

[0071] In step S13, the first pre-trained ophthalmological model is pre-trained in the second stage using the second data set to obtain a second pre-trained ophthalmological model as an image encoder;

[0072] In step S14, a unified multimodal framework is constructed, including the image encoder, the text encoder, the multimodal unified fusion module and the prediction module, wherein the multimodal unified fusion module fuses the image token output by the image encoder and the text token output by the text encoder to obtain a fused token;

[0073] In step S15, the prediction module is trained using the fusion token to obtain a trained unified multimodal framework;

[0074] In step S16, the data to be predicted is imported into the trained unified multimodal framework, and the predicted VF value is output.

[0075] In this Figure 1 In the method shown, step S10 can be used for data preparation. In this embodiment, the first data set and the second data set are both multimodal data sets, and the first data set includes images and biological information, wherein the biological information includes age, intraocular pressure, and visual field defects. The second data set includes color fundus photography images and region of interest images, wherein the region of interest includes the optic cup and optic disc areas. In this example, the second data set can be Gla-2K or GRAPE. Gla-2K is a hybrid data set designed for pre-training of glaucoma tasks in the field of ophthalmology, and the data comes from publicly available data sets including ARCIM, Drishti-GS, and ODIR. GRAPE is a data set designed for tracking and management of glaucoma, including information on visual fields, fundus images, intraocular pressure (IOP), OCT measurements, and other relevant clinical data.

[0076] Step S11 can construct a basic ophthalmology model based on the Vision Transformer model.

[0077] In step S12, the first data set is used to perform a first stage pre-training on the basic ophthalmology model to obtain a first pre-trained ophthalmology model. After the first stage pre-training, the basic ophthalmology model can acquire general ophthalmology knowledge. The first stage pre-training method may include the following steps:

[0078] S12.1. Initialize the parameters of the basic ophthalmology model, including the number of Transformer layers, the number of hidden units, and the number of attention heads. In this example, the model can be composed of 12 Transformer layers, each with 768 hidden units and 12 attention heads.

[0079] S12.2, normalize the first data set and input it into the basic ophthalmology model. In this example, the normalization process may be to adjust the input image to A x B pixels, preferably 224 x 224 pixels.

[0080] S12.3, update the basic ophthalmology model parameters using the Adam optimization algorithm. In this example, the Adam optimizer may be used for training, the initial learning rate is α=10-4, and the cosine annealing scheduler is used for decay.

[0081] S12.4, calculate the loss function and verify whether it meets the preset standard. If it meets the preset standard, stop training. In this example, weight decay can be applied and the model is pre-trained for 400 cycles.

[0082] If the preset standard is not met, the step of updating the basic ophthalmic model parameters using the Adam optimization algorithm is repeated.

[0083] In step S13, the second data set may be used to perform second-stage pre-training on the first pre-trained ophthalmology model. By using the second data set for the second-stage pre-training, the first pre-trained model may acquire glaucoma-related expertise, and the second-stage pre-training adopts selective pre-training enhancement (SPE), uses an attention-guided mask method to focus on areas with higher information density, and guides the model's attention to focus on features related to glaucoma-related tasks.

[0084] The second stage pre-training method may include the following steps:

[0085] S13.1. Divide the color fundus photography image into multiple patches. In this example, the input image may be divided into patches of fixed size, which will serve as the basic processing units of the model.

[0086] S13.2, set mask strategy. In this example, some parts of each region may be masked at a ratio of 50%-80% to guide the model to focus on key areas in the input, preferably at a ratio of 75%.

[0087] S13.3. Pass the input containing the patch embedding to the transformer encoder, which outputs a feature map. In this example, an image encoder is used to convert the image patch into an embedding representation. Image features are extracted through convolutional layers and batch normalization layers.

[0088] S13.4, pooling the feature map, passing the pooled feature map to the classification head for classification. In this example, the classification token is added to the patch embedding at the input stage, and after passing through the Transformer encoder, its representation is used for classification, and the final classification token embedding is passed to the multi-layer perceptron (MLP) head to produce the category logic, where the MLP consists of one or more fully connected layers with learnable parameters, followed by a softmax activation to produce the probability on the category label.

[0089] In step S14, a unified multimodal framework is constructed, including the image encoder, the text encoder, the multimodal unified fusion module and the prediction module, wherein the multimodal unified fusion module fuses the image token output by the image encoder and the text token output by the text encoder to obtain a fusion token. The image encoder converts the region of interest data, fundus image data, etc. into image tokens that can be processed by the Transformer network. This process usually involves image preprocessing, feature extraction and dimensionality reduction, so that high-dimensional image data can be effectively processed by the deep learning model. The image encoder can be constructed based on a convolutional neural network (CNN), and its powerful spatial feature extraction capability is used to convert image data into a series of feature vectors, i.e., image tokens. The text encoder is responsible for processing structured text data, such as the patient's age, clinical information, and intraocular pressure (IOP) data. Although these data are not as intuitive as images, they contain rich information. The text encoder converts text data into a series of text tokens. This process usually involves natural language processing (NLP) technology, such as word embedding and sentence encoding, so that text data can be input into the model in numerical form. The multimodal unified fusion module includes: self-attention mechanism and cross-attention mechanism. The self-attention mechanism allows the model to focus on different parts of the same modality when processing data of one modality, while the cross-attention mechanism enables the model to establish connections between different modalities and promote information interaction and fusion. The multimodal unified fusion module has a fusion function, which can fuse the text tokens output by the text encoder with the image tokens output by the image encoder through the self-attention mechanism and the cross-attention mechanism to obtain fusion tokens. These tokens contain the joint features of the image and text. This fusion not only retains the key information of each modality, but also can establish connections between different modalities, thereby supporting cross-modal tasks. The prediction module includes: a visual field reconstruction model. The prediction head in the visual field reconstruction model maps the fusion token to the final output through a convolution layer to obtain a predicted VF value. In this process, the role of the prediction head is to convert the fused features into the final prediction results. For example, in image super-resolution reconstruction, the role of the convolution layer is to extract image block features, nonlinear mapping and reconstruction. In the visual field reconstruction model, the prediction head uses a similar mechanism to predict and reconstruct the visual field using fusion features. In this example, the self-attention mechanism is expressed as formula (1):

[0090]

[0091] Among them, Q (query) is the query, K (key) is the key, V (value) is the value, T is the sequence length, d k is the dimension of the word vector;

[0092] The cross attention mechanism is expressed as formula (2) and (3):

[0093]

[0094] Among them, Q image is the image query, K text is the text key, V text is the text value, T is the sequence length, d k is the dimension of the word vector;

[0095]

[0096] Among them, Q text is a text query, K image is the image key, V image is the image value, T is the sequence length, d k is the dimension of the word vector.

[0097] In step S15, the prediction module is trained using the fusion token, including:

[0098] S15.1, inputting the fusion token into the model of the prediction module;

[0099] S15.2, the model of the prediction module extracts features from the input fusion token and predicts the VF value;

[0100] S15.3. Calculate the loss function between the predicted VF value and the true value; in this example, the loss function may be the mean square error loss (L_MSE), and the calculation formula thereof is as follows:

[0101]

[0102] VF i is the true VF value of the i-th group of samples, is the predicted VF value of the i-th group of samples.

[0103] S15.4, updating the model parameters of the prediction module according to the loss function result. In this example, the model parameters are adjusted by minimizing the square error between the predicted VF value and the true value, thereby improving the prediction accuracy of the model.

[0104] The VF value multimodal prediction method based on deep learning according to one embodiment of the present invention further includes: training a unified multimodal framework, including the following steps:

[0105] Get image and text tokens. In this example, an image encoder is used to extract the average value of tokens from the input image features as image tokens, and a text encoder is used to convert the input structured text data into text tokens.

[0106] The image and text tokens are input into the multimodal unified fusion module for fusion to obtain fused tokens.

[0107] Importing the fusion token into a prediction module and outputting a predicted VF value;

[0108] Iterate the unified multimodal framework n times.

[0109] The prediction performance of the VF value multimodal prediction method based on deep learning according to one embodiment of the present invention is evaluated:

[0110] Evaluation criteria: Root mean square error (RMSE), mean absolute error (MAE) and coefficient of determination (R 2 ) as an accuracy assessment metric. The root mean square error (RMSE) measures the average size of the prediction errors, and because the differences are squared, it penalizes larger errors more than smaller errors. The mean absolute error (MAE) calculates the average of the absolute differences between the predicted and true values. The coefficient of determination R 2 Evaluates the proportion of variance in the true visual field explained by the model predictions. R 2 Values ​​range from 0 to 1, with values ​​closer to 1 indicating a better model fit. Together, these metrics allow for an overall assessment of the model’s predictive accuracy, with RMSE and MAE providing insight into the magnitude of the error, while R 2 Provides a measure of how well the model explains the variance in the true horizon.

[0111] The root mean square error (RMSE) is expressed as formula (4):

[0112]

[0113] The mean absolute error (MAE) expression is as shown in formula (5):

[0114]

[0115] Coefficient of determination (R 2 ) is expressed as formula (6)

[0116]

[0117] in is the predicted field of view, VF i is the actual field of view, N is the total number of pixels or areas, It is the average value of the real field of view.

[0118] Comparison of evaluation results: The VF value multimodal prediction method of the present invention was used as the experimental group, and the MAE model pre-trained on the ImageNet-1K

[30] and mmOphth-v1 datasets was used as the control group.

[0119] Comparing the results of the experimental group and the control group, Figure 4 As shown in the table, on the CFP data, the two-stage pre-training multimodal fusion framework of the present invention has achieved significant improvements: RMSE is reduced by 0.568, MAE error is reduced by 0.592, R 2 The performance is significantly improved by 0.121 compared to the baseline. On the CFP-ROI data, the three indicators are improved by 0.571, 0.581 and 0.117 respectively. In addition, the model of the present invention outperforms the ImageNet-1K and mmOphth-v1 pre-trained models, as well as the single-stage training model, highlighting the superior performance of the present invention in predicting the field of view.

[0120] Through the above technical solution, the embodiment of the present invention provides a multimodal prediction method of VF value based on deep learning. The method constructs a pre-training model based on Vision Transformer, integrates multimodal data, and uses two-stage pre-training technology to enable the model to adapt from a wide range of ophthalmology fields to the professional field of glaucoma management, fill the gaps in the existing technology, and improve the accuracy and generalization ability of the model. At the same time, a unified multimodal framework is constructed, and the image and text data are fused through a multimodal unified fusion module to predict the VF value. Through this method, the present invention can significantly improve the efficiency and accuracy of visual field value prediction, and provide a new and effective means for glaucoma management.

[0121] The multimodal prediction method for VF values ​​provided by the present invention has broad application prospects. It can not only improve the efficiency of glaucoma diagnosis and better cover early glaucoma patients, but also improve patient experience, increase patient accessibility and convenience, and promote the development of precision medicine.

[0122] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0123] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0124] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0126] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0127] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0128] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0129] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0130] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A VF value multimodal prediction method based on deep learning, characterized in that: include, Data preparation, including the first data set and the second data set; Build a basic ophthalmology model based on the Vision Transformer model; Using the first data set to perform a first-stage pre-training on the basic ophthalmology model to obtain a first pre-trained ophthalmology model; Using the second data set to perform a second stage of pre-training on the first pre-trained ophthalmological model to obtain a second pre-trained ophthalmological model as an image encoder; Constructing a unified multimodal framework, including the image encoder, the text encoder, a multimodal unified fusion module and a prediction module, wherein the multimodal unified fusion module fuses the image token output by the image encoder and the text token output by the text encoder to obtain a fused token; The prediction module is trained using fusion tokens to obtain a trained unified multimodal framework; Import the data to be predicted into the trained unified multimodal framework and output the predicted VF value.

2. The VF value multimodal prediction method based on deep learning according to claim 1, characterized in that: The first dataset includes: Image and biometric information, including age, intraocular pressure, and visual field defects; The second dataset includes: Color fundus photography images and region of interest images, where the region of interest includes the optic cup and optic disc areas.

3. The VF value multimodal prediction method based on deep learning according to claim 1, characterized in that: The first data set is used to perform a first-stage pre-training on the basic ophthalmology model to obtain a first pre-trained ophthalmology model, including: Initialize the parameters of the basic ophthalmology model, including the number of Transformer layers, the number of hidden units, and the number of attention heads; Normalizing the first data set and inputting it into the basic ophthalmology model; Using the Adam optimization algorithm to update the parameters of the basic ophthalmology model; Calculate the loss function and verify whether it meets the preset standard. If it meets the preset standard, stop training to obtain the first training ophthalmology model; If the preset standard is not met, the step of updating the basic ophthalmic model parameters using the Adam optimization algorithm is repeated.

4. The VF value multimodal prediction method based on deep learning according to claim 2, characterized in that: Using the second data set to perform second-stage pre-training on the first pre-trained ophthalmology model includes: dividing the color fundus photography image into a plurality of patches; Set a masking strategy to mask some parts of each region at a ratio of 50%-80% to guide the model to focus on key areas in the input; Pass the input containing the patch embedding to the Transformer encoder, outputting a feature map; The feature map is pooled, and the pooled feature map is passed to a classification head for classification.

5. The VF value multimodal prediction method based on deep learning according to claim 1, characterized in that: Text encoders include: The text encoder converts structured text data into tokens that can be processed by the Transformer network; The multimodal unified fusion module includes: Self-attention mechanism and cross-attention mechanism; the multimodal unified fusion module has a fusion function, and the text tokens output by the text encoder are fused with the image tokens output by the image encoder through the self-attention mechanism and the cross-attention mechanism to obtain fused tokens; The prediction module includes: Field of view reconstruction model,The prediction head in the field of view reconstruction model maps the fusion token to the final output through a convolutional layer to obtain a predicted VF value.

6. The VF value multimodal prediction method based on deep learning according to claim 5, characterized in that: The self-attention mechanism is expressed as formula (1): Where Q is the query, K is the key, V is the value, T is the sequence length, and d k is the dimension of the word vector; The cross attention mechanism is expressed as formula (2) and (3): Among them, Q image is the image query, K text is the text key, V text is the text value, T is the sequence length, d k is the dimension of the word vector; Among them, Q text is a text query, K image is the image key, V image is the image value, T is the sequence length, d k is the dimension of the word vector.

7. The VF value multimodal prediction method based on deep learning according to claim 1, characterized in that: The prediction module is trained using fusion tokens, including: inputting the fused token into a model of a prediction module; The model of the prediction module extracts features from the input fusion tokens and predicts the reconstruction result of the field of view; Calculating a loss function between the predicted reconstruction result and the true value; The model parameters of the prediction module are updated according to the result of the loss function.

8. The VF value multimodal prediction method based on deep learning according to claim 1, characterized in that: It also includes training of a unified multimodal framework, including: Get image and text tokens; Input the image token and text token into the multimodal unified fusion module for fusion to obtain a fusion token; Importing the fusion token into a prediction module and outputting a predicted VF value; Iterate the unified multimodal framework n times.

9. The VF value multimodal prediction method based on deep learning according to claim 8, characterized in that: Getting image and text tokens involves: An image encoder is used to extract the average value of tokens from the input image features as image tokens, and a text encoder is used to convert the input structured text data into text tokens.

10. A VF value multimodal prediction system based on deep learning, characterized in that: The system includes a processor configured to execute a unified multimodal fusion framework obtained by the method according to any one of claims 1 to 9.

Images