An ophthalmic disease state prediction method based on ViT and neural network

Abstract

The application discloses an ophthalmic disease state prediction method based on a ViT and a neural network, relates to the technical field of ophthalmic disease state prediction, and comprises the following steps: inputting an eye fundus image data set of a preprocessed test eye fundus image into a residual convolution module to generate a global feature calculation model and into a ViT-based coding module to generate a local feature calculation model; inputting the preprocessed eye fundus image data set into the local feature calculation model to determine a global feature information atlas of the eye fundus image data set and into the local feature calculation model to determine a local feature information atlas of the eye fundus image data set; inputting the global feature information atlas and the local feature information atlas into a feature fusion layer to generate a multi-channel feature information atlas; and inputting the multi-channel feature information atlas into a state prediction module to determine an eye fundus image prediction result in the test eye fundus image. Global feature information and local feature information of the predicted eye fundus image are considered, and the accuracy of the prediction result is ensured.

Description

Technical Field

[0001] This invention relates to the field of ophthalmic disease state prediction technology, and in particular to an ophthalmic disease state prediction method based on ViT and neural networks. Background Technology

[0002] In existing technologies, when predicting the risk of diabetic retinopathy in fundus images, the focus is on using neural networks for iterative training to obtain global feature information in the image. However, local feature information of the image itself is easily overlooked, including but not limited to image location feature information and image edge feature information. Furthermore, the accuracy and efficiency of using neural networks for training alone no longer meet current practical needs.

[0003] In 2020, Google proposed the VisionTransformer (ViT) as a novel image processing model architecture, based on the characteristics of natural processing and image processing. Since its introduction, it has received widespread attention from the industry due to its superior performance compared to previous models. However, VisionTransformer lacks the ability to generalize bias in image processing, which makes it easy to ignore global feature information in images.

[0004] When predicting diabetes risk using fundus images, both global and local feature information of the fundus images are crucial. Existing technologies cannot take into account both global and local feature information of fundus images, resulting in low accuracy and incomplete prediction results. Summary of the Invention

[0005] The purpose of this invention is to provide a method for predicting the state of ophthalmic diseases based on ViT and neural networks, in order to solve the problem that the prediction results are inaccurate and incomplete in the prior art because it is impossible to take into account both global and local feature information of the fundus image.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A method for predicting the state of ophthalmic diseases based on ViT and neural networks, comprising:

[0008] The fundus image dataset to be tested is input into the data preprocessing module to determine the preprocessed fundus image dataset; the fundus image dataset includes multiple fundus images;

[0009] The preprocessed fundus image dataset is input into the residual convolution module to generate a global feature calculation model;

[0010] The preprocessed fundus image dataset is input into the ViT-based encoding module to generate a local feature calculation model;

[0011] The preprocessed fundus image dataset is input into the global feature calculation model to determine the global feature information map of the fundus image dataset; the global feature information map includes global texture features, global color features, global size features, and global edge features of the fundus image;

[0012] The preprocessed fundus image dataset is input into the local feature calculation model to determine the local feature information map of the fundus image dataset; the local feature information map includes local texture features, local color features, local size features, and local edge features of the fundus image;

[0013] The global feature information map and the local feature information map are input into the feature fusion layer to generate a multi-channel feature information map;

[0014] The multi-channel feature information map is input into the state prediction module to determine the fundus image prediction result in the fundus image to be tested; the fundus image prediction result includes those with ophthalmic diseases and those without ophthalmic diseases.

[0015] Optionally, the preprocessed fundus image dataset is input into the residual convolution module to generate a global feature calculation model, specifically including:

[0016] The residual convolution module is trained using the preprocessed fundus image dataset to generate global features of the preprocessed fundus image dataset;

[0017] The global feature information is learned and superimposed to determine the global feature calculation model.

[0018] Optionally, the global feature information is:

[0019] f = wx + bias;

[0020] In the formula, f represents the global feature information of the preprocessed fundus image dataset, x represents the preprocessed fundus image dataset, w represents the weight coefficient in iterative training, and bias represents the preset bias value.

[0021] Optionally, the preprocessed fundus image dataset is input into a ViT-based encoding module to generate a local feature calculation model, specifically including:

[0022] Using a ViT-based encoding module, the image information in the preprocessed first fundus image dataset is divided into multiple slices; the image information includes the pixel information of the image;

[0023] The multiple slices are position-encoded, and each slice is input into a 1*1 dimensionality reduction layer in the order of position encoding to perform dimensionality transformation and determine the first image feature information of each slice.

[0024] The position code and image information of each slice are input into the encoder to extract local feature information of the slice and determine the local feature information image; the local feature information includes: slice texture features, slice edge features, slice color features and slice spatial structure features;

[0025] Multiple local feature information pieces are spliced ​​together according to positional encoding to determine the coded feature information of the complete fundus image;

[0026] The complete fundus image encoded feature information is input into a 5*5 transposed convolutional layer. The size of the local feature information image is restored through the 5*5 transposed convolution operation to determine the local feature calculation model.

[0027] Optionally, the position code of each slice and the image information are input into an encoder to extract local feature information of the slices and determine the local feature information image, specifically including:

[0028] The position code of each input slice and the image information of the slice itself are normalized.

[0029] Through a multi-head attention layer, each slice after normalization is processed by linear combination of three types of vectors: value vector, key phasor, and query vector.

[0030] Each slice that has undergone normalization is combined with the information that has undergone linear combination processing by the multi-head attention layer to determine a combined information slice; the combined information slice is a slice that contains the information after linear combination processing.

[0031] Based on the positional encoding of each of the combined information slices, and arranging the combined information slices in the order of their positional encodings, a local feature information image is determined.

[0032] Optionally, through a multi-head attention layer, each slice after normalization is processed by linearly combining three types of vectors: value vector, key phasor, and query vector. Specifically, this includes:

[0033] For each slice after normalization, a linear combination of three types of vectors is performed: value vector, key phasor, and query vector.

[0034] Perform a 3x3 convolution operation and a 2x2 pooling operation on each slice after linear combination;

[0035] Multi-head tensor merging is performed on all slices after pooling to determine complete image feature information;

[0036] A linear transformation is performed on the complete image feature information.

[0037] Optionally, the preprocessed fundus image dataset is input into the global feature calculation model to determine the global feature information map of the fundus image dataset, specifically including:

[0038] The global feature calculation model includes: convolutional layers, pooling layers, activation layers, and fully connected layers;

[0039] The preprocessed fundus image dataset is input into the convolutional layer to generate a feature information map.

[0040] The feature map is input into the pooling layer to generate a pooling feature information map.

[0041] The pooled feature map is input into the activation layer to generate a nonlinear learning feature information map.

[0042] The nonlinear learning feature map is input into the fully connected layer to generate a global feature information map of the fundus image dataset.

[0043] Optionally, the global feature information map and the local feature information map are input into the feature fusion layer to generate a multi-channel feature information map, specifically including:

[0044] The global feature information map and the local feature information map are input into the feature fusion layer, and 256 3*3*1 convolutional kernels are used to extract global features from the global feature information map to generate a one-dimensional global feature map.

[0045] Local features in the local feature information map are extracted using 256 3*3*1 convolutional kernels to generate a one-dimensional local feature map;

[0046] The one-dimensional global feature map and the one-dimensional local feature map are superimposed to generate a 3*3*512 multi-channel feature information map.

[0047] Optionally, the multi-channel feature information map is input to the state prediction module to determine the fundus image prediction result in the fundus image to be tested, specifically including:

[0048] The multi-channel feature information map is input into the state prediction module to determine the state prediction value of the retina in the fundus image to be tested, which indicates that the retina has an ophthalmic disease; the state prediction value includes 0 and 1.

[0049] When the state estimate is 0, it is determined that the image under test does not have an eye disease;

[0050] When the state estimate is 1, it is determined that the image to be tested suffers from an ophthalmological disease.

[0051] Optionally, the multi-channel feature information map is input to the state prediction module to determine the state prediction value of the retina in the fundus image to be tested, which includes:

[0052] The state prediction module includes a vector transformation layer, a feature combination layer, a convolutional layer, a pooling layer, an activation function layer, and a fully connected layer.

[0053] The multi-channel feature information map is input into the vector transformation layer to generate a multi-channel feature information map after vector dimension transformation.

[0054] The multi-channel feature information map after the vector dimension transformation is input into the feature combination layer to generate multi-channel feature information;

[0055] The multi-channel feature information is sequentially input into the convolutional layer, the pooling layer, the activation function layer, and the fully connected layer to generate a state prediction of retinal disease in the fundus image to be tested.

[0056] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0057] The ophthalmic disease state prediction method based on ViT and neural networks provided by this invention involves inputting a preprocessed fundus image dataset of the test fundus image into a residual convolution module to generate a global feature calculation model, and into a ViT-based encoding module to generate a local feature calculation model. The preprocessed fundus image dataset is then input into the local feature calculation model to determine the global feature information map of the fundus image dataset, and into the local feature calculation model to determine the local feature information map of the fundus image dataset. The global and local feature information maps are then input into a feature fusion layer to generate a multi-channel feature information map. Finally, the multi-channel feature information map is input into a state prediction module to determine whether the retina in the test fundus image has an ophthalmic disease or not. Because the global feature information in the test fundus image is learned through the residual convolution module, and the local feature information is learned through the VisionTransformer-based encoding module, both global and local feature information of the fundus image are predicted simultaneously, ensuring the accuracy and comprehensiveness of the prediction results. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 A flowchart of the ophthalmic disease state prediction method based on ViT and neural networks provided by the present invention;

[0060] Figure 2 This is a schematic diagram of the working process of the residual module in this invention;

[0061] Figure 3 This is a schematic diagram illustrating the working process of the ViT-based encoding module in this invention;

[0062] Figure 4 This is a schematic diagram of the encoder's operation process in this invention;

[0063] Figure 5 This is a schematic diagram illustrating the working process of the multi-head attention layer in this invention;

[0064] Figure 6 This is a schematic diagram illustrating the working process of the feature fusion layer in this invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] The purpose of this invention is to provide a method for predicting the state of ophthalmic diseases based on ViT and neural networks. It aims to learn global feature information in the fundus image under test through a residual convolution module and local feature information in the fundus image under test through a VisionTransformer-based encoding module. It simultaneously predicts both global and local feature information of the fundus image, ensuring the accuracy and comprehensiveness of the prediction results.

[0067] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0068] Example 1

[0069] like Figure 1As shown, the ophthalmic disease state prediction method based on ViT and neural networks in this embodiment includes:

[0070] Step 101: Input the fundus image dataset of the fundus images to be tested into the data preprocessing module to determine the preprocessed fundus image dataset; the fundus image dataset includes multiple fundus images. Specifically, the fundus image dataset of the fundus images to be tested is input into the data preprocessing module for standardization processing to determine fundus image datasets with the same image size, image pixel density, and image format. The preprocessing process enhances the fundus image data.

[0071] Step 102: Input the preprocessed fundus image dataset into the residual convolution module to generate a global feature calculation model. Specifically, the residual convolution module is trained using the preprocessed fundus image dataset to generate global features of the preprocessed fundus image dataset; global feature information is learned and superimposed to determine the global feature calculation model.

[0072] Specifically, the global feature information is: f = wx + bias.

[0073] In the formula, f represents the global feature information of the preprocessed fundus image dataset, x represents the preprocessed fundus image dataset, w represents the weight coefficient in iterative training, and bias represents the preset bias value.

[0074] Specifically, the preprocessed fundus image dataset is fed into the residual convolution module, where multiple iterations of training are performed to learn the global features of the preprocessed fundus image dataset in order to preserve global feature information. Finally, all global feature information is superimposed to form a global feature calculation model.

[0075] Step 103: Input the preprocessed fundus image dataset into the ViT-based encoding module to generate a local feature calculation model. Specifically, using the ViT-based encoding module, the image information in the preprocessed first fundus image dataset is divided into multiple slices; the image information includes the pixel information of the image; multiple slices are positionally encoded, and each slice is input into a 1*1 dimensionality reduction layer in the order of its positional encoding to perform dimensionality transformation and determine the first image feature information of each slice; the positional encoding and image information of each slice are input into the encoder to extract the local feature information of the slice and determine the local feature information image; the local feature information includes: slice texture features, slice edge features, slice color features, and slice spatial structure features; multiple local feature information are concatenated according to their positional encoding to determine the coded feature information of the complete fundus image; the complete fundus image coded feature information is input into a 5*5 transposed convolutional layer, and the size of the local feature information image is restored through the 5*5 transposed convolution operation to determine the local feature calculation model.

[0076] Among them, the ViT-based encoding module is a self-built encoding module, and it is an encoding module optimized based on the existing VisionTransformer structure.

[0077] Furthermore, the positional encoding and image information of each slice are input into the encoder for local feature information extraction to determine the local feature information image. Specifically, this includes: normalizing the positional encoding and image information of each input slice; using a multi-head attention layer, linearly combining the normalized slice with value vectors, key phasors, and query vectors; combining the information of each normalized slice with the information obtained from the linear combination process of the multi-head attention layer to determine the combined information slice; the combined information slice is a slice containing the information obtained from the linear combination process; and arranging the combined information slices according to the positional encoding order to determine the local feature information image.

[0078] Furthermore, through a multi-head attention layer, each slice after normalization is linearly combined using three types of vectors: value vector, key phasor, and query vector. Specifically, this includes: linearly combining the value vector, key phasor, and query vector for each slice after normalization; performing a 3*3 convolution operation and a 2*2 pooling operation on each slice after linear combination; performing multi-head tensor merging on all pooled slices to determine complete image feature information; and performing a linear transformation on the complete image feature information.

[0079] Step 104: Input the preprocessed fundus image dataset into the global feature calculation model to determine the global feature information map of the fundus image dataset; the global feature information map includes global texture features, global color features, global size features and global edge features of the fundus image.

[0080] Specifically, the global feature computation model includes: convolutional layers, pooling layers, activation layers, and a fully connected layer. The preprocessed fundus image dataset is input into the convolutional layers to generate a feature information map; the feature map is input into the pooling layers to generate a pooled feature information map; the pooled feature map is input into the activation layer to generate a non-linear learning feature information map; and the non-linear learning feature map is input into the fully connected layer to generate the global feature information map of the fundus image dataset. The global feature computation model specifically includes 3 convolutional layers, 2 pooling layers, 2 activation layers, and 1 fully connected layer.

[0081] The method for running neural network feature learning is as follows: the entire preprocessed fundus image dataset is fed into the global feature calculation model, which yields a complete feature information map of the preprocessed fundus image dataset. Specifically, after the preprocessed fundus image dataset is input into the global feature calculation model, the global image feature information possessed by the global feature calculation model is iteratively trained again with the preprocessed fundus image dataset in the form of training weight parameters, and finally, the global feature information map of the preprocessed fundus image dataset can be obtained.

[0082] Step 105: Input the preprocessed fundus image dataset into the local feature calculation model to determine the local feature information map of the fundus image dataset; the local feature information map includes local texture features, local color features, local size features, and local edge features of the fundus image.

[0083] Specifically, the preprocessed fundus image dataset is input into the local feature calculation model to generate slices of the fundus image; the local feature calculation model is trained based on the local feature information in the form of training weight parameters and the slices of the fundus image to determine the local feature information map of the fundus image dataset.

[0084] The local feature computation model is used to calculate the local image feature information map of the input fundus image to be tested, and it is mainly based on the VisionTransformer framework. The VisionTransformer framework divides the input fundus image to be tested into multiple slices (patch) and performs iterative training in the form of patches. This can better capture the local image feature information in the fundus image to be tested and protect the key feature information from being lost. By feeding the entire preprocessed fundus image dataset into the local feature computation model, the feature information map of the preprocessed fundus image dataset can be obtained.

[0085] Specifically, after the preprocessed fundus image dataset is input into the local feature calculation model, the local image feature information in the local feature calculation model will be trained again along with the D3 dataset in the form of training weight parameters, and finally the local feature information map in the D3 dataset can be obtained.

[0086] Step 106: Input the global feature information map and the local feature information map into the feature fusion layer to generate a multi-channel feature information map.

[0087] Specifically, the global feature map and the local feature map are input into the feature fusion layer. 256 3*3*1 convolutional kernels are used to extract global features from the global feature map to generate a one-dimensional global feature map. 256 3*3*1 convolutional kernels are also used to extract local features from the local feature map to generate a one-dimensional local feature map. The one-dimensional global feature map and the one-dimensional local feature map are then superimposed to generate a 3*3*512 multi-channel feature map.

[0088] More specifically, the global and local feature maps are each converted into a 3*3*1 one-dimensional feature map composed of 256 convolutional kernels. Channel stacking of these maps yields a 3*3*512 multi-channel feature map. This multi-channel feature map addresses the issue of lost edge features due to varying input image sizes and inconsistencies between the local and global feature maps, allowing for better fusion and learning of feature information.

[0089] Step 107: Input the multi-channel feature information map into the state prediction module to determine the fundus image prediction result in the fundus image to be tested; the fundus image prediction result includes those with ophthalmic diseases and those without ophthalmic diseases.

[0090] Specifically, the multi-channel feature information map is input into the state prediction module to determine the state prediction value of the retina in the fundus image to be tested for having an ophthalmic disease; the state prediction value includes 0 and 1; when the state prediction value is 0, it is determined that the image to be tested does not have an ophthalmic disease; when the state prediction value is 1, it is determined that the image to be tested has an ophthalmic disease.

[0091] Specifically, the multi-channel feature information map is input into the state prediction module to determine the state prediction value of the retina in the fundus image under test for ophthalmic diseases, including:

[0092] The state prediction module includes a vector transformation layer, a feature combination layer, a convolutional layer, a pooling layer, an activation function layer, and a fully connected layer.

[0093] The multi-channel feature information map is input into the vector transformation layer to generate a multi-channel feature information map after vector dimension transformation.

[0094] The multi-channel feature information map after vector dimension transformation is input into the feature combination layer to generate multi-channel feature information.

[0095] Multi-channel feature information is sequentially input into convolutional layers, pooling layers, activation function layers, and fully connected layers to generate a predicted state of retinal disease in the fundus image under test. Specifically, the multi-channel feature information generates a feature map after passing through the convolutional layer, a reduced-parameter feature map after passing through the pooling layer, and feature information after passing through the activation function.

[0096] The ViT and neural network-based ophthalmic disease state prediction method is applied to the ViT and neural network-based ophthalmic disease state prediction system. The ViT and neural network-based ophthalmic disease state prediction system includes: a data preprocessing module, a residual convolution module, a ViT-based encoding module, a feature fusion layer, and a state prediction module.

[0097] The data preprocessing module is mainly used to perform unified processing operations on all image data, including: image pixel density unification, image format unification, and data augmentation operations.

[0098] like Figure 2 As shown, the residual convolution module is mainly used to perform multi-round iterative training on the preprocessed fundus image dataset in the form of a neural network, so as to fully obtain the fundus image feature information in the preprocessed fundus image dataset. During the model training stage, multiple cycles can be trained to generate the corresponding weight model, which is denoted as the global feature calculation model.

[0099] The residual convolution module mainly consists of multiple convolutional layers, multiple pooling layers, multiple activation function layers, and a fully connected layer. The activation function of the activation function layer is ReLU.

[0100] The global feature calculation model is mainly used to calculate the texture feature information, color feature information, size feature information, and edge feature information of a fundus image.

[0101] like Figure 3 As shown, the ViT-based encoding module is mainly used to calculate the local feature information map of the input fundus image, which is then used as input for the subsequent risk prediction module.

[0102] The ViT-based encoding module is optimized and constructed based on the existing VisionTransformer framework, specifically including:

[0103] (1) Divide the input fundus image used for testing into m rows and n columns of slices, for a total of m*n slices. Each slice is a small slice compared to the fundus image.

[0104] (2) The above m*n small slices are positionally encoded to obtain position information in advance. Each small slice is then passed to a 1*1 dimensionality reduction layer in the order of its position information to perform dimensionality transformation and obtain the first image feature information of each small slice. Compared with the existing VisionTransformer framework, this processing method greatly reduces computational complexity and reduces redundant information for the ViT-based encoding module.

[0105] (3) Input the obtained small slice position code and small slice information into the encoder to extract local feature information of the small slice. Local feature information includes: small slice texture features, small slice edge features, small slice color features, and small slice spatial structure features.

[0106] (4) Multiple local feature information are spliced ​​together according to the position encoding to form a complete fundus image encoded feature information.

[0107] (5) The complete fundus image encoded feature information obtained above is fed into a 5*5 transpose convolution operation. The 5*5 transpose convolution operation can restore the size of the image after the above encoding operation to be consistent with the input size, thus obtaining the local feature calculation model. Compared with the existing VisionTransformer framework, this processing method can better restore the image size and reduce the loss of some feature information caused by the encoding operation.

[0108] Among them, such as Figure 4 As shown, the encoder mainly encodes the position of the small slice and the information of the small slice itself, which mainly includes:

[0109] (1) Normalization processing. The position codes and information of each input slice are normalized to facilitate subsequent processing.

[0110] (2) Linear combination processing. The multi-head attention layer processes each normalized small slice using three types of vectors: value vector (V), key phasor (K), and query vector (Q) in a linear combination.

[0111] (3) Combine each small slice after normalization in the linear combination process with the information after linear combination processing by the multi-head attention layer. The information combination is specifically performed by arranging the order of positions according to the position information and restoring it to the form of m*n.

[0112] The operations from step (1) to step (3) above are recorded as one processing step; L processing steps can be performed in the encoder.

[0113] like Figure 5 As shown, the multi-head attention layer mainly includes convolutional pooling operations and information transformations on three types of vectors: V, K, and Q. Specifically, it includes:

[0114] a) Perform a linear combination of the three vectors V, K, and Q for each small slice after normalization.

[0115] b) Perform a 3x3 convolution operation and a 2x2 pooling operation on each small slice after linear combination.

[0116] c) Perform multi-head tensor merging on all the small slices after the above processing to make them into a complete image feature information.

[0117] d) Perform a linear transformation on the complete image feature information obtained after the above processing.

[0118] It should be noted that during model training, all fundus image data after the data preprocessing module is input into the ViT-based encoding module. This allows the module to fully learn all local feature information of the fundus image under test, ensuring that key feature information is not lost. After training, a corresponding weight model can be generated, denoted as the local feature calculation model.

[0119] The local feature calculation model can be used to calculate the local feature map information of the input fundus image data to be tested, which is then fed into the subsequent risk prediction module.

[0120] like Figure 6 As shown, the feature fusion layer is mainly used to generate multi-channel feature maps. A multi-channel feature map refers to the flattening of the global feature map and the local feature map into 3*3*1 one-dimensional global feature map and one-dimensional global feature map composed of 256 convolutional kernels. Channel superposition of these results in a 3*3*512 multi-channel feature map.

[0121] Compared to existing technologies that unify the size of all input images and lose image information, this multi-channel feature information map can solve the problem of loss of some edge features due to inconsistent input image sizes and the inconsistency between local and global feature information map channels, and can better integrate and learn feature information.

[0122] The state prediction module is mainly used to combine and parse the input global feature information map and local feature information map.

[0123] The state prediction module mainly consists of a vector transformation layer, a feature combination layer, a fully connected layer, a convolutional layer, a pooling layer, and an activation function layer. The vector transformation layer converts the input feature map information into vector dimensions. The feature combination layer performs channel mapping on the two input vector dimensions to obtain the feature information of the fundus image. In calculating the predicted lesion risk value, after obtaining the fundus image feature information, it is then fed into the convolutional and pooling layers for dimensionality reduction, followed by the activation function layer to increase its non-linearity, and finally into the fully connected layer to achieve the respective probabilities for the two categories.

[0124] The output state prediction value is divided into two cases: 0 and 1. 0 represents a good state, meaning the image under test does not have an eye disease; 1 represents a bad state, meaning the image under test has an eye disease.

[0125] In summary, this invention generates two feature calculation models through optimized VisionTransformer structure and neural network algorithm training, obtaining more accurate local and global feature information of images. The constructed state prediction module can more accurately and efficiently analyze the ophthalmic disease state values ​​of fundus images.

[0126] The technical solution of this invention can improve the prediction of whether an eye disease is present by using fundus images, and has good accuracy, efficiency, convenience and comprehensiveness.

[0127] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for predicting the state of ophthalmic diseases based on ViT and neural networks, characterized in that, include: The fundus image dataset to be tested is input into the data preprocessing module to determine the preprocessed fundus image dataset; the fundus image dataset includes multiple fundus images; The preprocessed fundus image dataset is input into the residual convolution module to generate a global feature calculation model; The preprocessed fundus image dataset is input into a ViT-based encoding module to generate a local feature calculation model. Specifically, this includes: using the ViT-based encoding module to segment the image information in the preprocessed first fundus image dataset into multiple slices; the image information includes pixel information; the multiple slices are position-encoded, and each slice is input into a local feature calculation model according to the position-encoded order. A dimensionality reduction layer is used to perform dimensionality transformation and determine the first image feature information of each slice. The positional encoding of each slice and the image information are input into an encoder to extract local feature information of the slice and determine the local feature information image. The local feature information includes: slice texture features, slice edge features, slice color features, and slice spatial structure features. Multiple local feature information are concatenated according to the positional encoding to determine the encoded feature information of the complete fundus image. The encoded feature information of the complete fundus image is input into a 5-dimensional reduction layer. A 5-transposed convolutional layer, through 5 The transpose convolution operation of step 5 restores the size of the local feature information image and determines the local feature calculation model; The preprocessed fundus image dataset is input into the global feature calculation model to determine the global feature information map of the fundus image dataset; the global feature information map includes global texture features, global color features, global size features, and global edge features of the fundus image; The preprocessed fundus image dataset is input into the local feature calculation model to determine the local feature information map of the fundus image dataset; the local feature information map includes local texture features, local color features, local size features, and local edge features of the fundus image; The global feature information map and the local feature information map are input into the feature fusion layer to generate a multi-channel feature information map. Specifically, this includes: inputting the global feature information map and the local feature information map into the feature fusion layer, and using 256 3D features... 3 A 1x1 convolutional kernel extracts global features from the global feature information map, generating a one-dimensional global feature map; using 256 3x3 convolutional kernels... 3 A convolutional kernel of 1 extracts local features from the local feature information map, generating a one-dimensional local feature map; the one-dimensional global feature map and the one-dimensional local feature map are then superimposed through channels to generate a 3D global feature map. 3 512 multi-channel feature information map; The multi-channel feature information map is input into the state prediction module to determine the fundus image prediction result in the fundus image to be tested; the fundus image prediction result includes those with ophthalmic diseases and those without ophthalmic diseases.

2. The method for predicting ophthalmic disease states based on ViT and neural networks according to claim 1, characterized in that, The preprocessed fundus image dataset is input into the residual convolution module to generate a global feature calculation model, which specifically includes: The residual convolution module is trained using the preprocessed fundus image dataset to generate global features of the preprocessed fundus image dataset; The global feature information is learned and superimposed to determine the global feature calculation model.

3. The ophthalmic disease state prediction method based on ViT and neural networks according to claim 2, characterized in that, The global feature information is: f = wx + bias; In the formula, f represents the global feature information of the preprocessed fundus image dataset, x represents the preprocessed fundus image dataset, w represents the weight coefficient in iterative training, and bias represents the preset bias value.

4. The method for predicting ophthalmic disease states based on ViT and neural networks according to claim 1, characterized in that, The position code and image information of each slice are input into the encoder to extract local feature information of the slices and determine the local feature information image, specifically including: The position code of each input slice and the image information of the slice itself are normalized. Through a multi-head attention layer, each slice after normalization is processed by linear combination of three types of vectors: value vector, key vector, and query vector. Each slice that has undergone normalization is combined with the information that has undergone linear combination processing by the multi-head attention layer to determine a combined information slice; the combined information slice is a slice that contains the information after linear combination processing. Based on the positional encoding of each of the combined information slices, and arranging the combined information slices in the order of their positional encodings, a local feature information image is determined.

5. The ophthalmic disease state prediction method based on ViT and neural networks according to claim 4, characterized in that, Through a multi-head attention layer, each slice after normalization is processed by linearly combining three types of vectors: value vector, key vector, and query vector. Specifically, this includes: For each slice after normalization, a linear combination of three vectors is performed: value vector, key vector, and query vector. For each slice after linear combination, perform a 3-layer process. 3 convolution operations and 2 2. Pooling process; Multi-head tensor merging is performed on all slices after pooling to determine complete image feature information; A linear transformation is performed on the complete image feature information.

6. The method for predicting ophthalmic disease states based on ViT and neural networks according to claim 1, characterized in that, The preprocessed fundus image dataset is input into the global feature calculation model to determine the global feature information map of the fundus image dataset, specifically including: The global feature calculation model includes: convolutional layers, pooling layers, activation layers, and fully connected layers; The preprocessed fundus image dataset is input into the convolutional layer to generate a feature information map. The feature information map is input into the pooling layer to generate a pooling feature information map; The pooled feature information map is input into the activation layer to generate a nonlinear learning feature information map; The nonlinear learning feature information map is input into the fully connected layer to generate a global feature information map of the fundus image dataset.

7. The method for predicting ophthalmic disease states based on ViT and neural networks according to claim 1, characterized in that, The multi-channel feature information map is input into the state prediction module to determine the fundus image prediction result in the fundus image to be tested, specifically including: The multi-channel feature information map is input into the state prediction module to determine the state prediction value of the retina in the fundus image to be tested, which indicates that the retina has an ophthalmic disease; the state prediction value includes 0 and 1. When the state estimate is 0, it is determined that the fundus image to be tested does not have an eye disease; When the state estimate is 1, it is determined that the fundus image to be tested has an ophthalmic disease.

8. The method for predicting ophthalmic disease states based on ViT and neural networks according to claim 7, characterized in that, The multi-channel feature information map is input into the state prediction module to determine the state prediction value of the retina in the fundus image to be tested, which includes: The state prediction module includes a vector transformation layer, a feature combination layer, a convolutional layer, a pooling layer, an activation function layer, and a fully connected layer. The multi-channel feature information map is input into the vector transformation layer to generate a multi-channel feature information map after vector dimension transformation. The multi-channel feature information map after the vector dimension transformation is input into the feature combination layer to generate multi-channel feature information; The multi-channel feature information is sequentially input into the convolutional layer, the pooling layer, the activation function layer, and the fully connected layer to generate a state prediction of retinal disease in the fundus image to be tested.

Images