An analysis method, system and device for fundus images

Abstract

The application relates to an analysis method, system and device of a fundus image. The method comprises the following steps: acquiring a fundus image; classifying the fundus image based on coarse-grained features to obtain a first classification result, wherein the first classification result comprises retinal vein obstruction, retinal artery obstruction and a normal sample; classifying the fundus image with the first classification result of retinal artery obstruction based on fine-grained features to obtain retinal branch artery obstruction and central retinal artery obstruction classification results, or classifying the fundus image with the first classification result of retinal vein obstruction based on fine-grained features to obtain retinal branch vein obstruction and central retinal vein obstruction classification results. The application aims to realize classification and prediction of retinal vascular obstruction by fusing coarse-grained features and fine-grained features through a constructed fundus image hierarchical model, and to explore the capability and potential application value of the fundus image analysis.

Description

Technical Field

[0001] This invention relates to the field of image analysis of retinal fundus images, and more specifically, to a method, system, device, computer-readable storage medium for analyzing fundus images and their applications. Background Technology

[0002] Retinal vascular occlusions (RVOs) are a common cause of visual impairment and loss, reflecting systemic vascular disease. Retinal vein occlusion and retinal artery occlusion are two forms of RVO. Depending on the location of the occlusion, retinal vein occlusion can be further divided into retinal branch vein occlusion and central retinal vein occlusion, while retinal artery occlusion is divided into retinal branch artery occlusion and central retinal artery occlusion. The pathophysiology, systemic effects, and management of different RVO subtypes differ significantly, and their prognoses and even local treatments also represent different clinical entities. Therefore, accurate identification and differentiated diagnosis of RVOs are crucial for developing appropriate treatment and healthcare plans to properly manage eye conditions and minimize unmet medical needs. Unlike the auxiliary diagnostic issues of other ophthalmic diseases such as diabetic retinopathy and glaucoma, the number of RVO subtypes is relatively limited, and their distribution is severely uneven. Currently, no studies can simultaneously screen for and diagnose all of the aforementioned RVO subtypes.

[0003] Furthermore, deep learning models typically require large amounts of data for training to learn effective discriminative features. However, due to the complex intrinsic characteristics of RVO and the limited sample size, it is impossible to extract sufficiently deep discriminative features from the limited data in one step to handle the hierarchical features between various retinal vascular occlusion disease and normal samples. VisionTransformer (ViT), as a recent milestone in computer vision, has great potential in medical image processing. Therefore, using high-performance deep learning models to automatically screen and diagnose various RVO subtypes in fundus images has become a promising solution. Summary of the Invention

[0004] The purpose of this application is to provide a method for constructing a fundus image grading model, a method for analyzing fundus images, a system and device, a computer-readable storage medium and its applications, in order to address the above-mentioned problems. The aim is to achieve secondary classification and prediction of retinal vascular occlusion by fusing coarse-grained features and fine-grained features through a fundus image grading model, and to explore its analytical capabilities and potential application value in fundus images.

[0005] According to a first aspect of this application, one embodiment of this application provides a method for constructing a fundus image grading model, which includes:

[0006] S1: Obtain fundus images and their classification labels, and use a sample generator based on generative adversarial network to generate synthetic fundus images to obtain sample-balanced fundus image data;

[0007] S2: Construct a first fundus image grading model based on an attention mechanism. The specific implementation steps are as follows: perform multi-scale feature extraction and feature selection on the sample-equalized fundus image data, obtain the first classification result through a classifier, compare the first classification result with the classification label, optimize the classifier based on the comparison result, and construct a first fundus image grading model based on an attention mechanism. The first classification result includes retinal vein occlusion, retinal artery occlusion, and normal samples.

[0008] S3: Construct a second fundus image grading model, which includes a retinal vein occlusion classification model and a retinal artery occlusion classification model. Specific implementation steps are as follows:

[0009] Constructing a retinal vein occlusion classification model: Learning deep association features on fundus images with the first classification result of retinal vein occlusion to obtain a second classification result of retinal branch vein occlusion and central retinal vein occlusion. The second classification result is compared with the classification label, and the classifier is optimized based on the comparison result to construct a retinal vein occlusion classification model.

[0010] Constructing a retinal artery occlusion classification model: Deep correlation features are learned for fundus images with the first classification result of retinal artery occlusion to obtain a second classification result of retinal branch artery occlusion and central retinal artery occlusion. The second classification result is compared with the classification label, and the classifier is optimized based on the comparison result to construct a retinal artery occlusion classification model.

[0011] Furthermore, the sample generator consists of two pairs of generative adversarial networks, namely the first pair of generative adversarial networks and the second pair of generative adversarial networks.

[0012] The network structure of the first pair of generative adversarial networks includes a first generator G. I and the first discriminator D I Used to generate the synthetic fundus image;

[0013] Specifically, the first pair of generative adversarial network construction processes involves: randomly generating a noise vector. z ~ N(0,1) serves as the first generator G, which consists of many transposed convolutional layers. I The input is used to obtain a synthesized image, which is then used by the first discriminator D. ITo distinguish between real and synthetic images, cross-entropy loss is used to optimize the generator and discriminator, ultimately generating synthetic fundus images.

[0014] Another point is that the network structure of the second pair of generative adversarial networks includes a second generator. Second discriminator This is used to optimize the resolution of the synthesized fundus images, resulting in high-resolution fundus images with sample equalization;

[0015] Optionally, the second generator The structure consists of a residual block and an upsampling block, and the second discriminator Using the fusion loss function Optimize the method

[0016] Optionally, the second discriminator Based on a convolutional neural network as the backbone network;

[0017] Preferably, the fusion loss function The calculation is as follows:

[0018]

[0019] Where r, W, and H represent the upsampling factor, image width, and image height, respectively; W i,j H i,j , respectively, refer to the feature map obtained after activation of convolutional layer j before max pooling layer i and its corresponding dimension in the VGG backbone network;

[0020] Preferably, the super-resolution size of the optimized synthetic fundus image is 1024×1024×3;

[0021] Preferably, the second pair of generative adversarial networks is constructed based on image super-resolution generative adversarial networks.

[0022] In some embodiments, the deep association features are learned by mining the deep associations between the first and second classification results using a masking strategy. The masking strategy is based on an attention-based fundus image hierarchical model that randomly samples and masks portions of the image after image patch segmentation, and then constructs a T-structure through linear embedding. m =[t1,…,tM2], k∈{1,…,M 2} represents the token for each local window, t k This refers to the token converted from image patch k, and then the multi-head self-attention W-MSA is calculated based on the window. The specific calculation of the multi-head self-attention W-MSA is as follows:

[0023]

[0024]

[0025]

[0026]

[0027] H = [h1,…,h] n ] T ,

[0028] W-MSA = W o H

[0029] Where n is the number of heads in the multi-head attention; k∈{1,…,n} are the weights used to obtain the query, key, and value matrices of the header. k∈{1,…,n}; d is the query and key dimension of the header k; k∈{1,…,n} is the relative position deviation of the head of i. i∈{1,…,n} is the attention value of head i. As weights, they are concatenated with all heads to obtain the multi-head attention W-MSA.

[0030] Furthermore, the fundus image grading model also includes a multi-head self-attention (SW-MSA) module based on a shifted window to construct the fundus image grading model. In the continuous attention network model, the SW-MSA module and the W-MSA module are used interchangeably. The SW-MSA module divides the window pixels... Remove the rule partition window from the previous layer while keeping the window configuration unchanged, and recalculate the self-attention. The preferred, continuous attention network module is calculated as follows:

[0031]

[0032]

[0033]

[0034]

[0035] in, and z l Let represent the output characteristics of the W-MSA and MLP modules of block l, respectively. The output characteristics of block l are then used as the input to block l+1, which contains a multi-head self-attention SW-MSA module based on a shift window, to obtain the output characteristics of the SW-MSA and MLP modules of block l+1. zl+1 .

[0036] According to a second aspect of this application, one embodiment of this application provides a method for analyzing fundus images, comprising:

[0037] Acquire fundus images;

[0038] The fundus images are classified based on coarse-grained features to obtain a first classification result, which includes retinal vein occlusion, retinal artery occlusion and normal samples.

[0039] Based on fine-grained features, fundus images with a first classification result of retinal artery occlusion are classified into second classification results of retinal branch artery occlusion and central retinal artery occlusion. Alternatively, fundus images with a first classification result of retinal vein occlusion are classified into second classification results of retinal branch vein occlusion and central retinal vein occlusion.

[0040] In some embodiments, the classification of the fundus image based on coarse-grained features is achieved by a first fundus image classification model constructed based on an attention mechanism. Specifically, the fundus image is input into the first fundus image classification model constructed based on an attention mechanism for classification, and a first classification result is output.

[0041] Preferably, the first fundus image grading model based on the attention mechanism extracts multi-scale features from the fundus image, then selects features by calculating the self-attention value of the fundus image, and then inputs the feature selection result into the classifier to output the first classification result;

[0042] Optionally, the first fundus image grading model is implemented based on any one or more of the following models: Swing Transformer, HaloNet, Point-BERT, ViT;

[0043] Preferably, the first fundus image grading model is implemented based on the Swing Transformer.

[0044] Furthermore, the feature selection process includes: inputting the fundus image into a first fundus image hierarchical model, the fundus image being uniformly divided into multiple image blocks, and then uniformly dividing the fundus image into multiple window regions in a non-overlapping manner, calculating self-attention based on the image blocks contained in each window region, and then performing feature selection based on the self-attention value;

[0045] Preferably, the self-attention calculation process involves: a first fundus image grading model built based on the Swing Transformer grading the fundus image... The image is uniformly divided into 4×4 blocks, each block is treated as a token, and its features are set as the concatenation of its original pixel RGB values. The feature dimension of each block is 4×4×3, or 48. Then, the fundus image I is processed in a non-overlapping manner. cfp The image is uniformly divided into multiple local windows of size M×M, and each image patch P contained in the local window is used as the basic unit to calculate the self-attention value:

[0046]

[0047] in, j∈{1,…,M 2},P i,j It refers to image block j in local window i.

[0048] In some embodiments, the fine-grained features are obtained by mining the deep correlation features between the first classification result and the second classification result through a masking strategy in the second fundus image grading model; the masking strategy is that after the fundus image grading model divides the image into blocks, it randomly samples some image blocks for masking, and then constructs the feature through linear embedding. k∈{1,…,M 2} represents the token for each local window, t k This refers to the token converted from image patch k, and then the multi-head self-attention W-MSA is calculated based on the window; preferably, the specific calculation of the multi-head self-attention W-MSA is as follows:

[0049]

[0050]

[0051]

[0052]

[0053] H = [h1,…,h] n ] T ,

[0054] W-MSA = W o H

[0055] Where n is the number of heads in the multi-head attention; k∈{1,…,n} are the weights used to obtain the query, key, and value matrices of the header. k∈{1,…,n}; d is the query and key dimension of the header k; k∈{1,…,n} is the relative position deviation of the head of i; i∈{1,…,n} is the self-attention value of head i; As weights, they are concatenated with all heads to obtain the multi-head self-attention W-MSA.

[0056] Furthermore, the second fundus image grading model also includes a continuous attention network model module constructed by introducing a multi-head self-attention SW-MSA based on a shifted window. In the continuous attention network model module, the SW-MSA and the W-MSA are used interchangeably, and the SW-MSA divides the window pixels... Remove the rule partition window from the previous layer while keeping the window configuration unchanged, and recalculate the self-attention value;

[0057] Preferably, the SW-MSA calculation of the continuous attention network model module is as follows:

[0058]

[0059]

[0060]

[0061]

[0062] in and z l Let represent the output characteristics of the W-MSA and MLP modules of block l, respectively. The output characteristics of block l are then used as the input to block l+1, which contains a multi-head self-attention SW-MSA module based on a shift window, to obtain the output characteristics of the SW-MSA and MLP modules of block l+1. z l+1 ...

[0063] In some embodiments, the second fundus image grading model is based on any one or more of the following models: deep relational network, deep self-attention model, graph feature attention neural network, and recursive attention convolutional neural network.

[0064] According to a third aspect of this application, one embodiment of this application provides a system for analyzing fundus images, comprising:

[0065] The module acquires fundus images;

[0066] The first prediction module classifies the fundus image based on coarse-grained features to obtain a first classification result, which includes retinal vein occlusion, retinal artery occlusion, and normal samples.

[0067] The second prediction module classifies fundus images with a first classification result of retinal artery occlusion based on fine-grained features to obtain classification results for retinal branch artery occlusion and central retinal artery occlusion, or classifies fundus images with a first classification result of retinal vein occlusion based on fine-grained features to obtain classification results for retinal branch vein occlusion and central retinal vein occlusion.

[0068] According to a fourth aspect of this application, one embodiment of this application provides a device for analyzing fundus images, mainly comprising:

[0069] Memory and / or processor;

[0070] The memory is used to store program instructions, which contain a computer program for image analysis of retinal vascular occlusion. When the computer program is executed by the processor, it implements the fundus image analysis method or constructs a fundus image grading model that integrates coarse-grained and fine-grained features using the fundus image grading model construction method, thereby achieving classification and prediction of retinal vascular occlusion.

[0071] The processor is used to call program instructions. When the program instructions are executed, the processor executes the fundus image analysis method or constructs a fundus image grading model that integrates coarse-grained features and fine-grained features using the fundus image grading model construction method described above, so as to achieve classification and prediction of retinal vascular occlusion.

[0072] According to a fourth aspect of this application, one embodiment of this application provides a computer-readable storage medium storing a computer program for analyzing fundus images. When the computer program is executed by a processor, it implements the above-described fundus image analysis method or the above-described fundus image grading model construction method.

[0073] The application of the aforementioned devices or systems in the intelligent classification and prediction of retinal vascular occlusion in fundus images;

[0074] The aforementioned devices or systems offer some new ideas for using few-shot learning to address ophthalmic problems and demonstrate the potential benefits of deep learning models for rare ophthalmic diseases in remote areas of the world.

[0075] The aforementioned equipment or systems help address issues such as insufficient ophthalmological services, a shortage of ophthalmologists, and a lack of experience in rural and remote areas, especially in developing countries, in order to provide more comprehensive and refined support for patients' treatment decisions.

[0076] The aforementioned devices or systems are used in the selection of prognostic analysis plans to assist in the formulation of treatment plans and closer follow-up plans; optionally, the assistance in formulating treatment plans includes the ability of ophthalmologists to develop appropriate treatment and care plans for patients through accurate identification and differential diagnosis of RVO in order to properly manage the eye condition and minimize destructive effects; optionally, the prognostic analysis includes in-depth research based on the fundus image grading and fundus image analysis results provided in this application, which has a positive impact and promoting effect.

[0077] This invention presents a fundus image grading model based on an attention mechanism. By fusing coarse-grained and fine-grained features, it performs secondary classification and prediction of retinal vascular occlusion, overcoming the influence of human subjectivity and the impact of limited samples on deep learning models. It achieves automatic fine classification of various subtypes of retinal vascular occlusion, demonstrating strong innovation and making a beneficial contribution to the analysis and research of fundus image data.

[0078] Advantages of this application:

[0079] 1. This application innovatively discloses a method for analyzing fundus images. This method uses a two-step strategy based on a mask self-attention mechanism to complete the phased processing of coarse-grained features and fine-grained features by the model, realizing secondary classification prediction of retinal vascular occlusion, especially automatic fine classification of various subtypes of retinal vascular occlusion, effectively reducing the interference caused by mixed features of different scales, and objectively improving the accuracy and depth of data analysis.

[0080] 2. This application innovatively constructs a sample generator based on generative adversarial networks to generate synthetic fundus images, obtaining fundus image data with balanced samples. This is an effective measure to compensate for data gaps and has significant time-saving benefits.

[0081] 3. This application creatively discloses a first fundus image grading model constructed based on multi-scale feature learning, used to process coarse-grained features and obtain the first classification result, and a first fundus image grading model constructed based on deep correlation features, used to process fine-grained features and obtain the second classification result. It considers mixed features of different scales and avoids the model being affected by local features. Given that the fundus image grading results are of great research significance for formulating reasonable treatment and health care plans to properly manage eye conditions, as well as for prognosis and prevention, this application can be more accurately applied to the auxiliary analysis of the occurrence and development of diseases related to fundus images. Attached Figure Description

[0082] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0083] Figure 1 This is a schematic flowchart of the fundus image analysis method provided in an embodiment of the present invention;

[0084] Figure 2 This is a schematic diagram of the construction method of the fundus image grading model provided in the embodiment of the present invention;

[0085] Figure 3 This is a schematic diagram of the framework structure of the fundus image grading model constructed according to an embodiment of the present invention;

[0086] Figure 4 This is a schematic diagram of the types of fundus images used in the fundus image analysis process provided in the embodiments of the present invention;

[0087] Figure 5 This is a schematic diagram illustrating the classification performance of various retinal vascular occlusion subtypes provided in an embodiment of the present invention;

[0088] Figure 6 This is a performance comparison chart between the fundus image grading model provided in this embodiment of the invention and existing deep learning models;

[0089] Figure 7 This is a schematic diagram illustrating the generalization ability of the fundus image grading model provided in an embodiment of the present invention;

[0090] Figure 8 This is a schematic diagram of the fundus image analysis device provided in an embodiment of the present invention. Detailed Implementation

[0091] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0092] In some of the processes described in the specification, claims, and accompanying drawings of this invention, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be performed in the order they appear herein, or may be performed in parallel. The operation numbers, such as S101, S102, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be performed sequentially or in parallel.

[0093] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0094] This application provides a method for constructing a fundus image grading model, a method for analyzing fundus images, a system for analyzing fundus images, a device for analyzing fundus images, a computer-readable storage medium, and their applications. The construction of the fundus image grading model, including its corresponding training device, can be integrated into a computer device, which can be a terminal or a server. The terminal can be a smartphone, tablet, laptop, personal computer, or other similar device. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0095] Please see Figure 1 , Figure 1 This is a schematic flowchart of the fundus image analysis method provided in the embodiments of the present invention. Specifically, as shown in the flowchart... Figure 1 The following operations are shown:

[0096] S101: Acquire fundus image.

[0097] In one embodiment, the acquired fundus images comprised 5,624 retinal fundus images recruited from a single-center and three multi-center cohorts for deep learning model development and clinical evaluation. One single-center cohort consisted of 1,529 fundus images from 914 patients retrospectively collected from hospitals (WMUEH cohort), randomly split into a training set (80%) for model training and a validation set (20%) for model training. The other three cohorts were collected from publicly available resources for multi-center, externally independent validation, including 839 fundus images from the Retinal Fundus Multiple Disease Image Database (RFMiD cohort); 3,152 fundus images from Ophthalmic Intelligent Recognition, a structured real-world ophthalmic database (ODIR cohort); and 104 fundus images from previous studies (JSIEC cohort).

[0098] Furthermore, to improve the model's usability, patients with RVO and other eye conditions (such as cataracts, diabetic retinopathy, and glaucoma) were also included. All fundus images were labeled collaboratively by three experienced ophthalmologists.

[0099] In one specific embodiment, acquiring fundus images also includes image enhancement of the acquired fundus images. For all fundus images of different resolutions captured by various devices, the eyeball portion is automatically located and preserved, and irrelevant backgrounds are removed. Then, an image enhancement method based on pixel color magnification theory is used to adjust the brightness of the fundus images to highlight lesion areas and eye structures.

[0100] S102: Classify the fundus image based on coarse-grained features to obtain a first classification result, which includes retinal vein occlusion, retinal artery occlusion, and normal samples.

[0101] In one embodiment, the fundus image is classified based on coarse-grained features using a first fundus image grading model constructed based on an attention mechanism. Specifically, the fundus image is input into the first fundus image grading model based on the attention mechanism for classification, and a first classification result is output. Preferably, the first fundus image grading model based on the attention mechanism extracts multi-scale features from the fundus image, then performs feature selection by calculating the self-attention of the fundus image, and finally inputs the selected feature result into a classifier to output the first classification result.

[0102] Optionally, the first fundus image grading model is implemented based on one or more of the following models: SwinTransformer, HaloNet, Point-BERT, ViT;

[0103] Preferably, the first fundus image grading model is implemented based on Swing Transformer.

[0104] Swin Transformer is a hierarchical visual Transformer that uses moving windows. By designing a hierarchical Transformer, it learns features of different sizes in an image and uses moving windows to interact between them, thereby performing global modeling of the image and presenting higher computational efficiency and model performance.

[0105] HaloNet, a novel self-attention model, incorporates two extensions to self-attention and combines them with a more efficient implementation of self-attention, improving the model's speed, memory usage, and accuracy.

[0106] Point-BERT helps standard Transformers learn both low-level structural information and high-level semantic information by constructing a Masked Point Modeling (MPM) task belonging to point clouds, and brings significant improvements to standard Transformers in 3D object-related tasks.

[0107] ViT (Vision Transformer) is a vision model that uses a Transformer encoder to divide the input image into blocks and quantize them, thus allowing the use of the same encoding model as word vectors.

[0108] Furthermore, the feature selection process includes: inputting the fundus image into the first fundus image classification model, dividing the fundus image into multiple image blocks, then uniformly dividing the fundus image into multiple window regions in a non-overlapping manner, calculating the self-attention value based on the image blocks contained in each window region, and then performing feature selection based on the self-attention value.

[0109] In one specific embodiment, the self-attention calculation process is as follows: A first-stage fundus image grading model built based on the Swing Transformer processes the fundus images... The image is uniformly divided into 4×4 blocks, each block is treated as a token, and its features are set as the concatenation of its original pixel RGB values. The feature dimension of each block is 4×4×3, or 48. Then, the fundus image I is processed in a non-overlapping manner. cfp The image is uniformly divided into multiple local windows of size M×M, and each image patch P contained in the local window is used as a basic unit to calculate self-attention:

[0110]

[0111] in, j∈{1,…,M 2},P i,j It refers to image j in local window i.

[0112] S103: Based on fine-grained features, classify fundus images with the first classification result of retinal artery occlusion to obtain a second classification result of retinal branch artery occlusion or central retinal artery occlusion; or based on fine-grained features, classify fundus images with the first classification result of retinal vein occlusion to obtain a second classification result of retinal branch vein occlusion or central retinal vein occlusion.

[0113] In one embodiment, fine-grained features are obtained by mining the deep correlation features between the first and second classification results using a masking strategy in the fundus image grading model. The masking strategy involves randomly sampling and masking portions of the image after the fundus image grading model has divided the image into blocks, and then using linear embedding to construct... k∈{1,…,M 2} represents the label for each local window, t k This refers to the token converted from image patch k, and then the multi-head self-attention W-MSA is calculated based on the window; preferably, the multi-head self-attention W-MSA is calculated as follows:

[0114]

[0115]

[0116]

[0117]

[0118] H = [h1,…,h] n ] T ,

[0119] W-MSA = W o H

[0120] Where n is the number of heads in the multi-head attention; k∈{1,…,n} are the weights used to obtain the query, key, and value matrices of the header. k∈{1,…,n}; d is the query and key dimension of the header k; k∈{1,…,n} is the relative position deviation of the head of i; i∈{1,…,n} is the self-attention value of head i; As weights, they are concatenated with all heads to obtain the multi-head self-attention W-MSA.

[0121] Furthermore, the masking strategy also includes constructing a second fundus image hierarchical model by introducing a continuous attention network model module based on multi-head self-attention (SW-MSA) with a shifted window. In the continuous attention network model module, SW-MSA and W-MSA are used interchangeably, with SW-MSA dividing the window pixels... Remove the rule partition window from the previous layer while keeping the window configuration unchanged, and recalculate the self-attention value;

[0122] Preferably, the SW-MSA calculation for the continuous attention network model module is as follows:

[0123]

[0124]

[0125]

[0126]

[0127] in and z l Let represent the output characteristics of the W-MSA and MLP modules of block l, respectively. The output characteristics of block l are then used as the input to block l+1, which contains a multi-head self-attention SW-MSA module based on a shift window, to obtain the output characteristics of the SW-MSA and MLP modules of block l+1. z l+1 .

[0128] In one embodiment, the second fundus image grading model is implemented based on any one or more of the following models: deep relational network model, deep self-attention model, graph feature attention neural network, and recursive attention convolutional neural network.

[0129] The first few layers of the deep relational network model are pre-trained, while the later layers are used to learn the connections between different tasks. Parameter sharing is achieved based on the normal prior of the parameter tensors of multiple task-specific layers.

[0130] The deep self-attention model uses a self-attention mechanism to model the preference part of user features at a granular level. Through multiple perceptron layers, it outputs the preference probability of each target task.

[0131] A graph feature attention neural network, which introduces the attention neural network representation model of node position features.

[0132] Recurrent attention convolutional neural networks optimize the localization of discriminative regions and the learning of refined features together, more accurately finding discriminative sub-regions in an image, and then using high-resolution, refined features to describe these regions, thereby greatly improving the accuracy of refined object classification.

[0133] Figure 2 This is a schematic flowchart of a method for constructing a fundus image grading model provided in an embodiment of the present invention, specifically including the following steps:

[0134] S201: Obtain fundus images and their classification labels, generate synthetic fundus images based on a sample generator constructed using a generative adversarial network, and obtain sample-balanced fundus image data.

[0135] In one embodiment, the classification labels for the fundus images include: retinal vein occlusion, retinal artery occlusion, normal sample, retinal branch vein occlusion, central retinal vein occlusion, retinal branch artery occlusion, and central retinal artery occlusion. The classification labels for the fundus images are designed for training specific grading models for each individual fundus image.

[0136] Furthermore, the sample generator is designed to generate synthetic images of fundus images from two types of retinal branch artery occlusion and central retinal artery occlusion, which have relatively few sample data, thus obtaining sample-balanced data. The sample generator consists of two pairs of generative adversarial networks: a first pair and a second pair.

[0137] The first pair of generative adversarial networks includes a first generator G. I and the first discriminator D I It is used to generate synthetic fundus images from fundus images;

[0138] Specifically, the first pair of generative adversarial network construction processes: randomly generate a noise vector. z ~ N(0,1) serves as the first generator G, which consists of many transposed convolutional layers. I The input is used to obtain a synthesized image, which is then used by the first discriminator D. I To distinguish between real and synthetic images, cross-entropy loss is used to optimize the generator and discriminator, ultimately generating synthetic fundus images.

[0139] Another point is that the network structure of the second pair of generative adversarial networks includes a second generator. Second discriminator Super-resolution is used to optimize the synthetic fundus images, resulting in sample-equalized fundus images;

[0140] Optional, second generator Its structure includes a residual block and an upsampling block, and a second discriminator. Using the fusion loss function Optimize the method

[0141] Optional, second discriminator Based on a convolutional neural network as the backbone network;

[0142] Preferably, the super-resolution size of the optimized synthetic fundus image is 1024×1024×3;

[0143] Preferably, the second pair of generative adversarial networks is constructed based on image super-resolution adversarial generative networks.

[0144] S202: Constructing a first-level fundus image grading model. The specific implementation steps are as follows: perform multi-scale feature extraction and feature selection on the balanced fundus image data, obtain the first classification result through a classifier, compare the first classification result with the classification label, optimize the classifier based on the comparison result, and construct a first-level fundus image grading model based on the attention mechanism.

[0145] The first classification results include retinal vein occlusion, retinal artery occlusion, and normal samples.

[0146] In one embodiment, when constructing the first fundus image classification model, the classification labels of the obtained fundus images are: retinal vein occlusion, retinal artery occlusion, and normal samples.

[0147] S203: Construct a second fundus image grading model, which includes a retinal vein occlusion classification model and a retinal artery occlusion classification model.

[0148] The specific implementation steps of step S203 are as follows:

[0149] Constructing a retinal vein occlusion classification model: Learning deep association features on fundus images with the first classification result of retinal vein occlusion to obtain a second classification result of retinal branch vein occlusion and central retinal vein occlusion. The second classification result is compared with the classification label, and the classifier is optimized based on the comparison result to construct a retinal vein occlusion classification model.

[0150] Constructing a retinal artery occlusion classification model: Deep correlation features are learned for fundus images with the first classification result of retinal artery occlusion to obtain a second classification result of retinal branch artery occlusion and central retinal artery occlusion. The second classification result is compared with the classification label, and the classifier is optimized based on the comparison result to construct a retinal artery occlusion classification model.

[0151] In one embodiment, when constructing a retinal vein occlusion classification model, the classification labels used for the fundus images are: retinal branch vein occlusion and central retinal vein occlusion.

[0152] In one embodiment, when constructing the retinal artery occlusion classification model, the classification labels used for the fundus images are: branch retinal artery occlusion and central retinal artery occlusion.

[0153] In one specific embodiment, a schematic diagram of the framework structure of the fundus image grading model constructed based on the fundus image grading model construction method proposed in this application is shown below. Figure 3As shown. In order to handle two different granularities of features, coarse-grained and fine-grained, and to improve the flexibility of the model, a modular two-step strategy is proposed for the analysis and prediction of retinal vascular occlusion in fundus images. Another advantage of using a modular framework is that certain modules in the framework can be selectively used according to different use cases.

[0154] For ease of description, Figure 3 The parameters mentioned in this article represent: RVO (retinal vascular occlusion); RvO (retinal vein occlusion); RaO (retinal artery occlusion); BRVO (branch retinal vein occlusion); CRVO (central retinal vein occlusion); BRAO (branch retinal arterial occlusion); CRAO (central retinal arterial occlusion); and Normal (normal sample).

[0155] Figure 3 The fundus image grading model proposed in Part A is based on the DeepDrRVO architecture and mainly includes the DeepDrVAN module, DeepDrVBC, and DeepDrABC. First, a module called DeepDrVAN is designed to focus on identifying and distinguishing coarse-grained features for early detection of retinal vein occlusion, retinal artery occlusion, and normal samples. Second, the DeepDrVBC and DeepDrABC modules learn fine-grained distinguishing features between retinal vein occlusion and retinal artery occlusion for differential diagnosis of retinal vein occlusion and retinal artery occlusion subtypes. During inference, preprocessed images are first fed into the DeepDrVAN module, and then diagnosed as retinal vein occlusion, retinal artery occlusion, or normal using a set threshold. Fundus images belonging to retinal vein occlusion or retinal artery occlusion are then fed into the DeepDrVBC or DeepDrABC modules, respectively. The DeepDrVBC and DeepDrABC modules classify fundus images into retinal branch vein occlusion, central retinal vein occlusion, retinal branch artery occlusion, and central retinal artery occlusion based on the set threshold. Finally, the overall prediction results are obtained.

[0156] Figure 3The fundus image grading model framework shown in Figure A also includes preprocessing of the fundus image data source and sample generation. Preprocessing of the fundus image data source includes image enhancement of the acquired fundus images. For fundus images of different resolutions captured by various devices, the eyeball is automatically located and preserved, and irrelevant background is removed. Then, an image enhancement method based on pixel color amplification theory is used to adjust the brightness and darkness of the fundus images, highlighting lesion areas and ocular structures. Another point is that the sample generation process mainly involves... Figure 3 The few-sample generator shown in Figure A is used to generate the sample data. Figure 3 As shown in B, the few-shot generator consists of two pairs of generative adversarial networks, namely the first pair of generative adversarial networks and the second pair of generative adversarial networks. The first pair of generative adversarial networks is used to generate low-resolution synthetic images, and the second pair of generative adversarial networks is used to convert the generated low-resolution synthetic images into high-resolution synthetic fundus images.

[0157] To effectively alleviate the data imbalance problem and improve the performance of the DeepDrRVO framework, a sample generator inspired by deep convolutional adversarial generative networks and image super-resolution generative adversarial networks was constructed to generate high-resolution fundus images of rare morphologies in the queue.

[0158] Furthermore, the network structure of the first generative adversarial network includes a first generator G. I and the first discriminator D I A first generative adversarial network (GAN) is used to generate low-resolution synthetic fundus images from the fundus images. A first pair of GANs is used to generate synthetic images of branch retinal artery occlusion and central retinal artery occlusion. Noise vectors are sampled as input to a generator consisting of many transposed convolutional layers, producing a synthetic image of size 256×256×3. A discriminator is then used to distinguish between real and synthetic images for adversarial training with the generator. Finally, cross-entropy loss is used to optimize the first generator and the first discriminator.

[0159] Specifically, the construction process of the first pair of generative adversarial networks: randomly generate a noise vector. z ~ N(0,1) serves as the first generator G, which consists of many transposed convolutional layers. I The input is used to obtain a synthesized image, which is then used by the first discriminator D. I To distinguish between real and synthetic images, cross-entropy loss is used to optimize the generator and discriminator, ultimately generating low-resolution synthetic fundus images.

[0160] For branch retinal artery occlusion and central retinal artery occlusion, high resolution of fundus images is required because some lesions often appear in extremely small areas. Therefore, using another pair of generative adversarial networks and performing image super-resolution to a size of 1024×1024×3 allows the synthesized image to retain finer and more realistic features. Composed of many residual blocks and upsampling blocks, the network structure is more complex than the upsampling blocks. The residual blocks increase the network depth, and it is proposed to use a deeper perceptron to learn better transformation functions and representations. That is, the network structure of the second pair of generative adversarial networks includes a second generator. Second discriminator This is used to optimize the super-resolution of the synthesized fundus image, resulting in a sample-equalized high-resolution fundus image;

[0161] Preferably, the second generator The structure consists of a residual block and an upsampling block, and the second discriminator Using the fusion loss function Optimize the method

[0162] In a more specific embodiment, the second discriminator A VGG16 network was used as the backbone. Specifically, optimization was performed using a fusion loss method, with the fusion loss function... The calculation is as follows:

[0163]

[0164] Where r, W, and H represent the upsampling factor, image width, and image height, respectively; W i,j H i,j , respectively, refer to the feature map obtained after activation of convolutional layer j before max pooling layer i and its corresponding dimension in the VGG backbone network.

[0165] After preprocessing the data source and generating samples, the following results are obtained: Figure 4 The types of fundus images shown mainly include raw fundus images, enhanced fundus images, and synthetic fundus images. Figure 4 In A, the original fundus images mainly include retinal branch vein occlusion, central retinal vein occlusion, retinal branch artery occlusion, central retinal artery occlusion, and normal samples; Figure 4 Image A also includes enhanced fundus images obtained through image enhancement. It can be observed that the enhanced images of retinal branch vein occlusion, central retinal vein occlusion, retinal branch artery occlusion, central retinal artery occlusion, and normal samples highlight the lesion areas and ocular structures compared to the original images. Figure 4In section B, the synthesized fundus images mainly include retinal branch artery occlusion and central retinal artery occlusion obtained through the sample generator, ultimately resulting in high-definition simulated synthetic fundus images.

[0166] In some embodiments, the learning of deep correlation features is achieved by using a masking strategy to help mine the deep correlation features between the first and second classification results. The masking strategy is as follows: Figure 3 C and Figure 3 As shown in D, the self-attention module of the fundus image grading model is implemented based on the Swing Transformer mask module after image block segmentation and linear embedding. Figure 3 C mainly involves learning coarse-grained features of fundus images by using a mask self-attention module built based on the Swin Transformer mask module to perform multi-scale feature learning, and then classifying and predicting to obtain the first classification results, including retinal vein occlusion, retinal artery occlusion, and normal samples.

[0167] Specifically, the masking strategy randomly samples a portion of the image patch for masking, and then constructs a linear embedding. k∈{1,…,M 2} represents the token for each local window, t k This refers to the token transformed from image patch k, and then the multi-head self-attention W-MSA is calculated based on the window. The specific calculation of the multi-head attention W-MSA is as follows:

[0168]

[0169]

[0170]

[0171]

[0172] H = [h1,…,h] n ] T ,

[0173] W-MSA = W o H

[0174] Where n is the number of heads in the multi-head attention; k∈{1,…,n} are the weights used to obtain the query, key, and value matrices of the header. k∈{1,…,n}; d is the query and key dimension of the header k; k∈{1,…,n} is the relative position deviation of the head of i. i∈{1,…,n} is the self-attention value of head i. As weights, they are concatenated with all heads to obtain the multi-head self-attention W-MSA.

[0175] Furthermore, fundus image grading models also include, for example, Figure 3 As shown in Figure D, a tiered model for fundus images is constructed by introducing a continuous attention network model module based on multi-head self-attention (SW-MSA) with a shifted window. Features are input into a mask module based on the Swing Transformer, and after forward feedback, layer normalization, and deep association learning through continuous attention, output features—i.e., fine-grained features—are obtained. Within the continuous attention network model module, SW-MSA and W-MSA are used interchangeably; SW-MSA divides the window pixels... Remove the rule partitioning window from the previous layer while keeping the window configuration unchanged, and recalculate the self-attention value. Preferably, the SW-MSA calculation for the continuous attention network model module is as follows:

[0176]

[0177]

[0178]

[0179]

[0180] in and z l Let represent the output characteristics of the W-MSA and MLP modules of block l, respectively. The output characteristics of block l are then used as the input to block l+1, which contains a multi-head self-attention SW-MSA module based on a shift window, to obtain the output characteristics of the SW-MSA and MLP modules of block l+1. z l+1 .

[0181] In a more specific embodiment, the construction and training process of the fundus image grading model involves: image augmentation (e.g., on 1529 fundus images from the WMUEH cohort) is performed. Figure 4A right-hand side shows an enhanced fundus image. 1529 fundus images were randomly divided into a training set of 1229 images (80%) for model training and a validation set of 300 images (20%) for internal testing. In the second stage, a sample generator was trained by expanding the fundus images of 28 branch retinal artery occlusion and 65 central retinal artery occlusion with different enhancement combinations in the training set to 896 branch retinal artery occlusion and 1950 central retinal artery occlusion fundus images, respectively. For the first pair of generative adversarial networks in the sample generator, the expanded fundus images of branch retinal artery occlusion and central retinal artery occlusion were converted to 256×256×3 as the training set and the network was optimized using the Adam optimizer with a learning rate, batch size, and training epoch set to 3×10⁻⁶. -4 36 and 5000. The training method for the second pair of generative adversarial networks is similar to that of the first pair, i.e., Adam is used as the optimizer with a learning rate of 10. -3 The batch size is 24, and the dataset is trained for 200 epochs on a public dataset. In this dataset, low-resolution images are obtained by downsampling high-resolution images using a bicubic kernel with a downsampling factor r=4. Finally, the sample generator will generate images of size 1024×1024×3 with a resolution similar to the original WMUEH queue (e.g., ...). Figure 4 (As shown in C). These fundus images derived from the sample generator are added to the training set at a certain ratio for auxiliary training. Then, the fundus images in the training set are randomly scaled and cropped to a size of 384×384×3. The input fundus images are then normalized and standardized, with the average values ​​in the RGB channels being 0.485, 0.456, and 0.406, and the deviations in the standard RGB channels being 0.229, 0.224, and 0.225, respectively.

[0182] Furthermore, the Adam optimizer was used to train three modules (DeepDrVAN, DeepDrVBC, and DeepDrABC) in DeepDrRVO with a learning rate of 10. -4 Weight decay 10 -3 The batch size is 10, and the training epochs are 50. During training, the best-performing fundus image grading model is selected based on the performance of the last 10 training epochs on the validation set.

[0183] Furthermore, the construction and training process of the fundus image grading model can be implemented using PyTorch or other frameworks or other programming languages, and it is also possible to choose to use a system with... On a CentOS 7.6.1810 server with a Tesla V100 GPU.

[0184] In one embodiment, the DeepDrRVO framework (i.e., a fundus image grading model) proposed in this application adopts a modular approach, breaking down traditional one-time inference into two steps and three modules. Furthermore, a masking strategy is used to guide the deep learning model in mining deeply correlated features by simulating the decision-making process of an ophthalmologist. Simultaneously, a generative adversarial network-based sample generator is constructed to generate rare synthetic fundus images of branch retinal artery occlusion and central retinal artery occlusion to aid model training. Figure 5 The diagram illustrates the classification performance of the DeepDrRVO framework provided in this embodiment of the invention for various retinal vascular occlusion subtypes. Finally, the robustness of the DeepDrRVO framework was measured using three multi-center cohorts, with a total of 4395 fundus images used as the test and validation set. The beneficial effects obtained are as follows: Figure 6 As shown, this provides insights into solving other problems in the auxiliary diagnosis of eye diseases with similar data distributions.

[0185] In one embodiment, model evaluation and validation are performed using Receiver Operating Characteristic (ROC) curves, overall accuracy, sensitivity, specificity, and F1 score to assess and compare model performance in binary classification predictions. The area under the ROC curve (AUC) and 95% confidence interval (CI) are calculated using a nonparametric guided method (2,000 random resampling and replacements). Considering the class imbalance in the RVO sample data, the area under the exact recall curve (AUPR) and Cohen's kappa statistic are used to measure the consistency of model predictions and the ability to identify rare classes. Multi-class classification performance is further tested for multiple binary classifications when evaluated using F1-score, AUC, and AUPR. In this setting, metrics are calculated for each label and weighted by the number of true instances per label for an overall comparison across all classes.

[0186] In one embodiment, the DeepDrRVO fundus image classification model was trained using fundus images from the training set and tested on the validation set. The classification performance of various retinal vascular occlusion subtypes based on the DeepDrRVO framework using the validation set is as follows: Figure 5 As shown. In the first task, as... Figure 5As shown in Figure A, the AUCs of the DeepDrVAN module in DeepDrRVO, used to identify retinal vein occlusion, retinal artery occlusion, and normal fundus, were 0.979 (95% confidence interval (CI): 0.967–0.991), 0.942 (95% CI: 0.884–1.000), and 0.970 (95% CI: 0.954–0.986), respectively. In the second task, the AUCs of the DeepDrVBC and DeepDrABC modules, used to further differentiate retinal vein occlusion and retinal artery occlusion subtypes, reached 0.962 (95% CI: 0.930–0.995) and 0.667 (95% CI: 0.380–0.954), respectively. Figure 5 As shown in B and 5C. Figure 5 D shows that DeepDrRVO, integrating the above three modules, achieves an accuracy of 0.863 for retinal vessel occlusion recognition. Furthermore, DeepDrRVO provides inherent interpretability based on Axiom-based grad class activation mapping, such as... Figure 5 As shown in E, DeepDrRVO constructs saliency maps for the four subtypes of retinal vascular occlusion—retinal branch vein occlusion, central retinal vein occlusion, retinal branch artery occlusion, and central retinal artery occlusion—in fundus images, and performs activation mapping on regions that play an important decision-making role in fundus image classification and prediction tasks.

[0187] Figure 6 This section compares the DeepDrRVO fundus image grading model with several traditional deep learning models commonly used in similar fundus image grading tasks. They were trained and tested in the manner described in the implementation details, and their performance was then evaluated across various datasets. Figure 6 As shown, the proposed DeepDrRVO outperforms traditional deep learning models. Compared to the proposed DeepDrRVO, other traditional deep learning models fail to effectively identify retinal vascular occlusion and all exhibit strong bias against normal inference samples. These comparative results demonstrate the superiority of DeepDrRVO in the early and differential diagnosis of retinal vascular occlusion compared to traditional convolutional neural network-based deep learning models.

[0188] In a specific embodiment, such as Figure 7 The diagram shows the generalization ability of the fundus image grading model DeepDrRVO proposed in this application in three heterogeneous center cohorts of patients with different ethnicities: RFMiD, ODIR, and JSIEC. DeepDrVAN ( Figure 7 A), DeepDrVBC Figure 7 B), DeepDrABC Figure 7C) and DeepDrRVO Figure 7 D) ROC curves and confusion matrices for the three independent cohorts above. To verify the generalization ability of the proposed fundus image grading model DeepDrRVO in detecting key lesion features and conditions, DeepDrRVO was also applied to these three multi-heterogeneous center cohorts of patients with different ethnicities. The accuracy, sensitivity, specificity, and AUC results for the corresponding cohorts are shown in [reference needed]. Figure 7 .

[0189] First, the DeepDrVAN module maintained good screening capabilities, with overall accuracies of 0.976, 0.942, and 1.000, and AUCs of 0.941, 0.879, and 0.982 (95% CI: 0.962-1.000), and AUCs of 0.941, 0.979, and 0.982 (95% CI: 0.962-1.000), and 1.000 (95% CI: 1.000-1.000), respectively, and an AUC of 0.842 (95% CI: 0.976-1.000), and an AUC of 0.842 (95% CI: 0.976-1.000), respectively, for detecting RvO in the RFMiD, ODIR, and JSIEC cohorts. The AUCs for detecting RFMiD, ODIR, and JSIEC normal samples in the cohorts were 0.987 (95% CI: 0.976-0.998), 0.953 (95% CI: 0.916-0.990), and 1.000 (95% CI: 1.000)-1.000, respectively (e.g., % CI: 0.661-1.000). Figure 7 A). For the differential diagnosis of retinal vascular occlusion, the DeepDrVBC module demonstrated sufficient capability to effectively diagnose RvO subclasses, achieving accuracies of 0.938, 0.911, and 1.000 in the RFMiD, ODIR, and JSIEC cohorts, respectively, with corresponding AUCs of 0.993 (95% CI: 0.986–1.000), 0.925 (95% CI: 0.845–1.000), and 1.000 (95% CI: 1.000–1.000), and AUPRs of 0.911, 0.969, and 1.000, respectively (e.g., retinal occlusion subclasses). Figure 7 (As shown in B). When diagnosing RaO as BRAO and CRAO, the DeepDrABC module demonstrates good performance in the differential diagnosis of RaO in the queue, such as... Figure 7As shown in Figure C, the AUC reached 1.000 (95% CI: 1.000–1.000) and 0.500 (95% CI: 0.000–1.000) in the RFMiD cohort and the ODIR cohort, respectively, while the AUPR was 1.000 and 0.599 in the RFMiD cohort and the ODIR cohort, respectively. Finally, the confusion matrix of the DeepDrRVO hierarchical model prediction results is shown in Figure C. Figure 7 As shown in Figure D, the overall diagnostic accuracy of the DeepDrRVO grading model on RFMiD, ODIR, and JSIEC reached 0.926, 0.908, and 1.000, respectively.

[0190] To further test the effectiveness and robustness of DeepDrRVO, the performance of DeepDrRVO under different parameters—mask ratio and the number of synthesized images—was explored. The optimal mask ratio was applied in increments of 0.1 within the range of 0.1–0.5. Enhanced images with different mask ratios had only a slight impact on the final diagnostic performance. More specifically, the DeepDrRVO hierarchical model achieved the best performance when the mask ratio was 0.3, with an AUC of 0.964 for the DeepDrVAN module, 0.962 for the DeepDrVBC module, and 0.724 for the DeepDrABC module. Therefore, 0.3 was chosen as the mask ratio to further analyze the impact of the number of generated images on assisted training. To verify the effectiveness of synthesized images for assisted training, four different scales of generated image sets were designed for the generation of two types of synthetic fundus images: retinal branch artery occlusion and central retinal artery occlusion, during the training of DeepDrRVO. The performance of DeepDrVAN and DeepDrABC under different configurations shows that the different configurations of the synthesized images have little impact on the final diagnostic performance. More specifically, BRAO50+CRAO20, used as auxiliary training, achieved the most robust performance, with AUCs of 0.964 and 0.667, respectively. The DeepDrVAN and DeepDrABC modules ranked first in the overall ranking.

[0191] Applying the above method to the classification and prediction of retinal vascular occlusion in fundus images is feasible. It shows that by fusing coarse-grained and fine-grained features through deep correlation, the performance of the model can be further improved, and more accurate information can be provided for patient assessment. It can better reflect the effect of the fundus image grading model based on attention, providing a practical solution for the automatic screening and diagnosis of retinal vascular occlusion. It also provides ideas for solving other problems in the auxiliary diagnosis of eye diseases with similar data distributions, making this application more advantageous in the auxiliary analysis of the occurrence and development of diseases related to fundus imaging data.

[0192] The fundus image analysis system provided in this embodiment of the invention includes:

[0193] The module acquires fundus images;

[0194] The first prediction module classifies the fundus image based on coarse-grained features to obtain a first classification result, which includes retinal vein occlusion, retinal artery occlusion, and normal samples.

[0195] The second prediction module classifies fundus images with a first classification result of retinal artery occlusion based on fine-grained features to obtain classification results for retinal branch artery occlusion and central retinal artery occlusion, or classifies fundus images with a first classification result of retinal vein occlusion based on fine-grained features to obtain classification results for retinal branch vein occlusion and central retinal vein occlusion.

[0196] Figure 8 This invention provides an analysis device for fundus images, comprising:

[0197] Memory and / or processor;

[0198] The device may also include input devices and output devices.

[0199] Memory, processor, input devices, and output devices can be connected via a bus or other means. Figure 8 The example shown is a bus connection; the memory is used to store program instructions; the processor is used to call program instructions. When the program instructions are executed, they are used to execute the above-mentioned fundus image analysis method or to construct a fundus image grading model that integrates coarse-grained features and fine-grained features using the above-mentioned fundus image grading model construction method, so as to realize the classification and prediction of retinal vascular occlusion.

[0200] In one embodiment, memory can be understood as any device for storing programs, and processor can be understood as a device for using programs.

[0201] The present invention provides a computer-readable storage medium storing a computer program for classifying and predicting retinal vascular occlusion in fundus images. When the computer program is executed by a processor, it implements the above-mentioned fundus image analysis method or constructs a fundus image classification model that integrates coarse-grained features and fine-grained features using the above-mentioned fundus image classification model construction method, thereby realizing the classification and prediction of retinal vascular occlusion.

[0202] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0203] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0204] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0205] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The aforementioned integrated modules can be implemented in hardware or as software functional modules.

[0206] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0207] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program that executes the relevant hardware instructions. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0208] The computer device provided by this invention has been described in detail above. For those skilled in the art, based on the ideas of the embodiments of this invention, there will be changes in specific implementation methods and application scope. Therefore, the content of this specification should not be construed as limiting the invention.

Claims

1. A method for constructing a fundus image classification model, the method being achieved by the following steps: S1: obtaining fundus images and their classification labels, generating synthetic fundus images by a sample generator constructed based on a generative adversarial network, and obtaining sample-balanced fundus image data; S2: constructing a first fundus image classification model based on an attention mechanism, the specific implementation steps being: performing multi-scale feature extraction and feature selection on the sample-balanced fundus image data, obtaining a first classification result by a classifier, comparing the first classification result with the classification labels, optimizing the classifier according to the comparison result, and constructing the first fundus image classification model based on the attention mechanism, the first classification result including retinal vein occlusion, retinal artery occlusion and normal samples; S3: constructing a second fundus image classification model, the second fundus image classification model including a retinal vein occlusion classification model and a retinal artery occlusion classification model, the specific implementation steps being: constructing the retinal vein occlusion classification model: learning deep correlation features of the fundus images with the first classification result of retinal vein occlusion, obtaining a second classification result of retinal branch vein occlusion and retinal central vein occlusion, comparing the second classification result with the classification labels, optimizing the classifier according to the comparison result, and constructing the retinal vein occlusion classification model; constructing the retinal artery occlusion classification model: learning deep correlation features of the fundus images with the first classification result of retinal artery occlusion, obtaining a second classification result of retinal branch artery occlusion and retinal central artery occlusion, comparing the second classification result with the classification labels, optimizing the classifier according to the comparison result, and constructing the retinal artery occlusion classification model.

2. The method of claim 1, wherein, The sample generator is composed of two pairs of generative adversarial networks, i.e., a first pair of generative adversarial networks and a second pair of generative adversarial networks; The network structure of the first pair of generative adversarial networks comprises a first generator G I and a first discriminator D I , for generating a synthetic fundus image from the fundus image; Specifically, the first pair of generative adversarial network construction process: acquiring the fundus image to sample a noise vector z ~ N(0, 1) as the first generator G consisting of many transpose convolutional layers I The input of the first discriminator D I to distinguish between real images and synthetic images, while using cross-entropy loss to optimize the generator and discriminator, ultimately generating synthetic fundus images; The network structure of the second pair of generative adversarial networks comprises a second generator G IHR and a second discriminator D IHR , for optimizing the super-resolution of the synthesized fundus image to obtain a sample equalized fundus image; Optionally, the second generator comprises a residual block and an up-sampling block, the second discriminator is optimized using a fusion loss function ​ Optionally, the second discriminator D I HR is based on a convolutional neural network as a backbone network; Preferably, The calculation is as follows: where r, W, H represent the up-sampling factor and the image width and height, respectively; W i,j ,H i,j , respectively, are the feature map obtained after the activation of the convolutional layer j before the max-pooling layer i and the corresponding dimension within the backbone network. Preferably, the super-resolution size of the synthetic fundus image is 1024x1024x3 after optimization. Preferably, the second pair of generative adversarial networks is constructed based on an image super-resolution adversarial generation network. 3.A fundus image analysis method, comprising: obtaining a fundus image; classifying the fundus image based on coarse-grained features to obtain a first classification result, the first classification result including retinal vein occlusion, retinal artery occlusion and normal samples; classifying the fundus image with the first classification result of retinal artery occlusion based on fine-grained features to obtain a second classification result of retinal branch artery occlusion and retinal central artery occlusion, or classifying the fundus image with the first classification result of retinal vein occlusion based on fine-grained features to obtain a second classification result of retinal branch vein occlusion and retinal central vein occlusion.

4. The method of analyzing an ocular fundus image according to claim 3, characterized by, The classification of the fundus image based on coarse-grained features is achieved by a first fundus image classification model based on an attention mechanism, the specific implementation process being that the fundus image is input into the first fundus image classification model based on the attention mechanism for classification, and the first classification result is output. Preferably, the first fundus image classification model based on the attention mechanism extracts multi-scale features from the fundus image, selects features by calculating the self-attention value of the fundus image, and then inputs the selected features into a classifier to output a first classification result. Optionally, the first fundus image classification model is implemented based on any one or several of the following models: SwinTransformer, HaloNet, Point-BERT, and ViT. Preferably, the first fundus image classification model is implemented based on SwinTransformer.

5. The method of analyzing an ocular fundus image according to claim 4, characterized by, The feature selection process includes: inputting the fundus image into the first fundus image classification model, dividing the fundus image into multiple image blocks, uniformly dividing the fundus image into multiple window regions in a non-overlapping manner, calculating the attention value based on the image blocks contained in each window region, and then selecting features according to the attention value. Preferably, the calculation process of the self-attention is: the first fundus image hierarchical model constructed based on SwinTransformer divides the fundus image I uniformly into 4x4 image blocks, each image block is regarded as a token, and the feature thereof is set as the concatenation of the original pixel RGB values, and the feature dimension of each image block is 4x4x3, i.e. 48; then, the fundus image I cfp is uniformly divided into multiple local windows of MxM size, and the self-attention is calculated by using each image block P contained in the local window as a basic unit: wherein P i,j denotes patch j in local window i.

6. The method of analyzing an ocular fundus image according to claim 3, characterized by, The fine-grained features are obtained by a mask strategy in the second fundus image classification model to mine deep correlation features of the first classification result and the second classification result, the mask strategy is that the second fundus image classification model randomly samples part of the image blocks for covering after image block division, and linear embedding is performed to form T m = t1, …, t M2 , to represent the label of each local window, t k is a token converted from the patch k, and then multi-head self-attention W-MSA is calculated based on the window; preferably, specific calculation of the multi-head self-attention W-MSA is as follows: H = [h1,..., h n ] T , W-MSA = W o H where n is the number of heads in multi-head attention; Q k ,K k , k e {1,..., n} is the weight of the query, key and value matrices used to obtain head k d is the dimension of the query, key of head k; is the relative position bias of head i; is the self-attention value of head i; The multi-head self-attention W-MSA is calculated after all the heads are concatenated with the weight 7. The method of analyzing an ocular fundus image according to claim 6, characterized in that, The second fundus image grading model further comprises a continuous attention network model module constructed by introducing a multi-head self-attention based on a shift window SW-MSA, in which the SW-MSA is used interchangeably with the W-MSA, and the SW-MSA calculates the window pixels The rule partition window of the previous layer is moved out while the window configuration remains unchanged, and the self-attention value is recalculated; preferably, the SW-MSA of the continuous attention network model module is calculated as follows: wherein and z l represent the output characteristics of the W-MSA and MLP modules of block l, respectively, and use the output characteristics of block l as input to block l+1 comprising a shift window-based multi-head self-attention, SW-MSA, module to obtain the output characteristics of the SW-MSA and MLP modules of block l+1 z l+1 .

8. The method of analyzing an ocular fundus image according to claim 6, characterized by, The second fundus image classification model is implemented based on any one or several of the following models: deep relationship network, deep self-attention model, graph feature attention neural network, and recursive attention convolutional neural network.

9. An analysis system for fundus images, comprising: an acquisition module configured to acquire a fundus image; a first prediction module configured to classify the fundus image based on coarse-grained features to obtain a first classification result, the first classification result including retinal vein occlusion, retinal artery occlusion, and normal samples; a second prediction module configured to classify the fundus image with the first classification result of retinal artery occlusion based on fine-grained features to obtain a classification result of retinal branch artery occlusion and retinal central artery occlusion, or to classify the fundus image with the first classification result of retinal vein occlusion based on fine-grained features to obtain a classification result of retinal branch vein occlusion and retinal central vein occlusion.

10. An apparatus for analyzing an ocular fundus image, characterized by comprising: The device comprises: a memory and / or a processor; the memory is configured to store program instructions, the program instructions store a computer program for analyzing images of retinal vascular occlusion, and when the computer program is executed by the processor, the analysis method for fundus images according to any one of claims 3-8 or the construction method of the fundus image classification model according to claims 1-2 is implemented to construct a fundus image classification model that integrates coarse-grained features and fine-grained features, and to realize classification prediction of retinal vascular occlusion; the processor is configured to call the program instructions, and when the program instructions are executed, the analysis method for fundus images according to any one of claims 3-8 or the construction method of the fundus image classification model according to claims 1-2 is implemented to construct a fundus image classification model that integrates coarse-grained features and fine-grained features, and to realize classification prediction of retinal vascular occlusion.

Images