Retinal OCT image classification method, system, computer device and storage medium based on multi-scale residual network

Abstract

The present invention relates to a retinal OCT image classification method, system, computer device and storage medium based on a multi-scale residual network, which belongs to the field of OCT image processing technology and solves the problem of low accuracy in retinal OCT image classification. The method of the present invention comprises: obtaining a retinal OCT image classification dataset and dividing it into a training set and a test set; preprocessing the training set and the test set; improving the baseline network using a dual-channel attention mechanism and a multi-scale residual module to establish a retinal OCT image classification model; the multi-scale residual module specifically comprises several branches, a residual module, a feature fusion module and an addition module, each branch comprising several convolutions; the feature fusion module is used to perform feature fusion on the output information of several branches; the addition module is used to add the fused information to the output information of the residual module; obtaining a trained retinal OCT image classification model; and classifying retinal OCT images. The present invention is suitable for the classification of retinal OCT images.

Description

Technical Field

[0001] The present application relates to the field of OCT image processing technology, and in particular to retinal OCT image classification. Background Art

[0002] Retinal diseases are the leading cause of vision loss. Due to their high morbidity and blindness rates, retinal diseases have long been a focus of research. Currently, the global population is aging, and as the population ages, retinal deterioration also occurs. Retinal eye diseases are among the most common conditions among the elderly. The main causes of blindness in my country include glaucoma, cataracts, diabetic macular edema (DME), and age-related macular degeneration (AMD). AMD can be divided into dry (DRUSEN) and wet (CNV) forms based on clinical manifestations and pathological changes. The key to treating retinal diseases lies in early screening and regular examinations.

[0003] Optical Coherence Tomography (OCT) can provide high-resolution cross-sectional images of the retina and is currently the most advanced technology used to detect retinal diseases. It has the advantages of being non-contact, non-invasive, and fast imaging. Currently, the common method for diagnosing retinal diseases is for ophthalmologists to manually examine the patient's fundus OCT images and make a judgment on the patient's disease. However, the disadvantage of this judgment method is that there are relatively few ophthalmologists and a large number of patients with eye diseases, which can cause doctor fatigue. OCT images are grayscale images, and some disease characteristics are not obvious, which can lead to misdiagnosis and missed diagnosis, further preventing patients from discovering eye diseases and receiving treatment in a timely manner, leading to worsening of the disease.

[0004] Deep learning technology has garnered widespread attention across many fields, owing to its advantages in large-scale data processing. In recent years, deep learning, a method within the discipline of machine learning, has garnered widespread attention across various fields. Its application in image classification primarily relies on feature learning using convolutional neural networks (CNNs), eliminating the need for manual feature design. The "deep" in deep learning refers to the fact that the network used for image classification consists of a large number of layers, including convolutional, pooling, and fully connected layers. Deep learning technology has advantages in large-scale data processing and has achieved significant success in the field of OCT image classification. Recently, a transfer learning-based method for retinal OCT image classification has been proposed. By fine-tuning the GoogleNet network, this method reduces the network's reliance on large amounts of data. With limited training data, it achieves classification accuracy of 86%, 89%, and 99% for DME, AMD, and normal images, respectively. Some approaches also use depthwise separable convolutions instead of standard convolutional layers to reduce network parameters, while also using global average pooling instead of fully connected layers to improve spatial robustness. Experiments have shown that classification accuracy can reach 97%. While many of the aforementioned image classification algorithms have achieved good results in terms of classification accuracy, the accuracy of most algorithms still falls short of the requirements of the medical field and remains relatively low in this area. Summary of the Invention

[0005] The purpose of the present invention is to solve the problem of low accuracy in the existing classification of retinal OCT images, and to provide a retinal OCT image classification method, system, computer device and storage medium based on a multi-scale residual network.

[0006] The present invention is achieved through the following technical solutions. In one aspect, the present invention provides a retinal OCT image classification method based on a multi-scale residual network, the method comprising:

[0007] Step 1: Obtain a retinal OCT image classification dataset and divide it into a training set and a test set;

[0008] Step 2: preprocessing the training set and the test set;

[0009] Step 3: Improve the baseline network using a dual-channel attention mechanism and a multi-scale residual module to establish a retinal OCT image classification model;

[0010] The multi-scale residual module specifically includes several branches, a residual module, a feature fusion module and an addition module, and each branch includes several convolutions;

[0011] The feature fusion module is used to perform feature fusion on the output information of the multiple branches;

[0012] The adding module is used to add the fused information and the output information of the residual module;

[0013] Step 4: training the retinal OCT image classification model using the preprocessed training set;

[0014] Testing the retinal OCT image classification model using the preprocessed test set;

[0015] Obtain the trained retinal OCT image classification model;

[0016] Step 5: Classify the retinal OCT images using the trained retinal OCT image classification model.

[0017] Furthermore, the preprocessing specifically includes:

[0018] Using a downsampling method, the number of retinal OCT images of several categories in the training set and the test set is unified;

[0019] unifying the sizes of the unified number of retinal OCT images;

[0020] The retinal OCT images after the size is unified are normalized.

[0021] Furthermore, the baseline network is a ResNet50 network.

[0022] Furthermore, step 3 specifically includes:

[0023] Add several dual-channel attention mechanism modules, maximum pooling layers, and average pooling layers to the baseline network;

[0024] Replacing the residual modules of the plurality of baseline networks with a plurality of the multi-scale residual networks;

[0025] Set the loss function to learn the model parameters;

[0026] According to the retinal OCT image classification task, the output dimension of the network is set.

[0027] Furthermore, there are two dual-channel attention mechanism modules, which are respectively set after the convolutional layer and after the last multi-scale residual module.

[0028] Furthermore, the maximum pooling layer is arranged before the first residual module.

[0029] Furthermore, the plurality of branches specifically include four branches;

[0030] The four branches are respectively composed of 1×1 convolution, 1×1 and 3×3 convolution, 1×1 and 5×5 convolution, and a maximum pooling layer and 1×1 convolution.

[0031] The second invention provides a retinal OCT image classification system based on a multi-scale residual network, the system comprising:

[0032] The dataset acquisition module is used to obtain the retinal OCT image classification dataset and divide it into a training set and a test set;

[0033] A data set processing module, used for preprocessing the training set and the test set;

[0034] A retinal OCT image classification model building module is used to improve the baseline network using a dual-channel attention mechanism and a multi-scale residual module to build a retinal OCT image classification model;

[0035] The multi-scale residual module specifically includes several branches, a residual module, a feature fusion module and an addition module, and each branch includes several convolutions;

[0036] The feature fusion module is used to perform feature fusion on the output information of the multiple branches;

[0037] The adding module is used to add the fused information and the output information of the residual module;

[0038] A model training and testing module, configured to train the retinal OCT image classification model using the preprocessed training set;

[0039] Testing the retinal OCT image classification model using the preprocessed test set;

[0040] Obtain the trained retinal OCT image classification model;

[0041] The model classification module is used to classify the retinal OCT images using the trained retinal OCT image classification model.

[0042] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the steps of the retinal OCT image classification method based on a multi-scale residual network as described in any one of the above items are executed.

[0043] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein a plurality of computer instructions are stored in the computer-readable storage medium, and the plurality of computer instructions are used to enable a computer to execute the retinal OCT image classification method based on the multi-scale residual network as described in any one of the above items.

[0044] Beneficial effects of the present invention:

[0045] The pathological features displayed by the original OCT image characteristics are not obvious enough, especially some small features are difficult to extract, such as the detailed information between different retinal membrane layers and the cross-sectional image feature information of the macular area. This leads to the inability to learn the image features of different categories of retinal diseases well during image classification.

[0046] First, the present invention uses a cascaded multi-kernel residual link method to provide training and classification results. It can obtain information of different sizes from the input information, which can solve the problems of unclear imaging of pathological features in the original retinal OCT images and difficulty in capturing the characteristics of small target lesions, thereby improving the accuracy of retinal OCT image classification.

[0047] Secondly, a dual-channel attention module is used to extract local feature areas distributed in space and channels from the input channels that are useful for model classification. By capturing more global and local information from the input images, it effectively learns the subtle details in the input, further improving the accuracy of retinal OCT image classification.

[0048] The present invention has been verified through experiments and has a classification accuracy rate of up to 97.3%.

[0049] The present invention is applicable to the classification of retinal OCT images. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0051] Figure 1 It is a retinal OCT image classification algorithm framework based on multi-scale residual network;

[0052] Figure 2 It is a dual-channel attention module framework;

[0053] Figure 3 It is the channel attention module framework;

[0054] Figure 4 It is the framework of spatial attention module;

[0055] Figure 5 is the residual module framework in the baseline network;

[0056] Figure 6 The invented multi-scale residual module framework;

[0057] Figure 7 This is a training flowchart for the retinal OCT image classification algorithm based on a multi-scale residual network;

[0058] Figure 8 This is a flowchart for testing the retinal OCT image classification algorithm based on the multi-scale residual network;

[0059] Figure 9 Confusion matrix diagrams obtained by the baseline network and the present invention for retinal OCT image classification tests ((a) is the ResNet50 test result, and (b) is the test result of the present invention). DETAILED DESCRIPTION

[0060] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but are not to be construed as limiting the present invention.

[0061] Embodiment 1: A retinal OCT image classification method based on a multi-scale residual network, the method comprising:

[0062] Step 1: Obtain a retinal OCT image classification dataset and divide it into a training set and a test set;

[0063] Step 2: preprocessing the training set and the test set;

[0064] Step 3: Improve the baseline network using a dual-channel attention mechanism and a multi-scale residual module to establish a retinal OCT image classification model;

[0065] The multi-scale residual module specifically includes several branches, a residual module, a feature fusion module and an addition module, and each branch includes several convolutions;

[0066] The feature fusion module is used to perform feature fusion on the output information of the multiple branches;

[0067] The adding module is used to add the fused information and the output information of the residual module;

[0068] Step 4: training the retinal OCT image classification model using the preprocessed training set;

[0069] Testing the retinal OCT image classification model using the preprocessed test set;

[0070] Obtain the trained retinal OCT image classification model;

[0071] Step 5: Classify the retinal OCT images using the trained retinal OCT image classification model.

[0072] First, this embodiment utilizes a cascaded multi-kernel residual link method to provide training and classification results. This method can obtain information of different sizes from the input information, addressing the issues of unclear imaging of pathological features in original retinal OCT images and difficulty capturing features of small target lesions, thereby improving the accuracy of retinal OCT image classification.

[0073] Secondly, a dual-channel attention module is used to extract local feature areas distributed in space and channels from the input channels that are useful for model classification. By capturing more global and local information from the input images, it effectively learns the subtle details in the input, further improving the accuracy of retinal OCT image classification.

[0074] It should be noted that the classification in the classification model can use the softmax function; the programming language for implementing the classification model in step 4 is Python, the software tool used is Pytorch, the operating system is Linux, and the graphics card used is NVIDIA GeForce RTX 3070.

[0075] It should be noted that the baseline network is an image classification network, which is used to learn and classify image features of different categories. After training, it can eventually classify test images.

[0076] Embodiment 2: This embodiment further limits the retinal OCT image classification method based on the multi-scale residual network described in embodiment 1. In this embodiment, the preprocessing is further limited, specifically including:

[0077] The preprocessing specifically includes:

[0078] Using a downsampling method, the number of retinal OCT images of several categories in the training set and the test set is unified;

[0079] unifying the sizes of the unified number of retinal OCT images;

[0080] The retinal OCT images after the size is unified are normalized.

[0081] It should be noted that, in this embodiment, the image matrix and image pixels of the retinal OCT image after the size is unified can be normalized. The specific normalization method formula is as follows:

[0082]

[0083] In formula (1), μ is the mean of the image, and X represents the image matrix;

[0084]

[0085] In formula (2), σ represents the standard deviation and N represents the number of pixels in the image.

[0086] In this embodiment, the retinal OCT images can be unified in size to 224×224, and the number of retinal OCT images of the four categories can be unified by downsampling, and the images can be standardized. The pre-processed images are used as input into the classification network to make the classification more accurate and precise. Image standardization is a process of centralizing the data by removing the mean. According to the knowledge related to convex optimization theory and data probability distribution, data centralization conforms to the law of data distribution, and it is easier to achieve generalization effect after training.

[0087] Embodiment 3: This embodiment further limits the retinal OCT image classification method based on the multi-scale residual network described in embodiment 1. In this embodiment, the baseline network is further limited, specifically including:

[0088] The baseline network is the ResNet50 network.

[0089] The ResNet50 network used in this embodiment generally extracts some shallow features for retinal OCT image features in shallow networks, which cannot be learned well. However, the method of enhancing the network learning ability by increasing the number of network layers is not always feasible. Because after the network depth reaches a certain number of layers, if the number of network layers is increased, the network will have the problem of random gradient disappearance, which will also cause the network accuracy to decrease. ResNet50 can effectively solve this problem. The residual units in ResNet50 are connected across layers, and the input information can directly cross the layer to the deeper layer, which can effectively alleviate the gradient disappearance problem, thereby helping to improve the accuracy of retinal OCT image classification.

[0090] Embodiment 4: This embodiment further limits the retinal OCT image classification method based on a multi-scale residual network described in any one of Embodiments 1 to 3. In this embodiment, step 3 is further limited, specifically including:

[0091] Step 3 specifically includes:

[0092] Add several dual-channel attention mechanism modules, maximum pooling layers, and average pooling layers to the baseline network;

[0093] Replacing the residual modules of the plurality of baseline networks with a plurality of the multi-scale residual networks;

[0094] Set the loss function to learn the model parameters;

[0095] According to the retinal OCT image classification task, the output dimension of the network is set.

[0096] It should be noted that the loss function in this embodiment may be a cross entropy loss function.

[0097] In this embodiment, if Figure 1 As shown, a retinal OCT image classification model based on a multi-scale residual network is designed. In general, the retinal OCT image classification network based on a multi-scale residual network in this embodiment is an end-to-end deep learning network architecture. This embodiment can combine the multi-scale residual module and dual-channel attention with ResNet50, replace some residual modules in the network with multi-scale residual modules, add a maximum pooling layer to the network to reduce the network computational complexity, and change the stride of the convolution layer after the pooling layer is added from 2 to 1; the output dimension of the network can be set to 4 for the four-category classification task of retinal disease images: CNV, DME, DRUSEN, and NORMAL.

[0098] in,

[0099] First, a dual-channel attention mechanism module is added to the baseline network. The dual-channel attention mechanism module structure is as follows: Figure 2 As shown in the figure, the dual-channel attention mechanism module combines the spatial attention mechanism module and the channel attention mechanism module, thus having the advantages of both the spatial attention mechanism module and the channel attention module. It can capture useful information distributed in space and channels, thereby improving the classification performance of the network.

[0100] Second, the pooling layer is placed before the residual module in the network, and the step size of the residual module after the pooling layer is changed from 2 to 1, so that the input and output parameters of the modified network are consistent;

[0101] Third, the multi-scale residual module structure is as follows Figure 6As shown in the figure, a multi-scale residual module with cascaded multi-cores is used, which combines the advantages of the residual module. In addition, there is a ReLU layer after each convolution layer to reduce the amount of calculation and alleviate the overfitting problem. Through this module, the advantages of the residual module can be obtained, which can effectively alleviate the gradient diffusion problem and the overfitting problem. At the same time, information of different sizes from the input information can be obtained, which can solve the problem that the pathological features in the original retinal OCT image are not clear and the characteristics of small target lesions are difficult to capture.

[0102] Fourth, add an average pooling layer after the network. This reduces the amount of parameter calculations and alleviates the overfitting problem.

[0103] Fifth, cross entropy is used as the loss function. The cross entropy loss function is often used in classification problems. Since it involves the probability of each category, in this embodiment, the cross entropy loss function appears together with the softmax function.

[0104] Sixth, the network's output dimension is modified to make it suitable for retinal OCT image classification tasks. The original baseline network has an output dimension of 100. To handle the four-category retinal OCT image classification task (CNV, DME, DRUSEN, NORMAL), this implementation can set the network's output dimension to 4.

[0105] Embodiment 5: This embodiment further limits the retinal OCT image classification method based on the multi-scale residual network described in embodiment 4. In this embodiment, the dual-channel attention mechanism module is further limited, specifically including:

[0106] There are two dual-channel attention mechanism modules, which are respectively set after the convolutional layer and after the last multi-scale residual module.

[0107] In this implementation, a dual-channel attention mechanism module is added to the baseline network. The dual-channel attention mechanism module structure is as follows: Figure 2 As shown in the figure, the dual-channel attention mechanism module combines the spatial attention mechanism module and the channel attention mechanism module, so it has the advantages of both the spatial attention mechanism module and the channel attention module, and can capture useful information distributed in space and channels of the input, thereby improving the classification performance of the network.

[0108] Among them, the structure diagrams of the channel attention module and the spatial attention module are as follows: Figure 3 、 Figure 4As shown, the channel attention module can generate a channel attention feature map by utilizing the channel relationship of features. Each channel of a feature map can be considered as a feature detector. Through the channel attention mechanism, the meaningful part of the input feature distribution on the channel can be obtained. In order to efficiently calculate the channel attention, the spatial dimension of the input feature map is compressed. In this embodiment, global average pooling and maximum pooling are used to aggregate the spatial information of a feature map to generate two different spatial context descriptors: and These two descriptors represent the average pooled features and the maximum pooled features respectively. The channel attention is calculated as shown in formula (3), where σ is the sigmoid function and the weight size of MLP is and The two inputs are shared by the average pooling feature and the maximum pooling feature, and after the ReLU activation function, it is W0, that is, the pooling feature needs to be processed by the ReLU function before being input into the MLP.

[0109]

[0110] After the channel attention module, the spatial attention module is introduced. The spatial relationship between features can generate a spatial attention map. Spatial attention focuses on the information-rich part, which is complementary to the channel attention. In order to calculate the spatial attention, the average pooling operation and the maximum pooling operation are first performed along the channel axis, and the two are spliced ​​together to generate an efficient feature descriptor, and finally a 2D spatial attention map is generated. In short, the spatial attention calculation method is shown in formula (4), where σ represents the sigmoid function, f 7×7 Represents a convolution operation with a convolution kernel size of 7×7.

[0111]

[0112] Embodiment 6: This embodiment further limits the retinal OCT image classification method based on the multi-scale residual network described in Embodiment 4. In this embodiment, the maximum pooling layer is further limited, specifically including:

[0113] The maximum pooling layer is arranged before the first residual module.

[0114] It should be noted that the maximum pooling layer can be placed before the residual module in the network, and the step value of the residual module after the pooling layer is changed from 2 to 1, so that the input and output parameters in the modified network are consistent.

[0115] Embodiment 7: This embodiment further limits the retinal OCT image classification method based on the multi-scale residual network described in Embodiment 1. In this embodiment, the several branches are further limited, specifically including:

[0116] The plurality of branches specifically include four branches;

[0117] The four branches are respectively composed of 1×1 convolution, 1×1 and 3×3 convolution, 1×1 and 5×5 convolution, and a maximum pooling layer and 1×1 convolution.

[0118] In this embodiment, the multi-scale residual module is used to replace the residual module in the baseline network. The residual module framework in the baseline network is as follows: Figure 5 As shown in the figure, in modern neural networks, as the network depth gradually deepens, the probability of gradient diffusion and gradient explosion will gradually increase. This problem makes deep neural networks difficult to train. The residual module contains multiple convolutional layers. Data passes through these convolutional layers. At the same time, the original input information skips these convolutional layers and is directly transmitted to the subsequent layers. Finally, the entire convolutional layer is used as input and activated by the activation function to obtain the output of the residual module. Through residual learning, during backpropagation, the initial signal can be directly transmitted to the bottom layer without passing through any intermediate weights. This can alleviate the problems of gradient diffusion and gradient explosion to a certain extent. Even if the weight of the intermediate layer is very small, the gradient will basically not disappear.

[0119] The multi-scale residual module structure is as follows Figure 6 As shown, a multi-scale residual module using cascaded multi-cores combines the advantages of the residual module and uses 1×1 convolution for dimensionality reduction to achieve parameter reduction while reducing the amount of parameter computation. Furthermore, convolution is performed simultaneously on multiple scales to re-aggregate. This module uses multi-core convolution (i.e., 1×1, 3×3, and 5×5) to capture more global and local information in the distributed re-input scan. However, because the 3×3 and 5×5 convolution operations are computationally expensive, additional 1×1 convolutions are used for dimensionality reduction. In addition, each convolution layer is followed by a ReLU layer to reduce computation and alleviate overfitting problems. This also allows the network to handle large changes in information positions in the input scan.

[0120] The multi-scale residual module structure is as follows Figure 6As shown, the multi-scale residual module designed in this embodiment, while combining the advantages of the residual module, can obtain more and more detailed information from the input information, and use 1×1 convolution to perform dimensionality reduction to achieve parameter reduction and reduce the amount of parameter calculation. The module contains a residual module and four branches. The residual module greatly improves the network depth that can be effectively trained. Its structure is implemented in the form of skip layer connection, that is, the input of the unit is directly added to the unit output, which further effectively alleviates the gradient disappearance problem. The four branches are composed of 1×1 convolution, 1×1 and 3×3 convolution, 1×1 and 5×5 convolution, and a maximum pooling layer and 1×1 convolution. The number of channels is changed by 1×1 convolution to reduce the amount of calculation. The input information is input with 16K channels and output with 4K channels after passing through the first branch. When passing through the second branch, it is output with 6K channels after 1×1 convolution and with 8K channels after 3×3 convolution. When passing through the third branch, After 1×1 convolution, it is output with K channels, after 5×5 convolution, it is output with 2K channels, and when passing through the fourth branch, it is output with 2K channels after 1×1 convolution. Feature extraction is performed through 1×1, 3×3, and 5×5 convolutions to obtain information of different scales. The output information is fused with multi-scale features in a concatenated form. The output information obtained after fusion is added to the output information of the residual module in a sum form. In addition, a ReLU layer is added after each convolution layer. The ReLU layer will make the output of some neurons 0, resulting in the sparsity of the network. At the same time, it can reduce the interdependence between parameters and alleviate the occurrence of overfitting problems. Through this module, the advantages of the residual module can be obtained, which can effectively alleviate the gradient diffusion problem and the overfitting problem. At the same time, information of different sizes from the input information can be obtained, which can solve the problem of unclear imaging of pathological features in the original retinal OCT image and difficulty in capturing the characteristics of small target lesions.

[0121] Embodiment 8: This embodiment is an embodiment of the retinal OCT image classification method based on the multi-scale residual network of this embodiment, specifically:

[0122] This example is an open-source dataset of retinal OCT lesion images downloaded from the Kaggle platform. This dataset was made public by the University of California, San Diego (UCSD) in 2017. The dataset contains four categories of retinal lesion images: wet age-related macular degeneration (CNV), diabetic macular edema (DME), dry age-related macular degeneration (DRUSEN), and normal images (NORMAL), as shown in the figure. The original dataset consists of a training set and a test set. The original training set of the dataset consists of 83,484 OCT images from 4,686 patients, including 37,205 CNV images, 8,616 DRUSEN images, 11,348 DME images, and 26,315 NORMAL images. The test set consists of 250 images per category, for a total of 1,000 OCT images.

[0123] Table 1 Information of the retinal OCT image dataset used

[0124] Dataset CNV DME DRUSEN NORMAL TOTAL OCT2017 37205 11348 8616 26315 83484

[0125] To verify the effectiveness of the method in this embodiment, this embodiment downsampled the OCT2017 dataset. Taking the data volume of macular lesions (8616) as a benchmark, 8600 images were randomly sampled for each category, and then the data was divided into training and test sets at a ratio of 9:1.

[0126] The technical solution of this embodiment is described in detail below with reference to the accompanying drawings and embodiments.

[0127] Step 1: Obtain a retinal OCT image classification dataset and divide it into a training set and a test set in a ratio of 9:1. Finally, four categories of retinal disease OCT images are obtained, with 8,600 images in each category, of which 7,740 are used for training and 860 are used for testing.

[0128] Step 2: Image preprocessing: resize the retinal OCT images to 224×224, use downsampling to unify the number of retinal OCT images of the four categories, and standardize the images. The images after image preprocessing can be used as input into the classification network to achieve higher classification accuracy and better precision. Image standardization is a process of centralizing the data by removing the mean. According to convex optimization theory and knowledge related to data probability distribution, data centralization conforms to the data distribution law and is more likely to achieve generalization effect after training. The formula of the standardization method used in this embodiment is as follows:

[0129]

[0130] In formula (1), μ is the mean of the image, and X represents the image matrix;

[0131]

[0132] In formula (2), σ represents the standard deviation and N represents the number of pixels in the image.

[0133] like Figure 1 As shown, step 3: design a retinal OCT image classification model based on a multi-scale residual network. In general, the retinal OCT image classification network based on a multi-scale residual network in this embodiment is an end-to-end deep learning network architecture. This embodiment combines the multi-scale residual module and dual-channel attention with ResNet50, replaces some residual modules in the network with the multi-scale residual module, adds a maximum pooling layer to the network to reduce the amount of network computation, and changes the stride of the convolution layer after the pooling layer from 2 to 1; sets the output dimension of the network to 4 for the four-category task of retinal disease images: CNV, DME, DRUSEN, and NORMAL; the specific implementation is as follows:

[0134] Step 3.1: Add a dual-channel attention mechanism module to the baseline network. The dual-channel attention mechanism module structure is as follows: Figure 2 As shown in the figure, the dual-channel attention mechanism module combines the spatial attention mechanism module and the channel attention mechanism module, so it has the advantages of both the spatial attention mechanism module and the channel attention module, and can capture useful information distributed in space and channels of the input, thereby improving the classification performance of the network.

[0135] Among them, the structure diagrams of the channel attention module and the spatial attention module are as follows: Figure 3 、 Figure 4 As shown, the channel attention module can generate a channel attention feature map by utilizing the channel relationship of features. Each channel of a feature map can be considered as a feature detector. Through the channel attention mechanism, the meaningful part of the input feature distribution on the channel can be obtained. In order to efficiently calculate the channel attention, the spatial dimension of the input feature map is compressed. In this embodiment, global average pooling and maximum pooling are used to aggregate the spatial information of a feature map to generate two different spatial context descriptors: and These two descriptors represent the average pooled features and the maximum pooled features respectively. The channel attention is calculated as shown in formula (3), where σ is the sigmoid function and the weight size of MLP is and The two inputs are shared by the average pooling feature and the maximum pooling feature, and after the ReLU activation function, it is W0, that is, the pooling feature needs to be processed by the ReLU function before being input into the MLP.

[0136]

[0137] After the channel attention module, the spatial attention module is introduced. The spatial relationship between features can generate a spatial attention map. Spatial attention focuses on the information-rich part, which is complementary to the channel attention. In order to calculate the spatial attention, the average pooling operation and the maximum pooling operation are first performed along the channel axis, and the two are spliced ​​together to generate an efficient feature descriptor, and finally a 2D spatial attention map is generated. In short, the spatial attention calculation method is shown in formula (4), where σ represents the sigmoid function, f 7×7 Represents a convolution operation with a convolution kernel size of 7×7.

[0138]

[0139] In step 3.2, add a pooling layer to the network. Place the pooling layer before the residual module in the network and change the step size of the residual module after the pooling layer from 2 to 1, so that the input and output parameters of the modified network are consistent.

[0140] Step 3.3, use the multi-scale residual module of the embodiment to replace the residual module in the baseline network. The residual module framework in the baseline network is as follows: Figure 5 As shown in the figure, in modern neural networks, as the network depth gradually deepens, the probability of gradient diffusion and gradient explosion will gradually increase. This problem makes deep neural networks difficult to train. The residual module contains multiple convolutional layers. Data passes through these convolutional layers. At the same time, the original input information skips these convolutional layers and is directly transmitted to the subsequent layers. Finally, the entire convolutional layer is used as input and activated by the activation function to obtain the output of the residual module. Through residual learning, during backpropagation, the initial signal can be directly transmitted to the bottom layer without passing through any intermediate weights. This can alleviate the problems of gradient diffusion and gradient explosion to a certain extent. Even if the weight of the intermediate layer is very small, the gradient will basically not disappear.

[0141] The multi-scale residual module structure is as follows Figure 6 As shown, a multi-scale residual module using cascaded multi-cores combines the advantages of the residual module and uses 1×1 convolution for dimensionality reduction to achieve parameter reduction while reducing the amount of parameter computation. Furthermore, convolution is performed simultaneously on multiple scales to re-aggregate. This module uses multi-core convolution (i.e., 1×1, 3×3, and 5×5) to capture more global and local information in the distributed re-input scan. However, because the 3×3 and 5×5 convolution operations are computationally expensive, additional 1×1 convolutions are used for dimensionality reduction. In addition, each convolution layer is followed by a ReLU layer to reduce computation and alleviate overfitting problems. This also allows the network to handle large changes in information positions in the input scan.

[0142] The multi-scale residual module structure is as follows Figure 6 As shown, the multi-scale residual module designed in this embodiment, while combining the advantages of the residual module, can obtain more and more detailed information from the input information, and uses 1×1 convolution to perform dimensionality reduction to achieve parameter reduction and reduce the amount of parameter calculation. The module contains a residual module and four branches. The residual module greatly improves the network depth that can be effectively trained. Its structure is implemented in the form of skip layer connection, that is, the input of the unit is directly added to the unit output, which further effectively alleviates the gradient disappearance problem. The four branches are composed of 1×1 convolution, 1×1 and 3×3 convolution, 1×1 and 5×5 convolution, and a maximum pooling layer and 1×1 convolution. The number of channels is changed by 1×1 convolution to reduce the amount of calculation. The input information is input with 16K channels and output with 4K channels after passing through the first branch. When passing through the second branch, it is output with 6K channels after 1×1 convolution and with 8K channels after 3×3 convolution. When passing through the third branch, After 1×1 convolution, it is output with K channels, after 5×5 convolution, it is output with 2K channels, and when passing through the fourth branch, it is output with 2K channels after 1×1 convolution. Feature extraction is performed through 1×1, 3×3, and 5×5 convolutions to obtain information of different scales. The output information is fused with multi-scale features in a concatenated form. The output information obtained after fusion is added to the output information of the residual module in a sum form. In addition, a ReLU layer is added after each convolution layer. The ReLU layer will make the output of some neurons 0, resulting in the sparsity of the network. At the same time, it can reduce the interdependence between parameters and alleviate the occurrence of overfitting problems. Through this module, the advantages of the residual module can be obtained, which can effectively alleviate the gradient diffusion problem, alleviate the overfitting problem, and obtain information of different sizes from the input information. It can solve the problem that the pathological features in the original retinal OCT image are not clear and the characteristics of small target lesions are difficult to capture.

[0143] In step 3.4, add an average pooling layer after the network. This reduces the amount of parameter calculations and alleviates the overfitting problem.

[0144] In step 3.5, cross entropy is used as the loss function. The cross entropy loss function is often used in classification problems. Since it involves the probability of each category, in this embodiment, the cross entropy loss function appears together with the softmax function, specifically:

[0145] When performing multi-classification tasks, the softmax function and the cross-entropy loss function are often used together. Because the cross-entropy describes the difference between two probability distributions, and the neural network outputs a vector, the softmax activation function is needed to first "normalize" a vector into the form of a probability distribution, and then use the cross-entropy loss function to calculate the final loss.

[0146] In step 3.6, the network's output dimension is changed to make it suitable for the retinal OCT image classification task. The original baseline network has an output dimension of 100. To handle the four-category retinal OCT image classification task (CNV, DME, DRUSEN, NORMAL), this example sets the network's output dimension to 4.

[0147] like Figure 7 As shown in Figure 4, step 4: Input the preprocessed retinal OCT images from the training set into the classification model for training. First, the preprocessed retinal OCT images are placed into the classification model. These images are then used to train the classification model, optimize the objective function, and update the model parameters. Finally, after training, the relevant parameter values ​​are saved.

[0148] like Figure 8 As shown, step 5: input the preprocessed retinal OCT images in the test set into the classification result to obtain the prediction result of the image. The preprocessed retinal OCT test set images are tested by the model saved after training, and the network performance is further tested. Since different evaluation indicators have different estimation significances for the experimental results, this embodiment uses the commonly used accuracy, precision, recall rate and F1-score to evaluate the test results. In order to make the experimental results more concise and easy to understand, the confusion matrix is ​​used to display the classification results of each type of data. The accuracy, precision, recall rate and F1-score are defined as shown in Formula 5-8:

[0149]

[0150]

[0151]

[0152]

[0153] In formula 5-8, TP represents the number of samples predicted as positive samples; TN represents the number of samples predicted as negative samples; FP represents the number of samples predicted as positive samples; FN represents the number of samples predicted as negative samples; TP+FP represents the total number of samples predicted as positive samples; TP+FN represents the total number of positive samples.

[0154] To verify the effectiveness of the present invention, we first compared it with the achievements of three human experts in the field of retinal OCT image classification. The results are shown in Table 2. As can be seen from Table 2, the classification accuracy of the expert method for DME and Drusen is not high, far lower than the classification accuracy of the other two categories. The algorithm used in the present invention, by referencing the dual-channel attention mechanism module and the multi-scale residual module, reduces the influence of redundant information in the original image, more efficiently improves the network's collection of input information, better avoids the loss of original input data features during network training, and further improves the network's feature learning ability for retinal OCT lesion images. The algorithm used in the present invention has significantly improved the classification accuracy of DME and Drusen. Compared with experts 1 and 2, the accuracy of the present invention for DME increased by 3.5% and 7.3%, and the accuracy for Drusen increased by 5.4% and 8.9%, and the overall classification accuracy reached 97.3%. Compared with the expert algorithm, the algorithm used by the present invention has the highest overall accuracy and the accuracy of each individual category, which fully verifies the advanced nature of the improved network of the present invention.

[0155] In addition, the present invention is also compared with ResNet50. The results are shown in Table 3. It can be seen from Table 3 that the method of the present invention performs better in terms of accuracy, precision, recall rate and F1-score.

[0156] Table 2 Comparison of retinal OCT image classification performance between the present invention and human experts

[0157] method CNV DME Drusen Normal Overall accuracy Expert 1 96.9％ 95.8％ 92.5％ 98.5％ 97％ Expert 2 —— —— —— —— 95.39％ Expert 3 98％ 92％ 89％ 96％ 96.5％ Algorithm of the present invention 98.6％ 99.3％ 97.9％ 98.8％ 97.3％

[0158] Table 3 Comparison of classification performance between the proposed method and ResNet50

[0159]

[0160] In order to further visualize the classification effect of the network on the four categories of retinal diseases, the confusion matrix diagrams of the baseline network and the network used in the present invention are drawn, as shown in Figure 2. Figure 9 As shown in the figure, the vertical axis is the predicted label and the horizontal axis is the true label. The data in the figure is the number of samples, which respectively indicate how many pictures of each category are used for testing, and how many pictures of each category are predicted correctly, how many pictures are predicted incorrectly, and what the predictions of the wrong pictures are, which can be used to calculate the accuracy. The confusion matrix uses blue as the base color. The darker the color, the higher the classification accuracy of the category. It can be seen that the network used in the present invention has a relatively significant improvement in the classification accuracy of CNV, DME, DRUSEN, and NORMAL categories.

Claims

1. A retinal OCT image classification method based on a multi-scale residual network, characterized by: The method comprises: Step 1: Obtain a retinal OCT image classification dataset and divide it into a training set and a test set; Step 2: preprocessing the training set and the test set; Step 3: Improve the baseline network using a dual-channel attention mechanism and a multi-scale residual module to establish a retinal OCT image classification model. The baseline network is an image classification network. The multi-scale residual module specifically includes several branches, the residual module of the baseline network, a feature fusion module and an addition module, and each branch includes several convolutions; The feature fusion module is used to perform feature fusion on the output information of the multiple branches; The adding module is used to add the fused information and the output information of the residual module; Step 4: training the retinal OCT image classification model using the preprocessed training set; Testing the retinal OCT image classification model using the preprocessed test set; Obtain the trained retinal OCT image classification model; Step 5: using the trained retinal OCT image classification model to classify the retinal OCT image; Add several dual-channel attention mechanism modules, maximum pooling layers, and average pooling layers to the baseline network; Replacing the residual modules of the plurality of baseline networks with a plurality of the multi-scale residual networks; Set the loss function to learn the model parameters; According to the retinal OCT image classification task, the output dimension of the network is set; Combine the multi-scale residual module and dual-channel attention with ResNet50, replace some residual modules in the network with the multi-scale residual module, add a maximum pooling layer to the network to reduce the amount of network computation, and change the stride of the convolution layer after the pooling layer from 2 to 1; The dual-channel attention mechanism module combines the spatial attention mechanism module and the channel attention mechanism module to capture useful information about the input distribution in space and channels; The channel attention mechanism module generates a channel attention feature map by utilizing the inter-channel relationship of features. Each channel of a feature map is considered as a feature detector. Global average pooling and maximum pooling are used to aggregate the spatial information of a feature map to generate two different spatial context descriptors: and , these two descriptors represent the features that are average pooled and the features that are maximum pooled respectively; After the channel attention mechanism module, the spatial attention mechanism module is introduced to generate a spatial attention map by using the spatial relationship between features. First, average pooling and maximum pooling operations are performed along the channel axis, and the two are concatenated to generate an efficient feature descriptor, and finally a 2D spatial attention map is generated. Add a pooling layer to the network; place the pooling layer before the residual module in the network, and change the step size of the residual module after the pooling layer from 2 to 1; Use multi-scale residual modules to replace the residual modules in the baseline network; The multi-scale residual module uses cascaded multi-core convolution to perform dimensionality reduction and reduction. The module uses multi-core convolution to capture more global and local information in the distribution input scan. The multi-core convolution is 1×1, 3×3 and 5×5. Add an average pooling layer after the network, use cross entropy as the loss function, and change the output dimension of the network to 4.

2. The retinal OCT image classification method based on a multi-scale residual network according to claim 1, characterized in that: The preprocessing specifically includes: Using a downsampling method, the number of retinal OCT images of several categories in the training set and the test set is unified; unifying the sizes of the unified number of retinal OCT images; The retinal OCT images after the size is unified are normalized.

3. The retinal OCT image classification method based on a multi-scale residual network according to claim 1, characterized in that: The baseline network is the ResNet50 network.

4. The retinal OCT image classification method based on a multi-scale residual network according to claim 1, characterized in that: There are two dual-channel attention mechanism modules, which are respectively set after the convolutional layer and after the last multi-scale residual module.

5. The retinal OCT image classification method based on a multi-scale residual network according to claim 1, characterized in that: The maximum pooling layer is arranged before the first residual module.

6. The retinal OCT image classification method based on a multi-scale residual network according to claim 1, characterized in that: The plurality of branches specifically include four branches; The four branches are respectively composed of 1×1 convolution, 1×1 and 3×3 convolution, 1×1 and 5×5 convolution, and a maximum pooling layer and 1×1 convolution.

7. A retinal OCT image classification system based on a multi-scale residual network, characterized by: The system comprises: The dataset acquisition module is used to obtain the retinal OCT image classification dataset and divide it into a training set and a test set; A data set processing module, used for preprocessing the training set and the test set; A retinal OCT image classification model building module is used to improve the baseline network using a dual-channel attention mechanism and a multi-scale residual module to build a retinal OCT image classification model; The multi-scale residual module specifically includes several branches, a residual module, a feature fusion module and an addition module, and each branch includes several convolutions; The feature fusion module is used to perform feature fusion on the output information of the multiple branches; The adding module is used to add the fused information and the output information of the residual module; A model training and testing module, configured to train the retinal OCT image classification model using the preprocessed training set; Testing the retinal OCT image classification model using the preprocessed test set; Obtain the trained retinal OCT image classification model; The model classification module is used to classify the retinal OCT images using the trained retinal OCT image classification model.

8. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor runs the computer program stored in the memory, the steps of the method according to any one of claims 1 to 6 are performed.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of computer instructions, and the plurality of computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 6.

Images