A method for classifying tubercular retinal OCT images based on feature fusion

Abstract

This invention discloses a feature fusion-based OCT image classification method for tuberculous retinal diseases, comprising the following steps: Step 1: acquiring tuberculous retinal OCT images; Step 2: image preprocessing; Step 3: constructing multiple baseline neural networks; Step 4: adding a multi-scale gated channel attention mechanism to the multiple baseline networks, utilizing the learning of the neural network to obtain the importance between feature channels, thereby strengthening features beneficial to the classification task and suppressing useless features; Step 5: training and testing: finding the optimal model under the optimal parameters of each network structure through multiple comparative experiments; Step 6: model fusion; This invention utilizes a multi-layer convolutional neural network and a gated attention mechanism to construct a basic learner that acts as a feature extractor, then fuses the image features extracted by different models, and inputs the fused features into a support vector machine for further classification and prediction, thereby further improving the accuracy of tuberculous retinal OCT image classification.

Description

Technical Field

[0001] This invention relates to the field of medical image processing and analysis technology in the field of computer vision, and in particular to a method for classifying tuberculous retinal OCT images based on model fusion. Background Technology

[0002] Tuberculosis (TB) is a chronic infectious disease caused by Mycobacterium tuberculosis (Mtb) and continues to pose a serious threat to human health. Therefore, rapid, effective, and accurate TB screening and analysis are essential.

[0003] All ocular tissues, except the lens, can be infected with Mycobacterium tuberculosis. The incidence of intraocular tuberculosis ranges from 1.4% to 18%. Early symptoms of tuberculosis affecting the eyes are often mild and similar to those of keratitis, nodular scleritis, trachoma, and retinal periphlebitis, frequently overlooked by patients and doctors, leading to misdiagnosis or missed diagnosis. When patients experience eye pain, redness, or decreased vision, most non-ophthalmologists are unable to make a reliable diagnosis and are unaware of when to seek help from an ophthalmologist. Treating only common eye diseases without anti-tuberculosis treatment is often ineffective and can ultimately lead to blindness. Current diagnosis of tuberculous retinopathy generally relies on the results of examinations such as wide-angle fundus photography, wide-angle fluorescein angiography, indocyanine green angiography, and optical coherence tomography (OCT). The images are then independently reviewed by two specialized fundus specialists to reach a consensus. As the only microvascular system in the human body that can be observed non-invasively, the eye allows for direct visualization of exposed nerves and blood vessels using simple instruments without surgical dissection. Ocular features can also provide a pathway for the diagnosis of potential systemic tuberculosis infection.

[0004] Optical coherence tomography (OCT) is a type of optical tomographic technique and one of the most promising new tomographic imaging technologies in recent years. It holds great promise for in vivo detection and imaging of biological tissues, and is used not only in clinical diagnosis in ophthalmology but also in interventional cardiology to aid in the diagnosis of coronary artery disease. Currently, based on OCT technology, the key to improving the diagnostic efficiency of tuberculous ocular lesions and alleviating diagnostic pressure lies in how artificial intelligence methods can be used to extract lesion features from OCT images and automatically identify tuberculosis lesions. Summary of the Invention

[0005] This invention provides a feature fusion-based OCT image classification method for tuberculous retina. It utilizes a multi-layer convolutional neural network and a gated attention mechanism to construct a basic learner that acts as a feature extractor. Then, it fuses image features extracted from different models and inputs the fused features into a support vector machine for further classification and prediction, thereby further improving the accuracy of OCT image classification for tuberculous retina.

[0006] This invention is achieved through the following technical solution:

[0007] A model fusion-based OCT image classification method for tuberculous retinal diseases includes the following steps:

[0008] Step 1: Obtain OCT images of tuberculous retina and divide them into training set, validation set and test set according to proportion;

[0009] Step 2: Image preprocessing: Denoise the original image and perform data augmentation operations on the denoised image, including scaling, random cropping, rotation, vertical flipping, horizontal flipping, and batch normalization, to ensure the generalization ability of the model.

[0010] Step 3: Construct ResNet50 and GoogLeNet network models as the base network models;

[0011] Step 4: Add a multi-scale gated channel attention mechanism to the ResNet50 network. This mechanism uses the neural network to learn the importance of feature channels, thereby strengthening features that are beneficial to the classification task and suppressing features that are not. The features that are not beneficial are irrelevant information such as noise in the image, while the important information in the image is the beneficial feature.

[0012] Step 5: Training and Testing: Through multiple comparative experiments, find the optimal model under the optimal parameters for each network structure;

[0013] Step 6: Model Fusion: Input the original image dataset into the trained optimal model to obtain image features under different models, then cascade them along the feature dimension to complete feature fusion, and construct a feature dataset. Finally, input it into the support vector machine for further classification training.

[0014] Furthermore, the noise reduction process in step 2 is as follows:

[0015] The nonlinear diffusion filtering method PM introduces a diffusion coefficient C based on the classical heat conduction equation. The value of C is determined by the gradient magnitude of a specific pixel. Areas with a large gradient magnitude are edges, where the C value tends to be 0, the conduction function value is small, and the diffusion speed is slow, thus preserving edge image features without filtering. Areas with a small gradient magnitude are non-edges, where the C value is large, the conduction function value is large, and the diffusion speed is fast, thus performing filtering. This achieves the noise reduction effect. PM is described by a nonlinear partial differential equation, as shown in formula (1):

[0016]

[0017] Where L is the gray value of the image, t is time, x and y are the spatial coordinates of the image, ▽L is the gradient vector of L, and div is the divergence of the gradient.

[0018] The transfer function c(x,y,t) is a function that controls the speed of the nonlinear diffusion filter, and its construction method is shown in formula (2):

[0019]

[0020] Among them ▽L σ The image L is Gaussian smoothed. σ The gradient of the function g() is usually expressed as a monotonically decreasing function, and there are two types:

[0021]

[0022]

[0023] Here, k is a constant used to control the value of the gradient response. The smaller the value of k, the steeper the curve of the transfer function, and the more smoothing is performed on the noise and detail information in the image. Therefore, the edge information is poorly preserved. When the value of k is larger, the curve of the transfer function is flatter, and the edge information is better preserved. However, more noise and detail information will also be preserved at the same time.

[0024] Furthermore, the specific process in step 4 is as follows:

[0025] The multi-scale gated channel attention mechanism is implemented through the multi-scale gated channel attention module GCT-A, which consists of three parts: a global context embedding module, a channel normalization module, and a gated adaptive module.

[0026] The global context embedding module uses three convolutional kernels of different sizes: 1×1, 3×3, and 5×5. The 1×1 kernel is used to adjust the number of channels to accommodate the needs of subsequent convolutional operations. The 3×3 and 5×5 kernels are used to capture receptive fields of different sizes to cover a wider range of global context information, thereby avoiding getting caught in local semantic ambiguity and aggregating the global context information in each channel. The specific definitions are as follows:

[0027]

[0028] The parameter α is defined as α = [α1, α2, ..., α...]. c When αn approaches 0, this channel does not participate in channel normalization. c H represents the channel of the feature map, H represents the height of the feature map, and W represents the width of the feature map. Let represent all channels in the i-th row and j-th column of the feature map, and p represents p-norm. A dual scale is formed by 1-norm and 2-norm. At the same time, in order to avoid the problem of differentiation at the zero point, a very small constant ε is introduced.

[0029] To establish a competitive relationship between neurons and channels, channel normalization is performed using BatchNorm normalization, as defined below:

[0030]

[0031] For each channel C, each pixel in the feature map is treated as a C-dimensional vector S. C Then, for vector S C BatchNorm normalization is performed, where C represents the number of channels in the feature map. The purpose of ε is to divide the BatchNorm-normalized feature vector by a scaling factor related to the number of channels C, in order to keep the magnitude of the feature vector constant. This is because the magnitude of the feature vector may change after BatchNorm normalization, and in order to maintain the representational power of the features, we need to scale it so that its magnitude is as close to 1 as possible, thereby enhancing the representational power of the features and the generalization ability of the network. ε is a very small number of 10. -5 To avoid the denominator being 0.

[0032] The channel importance calculated using the two methods is combined to obtain a final normalized vector, resulting in better performance and stability. The fusion formula is shown below.

[0033]

[0034] When a channel's gating mechanism is actively activated, it promotes competition between that channel and other channels; conversely, when it is passively activated, it promotes cooperation between that channel and other channels. The specific definitions are as follows:

[0035]

[0036] Where X C Here is the feature map of the original channel C, where ReLU is a linear rectified function, and the weights γ are defined as [γ1, γ2, ..., γ]. c The weight β is defined as [β1, β2, ..., β]. c Furthermore, when both the gate weight γ and the gate bias β are 0, the original features are passed to the next layer without loss. Therefore, GCT-A can effectively solve the degradation problem caused by deep networks.

[0037] Finally, the activation features of the GCT-A module are represented as follows:

[0038]

[0039] Here, α, γ, and β represent trainable parameters. α is responsible for the adaptive embedding output, while the gating weights γ and bias β control the activation of the gates. These three parameters determine the behavior of GCT-A in each channel. Under this formula, the computational complexity of GCT-A is only O(C), lower than SENet's O(C). 2 The reason for this is that SENet uses two fully connected layers, where C is the number of channels; therefore, GCT greatly accelerates the convergence speed and accuracy with a small increase in parameters.

[0040] Furthermore, in step 4, GCT-A is deployed before the ResNet50 convolutional layer to form the GCResNet network model, which can better suppress background and irrelevant noise and make the network's learning focus on beneficial features.

[0041] Furthermore, the specific process of step 5 is as follows:

[0042] The preprocessed training set is input into the base learner. The network parameters are continuously adjusted, and the trained model is saved. Then, the preprocessed test set is input into the saved model for testing, further evaluating the network performance. Since different evaluation metrics have different meanings for the experimental results, five evaluation metrics—Precision, Recall, Specificity, F1-Score, and Accuracy—are used to evaluate the performance of the experimental network model. The formulas are as follows:

[0043]

[0044]

[0045]

[0046]

[0047]

[0048] Where TP represents the number of images that are actually normal and the model predicts them as normal; FP represents the number of images that are actually normal but the model predicts them as tuberculosis; TN represents the number of images that are actually tuberculosis and the model predicts them as tuberculosis; and FN represents the number of images that are actually tuberculosis but the model predicts them as normal.

[0049] The advantages of this invention are as follows:

[0050] 1) To address the speckle noise problem in OCT images, a nonlinear diffusion filtering noise reduction method is adopted to reduce image noise while ensuring image quality, achieving the effect of "noise reduction" + "image enhancement".

[0051] 2) In response to the small location and indistinct shape of tuberculous ocular lesions, a multi-scale gating channel attention mechanism is introduced to enhance beneficial features and suppress useless features, so that the network attention is focused on effective information and the classification and recognition accuracy is improved.

[0052] 3) Using the CNN-SVM framework, the features extracted by CNN are fused together, combining the advantages of different models to further improve the overall recognition accuracy. Attached Figure Description

[0053] Figure 1 An experimental framework diagram for constructing the basic feature extractor of this invention;

[0054] Figure 2 This is a schematic diagram of the structure of the gated attention module GCT-A of the present invention;

[0055] Figure 3 This is a network structure diagram of the basic feature extractor of this invention;

[0056] Figure 4 This is a schematic diagram of the CNN-SVM feature fusion structure of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is described below with reference to specific embodiments shown in the accompanying drawings. However, it should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] like Figure 1 As shown, this application discloses a feature fusion-based OCT image classification method for tuberculous retinal diseases, comprising:

[0059] Step 1: Obtain tuberculous retinal OCT images and divide them into training, validation, and test sets in a 7:2:1 ratio. We collected 128 images per eye from 16 patients at an ophthalmology center in Hangzhou, totaling 256 images for both eyes, for a total of 4096 tuberculosis (TB) images, and 4224 normal images from 17 healthy medical staff. We shuffled the dataset and divided it into training, validation, and test sets in a 7:2:1 ratio, as shown in Table 1.

[0060]

[0061]

[0062] Step Two: Image Preprocessing: Noise Reduction: Because OCT equipment generates interferometric images based on the Michelson interferometer, the original OCT images contain a large amount of speckle noise, which can seriously affect medical judgment. Therefore, a nonlinear diffusion filtering method, PM, is used. Based on the classical heat conduction equation, a "diffusion coefficient C" is introduced. This C value is determined by the gradient magnitude of a specific pixel. Areas with a large gradient magnitude are edges, and the C value is 0. Areas with a small gradient magnitude are non-edges, and the C value is relatively large. Since the gradient in edge areas is larger, it is retained without filtering, while the gradient in non-edge areas is smaller and is filtered. This achieves the noise reduction effect. PM can be described by a nonlinear partial differential equation, as shown in Equation 1:

[0063]

[0064] The conduction function is constructed as shown in Formula 2:

[0065]

[0066] Among them ▽L σ The image L is Gaussian smoothed. σ The gradient of the function g() has the following two forms:

[0067]

[0068]

[0069] Where k is a constant used to control the value of the gradient response.

[0070] Image augmentation: Data augmentation transforms the training set, enriching it and thus improving the model's generalization ability. It also helps avoid overfitting, prevents the model from learning irrelevant information, enhances robustness, reduces sensitivity to images, and avoids sample imbalance. In this example, we use the following methods:

[0071] 1) Resize: Scales the smallest side of the image to 256 pixels, while scaling the longest side proportionally.

[0072] 2) Flipping: randomly crop the image to a new image of 224*224 pixels.

[0073] 3) Random Rotation: Rotates the image by randomly selecting an offset angle between 0° and 15°.

[0074] 4) Random Vertical Flip: Performs a vertical flip operation on the image with a 50% probability.

[0075] 5) Random Horizontal Flip: Performs a horizontal flip operation on the image with a 50% probability.

[0076] 6) Batch Normalization: Standardization process with a mean of 0 and a standard deviation of 1.

[0077] Step 3: Construct multiple benchmark neural networks. In this example, we use ResNet50 and GoogLeNet as benchmark networks for construction.

[0078] Step Four: As Figure 2 The image shows the Gated Attention Module (GCT-A), which comprises three parts: Global Context Embedding, Channel Normalization, and Gating Adaptation. It enhances the learning ability of the neural network by focusing on the relationships between feature channels. Simultaneously, it utilizes the neural network's learning to determine the importance of feature channels, thereby strengthening features beneficial to the classification task and suppressing those that are not. It explicitly models the relationships between channels using interpretable variables, thus determining whether they compete or cooperate.

[0079] The explanation for competition is as follows: In deep networks, high-level features often differ significantly and are directly and closely related to the classification task. To learn these high-level features, channels need to compete to acquire more relevant feature information. In shallow networks, learning low-level features requires cooperation between channels to extract features more broadly.

[0080] The explanation of cooperation: In shallow networks, learning low-level features requires cooperation between channels to extract features more broadly.

[0081] The Global Context Embedding module avoids getting caught in local semantic ambiguity by using a larger receptive field, aggregating global context information from each channel, and employing a dual-scale composition of 1-norm and 2-norm, as defined below:

[0082]

[0083] The parameter α is defined as α = [α1...αC]. When αn is close to 0, the channel does not participate in the channel normalization described below. To avoid the problem of finding the derivative at the zero point, a very small constant ε is introduced.

[0084] To establish a competitive relationship between neurons and channels, making channels with larger responses larger and suppressing channels with smaller responses, channel normalization is performed using BatchNorm normalization. The specific definition is as follows:

[0085]

[0086] The channel importance calculated using the two methods is combined to obtain a final normalized vector, resulting in better performance and stability. The fusion formula is shown below.

[0087]

[0088] When a channel's gating mechanism is actively activated, it promotes competition between that channel and other channels; conversely, when it is passively activated, it promotes cooperation between that channel and other channels. The specific definitions are as follows:

[0089]

[0090] Furthermore, when both the gate weight γ and the gate bias β are 0, the original features can be losslessly passed to the next layer. Therefore, GCT can effectively solve the degradation problem caused by deep networks.

[0091] Finally, the activation features of the GCT module can be expressed as:

[0092]

[0093] Here, α, γ, and β represent trainable parameters. α is responsible for the adaptive embedding output, while the gating weights γ and bias β control the activation of the gates. These three parameters determine the behavior of GCT-A in each channel. Under this formula, the parameter complexity of GCT-A is only O(C). Therefore, GCT-A significantly accelerates convergence speed and accuracy with only a small increase in parameters.

[0094] like Figure 3 As shown, deploying GCT-A before the ResNet50 convolutional layers to form the GCResNet network model can better suppress background and irrelevant noise, and make the network's learning focus concentrated on beneficial features.

[0095] Step 5: Through comparative experiments, the optimal model of the three networks is obtained. The preprocessed training set is input into the base learner, and the network parameters are continuously adjusted before the trained model is saved. The preprocessed test set is then input into the saved model for testing, further evaluating the network performance. Since different evaluation metrics have different meanings for the experimental results, this invention uses five evaluation metrics—precision, recall, specificity, F1-Score, and accuracy—to evaluate the performance of the experimental network models. The formulas are as follows:

[0096]

[0097]

[0098]

[0099]

[0100]

[0101] Where TP represents the number of images that are actually normal and the model predicts them as normal; FP represents the number of images that are actually normal but the model predicts them as tuberculosis; TN represents the number of images that are actually tuberculosis and the model predicts them as tuberculosis; and FN represents the number of images that are actually tuberculosis but the model predicts them as normal.

[0102] Step Six: As Figure 4As shown, a pre-trained convolutional neural network model is used as a feature extractor, and an SVM is used as a classifier. GoogLeNet, ResNet50, and GCResNet are selected to extract corresponding feature vectors, which are then concatenated to complete feature fusion. The fused features are then used as input to the SVM to generate a discriminant vector for classifying tuberculosis OCT, and the SVM is trained.

Claims

1. A model fusion-based OCT image classification method for tuberculous retinal diseases, characterized in that, Includes the following steps: Step 1: Obtain OCT images of tuberculous retina and divide them into training set, validation set and test set according to proportion; Step 2: Image preprocessing: Denoise the original image and perform data augmentation operations on the denoised image, including scaling, random cropping, rotation, vertical flipping, horizontal flipping, and batch normalization, to ensure the generalization ability of the model. Step 3: Construct the basic network model; Step 4: Add a multi-scale gated channel attention mechanism to the basic network model, and use the learning of the neural network to obtain the importance between feature channels, thereby strengthening the features that are beneficial to the classification task and suppressing the features that are not beneficial. The specific process in step 4 is as follows: The gated attention mechanism is implemented through the multi-scale gated attention channel module GCT-A, which consists of three parts: a global context embedding module, a channel normalization module, and a gated adaptive module. The global context embedding module uses three convolutional kernels of different sizes: 1×1, 3×3, and 5×5. The 1×1 kernel is used to adjust the number of channels to accommodate the needs of subsequent convolutional operations. The 3×3 and 5×5 kernels are used to capture receptive fields of different sizes to cover a wider range of global context information, thereby avoiding getting caught in local semantic ambiguity and aggregating the global context information in each channel. The specific definitions are as follows: (1) The parameter α is defined as α = [α1, α2, ..., α...]. c When α n When X approaches 0, this channel does not participate in channel normalization. c H represents the channel of the feature map, H represents the height of the feature map, and W represents the width of the feature map. Let represent all channels in the i-th row and j-th column of the feature map, and p represents p-norm. A dual scale is formed by 1-norm and 2-norm. At the same time, in order to avoid the problem of differentiation at the zero point, a very small constant ɛ is introduced. To establish a competitive relationship between neurons and channels, channel normalization is performed using BatchNorm normalization, as defined below: (2) For each channel C, each pixel in the feature map is treated as a C-dimensional vector S. C Then for vector S C BatchNorm normalization is performed, where C represents the number of channels in the feature map. The purpose of this is to divide the BatchNorm-normalized feature vector by a scaling factor related to the number of channels C, so as to keep the amplitude of the feature vector unchanged. The final result is the channel importance after aggregating global information through 1-norm and 2-norm. The channel importance calculated using the two methods is combined to obtain the final normalized vector, resulting in better performance and stability. The fusion formula is shown below: (3) When a channel's gating mechanism is actively activated, it promotes competition between that channel and other channels; conversely, when it is passively activated, it promotes cooperation between that channel and other channels. The specific definitions are as follows: (4) Where X C Here is the feature map of the original channel C, where ReLU is a linear rectified function, and the weights γ are defined as [γ1, γ2, ..., γ]. c The weight β is defined as [β1, β2, ..., β]. c Furthermore, when both the gate weight γ and the gate bias β are 0, the original features are passed to the next layer without loss. Therefore, GCT can effectively solve the degradation problem caused by deep networks. Finally, the activation features of the GCT-A module are represented as follows: (5) Here, α, γ, and β represent trainable parameters. α is responsible for the adaptive embedding output, while the gating weights γ and bias β control the activation of the gates. These three parameters determine the behavior of GCT-A in each channel. Under this formula, the computational complexity of GCT-A is only O(C), lower than SENet's O(C). 2 This is because SENet uses two fully connected layers, where C is the number of channels; therefore, GCT-A greatly accelerates the convergence speed and accuracy with a small increase in parameters. Step 5: Training and Testing: Through multiple comparative experiments, find the optimal model under the optimal parameters for each network structure; Step 6: Model Fusion: Input the original image dataset into the trained optimal model to obtain image features under different models, then cascade them along the feature dimension to complete feature fusion, and construct a feature dataset. Finally, input it into the support vector machine for further classification training.

2. The method for classifying tuberculous retinal OCT images based on model fusion according to claim 1, characterized in that, The noise reduction process in step 2 is as follows: The nonlinear diffusion filter method PM is adopted. PM is described by a nonlinear partial differential equation, as shown in equation (6): (6) Where L is the gray value of the image, t is time, x and y are the spatial coordinates of the image, ▽L is the gradient vector of L, and div is the divergence of the gradient. The transfer function c(x,y,t) is a function that controls the speed of the nonlinear diffusion filter, and its construction method is shown in formula (7): (7) Among them ▽L σ The image L is Gaussian smoothed. σ The gradient of the function g() is monotonically decreasing and has the following two forms: (8) (9) Where k is a constant used to control the value of the gradient response.

3. The method for classifying tuberculous retinal OCT images based on model fusion according to claim 1, characterized in that, In step 4, GCT-A is deployed before the convolutional layers, which can better suppress background and irrelevant noise and make the learning focus of the network concentrate on beneficial features.

4. The method for classifying tuberculous retinal OCT images based on model fusion according to claim 1, characterized in that, Step 5 is as follows: The preprocessed training set is input into the base learner. The network parameters are continuously adjusted, and the trained model is saved. Then, the preprocessed test set is input into the saved model for testing, further evaluating the network performance. Since different evaluation metrics have different meanings for the experimental results, five evaluation metrics—Precision, Recall, Specificity, F1-Score, and Accuracy—are used to evaluate the performance of the experimental network model. The formulas are as follows: (10) (11) (12) (13) (14) Where TN represents the number of images that are actually normal and the model predicts them as normal; FP represents the number of images that are actually normal but the model predicts them as tuberculosis; TP represents the number of images that are actually tuberculosis and the model predicts them as tuberculosis; and FN represents the number of images that are actually tuberculosis but the model predicts them as normal.

Images